by William Tunstall-Pedoe
True Knowledge is building one of the most exciting things on the internet: a platform that can automatically answer questions on any subject, phrased in the way that is most natural to the user.
Instead of guessing keywords and browsing web links, the user experience is a direct answer to a naturally phrased question (no more searching on external sites and ideal for mobile devices).
Behind this technology is a knowledge base we are building of the world's knowledge, stored in a structured form that computers can understand and reason with.
The project is ambitious (to put it mildly) but potentially of enormous value - and we are making solid progress. In this post I'm hoping to share some of the progress that we've made.
First, some background:
To answer a question, the system does two things:
First, it translates the question into internal semantic form (a language-independent machine-readable query). e.g.
what is the capital of france?
becomes
query a
[current time] [applies to] now
f: a [is the capital of] [france]
f [applies at timepoint] now
We call this "understood"
Secondly, we execute the query to produce an answer.
query a
[current time] [applies to] now
f: a [is the capital of] [france]
f [applies at timepoint] now
becomes
a=[paris]
"The french city of Paris"
We call this "answered"
Failure to understand is often due to missing lexical knowledge in the knowledge base (e.g. not knowing that the string "france" can denote the country).
Failure to answer (when understood) is almost always due to missing (factual) knowledge.
Both the company and our users have been building our knowledge base as fast as we can over the last year with the goal of improving our ability to answer questions.
The results are in the graph above.
The red line and red scale on right shows facts in our knowledge base ending at near 120 million.
The blue line and blue scale on left shows a 30-day average of the percentage of questions answered within our site.
These questions exclude the example questions and questions asked by staff. i.e. They are original questions on any subject asked by people external to the company of their own free will. The number bounces around a lot because people ask different questions every day - it isn't a fixed benchmark.
However, as you can see we recently had a record score for this metric of more than 45%. The understood percentage at this point was around 65%
The first question that may spring to mind is how this translates to other situations. These users are private beta testers, many of whom have an interest in the technology and have some familiarity with where it is strong.
The answer to this is undoubtedly that this positively affects the metric but by way of comparison we recently launched Quizbot: a public facing demo of our API which is available to anyone on the internet to play with. Taking stats from this service in the most pessimistic way (excluding example questions, auto-suggested questions and questions asked from heavy users of the service - which includes all staff members) we have an equivalent figure there of 30% - again these are random questions on any subject by people outside the company. (The sample is also high: tens of thousands of questions asked.)
Finally, people may ask what the value is of only being able to answer less than half of questions.
The answer is two-fold: first because our platform tries to understand the question and understands the knowledge needed to generate the answer, it knows when it doesn't know. It never guesses and wrong answers are rare. This means that ourselves or platform partners can quietly fail over to other methods: even keyword searching when a direct answer is unavailable. When our platform can answer the user gets the perfect direct answer to their question, when it can't, they still get a response.
The second answer is that these stats are for general questions. By comprehensively populating out a corner of the knowledge base in a vertical we can do far better and we will shortly be announcing a vertical application that we've built on our platform that demonstrates this.
Finally, we are improving the whole time. Even in the few days since I created the above graph, millions of new facts have gone in the knowledge base and each of those new facts can be used to answer new questions that the platform couldn't answer before (directly and through inference).
The percentage of questions that we answer will vary over time with the usage patterns and experience of our user base. However, as knowledge gets added and we continue to improve, the number of questions we can answer will continue to grow and the trend will continue up. It's an exciting time - please join us.
Where did the beta site go ?
I have tried to logon www.trueknowledge.com with my beta id but the "sign in" bottom
is grayed out and I can't logon.
Posted by: Equanimous | 09 March 2009 at 03:22 AM
Equanimous,
I don't know what happened here but I'll get the developers to fix it ASAP.
Beth
Posted by: Beth | 10 March 2009 at 06:05 PM