Search test: Google, Live, and Powerset. The winner is….

After TechCruch’s comments the other day about how terrific the new natural language aware PowerSet.com search would be I was eager to check it out. I was going to sign up for the beta and then I decided to wait for the launch. I didn’t have to wait long. It’s up now. Check it out.

I admit I am quite skeptical about the Powerset venture. The “core” of the product may be from Xerox Parc, but I’ve seen lots of people try to throw technology at search and see it come up short. My skepticism was telling me that this was to be another case.

Powerset is trying to leverage natural language processing to improve the quality of search. Rather than go for indexing tons of web pages they decided to focus on the semantics and what they could glean from Wikipedia (in one case). Half of this makes sense. The focusing on Wikipedia part. I’m completely guessing about the NLP side and from this part I’m guessing they focused too much on the NL and not enough on flat out the semantics regardless of any technique.

Anyway, so now that Powerset has launched I decided to do a 20 second test and I think many people will be surprised at the results, but not in the way you might think.

Here’s what I did. I searched for the difference between Tablet PCs and UMPCs:

(Click to enlarge)

The results were much like I expected. It’s hard to tell if any of the results targetted the query I gave.

Of course, I was being a bit unfair with my question. So I split it up into two parts. First, I asked what a Tablet PC is:

(Click to enlarge)

And then “What is a UMPC?”:

(Click to enlarge)

It’s just my opinion, but neither sets of results are that good. And what’s with the semantic summary at the top of the query results? What does it mean for UMPC features to be “system and low.” And worse, what does it mean for UMPC to “takes” “flight.” I can guess, knowing what I know about the market, but why am I guessing? I am performing the query supposedly because I don’t know the answer.

I’m not surprised by the poor quality of these results though, because Wikipedia has a small draw and being community driven it’s going to have a disproportionate voice that doesn’t “get” Microsoft’s efforts. And as a complete guess I wasn’t surprised to see the UMPC and Tablet PC to fall into this category.

So unimpressed with my 20-second Powerset search I decided to try Google. I asked it “What is the difference between a Tablet PC and a UMPC?” Not too bad. From what I see the third link is to an article entitled “How to buy a UMPC or Tablet PC”. Hmmm. That might give me a pretty good description of the differences I presume.

(Click to enlarge)

Not completely satisfied though, I decided to try Live. For the same query, here’s it’s results:

(Click to enlarge)

My. My. Look at this. The first link is to a forum post that is titled: “What is the difference between Convertable and Hybrid tablets.” Kind of close in terms of it being a comparison, but actually the link is of mediocre quality and a bit off target. I’m looking for a comparison between Tablet PCs and UMPCs.

The second link is a another so-so match. It’s titled “So what’s the difference between the Samsung Q1 and Q1B?” Both of these are UMPCs. Not qiute right.

Link 3 is getting warmer though–at least the title is more suggestive: “Define the Ultra-Mobile PC.” However, if you follow the link to Gottabemobile, it’s more about what UMPCs are and their differences with low-cost PCs, such as the Eee PC.

Scanning down the page though, you’ll see several articles with titles including the phrase “what is the difference between…”. Although none of them are exact matches, this does suggest that Live Search is placing greater sorted emphasis on content that also contains mention of at least UMPC or Tablet PC. Not bad. In fact the bottom two links on the page are “Difference between a MID vs UMPC” and “What is a UMPC.” If you read through this article, sure enough it compares in bits and pieces UMPCs to Tablet PCs. To me, although the artile is biased towards talking about a UMPC, it’s hands down a winner.

(Note: The Live query results aren’t very good if plural keywords are used, such as Tablet PCs and UMPCs. So the stemming logic in Live isn’t so hot. Not terrible. Just not as good as it could be.)

Yahoo search with the same query gives a valient effort too by strongly matching against the phrase “What is the difference between…” However I dont’ see articles that are strong matches, although I do see mentions of Tablet PCs and UMPCs which lead me to believe that with a little digging I might figure it out.

(Click to enlarge)

I also tried Ask.com, but the results weren’t that good though there is a sidebar in which you supposedly could narrow the search by clicking on “Definition of a Tablet PC” and “What is the Tablet PC used for.” Of course, the narrowing list doesn’t mention anything about UMPCs. So I left this out.

(Click to enlarge)

So, my conclusion after a couple of 20-second queries is this: I’ll stick with Google and Live with Yahoo in third place. Powerset? Well, maybe if I just want to search Wikipedia.

Was this a fair test? No. I really need to do more tests. However, it is in a domain I know something about and I’d expect any search engine to handle well. It’s not that obtuse a topic. Now, maybe I’m using the Powerset search engine “wrong” and another form of queries would do well. I’ll be watching out for the experiences of other bloggers.

An aside: Michael Arrington gives kudos to Powerset for returning good results for the query “when did earthquakes hit tokyo” and suggests that people try Google to see how good Powerset is. Well, he’s right. The results from Powerset return the first hit with “The special wards of Tokyo are as follows: ….Tokyo was hit by powerful earthquakes in 1703, 1782, 1812, 1855, and 1923.” A very good match for earthquakes in the last few hundred years. The results from Goole aren’t that spectacular. However, if you search for “wikipedia when did earthquakes hit tokyo,” you’ll be surprised. The third hit to “Dogpile” has the phrase…”Tokyo was hit by powerful earthquakes in 1703, 1782, 1812, 1855 and 1923. The 1923 earthquake , with an estimated.” This is the exact same phrase Powerset returned.

Yes, Google could tweak their results to take into account language more. You can see that in how Live and Yahoo appeared to have good results with my earlier queries. But is this a tweak to Google or a $100M business?

Update: Danny Sullivan does a much better job of explaining the potential value of Powerset. I don’t agree with him about the value of the semantic summaries (their value applies when you already know the meaning behind the sparse words) and in terms of the outline I think he’s right, it looks like there’s potential there. However, this means that the content will have to be contained within Powerset. That may work under Wikipedia’s license, but not other content. So I’m confused how far this is going to go. Now if Powerset wants to leave it at being a better host for Wikipedia content, that’s one thing. But a general search engine? That’s another.