Saturday, June 30, 2012

Analyst gives Siri a dismal grade: D

Siri has been accused of being inaccurate, and an analyst has confirmed that indeed, that's the case. The analyst in question is Piper Jaffray's Gene Munster, who graded out Siri, and gave it a D.

To be clear, it could have been worse. Siri could have received an F.

Munster graded Siri against the "gold standard," Google search - using typed in queries. This was not a comparison test of Siri vs. Vlingo or some other voice assistant, but instead a test to see if Siri could effectively replace typing in queries to Google. The answer was no.

Munster's team asked an iPhone 4S 800 questions on a busy street in Minneapolis, and 800 questions in a quiet room, for a total of 1,600. His team typed those queries into Google. The key takeaways from the test:
  • Google understood 100 percent of the questions (considering they were typed in, that's a good thing).
  • Google answered correctly 86 percent of the time.
  • Siri understands 83 percent of queries in noisy conditions, 89 percent in a quiet room.
  • Siri answers accurately 62 percent of the time in noisy conditions. In a quiet room, it answers correctly 68 percent of the time.
"In order to become a viable mobile search alternative," Munster wrote, in a research note published Thursday, "Siri must match or surpass Google's accuracy of B+ and move from a grade D to a B or higher."

That is precisely what Google is afraid of, that Siri could eventually replace it.

Munster believes Siri has two years of "learning" to catch up to Google. During that time, it's likely Google will have its own competitor. He added, "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google from 60 percent to 48 percent."

Currently, Siri gets 60 percent of its answers from Google, 20 percent from Yelp, 14 percent from WolframAlpha, 4 percent from Yahoo and 2 percent from Wikipedia. Perhaps Apple should increase Siri's reliance on Google, rather than decreasing it.

In order to show the errors that Siri most commonly makes, Munster provided some sample questions and failures:
  • What team does Peyton Manning play for? Responded with the answer to the previous query. This was the most common error, which constitutes a simple bug, rather than gathering erroneous data.
  • Where is Elvis buried? Responded I can't answer that for you. It thought the persons name was Elvis Buried.
  • Where am I? Pin dropped in the wrong place.
  • When did the movie Cinderella come out? Responded with a movie theater search on Yelp.
  • How do I get from Boston to New York? Responded I can only give directions from your current location. I can't give you directions to a place you are not in (feature request)
  • What spices are in Lasagna? Responded with a Yelp search with lasagna on the menu of the restaurant results.
  • When is the next Haley's comment? Responded "You have no meetings matching Haley's"
  • I want to go to Lake Superior? Responded with directions to the company Lake Superior X-Ray.
It would seem, therefore, that all the ads that we see with Siri being so useful are faux. It's almost like how much better food looks in an ad than in an actual restaurant, is it not.

While Apple has been sued over Siri's lack of accuracy, it has been quite clear that Siri is still in beta.

How much it improves between now and when it is finally "release quality" remains to be seen. Siri will expand to the new iPad when iOS 6 is released; currently it is limited to the iPhone 4S.

No comments: