To be clear, it could have been worse. Siri could have received an F.
Munster graded Siri against the "gold standard," Google search - using typed in queries. This was not a comparison test of Siri vs. Vlingo or some other voice assistant, but instead a test to see if Siri could effectively replace typing in queries to Google. The answer was no.
- Google understood 100 percent of the questions (considering they were typed in, that's a good thing).
- Google answered correctly 86 percent of the time.
- Siri understands 83 percent of queries in noisy conditions, 89 percent in a quiet room.
- Siri answers accurately 62 percent of the time in noisy conditions. In a quiet room, it answers correctly 68 percent of the time.
That is precisely what Google is afraid of, that Siri could eventually replace it.
Munster believes Siri has two years of "learning" to catch up to Google. During that time, it's likely Google will have its own competitor. He added, "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google from 60 percent to 48 percent."
Currently, Siri gets 60 percent of its answers from Google, 20 percent from Yelp, 14 percent from WolframAlpha, 4 percent from Yahoo and 2 percent from Wikipedia. Perhaps Apple should increase Siri's reliance on Google, rather than decreasing it.
In order to show the errors that Siri most commonly makes, Munster provided some sample questions and failures:
- What team does Peyton Manning play for? Responded with the answer to the previous query. This was the most common error, which constitutes a simple bug, rather than gathering erroneous data.
- Where is Elvis buried? Responded I can't answer that for you. It thought the persons name was Elvis Buried.
- Where am I? Pin dropped in the wrong place.
- When did the movie Cinderella come out? Responded with a movie theater search on Yelp.
- How do I get from Boston to New York? Responded I can only give directions from your current location. I can't give you directions to a place you are not in (feature request)
- What spices are in Lasagna? Responded with a Yelp search with lasagna on the menu of the restaurant results.
- When is the next Haley's comment? Responded "You have no meetings matching Haley's"
- I want to go to Lake Superior? Responded with directions to the company Lake Superior X-Ray.
While Apple has been sued over Siri's lack of accuracy, it has been quite clear that Siri is still in beta.
How much it improves between now and when it is finally "release quality" remains to be seen. Siri will expand to the new iPad when iOS 6 is released; currently it is limited to the iPhone 4S.