toggle

AAPL Stock: 111.78 ( -0.87 )

Printed from http://www.macnn.com

Siri tests suggest Google text search still best option

updated 10:51 am EDT, Fri June 29, 2012

Voice tech does only modestly better indoors than out

Apple's Siri voice command system is still probably an inferior choice for searches versus Google text entries, according to a study by researchers at Piper Jaffray. Analyst Gene Munster and his team tested Siri first with 800 queries in a quiet room, without using headphones, and then with another 800 queries out on a busy Minneapolis street, but using Skull Candy mic-equipped headphones. Indoors Siri understood 89 percent of queries, accurately answering 68 percent of questions. Outdoors it faired slightly worse, with rates of 83 and 62 percent, respectively.

By comparison, Google text searches returned answers with 86 percent accuracy. "We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app," Munster tells AppleInsider.

Some of the questions that confused Siri include "What team does Peyton Manning play for," "Where is Elvis buried," "Where am I," "When did the movie Cinderella come out," and "How do I get from Boston to New York." Some of these have exposed fundamental problems with the technology. With the Elvis question, for instance, Siri would search for a person named "Elvis Buried." Asking "where am I" would sometimes drop a map pin in the wrong place, and the Boston-to-New York question prompted Siri to answer with "I can only give directions from your current location. I can't give you directions to a place you are not in."

An irony is that Siri is thought to rely on Google for about 60 percent of its answers. Munster suggests that this ratio will drop to 48 percent with the launch of iOS 6, since Apple is moving away from Google for both mapping data and sports scores. "While Siri is two-plus years behind Google today, we are optimistic about its future," says Munster. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."




by MacNN Staff

POST TOOLS:

TAGS :

toggle

Comments

  1. facebook_Matt

    Via Facebook

    Joined: Jun 2012

    -2

    My favorite Siri fail

    This one happened to me a few days ago when trying to mail a package.. https://twitter.com/NerdMattBaker/status/214036290985930752/photo/1/large

  1. climacs

    Joined: Dec 1969

    -1

    not ready for prime time

    I can't help but assume that Siri was rolled out in order to give Apple a reason to sell a new revision of iPhone 4 because iPhone 5 was not ready. I also believe Jobs would never have allowed it to be released in its current form.

    Eventually it will be a significant technology, but clearly it's not ready yet. Half the time it's useful, the other half of the time it's frustrating, it doesn't really help, or I still end up having to finish manually the job it started.

    I am also disappointed that if you change Siri's voice to the UK or Australian version (the Australian Siri has a much more pleasing voice), you can't get US map results unless you switch back. WHY?

  1. coffeetime

    Joined: Dec 1969

    +5

    unfair comparison

    Why in world did they compare Siri with Google text search? It should be Siri vs. Google Voice Input. Will then Siri outperform others based on today's best technology that you can get. And how long Google Search had been on the market vs. Siri? Not fair!

  1. exca1ibur

    Joined: Dec 1969

    +5

    Indexing

    Google has over 15 years of page indexing under its belt. I better be more accurate

  1. FilmmakingFiasco

    Joined: Dec 1969

    +6

    comment title

    Siri will be greatly improved (especially for movies and sports) with iOS 6. Additional if you just preface 'Search the web for' in front of your query, you'll have as accurate results as it goes through your Safari search engine.

  1. facebook_Adam

    Via Facebook

    Joined: Jun 2012

    +9

    comment title

    I'm not sure why people seem to think that Steve Jobs would have not permitted Siri's release. Steve DID allow Siri's release. Do you honestly think for a second that Apple simply tossed in Siri several weeks prior to release? Steve was still in the position to influence direction right up until his passing. Siri was his purchase and he very much believed it's strategic importance.

    Also, I don't think Siri was intended to be used as a knowledge mining search tool. Siri is an assistant that helps us to complete tasks and it does quite well with that.

  1. TomMcIn

    Joined: Dec 1969

    +4

    The Obvious

    While this article is able to compare two technologies, it is really comparing "Apples to Oranges". Having all Google searches typed in while Siri used voice recognition as well as being front end to various search engines leaves too many uncontrolled variables. Comparing typed Google searches to Siri using just Google on the same searches would have been one option to compare the accuracy of the voice recognition. What we have in this result is a mish mash that does not separate Siri's voice recognition from Siri's use of multiple search engines. Rerunning the tests with the next version of Siri should be useful. I expect the Apple people have done similar studies on their own. All in all - a D grade article.
    And what would the whiners about Siri not being ready recommend. Developers need real world testing and this is what Apple is getting. If you don't want to help, don't use it, or at least remain silent.

  1. climacs

    Joined: Dec 1969

    -2

    @FB Adam

    Adam, I'm about as big an Apple fan as you can find, and I still assert that Siri was not ready for prime time and should not have been released in its current state. At the very least, if it needed to be beta-tested in the wild (certainly useful), it should not have been pitched as the Next Greatest Thing as it was, because it really isn't all that.

    Siri frequently has issues with company names and last names in my Contacts. So I have to manually go into Contacts to dial up our property management company or some of my friends, because Siri just does not know what the h*** I am talking about. No, I do not have a thick regional dialect and no I am not talking about obscure foreign names.

    Dictating emails and texts via Siri is fine so long as you get it right the first time or don't use the sorts of words and nouns I described above which Siri has issues with. If there's a way to get Siri to correct an email/text - let's say I wanted to strike the 2nd sentence or replace a couple words - I'd love to know how that works.

    It's useful for some stuff and I look forward to it developing and growing into a truly useful personal assistant. In its current state, it's still crawling but it was pitched to us as if it can walk and even run.

  1. DiabloConQueso

    Joined: Dec 1969

    +4

    Why?

    ...and then with another 800 queries out on a busy Minneapolis street, but using Skull Candy mic-equipped headphones.

    Would this bypass the dual-mic, noise-filtering technology that the iPhone 4S has? If so, could that have attributed to some of the inaccuracies -- maybe because by using an external mic, the recognition was marred by background noise?

    Maybe? Or does the iPhone 4S still use the noise-filtering/cancelling mic even when an external mic is used?

  1. testudo

    Joined: Dec 1969

    -2

    @climacs

    No one has stated siri is ready for prime-time (with the exception, maybe, of Apple, although they label it as 'beta', which should tell you something).

    Half the stuff I asked siri always came back with "Would you like me to look up on the web for..." questions anyway.

Login Here

Not a member of the MacNN forums? Register now for free.

toggle

Network Headlines

toggle

Most Popular

MacNN Sponsor

Recent Reviews

Dell AD211 Bluetooth speaker

For all of the high-priced, over-engineered Bluetooth speakers in the electronics market, there is still room for mass-market solution ...

VisionTek 128GB USB Pocket SSD

USB flash drives dealt the death blow to both the floppy and Zip drives. While still faster than either of the old removable media, sp ...

Kodak PixPro SL10 Smart Lens Camera

Smartphone imagery still widely varies. Large Megapixel counts don't make for a good image, and the optics in some devices are lackin ...

toggle

Most Commented