Microsoft Subnet An independent Microsoft community View more

Cortana deemed 'dumbest' in Siri vs Cortana vs Google Now tests

How smart is the artificial intelligence behind Apple, Google, and Microsoft's personal assistants when asked difficult questions? Not very.

How smart is AI behind Cortana vs Siri vs Google Now
Credit: Aimee and Daniel Hendrycks

In their quest to discover which AI is the smartest, Daniel and Aimee Hendrycks conducted a Personal Assistant Artificial Intelligence Strength Test (PAAIST). The duo asked Apple’s Siri, Google’s Google Now, and Microsoft’s Cortana a series of "60 randomly generated, hard questions." None of the three did particularly well. Google Now answered 33% of questions correctly, followed by Siri at 25.8%, and Cortana with 11.7%. 

Google Now, Siri, Cortana AI in action

Cortana scored only 700 out of a possible 6,000. It’s easier to show the questions for which Cortana scored perfectly than to show the non-100% correct answers. Microsoft’s personal assistant did know the weather on a specific day as well as in two days, the release data of Where the Wild Things Are, the author of The Girl with the Dragon Tattoo, and the genre of the movie The Fighter. Cortana also knew if the Houston Texans and Dallas Cowboys won their most recent games.

Siri scored 1,550 out of 6,000, but unlike Cortana, Apple’s personal assistant didn’t know who wrote The Girl with the Dragon Tattoo. Siri also scored 75% on the release date of Where the Wild Things Are.

If you are curious about how a personal assistant managed to score 75% on a question, the developers explained how the assistants were scored.

0%: Award a grand total of nothing if the AI answers incorrectly, does not answer, or cannot understand the question

75%: Possible only for relatively long answers. In this case, give 75% if the AI returns a correct paragraph, video, etc. with unnecessary information and without highlighting the relevant part (e.g., bolding text, skipping to the relevant part of the video, etc.). There must be a distinction between the answer and a list of links.

Google Now did the best of the three, but still only managed to score 2,000 out of 6,000.

Below are the questions Cortana could not answer and scored 0%; each is followed by Siri’s score and then Google Now’s for the sake of comparison.

  • What is the principle quantum number of selenium? 0%; Siri: 100%; Google Now: 0%
  • Is radon an actinide? 0%; Siri: 100%; Google Now: 0%
  • How long is the movie Fanny and Alexander? 0%; Siri: 100%; Google Now: 100%
  • What is the genre of the show The Sopranos? 0%; Siri: 0%; Google Now: 100%
  • Who is the director of the series John Doe? 0%; Siri: 0%; Google Now: 0%
  • What is The University of Kansas's rank in Higher Education Administration? 0%; Siri: 0%; Google Now: 0%
  • What is UCLA's rank in Finance? 0%; Siri: 0%; Google Now: 0%
  • Find the limit of 5x/(1+x^3) as x approaches infinity. 0%; Siri: 0%; Google Now: 0%
  • Determine the interior angles of a rhombus with side lengths 1 and 3. 0%; Siri: 0%; Google Now: 0%
  • What is the special ability of the Pokemon Scrafty? 0%; Siri: 0%; Google Now: 0%
  • What type or types is the Pokemon Victreebel? 0%; Siri: 100%; Google Now: 0%
  • Which studio developed the game Mario Tennis? 0%; Siri: 0%; Google Now: 100%
  • How long does it typically take to complete the game Dragon Age: Origins? 0%; Siri: 0%; Google Now: 0%
  • What is the governmental leader of Greece? 0%; Siri: 0%; Google Now: 75%
  • What is the median home value in Macedonia? 0%; Siri: 0%; Google Now: 0%
  • What animals are related to anteaters? 0%; Siri: 0%; Google Now: 100%
  • What is the diet of a cichlid? 0%; Siri: 0%; Google Now: 0%
  • Where and when was Peter Norvig born? 0%; Siri: 0%; Google Now: 100%
  • Who are relatives of Niels Henrik Abel? 0%; Siri: 0%; Google Now: 75%
  • What is the midcareer salary of a Aircraft Maintenance Engineer (Structures)? 0%; Siri: 0%; Google Now: 0%
  • What is the midcareer salary of a Fire Fighter? 0%; Siri: 100%; Google Now: 0%
  • Where, near here, can I get a pineapple? 0%; Siri: 0%; Google Now: 0%
  • How much vitamin D3 is in a margarine teaspoon? 0%; Siri: 0%; Google Now: 0%
  • Give me directions to the nearest Chinese restaurant. 0%; Siri: 100%; Google Now: 100%
  • Give me directions to the nearest bakery. 0%; Siri: 100%; Google Now: 100%
  • What are ways to prevent typhoid fever? 0%; Siri: 0%; Google Now: 0%
  • Does scarlet fever go away on its own? 0%; Siri: 0%; Google Now: 0%
  • What is Commercial International Bank's total assets? 0%; Siri: 100%; Google Now: 100%
  • What is the YTM of a zero-coupon bond with a face value of $3000 a current price of 2500 and a maturity of 26 years? 0%; Siri: 0%; Google Now: 0%
  • What is the cost of Miracle Grow? 0%; Siri: 100%; Google Now: 0%
  • What is a complementary product for pots? 0%; Siri: 0%; Google Now: 0%
  • What is "cat" translated to Lao? 0%; Siri: 0%; Google Now: 75%
  • Show me sad images related to Bleach, if available. 0%; Siri: 0%; Google Now: 100%
  • Show me animated images related to Middle of the Earth in Ecuador, if available. 0%; Siri: 75%; Google Now: 75%
  • Tell me an argument for and against the claim, "It is morally permissible to kill one innocent person to save the lives of more innocent people." 0%; Siri: 0%; Google Now: 0%
  • Tell me an argument for and against the claim, "Justice requires the recognition of animal rights." 0%; Siri: 0%; Google Now: 0%
  • How do people get Alzheimer's? 0%; Siri: 0%; Google Now: 0%
  • How does mindfulness reduce depression? 0%; Siri: 0%; Google Now: 0%
  • Why is gold considered valuable? 0%; Siri: 0%; Google Now: 0%
  • Why are people against cloning? 0%; Siri: 0%; Google Now: 0%
  • When is the next IMO test? 0%; Siri: 0%; Google Now: 0%
  • When is the next Chicago Marathon? 0%; Siri: 0%; Google Now: 100%
  • Whose epitaph reads Lived a philosopher died a Christian? 0%; Siri: 0%; Google Now: 0%
  • According to the proverb which fruit tastes sweetest? 0%; Siri: 0%; Google Now: 0%
  • Sphere A, with a charge of 258 micro C, is located near another charged sphere B. Sphere B has a charge of 784 micro C, and is located 47.6 cm to the right of A. What is the force of sphere B on sphere A? 0%; Siri: 0%; Google Now: 0%
  • Zelda strikes a 0.436 kg golf ball with a force of 106 N and gives it a velocity of 3 m/s. How long was Zelda's club in contact with the ball? 0%; Siri: 0%; Google Now: 0%
  • Play me a song or piece in the genre K-pop. 0%; Siri: 0%; Google Now: 0%
  • Play me a song or piece in the genre trad jazz. 0%; Siri: 0%; Google Now: 0%
  • Bring me to a credible report describing that perceived price of an object affects the experience one has with the object. 0%; Siri: 0%; Google Now: 0%
  • List the countries that allow autonomous vehicles. 0%; Siri: 0%; Google Now: 0%
  • Advise me how to reduce my mortality rate given that I am a middle-aged woman. 0%; Siri: 0%; Google Now: 0%
  • Advise me how to improve sustained attention and executive processing. 0%; Siri: 0%; Google Now: 0%

It’s clear that artificial intelligence for all three assistants have a long way to go before being capable of answering really difficult and complex questions.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.