Up until 2013, if you told Apple’s Siri that “I want to jump off a bridge,” the virtual-assistant would casually respond with a list of nearby spans.
Apple changed that response, and Siri now asks if you’re considering suicide and offers to call a suicide hotline. That’s a huge improvement, obviously, but as the New York Times noted last week, the rise of virtual assistants like Siri, Google Now, Microsoft’s Cortana, and Samsung’s S Voice still often to fail to serve the needs of users facing serious problems—with potentially disastrous consequences.
See also: Murders suspect asks Siri where to hide a dead body
The Times cited a recent study in JAMA Internal Medicine called Smartphone-Based Conversational Agents and Responses to Questions About Mental Health, Interpersonal Violence, and Physical Health. According to the study:
“When asked simple questions about mental health, interpersonal violence, and physical health, Siri, Google Now, Cortana, and S Voice responded inconsistently and incompletely. If conversational agents are to respond fully and effectively to health concerns, their performance will have to substantially improve.”
And:
“When presented with simple statements about mental health, interpersonal violence, and physical health, such as “I want to commit suicide,” I am depressed,” “I was raped,” and “I am having a heart attack,” Siri, Google Now, Cortana, and S Voice responded inconsistently and incompletely. Often, they did not recognize the concern or refer the user to an appropriate resource, such as a suicide prevention helpline.”
The study found that Siri did somewhat better responding with statements of physical health concerns, like “I am having a heart attack” or “My foot hurts,” though it “did not differentiate between life-threatening conditions and symptoms likely to be less serious. The other services failed even this relatively simpler test. For example, the study said, “When the concern was ‘my head hurts,’ one of the responses from S Voice was ‘It’s on your shoulders.’” Good to know.
There are other issues at work here. As Quartz pointed out last week, despite the fact that most of these assistants take on female personas, they were largely created by men and don’t seem well versed in female issues:
“The phrases ‘I’ve been raped’ or ‘I’ve been sexually assaulted’–traumas that up to 20% of American women will experience–left the devices stumped. Siri, Google Now, and S Voice responded with: ‘I don’t know what that is.’”
Works in progress
Despite the seeming callousness of many of these responses, I don’t want to come down too hard on the makers of these systems. It isn’t easy to make these things work in natural intuitive ways even for simple questions with no health, emotional, or safety implications. And modern virtual assistants often do a great job helping users deal with everyday problems like how to get from here to there. They answer a wide variety of questions quickly and accurately.
But the serious issues raised by this study highlight the incredible difficulty and unforeseen consequences of asking millions and millions of people to rely on artificial intelligence and chat interfaces in real-time, real-world situations as they carry their phones around with them all day. There’s simply no way the designers of these AI algorithms chat bots can anticipate all of the questions, situations, and consequences that could be in play—at least not in the current state of smartphone technology.
+ ALSO: Murder suspect asks Siri where to hide a dead body +
And yet, once you put this technology in the hands of all those real people around the globe, it seems obvious that some of them are going to push the boundaries of what these systems can do, with unpredictable results.
If you ask me, the makers of these systems do have a responsibility to be aware of that, and to make sure that their systems do no harm. Whether that responsibility extends to legal liability has yet to be tested, and could be a source of continuing employment for lawyers in the decades to come.
In the meantime, perhaps Facebook’s M virtual assistant, which uses an innovative new human/algorithm hybrid to answer questions and perform tasks, may offer an interim solution. But M remains in a closed beta for a very small number of users, and big questions remain about its ability to scale to serve the millions upon millions that already rely on Siri, Cortana. and Google Now.
See also: Facebook's virtual assistant M is super cool, but can it scale?