We fought the bots and the…bots won! 😉 Apparently, ChatGPT defeated doctors at diagnosing illness, even when the doctors used ChatGPT!

But first: We Need Your Help! The eDiscovery Today State of the Industry Report survey is currently live, with 12 questions about the state of the eDiscovery industry, including questions on generative AI, technology assisted review (TAR), mobile device and collaboration app discovery, eDiscovery use cases, hyperlinked files, and more! Takes two minutes to fill out. Please check it out here – doing so will get you a FREE copy of the report when it’s published in January.

So much for “human-in-the-loop”! According to Newser (ChatGPT Defeated Doctors at Diagnosing Illness, written by Jenn Gidman and available here), research published last month in the JAMA Network Open journal, which details an experiment involving 50 doctors, half paired with a ChatGPT-4 assistant from OpenAI and the other half without. Those without relied on conventional diagnostic methods, including Google and medical reference sites like UpToDate, per a release.

The bots alone were also given the opportunity to read the six case studies on hand and offer their diagnosis and reasoning for coming to their conclusion. Those who graded the entries had no clue if ChatGPT had any involvement. The results: The bots correctly identified the medical condition in question 90% of the time. However, there wasn’t a significant difference between the human doctors who used AI and those who didn’t, scoring 76% and 74%, respectively. “The LLM [large language model] alone outperformed physicians even when the LLM was available to them, indicating that further development in human-computer interactions is needed to realize the potential of AI in clinical decision support systems,” the researchers note.

Per the New York Times, the case studies in question, which were based on actual patients, had never been published before—meaning ChatGPT couldn’t have trained on them and was seeing them for the first time, just like the human doctors.

So why did doctors who used ChatGPT not have similar results to the bots working alone? The researchers note that many of the doctors may not have known how to fully maximize AI’s full capabilities—but also, they say that humans can be stubborn when it comes to their own opinions. “They didn’t listen to AI when AI told them things they didn’t agree with,” study co-author Adam Rodman tells the Times. Laura Zwaan of Erasmus Medical Center, who wasn’t involved with the study, concurs, noting, “People generally are overconfident when they think they are right.”

Gee, usually doctors are so humble! 😉 Essentially, each of the doctors should be singing: “We fought the bots and the…bots won!” 🤣 And apparently, sometimes, humans should stay out of the loop!

So, what do you think? Are you surprised that ChatGPT defeated doctors at diagnosing illness, even when the doctors used ChatGPT? Please share any comments you might have or if you’d like to know more about a particular topic.

Image created using GPT-4o’s Image Creator Powered by DALL-E, using the term “robot doctor looking at an email on a computer”.

Disclaimer: The views represented herein are exclusively the views of the authors and speakers themselves, and do not necessarily represent the views held by my employer, my partners or my clients. eDiscovery Today is made available solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Today should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.