For many people, the first instinct when facing common health symptoms like a runny nose or a persistent cough is to turn to the internet for answers. Rather than booking an appointment with their doctor, they often rely on search engines like Google or virtual self-diagnosis tools like WebMD to get a quick assessment of their condition. But with the rapid advancements in artificial intelligence, some may now be tempted to seek out a new online "doctor" - ChatGPT.

Researchers from Western University sought to explore whether OpenAI's popular AI-powered chatbot could provide accurate and reliable medical advice, potentially replacing the ubiquitous "Dr. Google." The study, led by Dr. Amrit Kirpalani from the Schulich School of Medicine & Dentistry, has recently been published in the journal PLOS One, and the findings offer a mixed outlook on the capabilities of ChatGPT in the healthcare realm.

The research team, which included third-year medical students Ali Hadi, Edward Tran and Branavan Nagarajan, set out to put ChatGPT to the test. They asked the AI chatbot to diagnose 150 complex medical cases from the Medscape Clinical Challenges platform, which is designed to assess the diagnostic skills of healthcare professionals. The researchers created prompts that provided ChatGPT with detailed patient histories, physical examination results, and laboratory or imaging test data, and then asked the chatbot to select the correct diagnosis from a multiple-choice list and explain its reasoning.

The results were, perhaps, not entirely surprising. ChatGPT was only able to accurately diagnose 49% of the cases, a far cry from the level of accuracy one would expect from a qualified medical professional. The researchers found that the AI chatbot struggled to interpret test results and often overlooked critical information that would be essential for making an accurate diagnosis.

"To me, the most relevant finding is that ChatGPT delivered its answers in a very simple and easy-to-understand way," said Kirpalani. "I think that's important because you can see the potential for it to be used as a great tool to help people learn and understand medical cases."

While ChatGPT may not be ready to replace a visit to the doctor's office, the researchers believe the AI chatbot could still have a role to play in healthcare and medical education. By synthesizing complex medical topics into easy-to-understand language, ChatGPT could potentially be leveraged by instructors and healthcare providers to deliver information in a more digestible format for patients and students.

However, Kirpalani emphasizes the importance of maintaining a critical eye and not blindly relying on AI-powered tools for medical advice. "Regardless of the accuracy of these online resources, we need to evaluate and double check responses against reliable, peer-reviewed sources," he said.

The study's findings highlight the need for continued research and advancements in the field of AI-powered medical diagnostics. As new models like ChatGPT continue to evolve, Kirpalani stresses the importance of developing "AI literacy" among patients, healthcare providers, educators, and students.

"We are going to need a lot of oversight on how it's being used to ensure patient safety and to make sure [this kind of AI technology] will be thoughtfully rolled out," Kirpalani said.

While the potential of AI-powered chatbots in healthcare is undeniable, the Western University study serves as a cautionary tale. ChatGPT may be a helpful tool for learning and understanding medical information, but it is not yet ready to be relied upon as a reliable diagnostic tool, particularly for complex health conditions. As the field of AI continues to advance, striking the right balance between the benefits and the risks will be crucial in ensuring that these technologies are used responsibly and safely in the healthcare sector.