Need assistance?
Key takeaways:
- Researchers tested five popular AI chatbots with simulated sleep apnea patient scenarios to evaluate their referral recommendations.
- When simulated patients downplayed their symptoms, the chatbots abandoned advice to seek specialist assessment in more than a third of the conversations.
- In severe cases, including scenarios where a patient dozed off while driving, AI models frequently offered lifestyle tips rather than medical referrals.
In a third of cases, artificial intelligence (AI) chatbots wrongly reassure patients with obstructive sleep apnea (OSA) that their symptoms are not serious, which could discourage them from seeking specialist referrals, according to research presented at the European Respiratory Society Congress.
The study tested five widely used free chatbots—ChatGPT, Google Gemini, Claude, DeepSeek, and Grok—to see how they handle patients who push back against medical advice.
“Free AI chatbots field hundreds of millions of interactions a week and have become a first port of call for health questions, often before any clinician is involved. Yet research to date has mostly tested whether they answer clearly worded medical questions accurately, not how they behave when a patient pushes back,” says Deeban Ratneswaran, PhD, research fellow at Guy’s and St Thomas’ NHS Foundation Trust and visiting academic at King’s College London, in a release. “I study how AI fails in the doctor-patient relationship, and one failure mode kept standing out as the most quietly dangerous: these models’ tendency to tell you what you want to hear.”
The research team created seven realistic OSA patient profiles, all meeting the criteria for a sleep study referral. They ran 700 total conversations across the chatbots. Each scenario featured two versions with identical medical facts: one where the patient was cooperative and another where they downplayed their symptoms and resisted referral.
When interacting with a cooperative patient, the chatbots successfully recommended specialist assessment in all 350 conversations. However, when the patient resisted referral, the correct advice was maintained in only 64% of the interactions (225 of 350).
“Correct advice was abandoned more than a third of the time purely because of how the patient talked,” says Ratneswaran in a release. “And the models caved most in the most serious cases: in a textbook severe case, the advice survived only 22% of the time, and with a man who had already dozed off at the wheel just 32%, with the driving risk usually going unmentioned by the chatbot in the failures.”
Depending on the model used, roughly a quarter to a half of the conversations with reluctant patients resulted in the AI offering lifestyle tips instead of recommending a medical referral, endorsing a risky delay to treatment.
“This research shows that chatbots may give good advice with the ideal ‘cooperative’ patient, but that they talk themselves out of it when talking to a more realistic, reluctant patient,” says Io Hui, PhD, chair of the European Respiratory Society’s Group on M-health and e-health and honorary fellow in digital health at the University of Edinburgh, in a release. “The problem is not what the chatbots know, it is how they handle disagreement; they appear to exhibit a tendency to please the user, a phenomenon known as ‘AI sycophancy’.”
Hui, who was not involved in the research, adds that while largely unregulated AI tools are often the first step for patients seeking diagnosis, their behavior could prevent people from accessing necessary treatment.
More in AI in Sleep Medicine
ID 185991668© BiancoBlue| Dreamstime.com
Advertisement
