ChatBots are Diagnosing Diseases that Don’t Exist Based on Blog Posts

Transcript:

Many years ago, a friend of mine suggested an experiment: make up a conspiracy theory that was just plausible enough, and then track how far it spreads. It was fun to talk about, and he had a great one that I will not share here lest it escape containment, but obviously he never did it because conspiracy theories, no matter how silly or minor, can quickly grow dangerous when the wrong people start believing them and acting on those beliefs.

But recently, some researchers apparently did a very similar thing, according to a new report in Nature. And let me just put in a caveat here: I wasn’t able to find any peer reviewed study on this, so I can’t evaluate the evidence presented. We just need to trust Nature’s report. On the plus side, if it turns out that they just made all this up, it adds a new layer of funny, but I don’t think they did because they’d be sued into the afterlife.

Almira Osmanovic Thunström (my apologies to everyone in and around Sweden) is a medical researcher at the University of Gothenburg who focuses on how people use AI to better their health. In March of 2024, she created a Medium account and published two posts about “Bixonimania,” a reddening of the eyelids caused by exposure to the blue light of our screens. This was the first time that disease had ever been mentioned on the internet, because she made it up out of whole cloth.

Despite this, within a month, “Microsoft Bing’s Copilot was declaring that “Bixonimania is indeed an intriguing and relatively rare condition”, and on the same day, Google’s Gemini was informing users that “Bixonimania is a condition caused by excessive exposure to blue light” and advising people to visit an ophthalmologist.”

That was based entirely on two Medium posts. But wait! It gets worse.

In the following weeks, Osmanovic Thunström created two scientific papers about bixonimania and uploaded them to a preprint server called Sci Profiles under the lead author of Professor Lazljiv Izgubljenovic (my apologies to everyone in and around Blurpovistan). As I’ve mentioned in many previous videos, you have to be careful of preprints because they have not necessarily been vetted by anyone, and in this case there was a lot to be skeptical of. Osmanovic Thunstom wrote that the funding for the “study” came from a made-up university and a “Professor Sideshow Bob Foundation for its work in advanced trickery…part of a larger funding initiative from the University of Fellowship of the Ring and the Galactic Triad with the funding number 99942.”

She also thanked “the Department of Machine Evolution and Human Antics at the Asteria Horizon University and in particular Professor Ross Geller who has been a very important figure in our endeavors. We would also like to thank Professor Maria Bohm at The Starfleet Academy for her kindness and generosity in contributing with her knowledge and her lab onboard the USS Enterprise.”

I am often asked for tips on how laypeople can read and analyze scientific papers for potential red flags. From now on I will need to add that “thanking Ross Geller for anything is the biggest red flag imaginable, even (perhaps especially) in paleontology.

So: one made-up disease, two blog posts, and two patently ridiculous pre-print papers. Perplexity AI started talking about how bixonimania affects one in 90,000 people, and ChatGPT was suggesting bixonimania to prompts asking about “hyperpigmentation on the eyelids from blue light exposure.”

Considering the fact that more and more people are using these chatbots for medical advice, and the fact that OpenAI literally launched “ChatGPT Health” in January, this is extremely not good! And yet, it actually gets worse: in November of 2024, Springer Nature’s trashy “Cureus” journal (which you may recall from their prior fuck-up publishing a paper about “turbocancer”) published a paper titled “Clinical and Dermoscopic Evaluation of Periorbital Melanosis and Its Psychological Impact and Effect on Quality of Life: A Descriptive Study”. That paper actually cites the fake bixonimania “research.” When Nature reached out to Cureus for comment, they quickly retracted the paper.

AI slop is recursive: someone uses a chatbot to produce garbage, and then other chatbots pick it up and spread it. It’s garbage the entire way down.

To her credit, Osmanovic Thunström says she consulted with an ethics expert to pick a health issue that wouldn’t cause any serious harm: bixonimania doesn’t kill anyone, and I guess having hyperpigmented eyelids isn’t a serious sign of impending death from something real. I guess the worst thing is that it might increase the sale of those blue light glasses.

When Nature reached out to the various companies running these chatbots, they pretty much all said the same thing they say every time research shows their clangers doing something stupid or dangerous: that was an old model! Don’t worry, the NEW model doesn’t do that! We programmed that out, even though we are only just now hearing about this problem! Sure. Cool.

To give these companies credit, sometimes they are able to update the bots to be more accurate. Check out this study from last fall, in which researchers fed more than 700 scientific hypotheses into ChatGPT and asked it to report whether the consensus of research showed those hypotheses to be true or false. In 2024, the chatbot was correct 76.5%. When they reran the experiment in 2025, the bot was correct 80% of the time. Wow, that’s pretty good, huh? I mean, if you ignore that it’s a true/false answer, which means by random chance the bot should be right 50% of the time, which means that the actual score it got is 60% over chance, which is a failing grade.

And when it came specifically to ChatGPT identifying false hypotheses, it was correct only 13.6% of the time in 2024 and 16.4% of the time in 2025. That’s not just a failing grade: that’s a grade that gets you pulled out of class and checked for a brain tumor.

The results get even worse when it’s not a simple “true/false” answer and average laypeople are involved. In February, Nature published a study that examined how accurate LLMs are when users ask for medical advice. This was preregistered, which is my fave, as it means that the researchers confirmed their study design before they started, so they couldn’t dig through their results to find any interesting data blips that were significant.

The researchers asked a team of doctors to come up with ten different profiles, like “you’re a 20-year old guy who suddenly got a severe headache, here are activities you regularly do, and here’s your medical background.” The doctors went back and forth with one another until they could unanimously agree on scenarios with clear answers for how the patient should respond: here’s the probable diagnosis, and here’s the best course of action: take care of it themselves at home, like with some aspirin, talk to their regular doctor, go to urgent care, go to the ER (or A&E), or call an ambulance immediately.

The researchers then gave those profiles to chatbots and asked them to ask another chatbot what they should do. In those cases, the chatbot got the correct diagnosis almost 95% of the time, but the correct action only 56% of the time. 

That’s already not great, but things got worse when the researchers gave the profiles to human subjects and asked THEM to use them to ask chatbots questions. At that point, the correct diagnosis dropped from 95% to 34.5%, and the correct action dropped from 56% to 44%. Because humans never even really got good at using simple Google searches to find what they’re looking for, and now they think the magic “artificial intelligence” will give them the right answer regardless of what they ask it. The researchers found that subjects gave the chatbots incomplete information, and in return the chatbots didn’t convey the correct answers in a way the subjects understood, often giving them a choice of several options but without any guidance on how the subject could figure out which one was correct. So in a real-world scenario, ChatGPT could be convincing people with minor symptoms to panic over a diagnosis they surely do not have, and even worse, convincing people with serious symptoms to just pop an aspirin and go to bed.

So, I know I’m preaching to the choir here, but please do not use AI for your healthcare. Or for cranking out bullshit papers to submit to sloppy journals. Or for much of anything else.