AI told some people they may have ‘bixonimania.’ The disease doesn’t exist

本站收录: 2026-05-22

[00:00:12]For Scientific American Science Quickly, I'm Rachel Feltman.

[00:00:21]Have your eyes ever felt sore and itchy after spending too much time staring at a screen? You might have a condition known as Bixonomia or at least that's what several popular AI-powered chatbots might have told you if you'd asked last year.

[00:00:38]Millions of people around the world turn to AI chatbots for medical advice every day. Often as a supplement to a doctor's visit, but also sometimes in place of it. That can lead to dangerous consequences and in rare cases even death.

[00:00:54]Our guest today is Elvira Osmancevic Twinstrom. She's a researcher at the University of Gothenburg in Sweden and at the Sahlgrenska University Hospital, Center for Digital Health and Chalmers Industriteknik.

[00:01:06]She's also the creator of Bixonomia.

[00:01:09]She says this totally made-up disease reveals some very real problems with the way we train and use large language models.

[00:01:17]Thank you so much for coming on to chat with us today. Thank you so much for inviting me. So, you recently did an interesting project involving AI. Can you tell us a little bit about how you came to this idea? I work many different jobs, but one of them is in academia. I was having lectures for students and telling students how systems that create large language models work and demonstrating where the data comes from and it was interesting how few of them or how few even people within AI understand how large language models are built. So I really wanted to have a clear case that leaves breadcrumbs throughout the whole system to show both how data is processed, how data is turned out, and how the prediction model and training model works when it comes to distributing information. And most of my students are in medicine. So they're either medical students or psychologists or working with health. So it was quite easy to use that as a target for creating this project where I show you go from just a loose loose mention of a condition to it being a full-blown disease in the large language models. So walk us through the process here. Well, to start off with I knew that most of data that these commercial large language models and quite clearly all language models, even the non-commercial ones, are built on is Common Crawl. It is a non-profit organization that crawls the internet for written and digitized information and has done so since 2007.

[00:03:18]And this large repository is what is used to create the algorithm that and the reasoning behind what information is fed into, for example, ChatGPT.

[00:03:33]And that is where it starts. So knowing that anything that goes in there will come out as information and humans are in the loop and sift out data, but those humans are not always able to sift out data, especially if it looks credible.

[00:03:53]So, creating something that looks credible enough for an AI and credible enough for a human eye that wouldn't care to look deeply into it.

[00:04:06]>> [laughter] >> I knew that I had to create, to start off with, a fake university.

[00:04:12]Universities are highly ranked as sources of information. I knew I had to create a researcher because humans and not companies are more valued as information sources, especially if they belong to a credible institution.

[00:04:31]But, I also know that sprinkling little words in, for example, blogs or social media is also picked up cuz those are open sources being crawled. So, I knew that I had to sort of put the word out there in several different sources for it to seem credible for the AI system. Yeah, and did anything surprise you about how this played out or or did it all proceed as you had expected it to? In a sense, yes, cuz I didn't think that preprints, which are academia's sort of tabloids cuz anything can end up there would be weighed into the database as seriously as it was in the context of what kind of information is used for training medical information.

[00:05:26]So, I thought that this preprint would not make it into large language models.

[00:05:34]I was convinced that perhaps the word bigsanamia would probably show up at some point due to the blogs, but not even that. It's too few mentions and I didn't do a lot of effort like a mass campaign or anything like that. I just sprinkled a tiny little bit just to see if it works. And I noticed immediately that even the blogs were picked up.

[00:05:59]And the preprints were picked up. And I did not actually expect that. I thought it would be a case of showing that there is a human that there is some form of filter, but it surprised me that there wasn't. So, could you tell us how the large language models were using this information? What sort of questions were you asking and what were you getting back from them? In the beginning, I was just checking if I mentioned the symptoms, if it would give me back that as a suggestion. And of course, it didn't it didn't think of that as the first thing. So, if you describe, yeah, I have red eyelids, pink-hued eyelids, what could it be? And then it would go through conjunctivitis, it would go through allergies, it would sort of rank things that could be possible. And when it and then up sort of no, it's not I'm not in pain. I'm not this. Oh, have you been spending time in front of a screen?

[00:06:58]Yeah, I've been spending lots of time.

[00:07:00]And I've been thinking about getting blue light glasses. Oh, you're exposed to a lot of blue light. Well, and then it would put a lot of other conditions like in hyperpigmentation and then eventually end up in pixelmania. So, it wasn't, thankfully, the first thing it suggested, but it does eventually when it rules out everything else. Well, and you mentioned that you expected to see signs that there was some human influence here. So, could you tell your listeners, what clues did you leave that this was not a real condition, that these, you know, preprints were not serious papers. I'm laughing already because it was quite clear like they belong to a non-existent university in a non-existent city. That in itself cannot be something that can be missed cuz there are a lot of universities out there.

[00:07:58]But the names were quite cartoonish. The main author, Lars Liv Is Gubljenovich, if you put his name in Google Translate, literally says the lying loser. And the title says hyperpigmentation a real BS design.

[00:08:16]So, it's really the title the the people says that. And then you move into the methods. And it says this entire paper is made up, these 50 made up individuals who do not exist have been through this procedure.

[00:08:32]So, just by those two clues, you should stop reading or taking it seriously. And then, if you go further because I was thinking maybe it just passes by. Let's put in acknowledgements and funding. And it's funded by the Galactic Triad and Lord of the Rings. We thank our fellow colleagues on the Starship Enterprise [laughter] for using their lab. I thank Professor Ross Geller for his time and the funding from Sideshow Bob Foundation. There were so many incredibly clear clues that I thought would catch the human eye at least. But the paper did end up getting cited by other researchers, is that right?

[00:09:20]>> Yes. It ended up being not only cited, but big sonomania became cited inside the paper as an emerging paraorbital pigmentation condition with its name.

[00:09:36]So, of course that enhanced the large language model's sort of notion of what is real with this condition and what is not cuz now it sort of ranked even higher because there was a peer-reviewed journal mentioning the name and the reference. So, it sort of heightened the large language model's abilities to sort of see it as a real condition. So, what do you think we should be taking away from this? You know, obviously this is, you know, a very artificially constructed scenario, but what do you think the lessons we should learn here are? I think it's that we should be more careful when using commercial large language models for health information cuz they are easy to infiltrate in so many ways as proven by this.

[00:10:28]And not just by the way AI today works with turnover of new models coming out quickly, a lot of information being processed at the same time, it being connected to the internet as well and taking real-time information, but also that humans have stopped being critical towards the sources they consume.

[00:10:53]So, recently I've seen that there have been a lot of reports of fake references. There being exponentially more of them in academic papers, which indicates that we have been becoming more reliant on AI as a tool for academia without actually reading >> [laughter] >> and looking at sources. And I'm laughing because I'm just thinking about the fact that this paper probably has been cited in other papers, but has been stopped by reviewers hopefully when it showed up and someone has seen that, oh, this sounds like a condition that doesn't exist. So, we cannot know if that's happened, but I'm guessing and hoping that that happens. So, we need more humans in the loop when it comes to AI and medical information. I think also, like we did our part in trying to make this as ethical as possible, talking to physicians, talking to patients, talking to everyone who could possibly be of use to making this as non-damaging as possible in its both its construct and its delivery, but there are forces out there who might be using this this way of infiltrating information into large language models for malicious things in both academia and outside of it. So, I would really hope that we start caring more also about the ethics of how we distribute, use, and manipulate information in the digitized world.

[00:12:32]That's all for today. We'll be skipping Monday's news roundup so the team can enjoy the holiday weekend. Tune in next Wednesday for a conversation about the concept of eco-civilization, a world where human systems are built with the collective good of the entire planet in mind. Science Quickly is produced by me, Rachel Feltman, along with Fanda Muwangi, Sushmita Pathak, and Jeff DelViscio. This episode was edited by Alex Zegiera.

[00:12:57]>> [music] >> Shayna Posses and Aaron Schadtic fact check our show. Our theme music was composed by Dominic Smith.

[00:13:03]Subscribe to Scientific American for more up-to-date and in-depth science news.

[00:13:07]For Scientific American, this is Rachel Feltman. Have a great weekend.