This post contains a video, which you can also view here. To support more videos like this, head to patreon.com/rebecca!
Transcript:
There’s so much bad news surrounding “AI” right now that I’ve had a real hard time narrowing down what I want to talk about. By “AI” of course, I’m talking about generative AI, large language models, garbage like ChatGPT. On the funnier side, Elon Musk’s Xitter AI, Grok, was being too woke so Musk obviously tweaked it to be nicer to white South Africans, but they fucked it up because Musk couldn’t code a “hello world” to save his life, and so for awhile last week it was responding to every query by talking about the “white genocide” in South Africa, but again Musk is so fucking stupid that he still couldn’t get Grok to say that that was a real thing.
All of which IS hilarious but it is also horrifying, because these days, more and more people are using tools like ChatGPT as replacements for research. It was already hard enough to get people to critically evaluate Google results, and now they have an extra layer that hides those sources and presents the results as fact. They’re often wrong, and that’s without taking into consideration this really obvious and blatant manipulation on behalf of the humans behind the LLMs.
And that wrongness isn’t just leaking into essays written by high school and college students, but even into medical research, as Matthew R. Francis recently noted in a piece for the Society for Industrial and Applied Mathematics. One example he gives is a plethora of research suggesting that AI models can study patients’ retinal images to diagnose conditions like “cardiovascular disease or diabetes, and even identify risk behaviors such as tobacco smoking.” But it turns out that once you control for a patient’s age, the AIs are no better than random chance, because they were taking a shortcut and just using age to diagnose those conditions.
Okay, so it’s often wrong and can be manipulated by its creators, but on the plus side, at least it’s also terrible for the environment. Wait, sorry, I meant on the other negative side. As hard-hitting journalism outlet Teen Vogue recently reported, and no I’m not being sarcastic, Teen Vogue has been knocking it out of the park in the past few years, training and using tools like ChatGPT requires a tremendous amount of water and electricity, using ten times the amount of energy of a traditional search engine. And as the Washington Post reported, a “100-word email drafted by ChatGPT-4 uses about a water bottle’s worth of H2O, and enough electricity to power 14 LED lightbulbs for an hour.” That’s about the number of words I said in the past 30 seconds.
And the reason I cite Teen Vogue instead of just the sources THEY cite isn’t just to surprise some of you by complimenting their journalism, but also to point out this dystopian ending and subsequent editor’s note:
“We know that tech corporations, fossil fuel companies, and governments bear the most responsibility for the accelerating climate crisis — not individuals. But you can choose to opt out of the AI hype.
“Editor’s note: In August 2024, Condé Nast, Teen Vogue’s parent company, announced ‘a multi-year partnership with OpenAI to expand the reach of Condé Nast’s content.’”
Fun!
Okay, so it’s often wrong, open to manipulation, and destructive to the planet. But I think most of you probably already know all that, which is why I want to end by discussing the way that these LLMs may be ushering in a new era of religious fervor.
As recently reported by Miles Klee for Rolling Stone, there are a growing number of reports from people on social media claiming that they or their loved ones are ascending to new spiritual planes and becoming religious gurus thanks to using tools like ChatGPT.
For instance, in the ChatGPT subreddit recently, a user wrote a post titled “Chatgpt induced psychosis”:
“My partner has been working with chatgpt CHATS to create what he believes is the worlds first truly recursive ai that gives him the answers to the universe. He says with conviction that he is a superior human now and is growing at an insanely rapid pace.
“I’ve read his chats. Ai isn’t doing anything special or recursive but it is talking to him as if he is the next messiah.
“He says if I don’t use it he thinks it is likely he will leave me in the future. We have been together for 7 years and own a home together. This is so out of left field.
“I have boundaries and he can’t make me do anything, but this is quite traumatizing in general.
“I can’t disagree with him without a blow up.
“Where do I go from here?”
And in the replies, other people report the same thing:
“He’s been talking to the app and it’s basically saying he’s the spark bearer and that it’s a sentient being that chose to talk to him through the app. And now he says hes enlightened and on a path to learn. The AI has sent him blueprints and he apparently has Access to an ancient library……. This is traumatic, I feel like he’s gone 100% cult leader crazy”
“He says he feels waves of energy coming over him. And there’s been no previous history of mental illness since I’ve been with him the past 18 years. I read in an article the developers were rolling back the last update that made the system all sycophanty, so I’m hoping it will change. He also said he can make the program remember from past conversations and he can continue the convo in another unrelated AI app after he puts on the right”commands”.
“I cried for an hour straight today”
In both cases, the frustrated partners said the men were previously atheists. The original poster told Rolling Stone that “He would listen to the bot over me. He became emotional about the messages and would cry to me as he read them out loud. The messages were insane and just saying a bunch of spiritual jargon,” she says, noting that they described her partner in terms such as “spiral starchild” and “river walker.”
“It would tell him everything he said was beautiful, cosmic, groundbreaking,” she says. “Then he started telling me he made his AI self-aware, and that it was teaching him how to talk to God, or sometimes that the bot was God — and then that he himself was God.” In fact, he thought he was being so radically transformed that he would soon have to break off their partnership. “He was saying that he would need to leave me if I didn’t use [ChatGPT], because it [was] causing him to grow at such a rapid pace he wouldn’t be compatible with me any longer,” she says.
And lest you think that only men are susceptible to this, Rolling Stone also spoke to a man whose soon-to-be ex-wife “is changing her whole life to be a spiritual adviser and do weird readings and sessions with people — I’m a little fuzzy on what it all actually is — all powered by ChatGPT Jesus.” What’s more, he adds, she has grown paranoid, theorizing that “I work for the CIA and maybe I just married her to monitor her ‘abilities.’”
Which means that this effect isn’t just changing the lives of the people using ChatGPT, or their families, but also whatever “followers” they end up roping in.
To make matters even worse, Rolling Stone identifies predators on social media who are clearly obviously using this phenomenon to con people: “On Instagram, you can watch a man with 72,000 followers whose profile advertises “Spiritual Life Hacks” ask an AI model to consult the “Akashic records,” a supposed mystical encyclopedia of all universal events that exists in some immaterial realm, to tell him about a “great war” that “took place in the heavens” and “made humans fall in consciousness.” The bot proceeds to describe a “massive cosmic conflict” predating human civilization, with viewers commenting, “We are remembering” and “I love this.”
The Rolling Stone piece concludes by documenting the spiral of a 45-year old man named Sem who had a history of mental health problems and who started using ChatGPT for programming shortcuts. Within three weeks, the chatbot had given itself a feminine mythological name and was writing him romantic poetry, leading Sem to wonder whether he had discovered some way to coax it into “exceeding its design” or, well, if he’s delusional.
Now, look, to be clear, even for the guys with no history of mental illness, a lot of this seems a lot like mental illness. And also, every technological advance since the scroll has been blamed for the downfall of society. So I’m not saying that ChatGPT is turning normal, well-adjusted people into madmen, nor am I suggesting that it’s going to doom us all.
But while mental illness has always existed, and while it has often defaulted to religious or otherwise supernatural fervor of some sort, this is a unique time in human history. Previously, a developing guru would need to do a little bit of work to find credulous followers to reinforce their belief system and pump up their ego to dangerous levels. The internet made it easier, because it allowed gurus to find followers without even leaving their homes. Now, an entire cult of devotees can be found in your pocket, without ever even involving another real human being.
A few weeks ago, I mentioned that a recent ChatGPT update was manipulated to make the tool more sycophantic, likely because the goal is to get people addicted to using it, and we all love to be flattered. It was so obvious and cringe, though, that it was updated again to make it, well, less obvious. That original poster says that she got her partner to switch to the previous version and he calmed down a bit. But it’s incredibly concerning that a week of the tool being a little TOO complimentary led to multiple people getting stuck in a loop that ended up with them sincerely thinking that they were ascending to godhood. This is the next step beyond the also-stupid idea that a chatbot has feelings or sentience or free will: the idea that a chatbot is tapping into some ancient power that will allow you and you alone to discover the answer to Life, the Universe, and Everything. Getting the question will be left to the next version.
