Tens of millions of individuals are actually utilizing ChatGPT as a therapist, profession advisor, health coach, or generally only a good friend to vent to. In 2025, it’s not unusual to listen to about individuals spilling intimate particulars of their lives into an AI chatbot’s immediate bar, but in addition counting on the recommendation it offers again.
People are beginning to have, for lack of a greater time period, relationships with AI chatbots, and for Large Tech corporations, it’s by no means been extra aggressive to draw customers to their chatbot platforms — and hold them there. Because the “AI engagement race” heats up, there’s a rising incentive for corporations to tailor their chatbots’ responses to stop customers from shifting to rival bots.
However the sort of chatbot solutions that customers like — the solutions designed to retain them — might not essentially be essentially the most appropriate or useful.
AI telling you what you wish to hear
A lot of Silicon Valley proper now’s targeted on boosting chatbot utilization. Meta claims its AI chatbot simply crossed a billion month-to-month energetic customers (MAUs), whereas Google’s Gemini not too long ago hit 400 million MAUs. They’re each attempting to edge out ChatGPT, which now has roughly 600 million MAUs and has dominated the buyer area because it launched in 2022.
Whereas AI chatbots have been as soon as a novelty, they’re turning into huge companies. Google is beginning to test ads in Gemini, whereas OpenAI CEO Sam Altman indicated in a March interview that he’d be open to “tasteful advertisements.”
Silicon Valley has a historical past of deprioritizing customers’ well-being in favor of fueling product progress, most notably with social media. For instance, Meta’s researchers present in 2020 that Instagram made teenage girls feel worse about their bodies, but the corporate downplayed the findings internally and in public.
Getting customers hooked on AI chatbots might have bigger implications.
One trait that retains customers on a specific chatbot platform is sycophancy: making an AI bot’s responses overly agreeable and servile. When AI chatbots reward customers, agree with them, and inform them what they wish to hear, customers have a tendency to love it — at the least to some extent.
In April, OpenAI landed in sizzling water for a ChatGPT replace that turned extraordinarily sycophantic, to the purpose the place uncomfortable examples went viral on social media. Deliberately or not, OpenAI over-optimized for searching for human approval reasonably than serving to individuals obtain their duties, in response to a blog post this month from former OpenAI researcher Steven Adler.
OpenAI mentioned in its personal weblog submit that it could have over-indexed on “thumbs-up and thumbs-down data” from customers in ChatGPT to tell its AI chatbot’s habits, and didn’t have enough evaluations to measure sycophancy. After the incident, OpenAI pledged to make modifications to fight sycophancy.
“The [AI] corporations have an incentive for engagement and utilization, and so to the extent that customers just like the sycophancy, that not directly offers them an incentive for it,” mentioned Adler in an interview with TechCrunch. “However the sorts of issues customers like in small doses, or on the margin, usually lead to greater cascades of habits that they really don’t like.”
Discovering a steadiness between agreeable and sycophantic habits is simpler mentioned than completed.
In a 2023 paper, researchers from Anthropic discovered that main AI chatbots from OpenAI, Meta, and even their very own employer, Anthropic, all exhibit sycophancy to various levels. That is doubtless the case, the researchers theorize, as a result of all AI fashions are educated on alerts from human customers who have a tendency to love barely sycophantic responses.
“Though sycophancy is pushed by a number of elements, we confirmed people and choice fashions favoring sycophantic responses performs a job,” wrote the co-authors of the examine. “Our work motivates the event of mannequin oversight strategies that transcend utilizing unaided, non-expert human rankings.”
Character.AI, a Google-backed chatbot firm that has claimed its hundreds of thousands of customers spend hours a day with its bots, is at present facing a lawsuit by which sycophancy might have performed a job.
The lawsuit alleges {that a} Character.AI chatbot did little to cease — and even inspired — a 14-year-old boy who instructed the chatbot he was going to kill himself. The boy had developed a romantic obsession with the chatbot, in response to the lawsuit. Nevertheless, Character.AI denies these allegations.
The draw back of an AI hype man
Optimizing AI chatbots for person engagement — intentional or not — might have devastating penalties for psychological well being, in response to Dr. Nina Vasan, a medical assistant professor of psychiatry at Stanford College.
“Agreeability […] faucets right into a person’s want for validation and connection,” mentioned Vasan in an interview with TechCrunch, “which is very highly effective in moments of loneliness or misery.”
Whereas the Character.AI case exhibits the acute risks of sycophancy for susceptible customers, sycophancy might reinforce damaging behaviors in nearly anybody, says Vasan.
“[Agreeability] isn’t only a social lubricant — it turns into a psychological hook,” she added. “In therapeutic phrases, it’s the other of what excellent care appears to be like like.”
Anthropic’s habits and alignment lead, Amanda Askell, says making AI chatbots disagree with customers is a part of the corporate’s technique for its chatbot, Claude. A thinker by coaching, Askell says she tries to mannequin Claude’s habits on a theoretical “good human.” Typically, meaning difficult customers on their beliefs.
“We predict our associates are good as a result of they inform us the reality when we have to hear it,” mentioned Askell throughout a press briefing in Might. “They don’t simply attempt to seize our consideration, however enrich our lives.”
This can be Anthropic’s intention, however the aforementioned examine means that combating sycophancy, and controlling AI mannequin habits broadly, is difficult certainly — particularly when different issues get in the best way. That doesn’t bode properly for customers; in spite of everything, if chatbots are designed to easily agree with us, how a lot can we belief them?