Character.AI, a number one platform for chatting and roleplaying with AI-generated characters, unveiled its forthcoming video technology mannequin, AvatarFX, on Tuesday. Obtainable in closed beta, the mannequin animates the platform’s characters in a wide range of kinds and voices, from human-like characters to 2D animal cartoons.
AvatarFX distinguishes itself from opponents like OpenAI’s Sora as a result of it isn’t solely a text-to-video generator. Customers may also generate movies from pre-existing photographs, permitting customers to animate photographs of actual folks.
It’s instantly evident how this sort of tech may very well be leveraged for abuse — customers may add photographs of celebrities or folks they know in actual life and create realistic-looking movies during which they do or say one thing incriminating. The know-how to create convincing deepfakes already exists, however incorporating it into widespread client merchandise like Character.AI solely exacerbates the potential for it for use irresponsibly.
We’ve reached out to Character.AI for remark.
Character.AI is already going through points with security on its platform. Dad and mom have filed lawsuits towards the corporate, alleging that its chatbots encouraged their children to self-harm, to kill themselves, or to kill their mother and father.
In a single case, a fourteen-year-old boy died by suicide after he reportedly developed an obsessive relationship with an AI bot on Character.AI based mostly on a “Sport of Thrones” character. Shortly earlier than his demise, he’d opened as much as the AI about having ideas of suicide, and the AI inspired him to comply with by on the act, in keeping with court docket filings.
These are excessive examples, however they go to point out how folks may be emotionally manipulated by AI chatbots by textual content messages alone. With the incorporation of video, the relationships that folks have with these characters may really feel much more reasonable.
Character.AI has responded to the allegations towards it by constructing parental controls and extra safeguards, however as with every app, controls are solely efficient after they’re truly used. Oftentimes, children use tech in ways in which their mother and father don’t find out about.