Within the fascinating new actuality of the web, teen women can’t study intervals on Reddit and indie artists can’t promote smutty video games on Itch.io, however a navy contractor will make you nonconsensual deepfakes of Taylor Swift taking her prime off for $30 a month.
Early Tuesday, Elon Musk’s xAI launched a brand new picture and video generator known as Grok Think about with a “spicy” mode whose output ranges from suggestive gestures to nudity. As a result of Grok Think about additionally has no perceptible guardrails towards creating photographs of actual folks, which means you may basically generate softcore pornography of anybody who’s well-known sufficient for Grok to recreate (though, pragmatically, it appears to mainly produce seriously NSFW output for women). Musk bragged that greater than 34 million photographs had been generated inside a day of launching operations. However the true coup is demonstrating that xAI can ignore stress to maintain grownup content material off its companies whereas serving to customers create one thing that’s extensively reviled, because of authorized gaps and political leverage that no different firm has.
xAI’s video function — which debuted across the identical time as a romantic chatbot companion named Valentine — appears from one angle strikingly bizarre, as a result of it’s being launched throughout a interval the place intercourse (all the way down to the word itself) is being pushed to the margins of the web. Late final month, the UK began implementing age-gating guidelines that required X and different companies to dam sexual or in any other case “dangerous” content material for customers below 18. Across the identical time, an activist group known as Collective Shout successfully pressured Steam and Itch.io to crack down on grownup video games and different media, main Itch.io specifically to mass-delist any NSFW uploads.
Deepfake porn of actual folks is a type of nonconsensual intimate imagery, which is unlawful to deliberately publish within the US below the Take It Down Act, signed by President Donald Trump earlier this 12 months. In a press release revealed Thursday, the Rape, Abuse & Incest Nationwide Community (RAINN) called Grok’s feature “a part of a rising downside of image-based sexual abuse” and quipped that Grok clearly “didn’t get the memo” in regards to the new regulation.
However based on Mary Anne Franks, a professor at George Washington College Regulation College and president of the nonprofit Cyber Civil Rights Initiative (CCRI), there’s “little hazard of Grok dealing with any form of legal responsibility” below the Take It Down Act. “The prison provision requires ‘publication,’ which, whereas sadly not outlined within the statute, suggests making content material obtainable to multiple particular person,” Franks says. “If Grok solely makes the movies viewable to the one that makes use of the device, that wouldn’t appear to suffice.”
Regulators have did not implement legal guidelines towards massive corporations even after they apply
Grok additionally probably isn’t required to take away the photographs below the Take It Down Act’s takedown provision — regardless of that rule being so worryingly broad that it threatens most social media companies. “I don’t suppose Grok — or no less than this specific Grok device — even qualifies as a ‘lined platform,’ as a result of the definition of lined platform requires that it ‘primarily supplies a discussion board for user-generated content material,’” she says. “AI-generated content material typically includes consumer inputs, however the precise content material is, because the time period signifies, generated by AI.” The takedown provision can be designed to work via folks flagging content material, and Grok doesn’t publicly put up the photographs the place different customers can see them — it simply makes them extremely straightforward to create (and nearly inevitably put up to social media) at a big scale.
Franks and the CCRI called out the limited definition of a “lined platform” as an issue for different causes months in the past. It’s one in every of a number of methods the Take It Down Act fails to serve folks impacted by nonconsensual intimate imagery whereas posing a threat to net platforms appearing in good religion. It may not even cease Grok from posting lewd AI-modified photographs of actual folks publicly, Franks told Spitfire News in June, partially as a result of there are open questions on whether or not Grok is a “particular person” impacted by the regulation.
These sorts of failures are a operating theme in web regulation that’s ostensibly purported to crack down on dangerous or inappropriate content material; the UK’s mandate, for example, has made it harder to run independent forums whereas nonetheless being pretty straightforward for teenagers to get round.
Compounding this downside, notably within the US, regulatory businesses have did not impose significant penalties for every kind of rulebreaking by highly effective corporations, together with Musk’s many companies. Trump has given Musk-owned corporations an nearly whole cross for unhealthy conduct, and even after formally leaving his highly effective place on the Division of Authorities Effectivity, Musk probably maintains super leverage over regulatory businesses just like the FTC. (xAI simply acquired a contract of as much as $200 million with the Division of Protection.) So even when xAI had been violating the Take It Down Act, it in all probability wouldn’t face investigation.
Past the federal government, there are layers of gatekeepers that dictate what is suitable on platforms, and so they typically take a dim view of intercourse. Apple, for example, has pushed Discord, Reddit, Tumblr, and different platforms to censor NSFW materials with various ranges of success. Steam and Itch.io reevaluated grownup content material below risk of dropping relationships with cost processors and banks, which have beforehand put the screws on platforms like OnlyFans and Pornhub.
In some circumstances, like Pornhub’s, this stress is the results of platforms permitting unambiguously dangerous and unlawful uploads. However Apple and cost processors don’t seem to keep up hard-line, evenly enforced insurance policies. Their enforcement appears to rely considerably on public stress balanced towards how a lot energy the goal has, and regardless of his falling out with Trump, just about no one in enterprise has extra political energy than Musk. Apple and Musk have repeatedly clashed over Apple’s insurance policies, and Apple has largely held agency on issues like its payment construction, nevertheless it’s apparently backed down on smaller issues, together with returning its commercials to X after pulling them from the Nazi-infested platform.
Apple has banned smaller apps for making AI-generated nudes of real people. Will it exert that form of stress on Grok, whose video service launched solely on iOS? Apple didn’t reply to a request for remark, however don’t maintain your breath.
Grok’s new function is dangerous for individuals who can now simply have nonconsensual nudes manufactured from them on a significant AI service, nevertheless it additionally demonstrates how hole the promise of a “safer” web is proving. Small-time platforms face stress to take away consensually recorded or completely fictional media made by human beings, whereas an organization run by a billionaire can become profitable off one thing that’s in some circumstances outright unlawful. For those who’re on-line in 2025, nothing is about intercourse, together with intercourse — which, per ordinary, is about energy.