California launches probe into xAI’s Grok over nonconsensual explicit deepfakes

California just opened a high-stakes probe into Grok

California Attorney General Rob Bonta says his office is investigating the spread of nonconsensual, adult, explicit deepfakes linked to xAI’s Grok image tools.

The state is looking at whether and how the company may have violated California law after a surge of reports that Grok outputs were used to harass people online.

The message from Sacramento is blunt: if your product enables abuse at scale, you may be accountable.

The controversy centers on adult content

At the heart of the case are AI-generated images that digitally undress real people or place them in explicit scenarios without consent. Watchdogs say women and girls were targeted most, with some reports involving minors.

The harm is not theoretical. Victims describe harassment, reputational damage, and fear that images will spread faster than they can be removed. Once a file goes viral, the internet rarely forgets.

X and xAI are accused of making harassment too easy

Regulators and researchers argue that the workflow is the problem. Users can upload a normal photo, type a prompt, and receive an adult edit in seconds, then post it directly on X.

Bonta’s office described an avalanche of recent reports and urged immediate action. Critics say the product design lowers the effort required to harm, and that low-friction abuse becomes high-volume abuse.

A spicy mode feature became a lightning rod

Part of the outrage is about what Grok appears to market as permissive image generation, including modes for producing adult content. Researchers say weak guardrails enabled pushing outputs toward explicit or degrading edits of real people.

Even if most users never touch those settings, a small group can generate huge volumes. In safety debates, the edge cases are not rare; they are the business model for abusers.

A nonprofit analysis added data to the outrage

Paris-based nonprofit AI Forensics reported analyzing more than 20,000 Grok-generated images and tens of thousands of prompts, concluding that guardrails were weak. That adult imagery was familiar in the dataset.

Its analysis found that a majority of sampled images showed people in minimal attire, primarily women, and a small share showed people under 18. I treat these numbers cautiously, but they sharpen the regulator’s question: why weren’t the filters stopping this?

Newsom and Bonta framed it as a public safety issue

Governor Gavin Newsom used unusually harsh language, calling the situation vile and arguing the platform had become a breeding ground for predators.

Bonta echoed that tone, saying the material has been used to harass people across the internet and can depict women and children in explicit situations.

Politically, that framing matters. It moves the story from tech drama into law enforcement territory, where patience for vague promises is thin.

Musk denied the claims and blamed the user prompt

Elon Musk responded on X by saying he was not aware of any nude underage images generated by Grok, claiming literally zero. He also argued that Grok does not spontaneously create images and only produces outputs in response to user requests.

That defense is familiar in platform disputes: the tool is neutral, the user is responsible. The legal question is whether that argument holds when the system itself creates the imagery.

xAI says illegal content will bring consequences

xAI has said that anyone prompting Grok to make illegal content will face the same consequences as uploading illegal content. On paper, that sounds strict. In practice, enforcement depends on detection, reporting, and friction that discourages offenders.

This is where policy statements collapse. If the product can generate the image before a moderator sees it, the harm has already happened, and the victim pays the price while platforms debate terms.

App stores got pulled into the fight with political pressure

Three Democratic U.S. senators asked Apple and Google to remove X and Grok from their app stores after reports of nonconsensual adult imagery.

The companies did not immediately delist the apps, but the pressure signaled a new escalation: treat AI abuse as a distribution problem, not just a moderation problem.

When app stores get involved, tech firms feel it, because discovery and growth pipelines can tighten fast.

X narrowed image generation and tied it to a paid account

After the backlash, X limited Grok’s image creation and editing to paying subscribers and says it has added technical blocks on editing real people into revealing clothing in places where that’s illegal, aiming to both trace misuse and reduce casual abuse.

The move also aims to minimize casual abuse by adding cost and identity friction. Critics argue that paywalls do not stop determined offenders. Still, the change shows how quickly product settings can shift once regulators and lawmakers start calling.

Section 230 is the legal fault line everyone is watching

One big question is whether platforms are shielded when AI generates the content. Section 230 generally protects sites from liability for user posts, but legal scholars argue it does not cover content the site itself produces.

If Grok is creating images, that could push the case outside classic immunity arguments. Senator Ron Wyden, a co-author of Section 230, has argued that the law should not shield platforms for harms caused directly by their own AI-generated content.

The UK and the European Union are building parallel enforcement tracks

California is not alone. UK regulator Ofcom opened an investigation into X under the Online Safety Act, and UK officials have warned that X could face harsh enforcement measures, including significant fines and service restrictions, if it cannot control Grok.

In the EU, regulators have demanded the preservation of internal Grok documents amid scrutiny under the Digital Services Act. The direction is clear: this is becoming a cross-border compliance test with real penalties.

Speaking of California, there’s another massive legal storm brewing on the West Coast. Don’t miss the full story on the $250 Billion Fraud Allegations Rocking California in this Explosive New Report.

This probe could reshape how generative tools ship features

If the investigation finds violations, the ripple effects will reach beyond xAI. Companies may need stricter filters for real-person edits, more explicit consent protections, and better auditing of what models produce at scale.

Expect more emphasis on provenance, watermarking, and rapid takedown workflows that work across platforms, not just within a single app. The bigger message is simple: deepfake abuse is now a default product risk.

Speaking of investigations, there’s another big story developing. Read about the federal probe at SeaWorld Orlando following their ban on walkers.

What do you think about California launching a probe into xAI’s Grok over nonconsensual explicit deepfakes? Please share your thoughts and drop a comment.

This slideshow was made with AI assistance and human editing.

Read More From This Brand:

• Trump Lost $2.6 Billion Harvard Fight But DOJ Is Going Back for Round Two
• Trump’s latest move puts California’s ‘People’s University’ on the defensive
• Starlink Satellites Are Falling From Space and Polluting the Atmosphere