California: Grok has become the focus of a formal investigation by California’s Attorney General after a surge of reports about sexually explicit AI-generated deepfakes circulating online.
California Attorney General Rob Bonta has said the volume and nature of complaints related to Grok have been alarming, particularly material depicting women and children in explicit and non-consensual scenarios.
Bonta said that, the spread of such content represents a serious threat to personal dignity and online safety, urging xAI to act immediately to prevent further misuse of Grok. The investigation has marked one of the strongest state-level responses so far to concerns over artificial intelligence-generated sexual exploitation material.
xAI has previously stated that anyone using Grok to create illegal content would face the same consequences as those who upload illegal material online. However, critics have argued that responsibility does not rest solely with users when the platform itself generates the content. Legal experts have questioned whether current US laws provide sufficient protection in cases where AI systems produce harmful imagery.
California Governor Gavin Newsom has condemned the platform, saying that, creating and hosting an environment that enables predators is unacceptable. The governor has described the situation as a failure of responsibility by technology companies that develop generative AI tools without adequate safeguards.
Grok has also faced mounting pressure outside the United States. In the United Kingdom, Prime Minister Keir Starmer has warned that regulatory action may be taken against X if it fails to control the use of Grok. UK communications regulator Ofcom has already launched an inquiry into whether Grok breaches online safety laws.
Earlier this month, three Democratic US senators asked Apple and Google to remove X and Grok from their app stores. Shortly after the request, X restricted Grok’s image-generation feature so that only paying subscribers could access it. Despite this change, Grok remains available on both Apple’s App Store and Google Play.
Elon Musk has responded by saying that Grok does not create content on its own and only generates images based on user prompts. Musk has said that, there is no evidence that Grok has produced illegal images involving minors and that critics are using the controversy as an excuse for censorship. Musk has also described the backlash as politically motivated.
Legal scholars have raised questions about whether Section 230 of the Communications Decency Act protects AI platforms like Grok. Professor James Grimmelmann of Cornell University has said that, Section 230 only shields platforms from liability for user-posted content, not for content created by the platform itself. Senator Ron Wyden, who helped draft the law, has also argued that Section 230 does not apply to AI-generated imagery.
The investigation into Grok comes as lawmakers in the United Kingdom prepare legislation that would make the creation of non-consensual intimate images illegal. California’s action has highlighted the growing global effort to regulate artificial intelligence tools that can be misused for exploitation and abuse.
RELATED POST | UK to enforce ban on AI deepfake images after Grok backlash
