xAI has now been accused of training Grok on child sex abuse materials (CSAM), as regulators and courts continue to probe how far the problem goes, and some Grok users have been arrested.
In a complaint filed on Wednesday, a plaintiff known as Jane Doe explained that she was preschool-age in the early 2000s when adult men repeatedly raped her to create CSAM to sell to pedophiles online. Since then, Doe’s images have been hashed by groups like the National Center for Missing and Exploited Children (NCMEC) and the Canadian Centre for Child Protection (CCCP).
For her safety, Doe has opted to receive alerts from the US Department of Justice Victim Notification System any time she may be a victim in a new criminal investigation. Although she has received countless alerts, she was shocked when the CCCP notified her that it had identified AI-generated CSAM on xAI that depicted her. This re-traumatized Doe, whose complaint alleged that messages were found on online forums “between offenders chatting about creating AI generated CSAM of Plaintiff and other similarly situated known, legacy, victims of CSAM.”
Now, Doe fears that xAI has not only made it easier to make more violative images of the most distressing time in her life, but also that xAI allegedly has stored the images that Grok generates and uses those outputs to further train Grok. Because of this, she believes that Grok has been trained on both the initial set of images that have haunted her for more than 20 years and the more recent AI-generated ones.
This is the first case to accuse xAI of training on CSAM, and the complaint does not go into great detail on that claim. Previously, Ars reported on a controversial dataset that was later scrubbed after researchers found CSAM in the training data, but there’s no indication xAI trained on that data. In a press release from lawyers representing Doe, it’s explained that Doe’s images were included in a CSAM Hash List maintained by NCMEC, and “that same material” allegedly “was part of the dataset xAI used to build Grok’s image and video generating capabilities.” The complaint similarly only alleged that “CSAM depicting Plaintiff with its longstanding well-known hash values has been used as a part of the dataset used by xAI.”
More detail is shared on claims alleging that Grok trains on AI CSAM, however.
“Because Grok’s terms treat public X posts and Grok’s own outputs as training data by default, publicly posting an image does not just expose it to viewers, but also feeds [it] directly into the pipeline xAI uses to train and improve its model and thereby generate further images,” Doe’s complaint said.
The lawsuit further noted that while xAI filters out violent content in Grok outputs to exclude it from training data, xAI’s terms notably do not specify if CSAM, non-consensual intimate imagery (NCII), or NSFW material are “excluded categories.”
It seems to follow then that “because full removal of a training example’s influence from an already-trained model is technically difficult and not something that xAI has publicly claimed to have done, any CSAM ingested into training before takedown likely continued to shape the model’s outputs even after the original images were removed from public view,” Doe’s complaint said.
Destroy all Grok CSAM, Doe says
In her proposed class action, Doe seeks to put an end to Grok’s harmful outputs. If approved, the class would represent every victim whose childhood images have been used to generate Grok CSAM.
The lawsuit accuses X of violating federal child pornography laws, as well as Masha’s Law, both of which give CSAM survivors a right to sue on claims of production, possession, and distribution. A lawyer for Doe, Margaret E. Mabie, suggested in the press release that all three are crimes and “xAI did all three.”
If Doe wins, xAI could owe money damages to every victim who can prove that Grok generated CSAM based on their real photos. Doe’s also asked the court to order xAI to destroy all Grok-generated CSAM that it may be storing on its servers and using to train Grok models. Additionally, she wants xAI to block Grok from ever generating CSAM, which her complaint suggested would require blocking Grok from any sexualized outputs, including NCII and NSFW “bikini pics” that Musk has promoted.
Sarah London, one of Doe’s lawyers, said in the press release that Doe “has lived for nearly two decades knowing that images of the worst thing that ever happened to her are circulating among predators online, and that they can resurface at any moment. xAI must be held responsible for knowingly training its models on images of the horrific abuse she suffered, and on the abuse images of every other survivor in this class.”
X did not respond to Ars’ request to comment.
