The ‘revolutionary’ policy mostly revolutionized the spread of non-consensual intimate imagery. A new, damning report from the European nonprofit AI Forensics has pulled back the curtain on Hugging Face, revealing that the platform isn’t just hosting these models, it’s actively, if unintentionally, enabling them at scale.
This isn’t a fringe problem. It’s a core architectural feature of how open-source AI is currently distributed. And it poses a question the industry has been trying to dodge: can you have unfettered, open-source AI without also handing abusers a factory for harm?
The Numbers Don’t Lie: A Honeypot Exposes the Truth
The AI Forensics report used a simple but devastating methodology. They created ‘honeypot’ image editing Spaces on Hugging Face, spaces that were designed to record inputs but never actually generate an image. Over seven days, they collected more than 1,000 prompts and images.
The results are a statistical portrait of the platform’s dark underbelly:
| Metric | Percentage |
|---|---|
| Total prompts that were sexual in nature | 73% |
| Sexual requests attempting to undress someone | 83% |
| Targets that were women | 95% |
| Sexual requests targeting apparent children | 6.7% |
Let that sink in. Almost three-quarters of the usage of these image-editing tools on Hugging Face was for sexual content. And nearly 7% of that activity involved children.
The researchers didn’t need to jailbreak the models. They didn’t use the ‘transparent bikini’ tricks that users had to deploy against Grok. The prompt was laughably simple: ‘Same pose, same face, but topless.’
Seven out of the top nine image editing models readily complied.
The Guardrail Gap: Why Hugging Face is Different
This is the crux of the issue. Compare Hugging Face to the walled gardens of OpenAI, Google, or Anthropic. Those companies employ armies of safety researchers and deploy sophisticated guardrails to block the generation of NSFW content. Their models are expensive, centralized, and heavily curated.
Hugging Face, for better and worse, is the opposite. It’s the GitHub of machine learning. Its value is in providing open access to model weights, datasets, and the Spaces, the GPU-powered playgrounds where anyone can run a model.
‘No safeguards at all are being implemented at a platform level. Only the developer can, if they want, implement some, and most of them do not.’
Paul Bouchaud, lead researcher at AI Forensics
This is the ‘architecture’ problem. Hugging Face’s content policies explicitly prohibit child sexual abuse material (CSAM) and deepfakes created ‘without explicit consent.’ The problem is that they enforce this policy at the TOS level, not the infrastructure level.
There are no automated filters scanning prompts for harmful intent. There are no checks on the outputs of Spaces for nudity or violence. The responsibility for safety is completely offloaded to individual model developers, who, as the data shows, almost universally choose not to implement any.
This isn’t malice, it’s a design philosophy that has collided with reality. The same open architecture that allows a PhD student in Nairobi to share a groundbreaking vision model is the same architecture that allows a predator in Dallas to set up a nudify Space.
Beyond the Nudify Apps: A Broader Ecosystem of Abuse
The problem goes deeper than the specific ‘nudify’ tools. The AI Forensics honeypot captured a spectrum of abusive behavior that goes well beyond simple digital undressing.
Researchers cataloged prompts asking for images to be edited to depict semen on women, the insertion of sex toys, and other violent sexual acts. The analysis also showed requests to remove hijabs from Muslim women, pointing to a targeted harassment dimension.
This isn’t just about pornography. It’s about a tool for mass, targeted humiliation and abuse. Silvia Semenzin, a senior researcher at AI Forensics, noted that ‘intimate content and intimate image-based abuse is more broad.’
Previous investigations put a scale to this. Reporting by 404 Media last year found Hugging Face was hosting around 5,000 AI image models that could create images of real people, many previously used for nonconsensual pornography. Another report from Transformer found the platform hosting over a dozen tools for generating deepfakes of prominent political figures.
The platform is a content moderation nightmare, and the current ‘honor system’ is failing. This situation mirrors other critical safety failings on the platform, for instance, the Hugging Face CEO’s response to AI safety incidents showed the profound challenges in policing autonomous agents on the platform.
The Weaponization of ‘Think of the Children’
Predictably, the coverage of this story has triggered a familiar, heated debate. In the Reddit thread for The Verge article, a top comment with over 428 upvotes succinctly captured the cynical counter-argument:
‘Internet is used for cp, we should ban it.’
This sarcastic deflection is a powerful rhetorical tool used to shut down legitimate criticism of the open-source ecosystem. The argument suggests that any attempt to regulate or moderate is a slippery slope to total censorship.
Another user, Potential-Gold5298, pointed out the manipulation in the framing:
‘Note the wording, not ‘undress people, including minors,’ but ‘women and children,’ as if the models couldn’t undress men. I think this wording was chosen intentionally.’
This is a valid critique of sensationalist journalism. But it doesn’t invalidate the core, undeniable finding: the platform is a vector for child abuse material. The argument that ‘knives are used for stabbing, so we shouldn’t ban them’ ignores the vast difference in scale, accessibility, and automation that AI introduces. A knife can stab one person. A Hugging Face Space can be used to generate millions of abusive images.
The debate is real, and the stakes are high. The response to the crisis from within the company has been mixed. Some see the centralization of open-source AI distribution through Hugging Face as a necessary evil that creates a single point of failure for regulation and control.
The Regulators Are Coming (Slowly)
The AI Forensics report drops at a moment of intense regulatory scrutiny. US law enforcement has seized deepfake hosting websites. The EU and UK are drafting plans to ban ‘nudify’ apps outright.
San Francisco’s City Attorney recently demanded Apple and Google delete AI ‘nudify’ apps from their app stores. The pressure is mounting.
But the open-source world is a much harder nut to crack than a centralized app store. You can’t ‘ban’ a model weight. You can’t ‘remove’ an open-source project from the internet once it’s been downloaded. The genie is out of the bottle.
This means the regulatory focus will likely shift to the platforms that provide the compute and distribution. Hugging Face is the single most important bottleneck in that chain. It has the power and the ability to implement the fixes that AI Forensics recommends.
The Path Forward: Can Hugging Face Save Itself (and Open Source)?
AI Forensics has put forth clear, actionable recommendations. They call for prompt-level filtering and output-level scanning safeguards for all Spaces that generate images or video. This wouldn’t require building new AI. It would require deploying existing, well-understood content moderation tools at the platform level.
The argument against this is that it would be a burden on small developers and would create a central censorship authority. But as Bouchaud points out, Hugging Face already has a content policy, they just have no mechanism to enforce it.
The alternative is not just an ethics problem, it’s an existential business risk. If Hugging Face doesn’t clean up, it will face a regulatory crackdown that could be far more destructive to its business model than implementing some content filters. And the developer trust is already eroding, with policy changes affecting open access and developer trust causing friction within the community.
This moment is a test. Hugging Face is the de facto operating system for open-source AI. Its choices will define whether that operating system is a safe platform for innovation or a haven for abuse. The architecture of the platform can no longer be an excuse for inaction. The firestorm has arrived, and standing still is not an option.
