Moral hazard: Can’t trust OpenAI claims on ‘Erotica’

I’ve read more smut at work than you can possibly imagine, all of it while working at OpenAI.

In the spring of 2021, I led the company’s product safety team and discovered a crisis involving erotic content. One prominent customer was running a text-based role-playing game that used our AI to generate interactive stories. These quickly turned into sexual fantasies — including encounters involving children and violent abductions — sometimes initiated by users, sometimes by the AI itself. One analysis found that more than 30 percent of players’ conversations were “explicitly lewd.”

After months of debate over user freedom, we prohibited erotic uses of our models. Erotica itself wasn’t the problem — it was the intensity of users’ emotional attachments to chatbots. For people struggling with