Hoax: Fake Russian “troll” error message

This case study was originally published in OpenAI’s October 2024⁠(opens in a new window) report.

Actor

On June 18, a post on X appeared to expose a Russian troll account whose credits for using GPT‑4o had expired. The post quickly went viral. Our investigation showed that this post was a hoax which could not have come from our models. However, earlier posts made by the same X account were generated using our models, apparently in an attempt to bait controversy. This activity likely originated in the United States. After further investigation, we banned this OpenAI account; public reporting shows that the X account has also been suspended.

Behavior

This hoax consisted of an OpenAI account and an account on X. Initially, the OpenAI account used our models to generate short, adversarial comments in reply to other people’s posts on X. These comments were then posted by the X account. This activity occurred in mid-June. On June 18, the X account posted a comment that appeared to be a JSON error message from a Russian-speaking user who was trying to generate content supportive of President Trump, but who had run out of credits. This message was not generated by our models, but appears to have been manually created. It included a snippet of apparent JSON code which was not valid JSON, and which mis-referenced our model’s name.

The hoax post quickly went viral, with tweets about it generating thousands of retweets, posts on LinkedIn and Reddit, multiple media queries, and even a debunk of the incident.