Upgrading the Moderation API with our new multimodal moderation model

OpenAI

We’re introducing a new model built on GPT‑4o that is more accurate at detecting harmful text and images, enabling developers to build more robust moderation systems.

Today we are introducing a new moderation model, omni-moderation-latest, in the Moderation API⁠(opens in a new window). Based on GPT‑4o⁠, the new model supports both text and image inputs and is more accurate than our previous model, especially in non-English languages. Like the previous version, this model uses OpenAI's GPT‑based classifiers to assess whether content should be flagged across categories such as hate, violence, and self-harm, while also adding the ability to detect additional harm categories. Additionally, it provides more granular control over moderation decisions by calibrating probability scores to reflect the likelihood of content matching the detected category. The new moderation model is free to use for all developers through the Moderation API.

Since we first

launched⁠ the Moderation API in 2022, the volume and variety of content that automated moderation systems need to handle has increased, especially as more AI apps have reached massive scale in production. We hope today’s upgrades help more developers benefit from the latest research and investments in our safety systems.

Companies across various sectors—from social media platforms and productivity tools to generative AI platforms—are using the Moderation API to build safer products for their users. For instance, Grammarly is using the Moderation API as part of the safety guardrails in its AI communications assistance to ensure its products outputs are safe and fair. Similarly, ElevenLabs utilizes the Moderation API along with in-house solutions to scan content generated by their audio AI products, preventing and flagging outputs that violate their policies.

The updated moderation model includes a number of major improvements:

• Calibrated scores: the new model’s scores now more accurately represent the probability that a piece of content violates the relevant policies and will be significantly more consistent across future moderation models.

• API Platform
• 2024

Authors

Ian Kivlichan, Justyn Harriman, Cameron Raymond, Meghan Shah, Shraman Ray Chaudhuri, Keren Gu-Lemberg