What is Content Moderation?
Also called Content Filtering, Safety Classification.
Content moderation is the classification and handling of text, images, or other media against a defined policy covering categories such as violence, sexual content, harassment, self-harm, and illegal activity. In AI systems it is applied to user input, to generated output, or to both, with actions ranging from blocking to rewriting to escalating for human review.
Moderation begins with a written policy, not with a model. The policy defines each category, states what falls inside and outside it, and settles edge cases such as clinical discussion of self-harm, historical description of violence, and quoted slurs in a news context. A classifier can only be as coherent as the definitions it was trained or prompted against.
Threshold setting is a value judgment expressed as a number. A strict threshold blocks more harmful content and also more legitimate content, which frustrates users and can disadvantage groups whose ordinary speech is misread. A permissive threshold does the reverse. Because the two error types fall on different people, the choice is a policy decision that engineering alone should not make.
Coverage is uneven in ways that matter. Classifiers generally perform best in English and in the domains most represented in their training data, and degrade on other languages, code-switching, regional slang, sarcasm, and coded terminology that shifts faster than models are retrained. Systems serving multiple regions should measure per-language rates rather than assume a single reported figure holds.
Moderation at scale is contested territory. Critics point to opaque appeals, inconsistent enforcement, cultural bias in what counts as harmful, and the psychological cost borne by human reviewers. Regulatory expectations differ across jurisdictions, so a single global policy rarely satisfies every requirement, and providers increasingly maintain regional variations of both policy and enforcement.
Key points
- Starts from a written policy, not from a classifier
- Threshold choice trades over-blocking against under-blocking
- Accuracy degrades outside well-represented languages and domains
- Different error types harm different groups
- Norms and legal expectations vary by jurisdiction
In practice
A wellness assistant applies a self-harm classifier to every incoming message. The policy separates two cases: a user describing distress, which triggers a supportive response and a crisis resource, and a request for method information, which is refused. Early tuning blocked clinical questions from professional users, so the policy added an explicit carve-out with a narrower trigger and a review path for disputed blocks.