Stop 19 · System Design for AI ApplicationsTopics · 7/9

Designing a Content Moderation System

In short

A platform with millions of posts a day can't have humans read everything, and can't trust a model with every call. Real moderation systems mix rules, models and people in layers.

Architecting a system that filters harmful content at scale — combining fast rule-based filters, ML classifiers, and human review escalation paths.

The blob in a security vest checks bags at a festive gate decorated with flowers and lights.

Think of it like this

Security at the college fest

At the fest gate, guards stop anyone on the banned list straight away. Bags get a quick check. Anything doubtful goes to the supervisor. Students who think they were wrongly stopped can appeal to the dean. Content moderation follows the same layers: fast rules, smart checks, humans, appeals.

Watch it happen

One uploaded post through the layers

Upload

Hash match

Classifier

Human queue

Decision

1/5A post with text and an image is uploaded.

The big ideas

What you need to know

Cheap layers first

Hash lists and rules are fast and cheap. Save ML and humans for what gets through.

Two thresholds

Auto-remove above a high score, auto-allow below a low score, send the middle to humans.

Feedback loop

Human decisions become labelled data to retrain the classifiers and tune thresholds.

Care for reviewers

Human moderators see disturbing content. Limit exposure, rotate tasks and provide support.

See it in code

Walk through the code

python

1def moderate(post, classify, known_bad_hashes):2    if post["image_hash"] in known_bad_hashes:3        return "remove"  # exact match to known banned content4 5    score = classify(post["text"])  # 0.0 = safe, 1.0 = clearly harmful6    if score >= 0.95:7        return "remove"8    if score >= 0.60:9        return "human_review"10    return "allow"

1/3The cheapest check runs first: is this already known banned content?

The routing core. The thresholds here are examples; real ones are tuned per policy using labelled data.

Save yourself the pain

Common mistakes

Oops

One global threshold for every kind of policy.

Do this instead

Spam, hate and self-harm need different thresholds and different responses. Tune each separately.

Oops

No way for users to appeal.

Do this instead

Models make mistakes. Appeals catch them and give you labelled examples to improve.

Your mission

Tune two thresholds

~35 min

Quick check

Why send middle-score posts to humans instead of auto-deciding?

Go deeper

1Practice designing a moderation pipeline with multiple filtering stages of increasing cost/accuracy.
2Consider the trade-off between moderation latency and thoroughness for real-time content.
3Think through how you'd handle appeals and human-in-the-loop review.