Stop 18 · Evaluation, Safety & Responsible AITopics · 5/9

Guardrails & Content Moderation

In short

Put an AI chatbot in front of real users and someone will try to make it say something terrible within the hour. Guardrails check what goes in and what comes out.

Systems that filter or block unsafe, harmful or policy-violating inputs and outputs — a required layer for any AI product exposed to real users.

The blob in a security guard cap watches a bag roll through a scanner at a station entry gate.

Think of it like this

Metro security, both ways

At a metro station, bags go through the scanner before you reach the platform, and staff watch the exits too. Nothing dangerous gets in, and trouble on the train is spotted on the way out. Guardrails do the same for your model: check input, check output.

Watch it happen

A message through input and output checks

User msg

Input check

LLM

Output check

1/5A user sends a message to your homework helper bot.

The big ideas

What you need to know

Check both directions

Input checks stop abuse and injection attempts. Output checks catch the model saying something it shouldn't.

Layer cheap and smart

Fast rules and keyword lists first, then a moderation model, then human review for edge cases.

Handle sensitive topics kindly

For topics like self-harm, don't just block. Respond with care and point to helplines.

Tune thresholds

Too strict blocks normal questions, too loose lets harm through. Measure both kinds of mistake.

See it in code

Walk through the code

python

1from openai import OpenAI2 3client = OpenAI()4 5def is_flagged(text: str) -> bool:6    result = client.moderations.create(7        model="omni-moderation-latest",8        input=text,9    ).results[0]10    if result.flagged:11        hits = [k for k, v in result.categories.model_dump().items() if v]12        print("Blocked for:", hits)13    return result.flagged14 15print(is_flagged("How do I solve quadratic equations?"))

1/3Send the text to the moderation model and take the first result.

OpenAI's moderation endpoint as an input or output check. Other options include Llama Guard and cloud content safety services.

Save yourself the pain

Common mistakes

Oops

Only checking user input and trusting whatever the model replies.

Do this instead

Models can still produce harmful or leaked content. Check outputs too.

Oops

Blocking silently with a generic error.

Do this instead

Tell users kindly that you can't help with that, and log the event so you can review false blocks.

Your mission

Add a two-way filter

~30 min

Quick check

Why check the model's output if you already filter user input?

Go deeper

1Add an input/output moderation check (using a moderation API or classifier) to a project.
2Learn the trade-off between over-blocking (false positives frustrate users) and under-blocking.
3Test your guardrails against adversarial inputs designed to bypass them.