Programmable checks that inspect user input before it reaches the model (jailbreak/prompt-injection, off-topic, banned requests) and model output before it reaches the user or a tool (schema, policy, hallucination, unsafe action).
How it works
Frameworks like NVIDIA NeMo Guardrails, Guardrails AI, and Llama Guard / Prompt Guard create a safety net around AI models by implementing a series of checks known as guardrails. These guardrails operate on both user input and model output. For input, they use a combination of regular expressions, classifiers, and small-model judges to evaluate and potentially modify the incoming data. If the input violates certain criteria, the system can either reject it outright or rewrite it to fit acceptable standards. This ensures that problematic prompts, like those attempting to bypass restrictions, never reach the model.
For output, these guardrails validate the generated responses before they are delivered to the user or any external tools. They check the structure of the output, run moderation classifiers to filter out inappropriate content, and verify that the response is grounded in retrieved context. Depending on the results of these checks, the system can block the output, redact sensitive information, or trigger a re-generation of the response. This multi-layered approach ensures that the model's outputs are safe and relevant, maintaining a high standard of interaction.
The policies governing these guardrails are expressed in formats like Colang, YAML, or even custom code. This allows developers to version and test these policies independently from the prompts themselves. By separating the guardrails from the model's training data, we create a more flexible and auditable system, which can adapt to new threats or requirements without altering the core model.
- Inspect user input before model processing.
- Validate model output before user delivery.
- Use regex and classifiers for checks.
- Version guardrails separately from prompts.
- Tune thresholds based on risk tolerance.
The software-engineering analogue
You can think of guardrails as similar to input validation combined with an API gateway or Web Application Firewall (WAF) that protects a service. In traditional software, you define strict rules that incoming requests must follow. However, the key difference here is that the 'attacker' input is in natural language, and the 'service'—the AI model—is nondeterministic. This means that instead of having exact rules, the guardrails rely on probabilistic classifiers that can assess the likelihood of input or output being harmful or inappropriate. This introduces its own challenges with false positives and negatives, making the guardrails a more complex system to manage.
Why not the obvious alternative
A common initial approach is to embed rules directly into the system prompt, such as stating 'never do X.' However, this method is easily bypassable through prompt injection attacks, where users can manipulate the input to circumvent these directives. Rules embedded in prompts are not easily auditable, making it difficult to track what was allowed or rejected. This lack of transparency can lead to significant security risks.
In contrast, external guardrails operate deterministically, meaning they apply the same checks regardless of user input. They cannot be argued out of by the user, as they are independent systems that enforce rules. These guardrails generate logs that provide an audit trail of what was blocked and why, enhancing accountability and allowing for better oversight of interactions with the AI.
When to use which
For public-facing applications or systems that have agentic capabilities, it's crucial to implement comprehensive input and output guardrails. This ensures that all interactions are thoroughly vetted for safety and relevance. In contrast, for internal tools that operate in low-risk environments, a lighter touch may be sufficient. In such cases, simple output schema validation might be enough to ensure that generated responses meet basic criteria. It's essential to tune the thresholds for these checks based on your organization’s tolerance for over-blocking versus the risk of allowing harmful content.
In production: when it breaks
Why it happens. This can occur when the guardrails are either too strict or misconfigured, resulting in legitimate user inputs being incorrectly classified. If the policy engine is not updated with the latest guidelines or context, it may fail to recognize valid inputs, causing false positives in filtering.
- Check the guardrail configuration for accuracy and relevance.
- Log the flagged inputs to identify patterns or commonalities.
- Analyze the policy engine's update frequency and ensure it aligns with current guidelines.
What it delivers
By implementing guardrails, you create a defensible and testable safety boundary that operates independently of how prompts are worded. This not only enhances the reliability of the AI's interactions but also provides an audit trail that details what was blocked and the rationale behind those decisions. This level of oversight is critical for maintaining trust and safety in AI applications.