Contextual Evaluation of LLM Guardrails Across Languages and Agentic Systems

Guardrails - mechanisms that filter, flag, or constrain Large Language Model (LLM) inputs and outputs - have become a core component of the AI safety stack, shaping how AI systems behave in chatbots, social media platforms, and public-facing services. Yet while the research community devotes significant attention to evaluating the outputs and capabilities of LLMs themselves, the guardrail models that govern what users actually see and experience have received less scrutiny. We argue that evaluating guardrails is as important as evaluating the LLMs they protect.

Historically, guardrails have been difficult to examine: often they performed as proprietary classifiers embedded in content moderation systems, visible to users only through refusals or minimally explained filtering decisions. The recent proliferation of open-source guardrail models, with accessible architectures, chain-of-thought reasoning traces, and open safety taxonomies, has created an opportunity for independent evaluation.

This tutorial invites FAccT researchers to critically examine guardrails across languages, domains, and deployment contexts. Drawing on our multilingual, context-aware evaluation of guardrails in a humanitarian setting and on the design of Mozilla AI's open-source any-guardrail framework, we show how guardrails can be systematically and contextually evaluated and that others can do this too, in their own context and language of interest.

We also identify key research frontiers in LLM guardrails design: dynamic safety taxonomies, agentic capabilities such as tool use and retrieval for fact-checking and trustworthiness, multimodal coverage, and feedback loops between evaluation results and guardrail improvement.