Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI

Content moderation systems are typically evaluated by measuring agreement with human labels. In rule-governed environments, this assumption fails: multiple decisions may be logically consistent with the governing policy, and agreement metrics penalize AI systems for being too cautious. For instance, consider a moderation system that flags all posts containing profanity, and a human moderator who flags only some of those posts but also flags additional posts without profanity. In this case, the AI system and the human moderator have low agreement, but the AI system is still correct in its application of the policy. This highlights the need for a more nuanced evaluation approach. The proposed defensibility signals aim to address this issue by providing a more comprehensive understanding of an AI system's decision-making process. By considering both the correctness and confidence of AI decisions, defensibility signals can provide a more accurate evaluation of the system's performance. This is particularly important in high-stakes applications, where the consequences of incorrect decisions can be severe. By escaping the agreement trap, defensibility signals can help improve the development and deployment of AI systems in rule-governed environments.