AgentJudge
Agent-Judge-Framework
Automated Evaluation and Reliability Assessment of Tool-Using Autonomous Agents
Evaluating Agents is Not Enough
Evaluating an agent solely through its final textual response may fail to capture important dimensions of its actual performance, including tool execution reliability, evidence validity, and consistency. LLM-as-a-Judge offers scalability, but is limited by position bias, verbosity bias, self-enhancement bias, and imperfect human agreement.
No Standard
Existing methods rely on human evaluation (costly, slow) or LLM-as-a-Judge (fast but biased).
No Reliability
An agent's score cannot be interpreted as a reliable measurement of its actual performance.
No Verification
Judges may accept citations that appear convincing but do not support the associated claims.
Four Interconnected Layers
Layer 1 — Knowledge
Reference documents, structured datasets, ground-truth annotations.
Layer 2 — Agent Execution
Three configurations: Direct, RAG, Tool-Using.
Layer 3 — Independent Judge
Two strategies: Zero-Shot, Structured Evidence.
Layer 4 — Reliability
Bias, Calibration, Agreement.
All Hypotheses Supported
View the Full Analysis
Live dashboard, detailed documentation, and the full manuscript.
Open Dashboard →