AJF Documentation

A complete reference for AJF — Agent-Judge-Framework: the architecture, the agents, the judges, the reliability assessment, the repository structure, the evaluation pipeline, and the validation checks.

Overview

AJF (Agent-Judge-Framework) is a reliability-oriented framework designed to evaluate tool-using autonomous agents while simultaneously assessing the reliability of the evaluation process itself. It separates agent performance from evaluator reliability across four interconnected layers.

"The ability to generate a modified program is separated from the ability to establish its admissibility."

Architecture

The AJF architecture contains four interconnected layers:

Layer 1: Knowledge & Evidence Base
        ↓
Layer 2: Agent Execution (3 configurations)
        ↓
Layer 3: Independent Judge (2 models × 2 strategies)
        ↓
Layer 4: Reliability Assessment (Bias, Calibration, Agreement)

Layer 1 — Knowledge & Evidence Base

Reference documents, structured datasets, verified API responses, evidence records, ground-truth annotations.

Layer 2 — Agent Execution

Three configurations:

• Agent A — Direct Answer: Model-knowledge only, no external retrieval.
• Agent B — RAG-Based: Retrieves from a knowledge base before responding.
• Agent C — Tool-Using: Calls APIs, executes code, retrieves from external sources.

Layer 3 — Independent Judge

Two LLM judge models from independent families, evaluated under two strategies:

• Strategy A — Zero-Shot Judge: Standard evaluation without mandatory evidence verification.
• Strategy B — Structured Evidence Judge: Requires claim identification, evidence location, correspondence verification, and confidence estimation.

Layer 4 — Reliability Assessment

Human-Judge Agreement, Judge Consistency, Position Bias Detection, Verbosity Bias Detection, Evidence-Related Bias Detection, Confidence Calibration, Score Stability.

Agents (3 Configurations)

Judges (2 Strategies)

Reliability Assessment

The reliability layer measures four dimensions:

Experimental Results

Five hypotheses tested. All supported (H3 and H5 partially).

Key Numbers

• Position Bias: 93 points (severe).
• Calibration (before): ECE = 0.3694.
• Calibration (after): ECE = 0.0528.
• Consistency (RAG): StDev = 0.00 (Evidence) vs 2.24 (ZeroShot).

Reproducibility

To reproduce the evaluation:

# 1. Install the package
pip install agent-judge-framework

# 2. Set the Groq API key
export GROQ_API_KEY="gsk_..."

# 3. Run the experiments
python examples/run_calibration.py
python examples/run_consistency.py
python examples/run_bias.py
python examples/run_calibration_fix.py

The full manuscript, all code, and the experimental results are available in the repository and on Zenodo/OSF/Figshare.

Repository Structure

AJF/
├── README.md
├── AUTHORS.md
├── CHANGELOG.md
├── CITATION.cff
├── CONTRIBUTING.md
├── LICENSE
├── MANIFEST.in
├── VERSION
├── pyproject.toml
├── requirements.txt
├── setup.py
├── setup.cfg
├── upload_ajf.py
│
├── ajf/                      # Python package
│   ├── core/
│   ├── agents/
│   ├── judges/
│   ├── evidence/
│   ├── reliability/
│   ├── human_eval/
│   ├── statistics/
│   └── benchmark/
│
├── DATA/
│   └── knowledge_base.json
│
├── ANALYSIS/
├── PUBLICATION/
│   ├── Agent_Judge_Framework.pdf
│   ├── figures/
│   └── Netlify/
│       └── public/
│
├── REPORTS/
├── EXTERNAL_REVIEW/
├── docs/
├── examples/
│   ├── run_experiment.py
│   ├── run_consistency.py
│   ├── run_bias.py
│   └── run_calibration.py
└── tests/

Evaluation Pipeline

The Python package agent-judge-framework includes the full evaluation pipeline. Six stages:

1. Task Definition    →  Benchmark task (T1, T2, T3)
2. Agent Execution    →  Direct / RAG / Tool-Using
3. Output Logging     →  response, tool_calls, evidence
4. Judge Evaluation   →  Zero-Shot / Evidence Judge
5. Reliability        →  Bias, Calibration, Consistency
6. Reporting          →  JSON results + README

The package includes:

• ajf/agents/ — three agent configurations
• ajf/judges/ — two judge strategies (Groq API)
• ajf/reliability/ — bias, calibration, consistency
• examples/ — reproducible experiment scripts

Validation

Five validation checks are executed at v1.0.0. All pass.

Scope & Limits

What AJF supports

• Tool-using agent evaluation (Direct, RAG, Tool-Using).
• Two judge strategies (Zero-Shot, Structured Evidence).
• Position bias measurement (reversed order).
• Confidence calibration (ECE, Temperature Scaling).
• Consistency analysis (5 repetitions).
• Open manuscript (CC-BY-4.0) and open code (MIT).

What AJF does not support

• Human expert evaluation (deferred).
• Multi-judge consensus (deferred).
• 60-task benchmark (planned for future work).
• Cross-domain generalization.
• Causal inference (descriptive only).

Citation

Zenodo Archive

@software{baladi2026ajf_zenodo,
  author       = {Baladi, Samir},
  title        = {{Agent-Judge-Framework}: A Reliability-Oriented Framework
                  for Automated Evaluation of Tool-Using Autonomous Agents},
  year         = {2026},
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.23907823},
  url          = {https://doi.org/10.5281/zenodo.23907823},
  note         = {Ronin Institute / Rite of Renaissance}
}

APA (inline)

Baladi, S. (2026). Agent-Judge-Framework: A reliability-oriented framework for automated evaluation of tool-using autonomous agents (Version 1.0.0). Zenodo. https://doi.org/10.5281/zenodo.23907823

Status