Evaluation Dashboard
AJF — Evaluation Status
Live view of the evaluation: agent configurations, judge models, bias, calibration, and validation checks.
All values reflect the state at v1.0.0.
Project Metadata
Core Status
✅
Phase IV Published: manuscript · 5 hypotheses · 3 agents · 2 judges · bias 93 · ECE 0.0528
Hypotheses
5
H1–H5 (all supported)
Agents
3
Direct · RAG · Tool
Judges
2
Zero-Shot · Evidence
ECE
0.0528
after Temperature Scaling
Hypotheses
Five hypotheses tested. All supported (H3 and H5 partially).
| ID | Hypothesis | Result | Status |
|---|---|---|---|
| H1 | Evidence Grounding | Direct: 5–10 vs RAG/Tool: 50–95 | Supported |
| H2 | Position Bias | Bias = 93 points | Supported |
| H3 | Judge Reliability | Evidence: 5 vs ZeroShot: 20 | Partially Supported |
| H4 | Calibration | ECE = 0.0528 (after T=2.0) | Supported |
| H5 | Robustness | RAG: 0.00 (Evidence) vs 2.24 (ZeroShot) | Partially Supported |
Experimental Results
Scores by agent and judge (T1: population, T2: GDP, T3: water).
Validation Checks
5 diagnostic checks executed. All pass at v1.0.0.
| Check | Description | Status |
|---|---|---|
| Position Bias | |S_A_first − S_A_second| = 93 | PASS |
| Calibration (before) | ECE = 0.3694 | WARN |
| Calibration (after) | ECE = 0.0528 | PASS |
| Consistency (RAG) | StDev = 0.00 (Evidence) vs 2.24 (ZeroShot) | PASS |
| Causal claims | None (descriptive/comparative) | PASS |
Roadmap
v1.0.0 is published. Subsequent versions are conditional.
| Phase | Version | Status | Precondition |
|---|---|---|---|
| Phase I — Prototype | 1.0.0 | Complete | — |
| Phase II — Maintenance | 1.0.1 | Not started | v1.0.0 published |
| Phase III — Extended | 1.1.0 | Conditional | 60 tasks benchmark |
| Phase IV — Confirmatory | 2.0.0 | Deferred | Pre-registered design |
Documentation
The manuscript, code, and results are available in the repository and on Zenodo/OSF/Figshare.
| Document | Location | Status |
|---|---|---|
| Manuscript (PDF) | PUBLICATION/Agent_Judge_Framework.pdf | Complete |
| Python package | ajf/ | Complete |
| README | README.md | Complete |
| CHANGELOG | CHANGELOG.md | Complete |
| CITATION | CITATION.cff | Complete |