Tag: evaluation

Showing 1-3 of 3

Apr 18, 20268 min readAgent PlaybookIntermediate180 min build

VAKRA benchmark: reproducible execution traces for diagnosing multi-step agent tool use

Guides running VAKRA's runnable benchmark—8,000+ local APIs across 62 domains—to record full execution traces, reproduce common multi‑step agent failures, and guide focused fixes.

VAKRA agents benchmarking tool-use failure-modes

+3 more

TutorialsFrance

Open

Feb 18, 20267 min readTooling Deep DiveAdvanced240 min build

Building an AI-Chat Evaluation Harness for He Xin's PEPC Formal Language (Part 2)

Build an AI-chat evaluation harness for He Xin’s PEPC formal language to test expressiveness, contradiction handling, and alignment with wargame baselines — includes artifacts and metrics.

PEPC He Xin formal-language evaluation LLM

+3 more

wargaming knowledge-graphs concept-drift

TutorialsFrance

Open

Feb 09, 20267 min readAgent PlaybookIntermediate240 min build

Build an APEX-Agents-style harness to evaluate AI agents' multi-domain performance

Reproducible tutorial to build an APEX-Agents-style test harness measuring AI agents' ability to stitch context across Slack and Google Drive. Includes configs, logs and rollout gates.

ai-agents benchmarking APEX-Agents Mercor knowledge-work

+3 more

evaluation production-readiness reliability