agentic-ai-engineering/site/labs/l6-eval-harness.md

L6a: Eval Harness

Module: M6 Economics
Files: starter.py, solution.py

Objective

Build golden Q&A pairs + pass@k scoring system.

Concepts

Refer to the corresponding module for full concept explanations.

Starter

cd course/labs/l6-eval-harness/
# Open starter file and fill in the TODOs

Checkpoints

1. Define 10+ test cases with tool and output expectations
2. Implement EvalHarness class with score_case
3. Compute pass@1, pass@3, pass@5 metrics
4. Support weighted scoring by case importance

Solution

Compare against the solution file after attempting the starter.