31 lines
614 B
Markdown
31 lines
614 B
Markdown
# L6a: Eval Harness
|
|
|
|
**Module**: M6 Economics
|
|
**Files**: starter.py, solution.py
|
|
|
|
## Objective
|
|
|
|
Build golden Q&A pairs + pass@k scoring system.
|
|
|
|
## Concepts
|
|
|
|
Refer to the corresponding module for full concept explanations.
|
|
|
|
## Starter
|
|
|
|
```bash
|
|
cd course/labs/l6-eval-harness/
|
|
# Open starter file and fill in the TODOs
|
|
```
|
|
|
|
## Checkpoints
|
|
|
|
1. Define 10+ test cases with tool and output expectations
|
|
2. Implement EvalHarness class with score_case
|
|
3. Compute pass@1, pass@3, pass@5 metrics
|
|
4. Support weighted scoring by case importance
|
|
|
|
## Solution
|
|
|
|
Compare against the solution file after attempting the starter.
|