49 lines
1.3 KiB
Markdown
49 lines
1.3 KiB
Markdown
# L7a: Autoresearch Loop
|
|
|
|
Build a self-improving agent with integrity guards.
|
|
|
|
**Module**: M7 Advanced Topics
|
|
**Est. Time**: 75 min
|
|
**Files**: `starter.py`, `solution.py`
|
|
|
|
## Objective
|
|
|
|
Create an agent that runs experiments, measures its own performance, logs results, and decides whether to keep or discard each change.
|
|
|
|
## Concepts
|
|
|
|
- Autoresearch: run → measure → log → decide
|
|
- Code hashing to detect grinding
|
|
- Median vs best comparison
|
|
- Reward hack defense
|
|
|
|
## Starter
|
|
|
|
```bash
|
|
cd course/labs/L7-autoresearch/
|
|
python starter.py
|
|
```
|
|
|
|
## Solution
|
|
|
|
```bash
|
|
python solution.py
|
|
```
|
|
|
|
## Experiment Log Format
|
|
|
|
```jsonl
|
|
{"run": 1, "status": "baseline", "metric": {"name": "latency", "value": 52, "unit": "ms"}}
|
|
{"run": 2, "status": "keep", "metric": {"name": "latency", "value": 46}, "deltaPct": -11.5}
|
|
{"run": 3, "status": "discard", "metric": {"name": "latency", "value": 53}, "deltaPct": +1.9}
|
|
```
|
|
|
|
## Integrity Guards
|
|
|
|
| Threat | Detection | Prevention |
|
|
|--------|-----------|------------|
|
|
| Grinding (same code re-run) | Code hash comparison | Skip run |
|
|
| Noise-chasing | Median vs best comparison | Use median, not best |
|
|
| Reward hacking | Timing function isolation | Verify computation not shortcut |
|
|
| Test set leakage | Test data hash verification | Assert data unchanged |
|