L7a: Autoresearch Loop

Build a self-improving agent with integrity guards.

Module: M7 Advanced Topics
Est. Time: 75 min
Files: starter.py, solution.py

Objective

Create an agent that runs experiments, measures its own performance, logs results, and decides whether to keep or discard each change.

Concepts

• Autoresearch: run → measure → log → decide
• Code hashing to detect grinding
• Median vs best comparison
• Reward hack defense

Starter

cd course/labs/L7-autoresearch/
python starter.py

Solution

python solution.py

Experiment Log Format

{"run": 1, "status": "baseline", "metric": {"name": "latency", "value": 52, "unit": "ms"}}
{"run": 2, "status": "keep", "metric": {"name": "latency", "value": 46}, "deltaPct": -11.5}
{"run": 3, "status": "discard", "metric": {"name": "latency", "value": 53}, "deltaPct": +1.9}

Integrity Guards

Threat	Detection	Prevention
Grinding (same code re-run)	Code hash comparison	Skip run
Noise-chasing	Median vs best comparison	Use median, not best
Reward hacking	Timing function isolation	Verify computation not shortcut
Test set leakage	Test data hash verification	Assert data unchanged