72 lines
2.3 KiB
Markdown
72 lines
2.3 KiB
Markdown
# Autoresearch Kit
|
|
|
|
Agents that improve themselves. Run experiments, measure results, keep what works.
|
|
|
|
## Included Skills
|
|
|
|
1. **Experiment Loop** — Run → Measure → Log → Decide
|
|
2. **Integrity Guard** — Code hashing to detect grinding
|
|
3. **Median Over Best** — Compare against median, not best
|
|
4. **Reward Hack Defense** — Detect timing manipulation
|
|
5. **Log to JSONL** — Structured experiment logging
|
|
|
|
## The Experiment Loop
|
|
|
|
```
|
|
1. RUN: Execute agent with current configuration
|
|
2. MEASURE: Collect metrics (latency, cost, success rate)
|
|
3. LOG: Record to experiments.jsonl
|
|
4. DECIDE: Keep if improvement, discard if regression
|
|
5. REPEAT: Try next experiment
|
|
```
|
|
|
|
## Integrity Guards
|
|
|
|
### Code Hashing (against grinding)
|
|
|
|
```python
|
|
code_hash = sha256(open("agent.py").read()).hexdigest()[:12]
|
|
if last_run_code_hash == code_hash:
|
|
flag_grind_run() # Same code re-run — discard result
|
|
```
|
|
|
|
### Median Over Best (against noise-chasing)
|
|
|
|
```python
|
|
scores = [52, 48, 47, 46, 46, 51, 53]
|
|
median = sorted(scores)[len(scores)//2] # 48
|
|
best = max(scores) # 46 — don't use this
|
|
improvement = (baseline - median) / baseline # Use median
|
|
```
|
|
|
|
### Test Set Integrity (against data leakage)
|
|
|
|
```python
|
|
test_data_hash_before = hash(open("benchmark.json").read())
|
|
# ... run experiment ...
|
|
test_data_hash_after = hash(open("benchmark.json").read())
|
|
assert test_data_hash_before == test_data_hash_after
|
|
```
|
|
|
|
## Experiment Log Format
|
|
|
|
```jsonl
|
|
{"run": 1, "status": "baseline", "metric": {"name": "latency", "value": 52, "unit": "ms"}, "description": "Initial measurement"}
|
|
{"run": 2, "status": "keep", "metric": {"name": "latency", "value": 46, "unit": "ms"}, "deltaPct": -11.5, "description": "Remove N+1 count queries", "code_hash": "a1b2c3d4"}
|
|
{"run": 3, "status": "discard", "metric": {"name": "latency", "value": 53, "unit": "ms"}, "deltaPct": +1.9, "description": "Add connection pool", "code_hash": "e5f6g7h8"}
|
|
```
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python autoresearch.py --max-iterations 10
|
|
# Results in experiments.jsonl
|
|
cat experiments.jsonl | python -m json.tool
|
|
```
|
|
|
|
## Related Course Material
|
|
|
|
- M7 Advanced Topics — Autoresearch lesson + lab
|
|
- `mythos-learnings.md` — original research on reward hacking, grinding
|
|
- `brand-monitor/autoresearch.jsonl` — real experiment data (52→46ms)
|