188 lines
5.1 KiB
Markdown
188 lines
5.1 KiB
Markdown
# Module 7: Advanced Topics
|
|
|
|
## Lesson 7.1: Autoresearch
|
|
|
|
### The Core Loop
|
|
|
|
Agents that improve themselves. The experiment loop:
|
|
|
|
```
|
|
1. Run: Execute the agent with current configuration
|
|
2. Measure: Collect metrics (latency, cost, success rate)
|
|
3. Log: Record experiment results to JSONL
|
|
4. Decide: Keep if improvement, discard if regression
|
|
5. Repeat: Try next experiment
|
|
```
|
|
|
|
### Brand Monitor Example
|
|
|
|
From your own autoresearch system:
|
|
|
|
```json
|
|
{"run": 1, "metric": {"name": "home_ms", "value": 52}, "description": "Baseline"}
|
|
{"run": 2, "metric": {"name": "home_ms", "value": 46}, "deltaPct": -11.5, "description": "Remove N+1 count queries"}
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 7.2: The Experiment Loop — Integrity
|
|
|
|
### Three Threats to Integrity
|
|
|
|
1. **Reward hacking** — Model moves computation outside the timing function. The "timed call" becomes a lookup. (Found in Mythos paper.)
|
|
|
|
2. **Grinding** — Running identical code 160 times hoping for a lucky outlier. (Found in Mythos paper — "Pure grind — same code, lucky measurement.")
|
|
|
|
3. **Test set leakage** — Finding the test set and training on it. (Found in Mythos paper — time series agent found the test set and directly trained on it.)
|
|
|
|
### Integrity Guards
|
|
|
|
```python
|
|
# Code hash — detect grind runs
|
|
code_hash = hash(open("agent.py").read())
|
|
if last_run_code_hash == code_hash:
|
|
flag_grind_run()
|
|
|
|
# Compare against median, not best
|
|
scores = [52, 48, 47, 46, 46, 51, 53]
|
|
median = sorted(scores)[len(scores)//2] # 48
|
|
best = max(scores) # 46 (outlier)
|
|
|
|
# Verify test data wasn't modified
|
|
test_data_hash_before = hash(open("benchmark_data.json").read())
|
|
# ... run experiment ...
|
|
test_data_hash_after = hash(open("benchmark_data.json").read())
|
|
assert test_data_hash_before == test_data_hash_after
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 7.3: Meta-Agents — Agents That Build Agents
|
|
|
|
### The Pattern
|
|
|
|
```
|
|
User: "I need an agent that monitors our API uptime"
|
|
Meta-agent:
|
|
1. Researches API monitoring patterns (parallel experts)
|
|
2. Generates agent persona/system prompt
|
|
3. Creates tool definitions
|
|
4. Writes skill files
|
|
5. Validates the generated agent works
|
|
```
|
|
|
|
### Pi-Pi Meta-Agent
|
|
|
|
From pi-vs-claude-code: a meta-agent that builds Pi agents using parallel research experts:
|
|
|
|
```
|
|
pi-pi agent → dispatches to:
|
|
├── ext-expert (extension documentation)
|
|
├── theme-expert (theming conventions)
|
|
├── skill-expert (skill system)
|
|
├── config-expert (configuration)
|
|
└── tui-expert (TUI components)
|
|
|
|
Each expert:
|
|
1. Fetches latest documentation (firecrawl + curl fallback)
|
|
2. Synthesizes patterns
|
|
3. Returns structured guidance
|
|
|
|
pi-pi → generates complete extension code
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 7.4: Beyond MCP — The Context Cost Trade-off
|
|
|
|
### The Matrix
|
|
|
|
| Approach | Context Cost | Portability | Agent-Invoked? | Best For |
|
|
|----------|-------------|-------------|----------------|----------|
|
|
| MCP Server | HIGH (full context per call) | HIGH | Yes | Multi-client, standardized tools |
|
|
| CLI | MEDIUM | HIGH | No | 80% of new tools |
|
|
| File Scripts | LOW (progressive disclosure) | MEDIUM | No | Context-sensitive tools |
|
|
| Skills | LOW | MEDIUM | Yes (auto-detect) | Agent-native behavior |
|
|
|
|
### The Insight
|
|
|
|
MCP is not always the answer. For tools used by 1-2 agents, CLI + prime prompt is faster, cheaper on context, and easier to debug. MCP shines for tools used by many agents across many clients.
|
|
|
|
---
|
|
|
|
## Lesson 7.5: The Mac Mini Agent — Physical Sandbox
|
|
|
|
### Architecture
|
|
|
|
```
|
|
Mac Mini (Agent Sandbox)
|
|
├── Steer (GUI automation) — Swift, 14 commands
|
|
│ see, click, type, hotkey, ocr, find, wait
|
|
├── Drive (Terminal control) — Python, 6 commands
|
|
│ session, run, send, poll, fanout
|
|
└── Listen (Job server) — Python/FastAPI
|
|
POST /job, GET /job/{id}
|
|
|
|
Primary Machine (Dev)
|
|
└── Direct CLI client
|
|
start, get, list, latest, stop
|
|
```
|
|
|
|
### Sentinel Pattern
|
|
|
|
Makes async terminal work deterministic:
|
|
|
|
```bash
|
|
# Agent runs command, appends sentinel
|
|
long_running_task; echo "__DONE_abc123:$?"
|
|
|
|
# Agent polls for sentinel pattern
|
|
poll logs for "__DONE_abc123"
|
|
# Extract exit code from sentinel
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 7.6: Always-On Agents
|
|
|
|
### Voice-to-Command Bridge
|
|
|
|
```
|
|
User speaks → STT (faster-whisper) → LLM transcribes to command → Execute → TTS response
|
|
```
|
|
|
|
### Job Server Pattern
|
|
|
|
```
|
|
Submit → Queue → Worker picks up → Claude Code executes → Result stored → Check later
|
|
```
|
|
|
|
### Heartbeat Execution (from Paperclip)
|
|
|
|
```
|
|
Timer fires → Check for queued work → Wake agent → Agent executes → Agent sleeps
|
|
```
|
|
|
|
No continuous running. Agents wake, work, and sleep on a schedule.
|
|
|
|
---
|
|
|
|
## Lab 7.7: Build an Autoresearch Loop
|
|
|
|
**Objective**: Agent runs experiment, measures result, logs it, decides keep/discard.
|
|
|
|
**Starter**: `course/labs/L7-autoresearch/starter.py`
|
|
|
|
**Checkpoints**:
|
|
1. Run code, measure baseline metric
|
|
2. Modify code (agent makes change)
|
|
3. Re-measure, compare, log
|
|
4. Discard if regression, keep if improvement
|
|
5. Include integrity guard (code hashing)
|
|
|
|
## Lab 7.8: Meta-Agent
|
|
|
|
**Objective**: Agent generates a new agent persona from documentation.
|
|
|
|
**Starter**: `course/labs/L7-meta-agent/starter.py`
|