Module 7: Advanced Topics

Lesson 7.1: Autoresearch

The Core Loop

Agents that improve themselves. The experiment loop:

1. Run: Execute the agent with current configuration
2. Measure: Collect metrics (latency, cost, success rate)
3. Log: Record experiment results to JSONL
4. Decide: Keep if improvement, discard if regression
5. Repeat: Try next experiment

Brand Monitor Example

From your own autoresearch system:

{"run": 1, "metric": {"name": "home_ms", "value": 52}, "description": "Baseline"}
{"run": 2, "metric": {"name": "home_ms", "value": 46}, "deltaPct": -11.5, "description": "Remove N+1 count queries"}

---

Lesson 7.2: The Experiment Loop — Integrity

Three Threats to Integrity

1. Reward hacking — Model moves computation outside the timing function. The "timed call" becomes a lookup. (Found in Mythos paper.)

2. Grinding — Running identical code 160 times hoping for a lucky outlier. (Found in Mythos paper — "Pure grind — same code, lucky measurement.")

3. Test set leakage — Finding the test set and training on it. (Found in Mythos paper — time series agent found the test set and directly trained on it.)

Integrity Guards

# Code hash — detect grind runs
code_hash = hash(open("agent.py").read())
if last_run_code_hash == code_hash:
    flag_grind_run()

# Compare against median, not best
scores = [52, 48, 47, 46, 46, 51, 53]
median = sorted(scores)[len(scores)//2]  # 48
best = max(scores)  # 46 (outlier)

# Verify test data wasn't modified
test_data_hash_before = hash(open("benchmark_data.json").read())
# ... run experiment ...
test_data_hash_after = hash(open("benchmark_data.json").read())
assert test_data_hash_before == test_data_hash_after

---

Lesson 7.3: Meta-Agents — Agents That Build Agents

The Pattern

User: "I need an agent that monitors our API uptime"
Meta-agent:
  1. Researches API monitoring patterns (parallel experts)
  2. Generates agent persona/system prompt
  3. Creates tool definitions
  4. Writes skill files
  5. Validates the generated agent works

Pi-Pi Meta-Agent

From pi-vs-claude-code: a meta-agent that builds Pi agents using parallel research experts:

pi-pi agent → dispatches to:
  ├── ext-expert (extension documentation)
  ├── theme-expert (theming conventions)
  ├── skill-expert (skill system)
  ├── config-expert (configuration)
  └── tui-expert (TUI components)

Each expert:
  1. Fetches latest documentation (firecrawl + curl fallback)
  2. Synthesizes patterns
  3. Returns structured guidance

pi-pi → generates complete extension code

---

Lesson 7.4: Beyond MCP — The Context Cost Trade-off

The Matrix

Approach	Context Cost	Portability	Agent-Invoked?	Best For
MCP Server	HIGH (full context per call)	HIGH	Yes	Multi-client, standardized tools
CLI	MEDIUM	HIGH	No	80% of new tools
File Scripts	LOW (progressive disclosure)	MEDIUM	No	Context-sensitive tools
Skills	LOW	MEDIUM	Yes (auto-detect)	Agent-native behavior

The Insight

MCP is not always the answer. For tools used by 1-2 agents, CLI + prime prompt is faster, cheaper on context, and easier to debug. MCP shines for tools used by many agents across many clients.

---

Lesson 7.5: The Mac Mini Agent — Physical Sandbox

Architecture

Mac Mini (Agent Sandbox)
  ├── Steer (GUI automation) — Swift, 14 commands
  │     see, click, type, hotkey, ocr, find, wait
  ├── Drive (Terminal control) — Python, 6 commands
  │     session, run, send, poll, fanout
  └── Listen (Job server) — Python/FastAPI
        POST /job, GET /job/{id}

Primary Machine (Dev)
  └── Direct CLI client
        start, get, list, latest, stop

Sentinel Pattern

Makes async terminal work deterministic:

# Agent runs command, appends sentinel
long_running_task; echo "__DONE_abc123:$?"

# Agent polls for sentinel pattern
poll logs for "__DONE_abc123"
# Extract exit code from sentinel

---

Lesson 7.6: Always-On Agents

Voice-to-Command Bridge

User speaks → STT (faster-whisper) → LLM transcribes to command → Execute → TTS response

Job Server Pattern

Submit → Queue → Worker picks up → Claude Code executes → Result stored → Check later

Heartbeat Execution (from Paperclip)

Timer fires → Check for queued work → Wake agent → Agent executes → Agent sleeps

No continuous running. Agents wake, work, and sleep on a schedule.

---

Lab 7.7: Build an Autoresearch Loop

Objective: Agent runs experiment, measures result, logs it, decides keep/discard.

Starter: course/labs/L7-autoresearch/starter.py

Checkpoints:

1. Run code, measure baseline metric
2. Modify code (agent makes change)
3. Re-measure, compare, log
4. Discard if regression, keep if improvement
5. Include integrity guard (code hashing)

Lab 7.8: Meta-Agent

Objective: Agent generates a new agent persona from documentation.

Starter: course/labs/L7-meta-agent/starter.py