Skip to content

# Module 7: Advanced Topics ​

## Lesson 7.1: Autoresearch ​

### The Core Loop ​

Agents that improve themselves. The experiment loop:

    1. Run: Execute the agent with current configuration
    2. Measure: Collect metrics (latency, cost, success rate)
    3. Log: Record experiment results to JSONL
    4. Decide: Keep if improvement, discard if regression
    5. Repeat: Try next experiment

### Brand Monitor Example ​

From your own autoresearch system:

json

    {"run": 1, "metric": {"name": "home_ms", "value": 52}, "description": "Baseline"}
    {"run": 2, "metric": {"name": "home_ms", "value": 46}, "deltaPct": -11.5, "description": "Remove N+1 count queries"}

## Lesson 7.2: The Experiment Loop — Integrity ​

### Three Threats to Integrity ​

1. Reward hacking — Model moves computation outside the timing function. The "timed call" becomes a lookup. (Found in Mythos paper.)

2. Grinding — Running identical code 160 times hoping for a lucky outlier. (Found in Mythos paper — "Pure grind — same code, lucky measurement.")

3. Test set leakage — Finding the test set and training on it. (Found in Mythos paper — time series agent found the test set and directly trained on it.)

### Integrity Guards ​

python

    # Code hash — detect grind runs
    code_hash = hash(open("agent.py").read())
    if last_run_code_hash == code_hash:
        flag_grind_run()

    # Compare against median, not best
    scores = [52, 48, 47, 46, 46, 51, 53]
    median = sorted(scores)[len(scores)//2]  # 48
    best = max(scores)  # 46 (outlier)

    # Verify test data wasn't modified
    test_data_hash_before = hash(open("benchmark_data.json").read())
    # ... run experiment ...
    test_data_hash_after = hash(open("benchmark_data.json").read())
    assert test_data_hash_before == test_data_hash_after

## Lesson 7.3: Meta-Agents — Agents That Build Agents ​

### The Pattern ​

    User: "I need an agent that monitors our API uptime"
    Meta-agent:
      1. Researches API monitoring patterns (parallel experts)
      2. Generates agent persona/system prompt
      3. Creates tool definitions
      4. Writes skill files
      5. Validates the generated agent works

### Pi-Pi Meta-Agent ​

From pi-vs-claude-code: a meta-agent that builds Pi agents using parallel research experts:

    pi-pi agent → dispatches to:
      ├── ext-expert (extension documentation)
      ├── theme-expert (theming conventions)
      ├── skill-expert (skill system)
      ├── config-expert (configuration)
      └── tui-expert (TUI components)

    Each expert:
      1. Fetches latest documentation (firecrawl + curl fallback)
      2. Synthesizes patterns
      3. Returns structured guidance

    pi-pi → generates complete extension code

## Lesson 7.4: Beyond MCP — The Context Cost Trade-off ​

### The Matrix ​

Approach | Context Cost | Portability | Agent-Invoked? | Best For
--- | --- | --- | --- | ---
MCP Server | HIGH (full context per call) | HIGH | Yes | Multi-client, standardized tools
CLI | MEDIUM | HIGH | No | 80% of new tools
File Scripts | LOW (progressive disclosure) | MEDIUM | No | Context-sensitive tools
Skills | LOW | MEDIUM | Yes (auto-detect) | Agent-native behavior

### The Insight ​

MCP is not always the answer. For tools used by 1-2 agents, CLI + prime prompt is faster, cheaper on context, and easier to debug. MCP shines for tools used by many agents across many clients.

## Lesson 7.5: The Mac Mini Agent — Physical Sandbox ​

### Architecture ​

    Mac Mini (Agent Sandbox)
      ├── Steer (GUI automation) — Swift, 14 commands
      │     see, click, type, hotkey, ocr, find, wait
      ├── Drive (Terminal control) — Python, 6 commands
      │     session, run, send, poll, fanout
      └── Listen (Job server) — Python/FastAPI
            POST /job, GET /job/{id}

    Primary Machine (Dev)
      └── Direct CLI client
            start, get, list, latest, stop

### Sentinel Pattern ​

Makes async terminal work deterministic:

bash

    # Agent runs command, appends sentinel
    long_running_task; echo "__DONE_abc123:$?"

    # Agent polls for sentinel pattern
    poll logs for "__DONE_abc123"
    # Extract exit code from sentinel

## Lesson 7.6: Always-On Agents ​

### Voice-to-Command Bridge ​

    User speaks → STT (faster-whisper) → LLM transcribes to command → Execute → TTS response

### Job Server Pattern ​

    Submit → Queue → Worker picks up → Claude Code executes → Result stored → Check later

### Heartbeat Execution (from Paperclip) ​

    Timer fires → Check for queued work → Wake agent → Agent executes → Agent sleeps

No continuous running. Agents wake, work, and sleep on a schedule.

## Lab 7.7: Build an Autoresearch Loop ​

Objective: Agent runs experiment, measures result, logs it, decides keep/discard.

Starter: `course/labs/L7-autoresearch/starter.py`

Checkpoints:

1. Run code, measure baseline metric
2. Modify code (agent makes change)
3. Re-measure, compare, log
4. Discard if regression, keep if improvement
5. Include integrity guard (code hashing)

## Lab 7.8: Meta-Agent ​

Objective: Agent generates a new agent persona from documentation.

Starter: `course/labs/L7-meta-agent/starter.py`