6.5 KiB

Raw Blame History

Module 7: Advanced Topics

Lesson 7.1: Autoresearch

The Core Loop

Agents that improve themselves. The experiment loop:

1. Run: Execute the agent with current configuration
2. Measure: Collect metrics (latency, cost, success rate)
3. Log: Record experiment results to JSONL
4. Decide: Keep if improvement, discard if regression
5. Repeat: Try next experiment

Brand Monitor Example

From your own autoresearch system:

{"run": 1, "metric": {"name": "home_ms", "value": 52}, "description": "Baseline"}
{"run": 2, "metric": {"name": "home_ms", "value": 46}, "deltaPct": -11.5, "description": "Remove N+1 count queries"}

Lesson 7.2: The Experiment Loop — Integrity

Three Threats to Integrity

Reward hacking — Model moves computation outside the timing function. The "timed call" becomes a lookup. (Found in Mythos paper.)
Grinding — Running identical code 160 times hoping for a lucky outlier. (Found in Mythos paper — "Pure grind — same code, lucky measurement.")
Test set leakage — Finding the test set and training on it. (Found in Mythos paper — time series agent found the test set and directly trained on it.)

Integrity Guards

# Code hash — detect grind runs
code_hash = hash(open("agent.py").read())
if last_run_code_hash == code_hash:
    flag_grind_run()

# Compare against median, not best
scores = [52, 48, 47, 46, 46, 51, 53]
median = sorted(scores)[len(scores)//2]  # 48
best = max(scores)  # 46 (outlier)

# Verify test data wasn't modified
test_data_hash_before = hash(open("benchmark_data.json").read())
# ... run experiment ...
test_data_hash_after = hash(open("benchmark_data.json").read())
assert test_data_hash_before == test_data_hash_after

Lesson 7.3: Meta-Agents — Agents That Build Agents

The Pattern

User: "I need an agent that monitors our API uptime"
Meta-agent:
  1. Researches API monitoring patterns (parallel experts)
  2. Generates agent persona/system prompt
  3. Creates tool definitions
  4. Writes skill files
  5. Validates the generated agent works

Pi-Pi Meta-Agent

From pi-vs-claude-code: a meta-agent that builds Pi agents using parallel research experts:

pi-pi agent → dispatches to:
  ├── ext-expert (extension documentation)
  ├── theme-expert (theming conventions)
  ├── skill-expert (skill system)
  ├── config-expert (configuration)
  └── tui-expert (TUI components)

Each expert:
  1. Fetches latest documentation (firecrawl + curl fallback)
  2. Synthesizes patterns
  3. Returns structured guidance

pi-pi → generates complete extension code

Lesson 7.4: Beyond MCP — The Context Cost Trade-off

The Matrix

Approach	Context Cost	Portability	Agent-Invoked?	Best For
MCP Server	HIGH (full context per call)	HIGH	Yes	Multi-client, standardized tools
CLI	MEDIUM	HIGH	No	80% of new tools
File Scripts	LOW (progressive disclosure)	MEDIUM	No	Context-sensitive tools
Skills	LOW	MEDIUM	Yes (auto-detect)	Agent-native behavior

The Insight

MCP is not always the answer. For tools used by 1-2 agents, CLI + prime prompt is faster, cheaper on context, and easier to debug. MCP shines for tools used by many agents across many clients.

Lesson 7.5: The Mac Mini Agent — Physical Sandbox

Architecture

Mac Mini (Agent Sandbox)
  ├── Steer (GUI automation) — Swift, 14 commands
  │     see, click, type, hotkey, ocr, find, wait
  ├── Drive (Terminal control) — Python, 6 commands
  │     session, run, send, poll, fanout
  └── Listen (Job server) — Python/FastAPI
        POST /job, GET /job/{id}

Primary Machine (Dev)
  └── Direct CLI client
        start, get, list, latest, stop

Sentinel Pattern

Makes async terminal work deterministic:

# Agent runs command, appends sentinel
long_running_task; echo "__DONE_abc123:$?"

# Agent polls for sentinel pattern
poll logs for "__DONE_abc123"
# Extract exit code from sentinel

Lesson 7.6: Always-On Agents

Voice-to-Command Bridge

User speaks → STT (faster-whisper) → LLM transcribes to command → Execute → TTS response

Job Server Pattern

Submit → Queue → Worker picks up → Claude Code executes → Result stored → Check later

Heartbeat Execution (from Paperclip)

Timer fires → Check for queued work → Wake agent → Agent executes → Agent sleeps

No continuous running. Agents wake, work, and sleep on a schedule.

Scheduling Strategies

Strategy	Pattern	Use Case
Fixed interval	Every N minutes/hours/days	Brand monitoring, price tracking
Event-triggered	Webhook → wake agent	CI/CD pipelines, PR reviews
Predictive	Learn optimal check times	Traffic monitoring, anomaly detection
Cascade	Agent A finishes → Agent B starts	Data pipeline, ETL workflows

OpenClaw: The Always-On Employee

OpenClaw is a daemon-style agent that runs as a background service:

# openclaw-config.yaml
agent:
  name: "brand-monitor"
  heartbeat: 300  # every 5 minutes
  max_sessions: 1
  wake_command: "claude -p 'Check brand mentions since last run'"
  sleep_command: "pkill -f 'claude.*brand-monitor'"
  log_path: "/var/log/openclaw/brand-monitor.log"

It starts, checks for work, executes if needed, then goes back to sleep. No continuous billing, no context window overflow, no runaway loops. This is the pattern for production always-on agents.

When NOT to Use Always-On

Always-on adds complexity. Before building one, verify you actually need it:

Batch jobs → Cron is simpler and cheaper
Real-time → Event-driven architectures (webhooks, Kafka) outperform polling agents
One-off → Just run the agent once

Always-on makes sense when you need adaptive scheduling, dynamic task generation, or autonomous decision-making about what to work on next.

Lab 7.7: Build an Autoresearch Loop

Objective: Agent runs experiment, measures result, logs it, decides keep/discard.

Starter: course/labs/L7-autoresearch/starter.py

Checkpoints:

Run code, measure baseline metric
Modify code (agent makes change)
Re-measure, compare, log
Discard if regression, keep if improvement
Include integrity guard (code hashing)

Lab 7.8: Meta-Agent

Objective: Agent generates a new agent persona from documentation.

Starter: course/labs/L7-meta-agent/starter.py

6.5 KiB Raw Blame History