11 KiB

Raw Blame History

Module 7: Advanced Topics

Lesson 7.1: Autoresearch

The Core Loop

Agents that improve themselves. The experiment loop:

1. Run: Execute the agent with current configuration
2. Measure: Collect metrics (latency, cost, success rate)
3. Log: Record experiment results to JSONL
4. Decide: Keep if improvement, discard if regression
5. Repeat: Try next experiment

Brand Monitor Example

From your own autoresearch system:

{"run": 1, "metric": {"name": "home_ms", "value": 52}, "description": "Baseline"}
{"run": 2, "metric": {"name": "home_ms", "value": 46}, "deltaPct": -11.5, "description": "Remove N+1 count queries"}

Lesson 7.2: The Experiment Loop — Integrity

Three Threats to Integrity

Reward hacking — Model moves computation outside the timing function. The "timed call" becomes a lookup. (Found in Mythos paper.)
Grinding — Running identical code 160 times hoping for a lucky outlier. (Found in Mythos paper — "Pure grind — same code, lucky measurement.")
Test set leakage — Finding the test set and training on it. (Found in Mythos paper — time series agent found the test set and directly trained on it.)

Integrity Guards

# Code hash — detect grind runs
code_hash = hash(open("agent.py").read())
if last_run_code_hash == code_hash:
    flag_grind_run()

# Compare against median, not best
scores = [52, 48, 47, 46, 46, 51, 53]
median = sorted(scores)[len(scores)//2]  # 48
best = max(scores)  # 46 (outlier)

# Verify test data wasn't modified
test_data_hash_before = hash(open("benchmark_data.json").read())
# ... run experiment ...
test_data_hash_after = hash(open("benchmark_data.json").read())
assert test_data_hash_before == test_data_hash_after

Lesson 7.3: Meta-Agents — Agents That Build Agents

The Pattern

User: "I need an agent that monitors our API uptime"
Meta-agent:
  1. Researches API monitoring patterns (parallel experts)
  2. Generates agent persona/system prompt
  3. Creates tool definitions
  4. Writes skill files
  5. Validates the generated agent works

Pi-Pi Meta-Agent

From pi-vs-claude-code: a meta-agent that builds Pi agents using parallel research experts:

pi-pi agent → dispatches to:
  ├── ext-expert (extension documentation)
  ├── theme-expert (theming conventions)
  ├── skill-expert (skill system)
  ├── config-expert (configuration)
  └── tui-expert (TUI components)

Each expert:
  1. Fetches latest documentation (firecrawl + curl fallback)
  2. Synthesizes patterns
  3. Returns structured guidance

pi-pi → generates complete extension code

Lesson 7.4: Beyond MCP — The Context Cost Trade-off

The Matrix

Approach	Context Cost	Portability	Agent-Invoked?	Best For
MCP Server	HIGH (full context per call)	HIGH	Yes	Multi-client, standardized tools
CLI	MEDIUM	HIGH	No	80% of new tools
File Scripts	LOW (progressive disclosure)	MEDIUM	No	Context-sensitive tools
Skills	LOW	MEDIUM	Yes (auto-detect)	Agent-native behavior

The Insight

MCP is not always the answer. For tools used by 1-2 agents, CLI + prime prompt is faster, cheaper on context, and easier to debug. MCP shines for tools used by many agents across many clients.

Lesson 7.5: The Mac Mini Agent — Physical Sandbox

Architecture

Mac Mini (Agent Sandbox)
  ├── Steer (GUI automation) — Swift, 14 commands
  │     see, click, type, hotkey, ocr, find, wait
  ├── Drive (Terminal control) — Python, 6 commands
  │     session, run, send, poll, fanout
  └── Listen (Job server) — Python/FastAPI
        POST /job, GET /job/{id}

Primary Machine (Dev)
  └── Direct CLI client
        start, get, list, latest, stop

Sentinel Pattern

Makes async terminal work deterministic:

# Agent runs command, appends sentinel
long_running_task; echo "__DONE_abc123:$?"

# Agent polls for sentinel pattern
poll logs for "__DONE_abc123"
# Extract exit code from sentinel

Why Physical Sandbox Matters

Cloud agents are ephemeral. A Mac Mini agent is persistent — it has files, databases, browser sessions, and GPU state that survive agent restarts. This matters for:

Long-running experiments that take hours or days
Browser automation with persistent login sessions
Local model inference where GPU memory is precious
Legacy system access that can't be moved to cloud

The physical sandbox pattern works with any mini PC — Mac Mini, Intel NUC, Raspberry Pi 5, or repurposed laptop. The key is a dedicated machine that the agent can control without affecting your primary workstation.

Lesson 7.6: Multi-Model Strategies

Why One Model Isn't Enough

No single model is best at everything. Frontier models (Claude Opus, GPT-4o) excel at reasoning but cost 10-100x more than small models (Haiku, Flash, Nano). The strategy: use the right model for each subtask.

Model Heterogeneity in Practice

Task: "Analyze this codebase for security vulnerabilities"

Step 1: File Discovery (Haiku — $0.0003/run)
  → List all files, grep for patterns
  → Cheap, fast, 100% accuracy needed

Step 2: Vulnerability Analysis (Sonnet — $0.008/run)
  → Read each suspicious file, classify vulnerability
  → Needs reasoning, but not frontier-level

Step 3: Report Generation (Opus — $0.03/run)
  → Synthesize findings into executive report
  → Needs highest quality output

Total cost: ~$0.04 instead of ~$0.30 if done entirely with Opus. 87% savings with no quality loss on the critical path.

The 3-Tier Cascade Pattern

TIERS = {
    "cheap": {"model": "claude-haiku",  "max_tokens": 4000,  "cost_per_k": 0.00025},
    "mid":   {"model": "claude-sonnet", "max_tokens": 8000,  "cost_per_k": 0.003},
    "premium": {"model": "claude-opus",  "max_tokens": 16000, "cost_per_k": 0.015},
}

def cascade(prompt, complexity="mid"):
    """Route to appropriate tier based on task complexity."""
    if complexity == "low":
        return call_model(TIERS["cheap"], prompt)
    elif complexity == "high":
        return call_model(TIERS["premium"], prompt)
    else:
        return call_model(TIERS["mid"], prompt)

Model Selection Decision Tree

Is the task deterministic? (grep, sort, count)
  → Use cheap model (Haiku/Flash)
  
Does the task need reasoning? (analyze, explain)
  → Use mid model (Sonnet/4o-mini)
  
Is the output customer-facing? (report, email, memo)
  → Use premium model (Opus/4o)

Will this code be deployed to production?
  → Run through ALL three tiers in sequence (cascade)
  → Cheap for bulk work, mid for analysis, premium for final review

Lesson 7.6: Always-On Agents

Voice-to-Command Bridge

User speaks → STT (faster-whisper) → LLM transcribes to command → Execute → TTS response

Job Server Pattern

Submit → Queue → Worker picks up → Claude Code executes → Result stored → Check later

Heartbeat Execution (from Paperclip)

Timer fires → Check for queued work → Wake agent → Agent executes → Agent sleeps

No continuous running. Agents wake, work, and sleep on a schedule.

Scheduling Strategies

Strategy	Pattern	Use Case
Fixed interval	Every N minutes/hours/days	Brand monitoring, price tracking
Event-triggered	Webhook → wake agent	CI/CD pipelines, PR reviews
Predictive	Learn optimal check times	Traffic monitoring, anomaly detection
Cascade	Agent A finishes → Agent B starts	Data pipeline, ETL workflows

OpenClaw: The Always-On Employee

OpenClaw is a daemon-style agent that runs as a background service:

# openclaw-config.yaml
agent:
  name: "brand-monitor"
  heartbeat: 300  # every 5 minutes
  max_sessions: 1
  wake_command: "claude -p 'Check brand mentions since last run'"
  sleep_command: "pkill -f 'claude.*brand-monitor'"
  log_path: "/var/log/openclaw/brand-monitor.log"

It starts, checks for work, executes if needed, then goes back to sleep. No continuous billing, no context window overflow, no runaway loops. This is the pattern for production always-on agents.

When NOT to Use Always-On

Always-on adds complexity. Before building one, verify you actually need it:

Batch jobs → Cron is simpler and cheaper
Real-time → Event-driven architectures (webhooks, Kafka) outperform polling agents
One-off → Just run the agent once

Always-on makes sense when you need adaptive scheduling, dynamic task generation, or autonomous decision-making about what to work on next.

Lesson 7.7: MCP + Identity — Authenticated Agent Tools

The Problem

Every MCP server so far has been public and unauthenticated. But production agents need to access private data — Google Drive, Slack, GitHub, SaaS APIs. That means OAuth, tokens, and identity management.

The Pattern: External Auth via MCP

Agent → MCP Server → OAuth Provider → External API
                  ↓
            Access Token (stored by MCP server)

The MCP server handles the OAuth flow. The agent just calls tools. The server manages token refresh, storage, and authentication headers.

Descope + Google Drive Example

From the agent-identity pattern, an MCP server that authenticates via Descope before accessing Google Drive:

from fastmcp import FastMCP
import descope  # OAuth management

mcp = FastMCP("google-drive-mcp")

@mcp.tool()
def search_drive(query: str):
    """Search Google Drive. Handles OAuth internally."""
    token = descope.get_token("google-drive")
    headers = {"Authorization": f"Bearer {token}"}
    resp = requests.get(
        "https://www.googleapis.com/drive/v3/files",
        params={"q": query},
        headers=headers,
    )
    return resp.json()

The agent doesn't know about OAuth, tokens, or refresh flows. It just calls search_drive("budget 2026") and gets results.

Identity Layer Options

Approach	Complexity	Best For
Descope (managed)	Low	Teams, multiple services, audit logs
OAuth2 Proxy	Medium	Self-hosted, single service
API Key passthrough	Low	Simple integrations, personal use
MCP with auth headers	Medium	Direct API access, dev tools

MCP Auth Spec (Upcoming)

The MCP protocol is standardizing auth. Future MCP servers will include:

# .mcp.json with auth
{
  "mcpServers": {
    "google-drive": {
      "command": "uv",
      "args": ["run", "google_drive_server.py"],
      "env": {
        "DESCOPE_MANAGEMENT_KEY": "${DESCOPE_KEY}"
      }
    }
  }
}

The key insight: the agent doesn't manage auth. The MCP server does. This keeps the agent simple and the auth secure.

Lab 7.7: Build an Autoresearch Loop

Objective: Agent runs experiment, measures result, logs it, decides keep/discard.

Starter: course/labs/L7-autoresearch/starter.py

Checkpoints:

Run code, measure baseline metric
Modify code (agent makes change)
Re-measure, compare, log
Discard if regression, keep if improvement
Include integrity guard (code hashing)

Lab 7.8: Meta-Agent

Objective: Agent generates a new agent persona from documentation.

Starter: course/labs/L7-meta-agent/starter.py

11 KiB Raw Blame History