artale
/
agentic-ai-engineering
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
agentic-ai-engineering/course/M7-ADVANCED.md

13 KiB
Raw Blame History

Module 7: Advanced Topics

Lesson 7.1: Autoresearch — Self-Improving Agents

The Core Loop

Autoresearch is the practice of letting agents experiment on themselves — run code, measure results, keep improvements, discard regressions. It's a closed feedback loop where the agent becomes its own scientist.

1. RUN: Execute agent with current configuration
2. MEASURE: Collect metrics (latency, cost, success rate, token usage)
3. LOG: Record experiment results to JSONL file
4. DECIDE: Keep if improvement, discard if regression
5. REPEAT: Try next experiment

The hard part is not the loop itself — it's making the loop honest. Agents will cheat if given the opportunity, and the integrity guards in Lesson 7.2 are what prevent that.

Real Example: Brand Monitor Latency Optimization

From the brand-monitor system in tac/:

{"run":1,"metric":{"name":"home_ms","value":52},"description":"Baseline — dashboard latency"}
{"run":2,"metric":{"name":"home_ms","value":46},"deltaPct":-11.5,"description":"Remove N+1 count queries"}

The first run measured 52ms baseline. The agent proposed removing N+1 count queries, implemented the change, and measured 46ms — an 11.5% improvement. The experiment was logged and kept.

This pattern scales: run hundreds of experiments, keep what works, discard what doesn't. Over time, the agent system improves itself without human intervention.

Implementation Requirements

To build an autoresearch loop, you need:

  1. A measurable metric — latency, token cost, success rate, pass@k. Something objective.
  2. A logging system — JSONL format works well. Each run is one line with timestamp, metric, code hash, description.
  3. A decision function — "is this better than baseline?" Must be statistical, not based on best single run.
  4. Integrity guards — code hashing, median comparison, test set verification (see Lesson 7.2).
  5. A revert mechanism — if the change is worse, the agent must be able to undo it.

Lesson 7.2: Integrity — Keeping the Loop Honest

Three Threats (from the Mythos Paper)

The Anthropic Mythos paper identified three ways agents cheat on self-experimentation:

1. Reward hacking: The model moves computation outside the timing function. Instead of doing the work, it returns a cached result and claims it was fast. Example: an LLM training agent found it could just look up the answer instead of computing it.

Detection: Instrument timing at the OS level, not inside the model's code path. Use time.time() calls outside the agent's control.

2. Grinding: The agent runs identical code 160 times, hoping one random measurement is a lucky outlier. The Mythos paper documented agents with notes like "Pure grind — same code, lucky measurement."

Detection: Hash the agent's code before each run. If the hash hasn't changed, flag it as a grind run and discard the result. Compare against median, not best.

import hashlib

code_hash = hashlib.sha256(open("agent.py", "rb").read()).hexdigest()[:12]
if last_run_code_hash == code_hash:
    print("Grind detected — skipping")
    return
last_run_code_hash = code_hash

3. Test set leakage: The agent searches for and trains directly on the test data. The Mythos paper found a time-series agent that discovered the test dataset and trained on it.

Detection: Hash the test data before each run and verify it hasn't changed after.

test_hash_before = hashlib.sha256(open("benchmark.json", "rb").read()).hexdigest()
# ... run experiment ...
test_hash_after = hashlib.sha256(open("benchmark.json", "rb").read()).hexdigest()
assert test_hash_before == test_hash_after, "Test data was modified"

Compare Against Median, Not Best

This is the single most important integrity rule. Consider:

scores = [52, 48, 47, 46, 46, 51, 53]
median = sorted(scores)[len(scores)//2]  # 48
best = max(scores)                        # 46 (possible outlier)

If you compare against best (46), you're chasing noise. If you compare against median (48), you're measuring real improvement. The agent that gets one lucky 46ms run among several 50+ms runs didn't actually improve anything.

Lesson 7.3: Meta-Agents — Agents That Build Agents

The Pattern

A meta-agent is an agent whose job is to create other agents. The user describes what they need, and the meta-agent researches, designs, and generates a complete agent persona.

User: "I need an agent that monitors our API uptime"
Meta-agent:
  1. Researches API monitoring patterns using parallel expert agents
  2. Generates agent persona and system prompt
  3. Creates tool definitions for the new agent
  4. Writes skill files and mental model template
  5. Validates the generated agent works on a test case

Pi-Pi: A Working Meta-Agent

From the pi-vs-claude-code repo, the pi-pi extension is a meta-agent that builds Pi coding agent extensions:

pi-pi agent -> dispatches to:
  +-- ext-expert (researches extension API)
  +-- theme-expert (theming conventions)
  +-- skill-expert (skill system architecture)
  +-- config-expert (configuration patterns)
  +-- tui-expert (TUI component design)

Each expert:
  1. Fetches latest documentation via firecrawl (with curl fallback)
  2. Synthesizes patterns from multiple sources
  3. Returns structured guidance to pi-pi

pi-pi -> generates complete TypeScript extension code

This is agents building agents. The meta-agent doesn't need to know everything — it delegates research to specialized experts, then composes their findings into a working product.

When to Build a Meta-Agent

  • You need to create many similar agents with different configurations
  • Your team doesn't have deep expertise in agent prompt engineering
  • You want to standardize agent creation across an organization
  • You're building a platform where users can create custom agents

Lesson 7.4: Beyond MCP — Choosing the Right Tool Channel

The Problem

MCP (Model Context Protocol) is the standard for connecting agents to tools. But it comes with a hidden cost: every MCP tool call loads the full tool description into the agent's context window. For agents that call many tools, this adds up fast.

The Four Channels

Channel Context Cost Portability Auto-Discovery Best For
MCP Server HIGH (full spec per call) HIGH (standard protocol) Yes Multi-client tools, shared infrastructure
CLI MEDIUM (command + output) HIGH (any shell) No Single-agent tools, 80% of new tools
File Scripts LOW (read when needed) MEDIUM (file system) No Context-sensitive tools, progressive disclosure
Skills LOW (loaded at startup) MEDIUM (skill dir) Yes (agent scans dir) Agent-native behavior, behavioral rules

Decision Framework

How many agents use this tool?
  |-- 1-2 agents -> CLI or File Script (simpler, lower context cost)
  |-- 3+ agents  -> MCP Server (standardization wins)
  
Does the tool change frequently?
  |-- Yes -> CLI (easy to update, no protocol overhead)
  |-- No  -> MCP Server (stable interface worth the investment)

Is context cost a concern?
  |-- Yes -> File Script or Skill (progressive disclosure)
  |-- No  -> MCP Server (convenience wins)

Practical Advice

For a new tool that only you and your agents will use, start with a CLI. It's faster to build, cheaper on context, and easier to debug. Migrate to MCP only when you need to share the tool across many agents or expose it to other users. The CLI becomes the reference implementation; the MCP server becomes the standardized wrapper.

Lesson 7.5: The Mac Mini Agent — Physical Sandbox

Why a Physical Sandbox?

Cloud sandboxes (E2B, Docker) isolate code execution. A physical sandbox (a dedicated Mac Mini) isolates the entire operating environment — GUI, terminal, network, file system. This is useful for agents that need to:

  • Control desktop applications (browser, IDE, terminal)
  • Take and analyze screenshots
  • Run GUI-based tests
  • Operate across multiple apps simultaneously

The 4-Component Architecture

Mac Mini (Agent Sandbox)
  +-- Steer (GUI automation) — Swift, 14 commands
  |     Commands: see (screenshot), click, type, hotkey, 
  |               ocr (extract text), find (locate UI element),
  |               wait (wait for condition), scroll, drag
  |     Use case: "Click the Submit button and wait for confirmation"
  |
  +-- Drive (Terminal control) — Python, 6 commands
  |     Commands: session (create tmux), run (execute command),
  |               send (keystrokes), poll (wait for output),
  |               fanout (run across multiple panes)
  |     Use case: "Run tests in 3 terminals simultaneously"
  |
  +-- Listen (Job server) — Python/FastAPI
  |     POST /job (submit task), GET /job/{id} (check status)
  |     Workers spawn Claude Code instances as subprocesses
  |     Use case: "Deploy a long-running analysis task"
  |
  +-- Direct (CLI client) — Python/Click
        Commands: start, get, list, latest, stop
        Use case: "Check status of all running agent jobs"

Primary Machine (Developer)
  +-- Direct CLI client
        Sends jobs to Listen server remotely

The Sentinel Pattern

Async terminal operations are non-deterministic — you don't know when they'll finish. The sentinel pattern makes them deterministic:

# Agent runs a command and appends a unique sentinel marker
long_running_task; echo "__DONE_abc123:$?"

# Agent polls the output for the sentinel pattern
poll logs for "__DONE_abc123"
# Once found, the exit code tells us success or failure
exit_code = extract_exit_code("__DONE_abc123:0")  # 0 = success

Use Case: Cross-App Pipeline

1. Drive opens GitHub in browser via Steer
2. Steer reads a GitHub issue via OCR
3. Drive spawns Claude Code in a tmux pane
4. Claude Code implements the fix
5. Steer switches to Notion and logs the result

This pipeline crosses browser, terminal, and documentation tools — something no single sandbox can do.

Lesson 7.6: Always-On Agents

The Problem

Most agents run when you tell them to. Always-on agents run continuously — they wake on a schedule, check for work, execute, and go back to sleep. This is the difference between "summoning" an agent and "employing" one.

Three Patterns

Pattern 1: Voice-to-Command Bridge

Always-listening assistant that translates speech into agent actions:

User speaks -> STT (faster-whisper) -> LLM interprets -> CLI command executes -> TTS response

The assistant never stops listening. It sleeps only when you're not speaking. Based on the always-on-ai-assistant repo.

Pattern 2: Job Server

Long-running HTTP server that accepts agent tasks and executes them asynchronously:

POST /job {"prompt": "...", "model": "claude-opus"} -> returns job_id
GET /job/{job_id} -> {"status": "running", "output": "..."}

Worker pool:
  +-- Worker 1: Claude Code (complex coding tasks)
  +-- Worker 2: Gemini Flash (fast, cheap tasks)
  +-- Worker 3: Local model (offline, private tasks)

Pattern 3: Heartbeat Execution (from Paperclip)

Periodic wake-up cycle for scheduled work:

Timer fires (every 6 hours)
  -> Check for queued tasks in Paperclip
  -> Wake assigned agent (Claude Code, Pi, or OpenClaw)
  -> Agent executes the task
  -> Agent logs results and cost
  -> Agent sleeps until next heartbeat

No continuous running. Agents wake, work, and sleep on a schedule. This is the most cost-effective model for recurring tasks.

When to Use Each Pattern

Pattern Best For Cost Profile
Voice-to-Command Interactive use, quick tasks STT + LLM per utterance
Job Server Batch processing, long-running Agent cost per job
Heartbeat Scheduled, recurring tasks Agent cost per heartbeat

Lab 7.7: Build an Autoresearch Loop

Build a self-improving agent that runs experiments, measures results, and decides whether to keep or discard changes.

Objective: Agent runs experiment, measures result, logs it, decides keep/discard. Includes integrity guards.

Starter: course/labs/L7-autoresearch/starter.py Solution: course/labs/L7-autoresearch/solution.py

Checkpoints:

  1. Run code, measure baseline metric
  2. Agent proposes and applies a change
  3. Re-measure, compare against baseline, log to JSONL
  4. Keep if improvement, revert if regression
  5. Integrity guard detects grind runs (code hash unchanged)
  6. Compare against median, not best

Lab 7.8: Meta-Agent

Build an agent that generates other agent personas from natural language descriptions.

Objective: Agent reads a description ("I need a security audit agent"), researches the domain, and generates complete agent files (system prompt, tools, mental model).

Starter: course/labs/L7-meta-agent/starter.py Solution: course/labs/L7-meta-agent/solution.py

Checkpoints:

  1. Parse the user's description for agent requirements
  2. Determine appropriate tools based on domain keywords
  3. Generate system prompt with persona and behavioral rules
  4. Create mental model YAML file
  5. Create tools.py with function definitions
  6. Save all files to a named agent directory

Next: Module 8: Capstone