Skip to content

# Module 5: Production Patterns ​

## Lesson 5.1: What Production Means for Agents ​

Production for agents is fundamentally different from traditional software:

Traditional Software | Agent Systems
--- | ---
Deterministic output | Non-deterministic behavior
Fixed cost per operation | Variable cost per session
Error = known exception | Error = unexpected behavior
Rollback = revert code | Rollback = revert prompt + pin model
Monitoring = latency + errors | Monitoring = token usage + loop depth
Testing = unit + integration | Testing = golden datasets + evals

### Production Readiness Checklist ​

    [ ] Version-locked prompts (hashed, diffed, reviewed)
    [ ] Model pinning (not "latest", specific version)
    [ ] Cost budgets per session/task/user
    [ ] Iteration limits (max tool calls per session)
    [ ] Observability (every tool call logged)
    [ ] Shadow deployment capability
    [ ] Rollback plan (prompt + model + env)
    [ ] Security review (L3+ minimum)

## Lesson 5.2: CI/CD for Agents ​

### Golden Dataset ​

A curated set of input/output/behavior pairs that define correct agent behavior:

json

    [
      {
        "input": "Find all users created in the last 24 hours",
        "expected_tools": ["query_database"],
        "expected_tool_params": {"query": "SELECT * FROM users WHERE created_at > now() - interval '24 hours'"},
        "expected_output_contains": ["users", "24 hours"]
      }
    ]

### Pipeline ​

    1. Agent runs against golden dataset
    2. Compare: tool selections match expected?
                params match expected?
                output contains expected strings?
    3. Calculate pass@k
    4. If pass@k < threshold (e.g., 80%), FAIL build
    5. If pass, deploy new prompt/config

## Lesson 5.2b: Case Study — The 5-Tool Production Stack ​

A real production multi-agent deployment uses multiple agent tools together, each for its strength. See `TOOL-REFERENCE.md` for full command references.

### The Stack ​

    agent-mux (Tauri UI) —— Meta-agent control plane
        │   └── cc agent SDK, pi-coding-agent SDK, opencode SDK via sidecar
        ▼
    mprocs (process monitor) —— Launches all agents
        │   mprocs -c ~/mprocs-teams.yaml
        │
        ├── claude-lead (Claude Code)
        │     └── psmux → tmux.exe → --teammate-mode → split panes
        │           └── Each teammate = separate Claude session
        │
        ├── pi-agent (Pi Coding Agent)
        │     └── Extensions: damage-control, tilldone, coms
        │
        ├── opencode (OpenCode CLI)
        │     └── Model: opencode-go/deepseek-v4-flash (via proxy)
        │
        ├── hermes (Hermes Agent)
        │     └── TypeScript-native, MCP-first workflows
        │
        ├── openclaw (OpenClaw daemon)
        │     └── Always-on employee, heartbeat-driven
        │
        ├── gemini (Gemini fallback)
        │     └── Fast/cheap tasks, cascade routing
        │
        ├── qwen (Qwen specialist)
        │     └── Chinese + structured tasks
        │
        └── sidecar (agent-mux IPC proxy)
              └── RPC bridge between Tauri UI and agent processes
              │
              ▼
        psmux (tmux session manager)
        │   tmux.exe at ~/.cargo/bin/tmux
        │   Requires: start agent-teams first, then dmux inside it
        │
        ▼
        dmux (git worktree isolation)
            Each task gets an isolated worktree
            Rollback = delete worktree

### Tool Roles and Selection Logic ​

Tool | Role | When to Use | Stack Position
--- | --- | --- | ---
Claude Code | Primary coding agent | Complex multi-step tasks, general development | `claude-lead` in mprocs
Pi Agent | Customizable harness | Custom workflows, safety-critical ops, P2P | Side agent with extensions
OpenCode | OSS alternative | Budget tasks, CI/CD, when license matters | Backup in mprocs
Hermes | TypeScript pipelines | MCP-native workflows, structured output | Specialist in mprocs
OpenClaw | Always-on employee | Scheduled tasks, heartbeats, recurring | Daemon (always running)
Gemini | Fast/cheap fallback | High-volume simple tasks | Cascade routing tier 1
Qwen | Specialist model | Chinese content, structured generation | Cascade routing tier 2

### How They Work Together (Real Session Flow) ​

    1. Human opens agent-mux Tauri UI
    2. mprocs launches all agents from mprocs-teams.yaml
    3. Claude Code (lead) runs in tmux via psmux
         └── --teammate-mode creates split panes:
               ├── pane 1: lead (primary coder)
               ├── pane 2: worker (sub-tasks)
               ├── pane 3: reviewer (code review)
               └── pane 4: verifier (read-only checks)
    4. OpenCode runs alongside as budget-aware backup
    5. OpenClaw daemon handles scheduled background tasks
    6. dmux isolates each task in its own git worktree
    7. agent-mux sidecar collects status from all agents
    8. Human monitors via Tauri UI, intervenes when needed

### Key Production Patterns ​

1. Model heterogeneity — Different models for different roles. Cascade routing in practice (M6).
2. Tool heterogeneity — Five CLIs, each with different strengths. No single point of failure.
3. Process management — mprocs supervises. If one agent crashes, the stack keeps running.
4. Session isolation — psmux (terminal sessions) + dmux (git worktrees) = two layers.
5. Meta-control plane — agent-mux Tauri UI. Human watches and intervenes, not drives.
6. Defense in depth — tool-level (damage-control), session-level (psmux), filesystem-level (dmux).

## Lesson 5.2c: The Agent Manager Role ​

In enterprise deployments, someone owns the agent harness. This is the Agent Manager (or DevEx Lead for AI).

### Responsibilities ​

    Agent Manager
    ├── Harness design (CLAUDE.md, skills, hooks, MCPs)
    ├── Tool selection (which agent CLIs, which models)
    ├── Security policy (damage-control rules, access levels)
    ├── Cost management (budgets per agent/task, optimization)
    ├── Quality gates (golden datasets, regression testing)
    ├── Update cadence (prompt versioning, model pinning)
    └── Incident response (tool loops, cost spikes, failures)

### 90-Day Setup Playbook ​

Month 1: Foundation

- Set up agent CLI (Claude Code, Pi, or OpenCode)
- Create CLAUDE.md with project context
- Install damage-control with 3 access levels
- Set up basic observability (tool call logging)

Month 2: Scale

- Add multi-agent teams (lead + workers)
- Create golden dataset (10+ test cases)
- Implement CI/CD gate
- Set up cost tracking and budgets

Month 3: Production

- Shadow deployment pipeline
- Rollback procedures documented
- Monitoring dashboard live
- Team trained on agent interaction patterns

## Lesson 5.3: Shadow Deployments ​

### How It Works ​

    Production agent: serves user traffic
    Shadow agent: runs IDENTICAL inputs, but outputs are NOT served
    Compare: did shadow make same decisions as production?
             did shadow cost more/less?
             did shadow hit any errors?
    Decision: if shadow improves on all metrics, swap them

### When to Shadow Deploy ​

- New prompt version
- New model version
- New tool addition
- Agent architecture change

## Lesson 5.4: Rollback Strategies ​

### What Rollback Means for Agents ​

You can't just revert a Git commit. Agent behavior depends on:

1. Prompt — the text of the system prompt + tools
2. Model — which model version
3. Parameters — temperature, top_p, etc.
4. Configuration — tool list, iteration limits, budget

A proper rollback restores ALL four.

### Implementation ​

yaml

    # agent-config-v42.yaml
    prompt_hash: "a1b2c3d4"
    model: "claude-sonnet-4-20260501"  # pinned, not "latest"
    temperature: 0.0
    max_iterations: 25
    tools: ["read", "write", "bash", "search"]
    budget_per_session: 0.50

Rollback = `cp agent-config-v41.yaml agent-config.yaml` + reload.

## Lesson 5.5: Observability & Monitoring ​

### What to Trace (Every Single Turn) ​

1. Input prompt (full, including system prompt)
2. LLM response (including tool call choices)
3. Tool calls (name, params, timestamp)
4. Tool results (output, error status, duration)
5. Token counts (input, output, cached)
6. Cost (per-call and running total)
7. Loop depth (current turn number)

### Decision Tracing ​

Standard APM (Datadog, Grafana) captures latency and errors. Agents need decision tracing — the full chain of reasoning and actions:

json

    {
      "session_id": "sess_abc123",
      "turn": 5,
      "input_tokens": 12400,
      "output_tokens": 350,
      "tool_calls": [
        {"tool": "search_web", "params": {"query": "latest pricing"}, "duration_ms": 1200}
      ],
      "decision": "Found pricing page, will extract"
    }

### Key Metrics ​

Metric | Warning | Critical
--- | --- | ---
Tool calls per session | >20 | >50
Cost per session | >$0.50 | >$2.00
Loop depth | >15 | >30
Same tool >5x in row | Investigate loop | Kill session
Context utilization | >80% | >95%

## Lesson 5.6: Alerting on Agent-Specific Signals ​

### What to Alert On ​

1. Tool loop detected — same tool called 5+ times with same params
2. Cost spike — session cost > 3x average
3. Context overflow imminent — token count within 10% of limit
4. Permission escalation — agent attempting blocked operations
5. Error cascade — 3+ tool failures in a row
6. Grinding detected — identical code rerun without changes

### Alert Routing ​

    P0 (immediate): Cost spike > $10, permission escalation, data exfil attempt
    P1 (within 5 min): Tool loop, error cascade, grinding
    P2 (within 1 hour): Context utilization high, cost trending up
    P3 (daily report): Average session cost, success rate, failure modes

## Lesson 5.6b: Cross-Provider Session Search ​

When you run agents across 5+ tools (Claude Code, Pi, OpenCode, Gemini, OpenClaw), session history is scattered across different directories and formats.

### The Problem ​

bash

    ~/.claude/sessions/*.jsonl      # Claude Code format
    ~/.pi/sessions/*.jsonl          # Pi format
    ~/.opencode/sessions/*          # OpenCode format
    ~/.gemini/sessions/*.jsonl      # Gemini format
    mprocs-logs/*.log               # mprocs supervisor logs

Searching across all of them is impossible without a unified index.

### The Solution ​

Reference implementation: Jeff Emanuel's `coding_agent_session_search` (783★)

    Indexer:
      ├── Watches all session directories
      ├── Normalizes into canonical format
      ├── Full-text indexes prompts, responses, tool calls
      └── Stores in SQLite with FTS5

    Search CLI:
      ├── Search across ALL providers from one command
      ├── Filter by: provider, date, model, tool, token count
      ├── Replay any session from any provider
      └── Export sessions as markdown or JSON

### Why This Matters for Production ​

1. Debugging: Find what went wrong across all agents in one search
2. Audit: Full history of every agent action, unified format
3. Learning: Search past solutions instead of re-solving problems
4. Cost analysis: Aggregate costs across all providers

### Integration with Our Stack ​

bash

    # Add to mprocs-teams.yaml as a sidecar service
    session-indexer:
      cmd: ["python", "session-indexer.py", "--watch-dirs",
            "~/.claude/sessions/", "~/.pi/sessions/",
            "~/.opencode/sessions/", "logs/"]

    # Search across everything
    session-search "deployment error" --providers claude,pi,opencode
    # Returns results from all providers in unified format

## Lesson 5.7: Deployment Modes ​

From Paperclip's deployment model:

Mode | Auth | Exposure | Use Case
--- | --- | --- | ---
Local trusted | None | localhost only | Single dev machine
Authenticated private | Login required | LAN/Tailscale/VPN | Team on private network
Authenticated public | Login required | Internet (behind reverse proxy) | Production cloud deployment

### Reachability ​

Bind | What It Means
--- | ---
`loopback` | localhost only (default)
`lan` | All interfaces (0.0.0.0)
`tailnet` | Tailscale IP only
`custom` | Specific host/IP

## Lesson 5.8: Cost Control ​

### Budget Architecture ​

    Company budget → Agent budgets → Session budgets → Per-call tracking

### Budget Policies ​

yaml

    budget_policies:
      - metric: "monthly_cost_cents"
        scope: "agent:backend-dev"
        amount: 50000  # $500/month
        warn_at: 80%   # warn at $400
        hard_stop: true # kill at $500
      - metric: "session_cost_cents"
        scope: "global"
        amount: 200     # $2/session max
        hard_stop: true

### Warning vs Hard Stop ​

- Warning (80%) — Notify operator, agent keeps running
- Hard Stop (100%) — Agent paused, new tasks queued, running task cancelled

## Lab 5.9: Set Up Agent Observability ​

Objective: Trace every tool call + LLM completion to a local SQLite database.

Starter: `course/labs/L5-observability/starter/`

## Lab 5.10: CI/CD Pipeline ​

Objective: Create a golden dataset and automated regression gate.

Starter: `course/labs/L5-cicd/starter/`