Module 5: Production Patterns

Lesson 5.1: What Production Means for Agents

Production for agents is fundamentally different from traditional software:

Traditional Software	Agent Systems
Deterministic output	Non-deterministic behavior
Fixed cost per operation	Variable cost per session
Error = known exception	Error = unexpected behavior
Rollback = revert code	Rollback = revert prompt + pin model
Monitoring = latency + errors	Monitoring = token usage + loop depth
Testing = unit + integration	Testing = golden datasets + evals

Production Readiness Checklist

[ ] Version-locked prompts (hashed, diffed, reviewed)
[ ] Model pinning (not "latest", specific version)
[ ] Cost budgets per session/task/user
[ ] Iteration limits (max tool calls per session)
[ ] Observability (every tool call logged)
[ ] Shadow deployment capability
[ ] Rollback plan (prompt + model + env)
[ ] Security review (L3+ minimum)

Lesson 5.2: CI/CD for Agents

Golden Dataset

A curated set of input/output/behavior pairs that define correct agent behavior:

json

[
  {
    "input": "Find all users created in the last 24 hours",
    "expected_tools": ["query_database"],
    "expected_tool_params": {"query": "SELECT * FROM users WHERE created_at > now() - interval '24 hours'"},
    "expected_output_contains": ["users", "24 hours"]
  }
]

Pipeline

1. Agent runs against golden dataset
2. Compare: tool selections match expected?
            params match expected?
            output contains expected strings?
3. Calculate pass@k
4. If pass@k < threshold (e.g., 80%), FAIL build
5. If pass, deploy new prompt/config

Lesson 5.2b: Case Study — The 5-Tool Production Stack

A real production multi-agent deployment uses multiple agent tools together, each for its strength. See TOOL-REFERENCE.md for full command references.

The Stack

agent-mux (Tauri UI) —— Meta-agent control plane
    │   └── cc agent SDK, pi-coding-agent SDK, opencode SDK via sidecar
    ▼
mprocs (process monitor) —— Launches all agents
    │   mprocs -c ~/mprocs-teams.yaml
    │
    ├── claude-lead (Claude Code)
    │     └── psmux → tmux.exe → --teammate-mode → split panes
    │           └── Each teammate = separate Claude session
    │
    ├── pi-agent (Pi Coding Agent)
    │     └── Extensions: damage-control, tilldone, coms
    │
    ├── opencode (OpenCode CLI)
    │     └── Model: opencode-go/deepseek-v4-flash (via proxy)
    │
    ├── hermes (Hermes Agent)
    │     └── TypeScript-native, MCP-first workflows
    │
    ├── openclaw (OpenClaw daemon)
    │     └── Always-on employee, heartbeat-driven
    │
    ├── gemini (Gemini fallback)
    │     └── Fast/cheap tasks, cascade routing
    │
    ├── qwen (Qwen specialist)
    │     └── Chinese + structured tasks
    │
    └── sidecar (agent-mux IPC proxy)
          └── RPC bridge between Tauri UI and agent processes
          │
          ▼
    psmux (tmux session manager)
    │   tmux.exe at ~/.cargo/bin/tmux
    │   Requires: start agent-teams first, then dmux inside it
    │
    ▼
    dmux (git worktree isolation)
        Each task gets an isolated worktree
        Rollback = delete worktree

Tool Roles and Selection Logic

Tool	Role	When to Use	Stack Position
Claude Code	Primary coding agent	Complex multi-step tasks, general development	`claude-lead` in mprocs
Pi Agent	Customizable harness	Custom workflows, safety-critical ops, P2P	Side agent with extensions
OpenCode	OSS alternative	Budget tasks, CI/CD, when license matters	Backup in mprocs
Hermes	TypeScript pipelines	MCP-native workflows, structured output	Specialist in mprocs
OpenClaw	Always-on employee	Scheduled tasks, heartbeats, recurring	Daemon (always running)
Gemini	Fast/cheap fallback	High-volume simple tasks	Cascade routing tier 1
Qwen	Specialist model	Chinese content, structured generation	Cascade routing tier 2

How They Work Together (Real Session Flow)

1. Human opens agent-mux Tauri UI
2. mprocs launches all agents from mprocs-teams.yaml
3. Claude Code (lead) runs in tmux via psmux
     └── --teammate-mode creates split panes:
           ├── pane 1: lead (primary coder)
           ├── pane 2: worker (sub-tasks)
           ├── pane 3: reviewer (code review)
           └── pane 4: verifier (read-only checks)
4. OpenCode runs alongside as budget-aware backup
5. OpenClaw daemon handles scheduled background tasks
6. dmux isolates each task in its own git worktree
7. agent-mux sidecar collects status from all agents
8. Human monitors via Tauri UI, intervenes when needed

Key Production Patterns

Model heterogeneity — Different models for different roles. Cascade routing in practice (M6).
Tool heterogeneity — Five CLIs, each with different strengths. No single point of failure.
Process management — mprocs supervises. If one agent crashes, the stack keeps running.
Session isolation — psmux (terminal sessions) + dmux (git worktrees) = two layers.
Meta-control plane — agent-mux Tauri UI. Human watches and intervenes, not drives.
Defense in depth — tool-level (damage-control), session-level (psmux), filesystem-level (dmux).

Lesson 5.2c: The Agent Manager Role

In enterprise deployments, someone owns the agent harness. This is the Agent Manager (or DevEx Lead for AI).

Responsibilities

Agent Manager
├── Harness design (CLAUDE.md, skills, hooks, MCPs)
├── Tool selection (which agent CLIs, which models)
├── Security policy (damage-control rules, access levels)
├── Cost management (budgets per agent/task, optimization)
├── Quality gates (golden datasets, regression testing)
├── Update cadence (prompt versioning, model pinning)
└── Incident response (tool loops, cost spikes, failures)

90-Day Setup Playbook

Month 1: Foundation

Set up agent CLI (Claude Code, Pi, or OpenCode)
Create CLAUDE.md with project context
Install damage-control with 3 access levels
Set up basic observability (tool call logging)

Month 2: Scale

Add multi-agent teams (lead + workers)
Create golden dataset (10+ test cases)
Implement CI/CD gate
Set up cost tracking and budgets

Month 3: Production

Shadow deployment pipeline
Rollback procedures documented
Monitoring dashboard live
Team trained on agent interaction patterns

Lesson 5.3: Shadow Deployments

How It Works

Production agent: serves user traffic
Shadow agent: runs IDENTICAL inputs, but outputs are NOT served
Compare: did shadow make same decisions as production?
         did shadow cost more/less?
         did shadow hit any errors?
Decision: if shadow improves on all metrics, swap them

When to Shadow Deploy

New prompt version
New model version
New tool addition
Agent architecture change

Lesson 5.4: Rollback Strategies

What Rollback Means for Agents

You can't just revert a Git commit. Agent behavior depends on:

Prompt — the text of the system prompt + tools
Model — which model version
Parameters — temperature, top_p, etc.
Configuration — tool list, iteration limits, budget

A proper rollback restores ALL four.

Implementation

yaml

# agent-config-v42.yaml
prompt_hash: "a1b2c3d4"
model: "claude-sonnet-4-20260501"  # pinned, not "latest"
temperature: 0.0
max_iterations: 25
tools: ["read", "write", "bash", "search"]
budget_per_session: 0.50

Rollback = cp agent-config-v41.yaml agent-config.yaml + reload.

Lesson 5.5: Observability & Monitoring

What to Trace (Every Single Turn)

Input prompt (full, including system prompt)
LLM response (including tool call choices)
Tool calls (name, params, timestamp)
Tool results (output, error status, duration)
Token counts (input, output, cached)
Cost (per-call and running total)
Loop depth (current turn number)

Decision Tracing

Standard APM (Datadog, Grafana) captures latency and errors. Agents need decision tracing — the full chain of reasoning and actions:

json

{
  "session_id": "sess_abc123",
  "turn": 5,
  "input_tokens": 12400,
  "output_tokens": 350,
  "tool_calls": [
    {"tool": "search_web", "params": {"query": "latest pricing"}, "duration_ms": 1200}
  ],
  "decision": "Found pricing page, will extract"
}

Key Metrics

Metric	Warning	Critical
Tool calls per session	>20	>50
Cost per session	>$0.50	>$2.00
Loop depth	>15	>30
Same tool >5x in row	Investigate loop	Kill session
Context utilization	>80%	>95%

Lesson 5.6: Alerting on Agent-Specific Signals

What to Alert On

Tool loop detected — same tool called 5+ times with same params
Cost spike — session cost > 3x average
Context overflow imminent — token count within 10% of limit
Permission escalation — agent attempting blocked operations
Error cascade — 3+ tool failures in a row
Grinding detected — identical code rerun without changes

Alert Routing

P0 (immediate): Cost spike > $10, permission escalation, data exfil attempt
P1 (within 5 min): Tool loop, error cascade, grinding
P2 (within 1 hour): Context utilization high, cost trending up
P3 (daily report): Average session cost, success rate, failure modes

Lesson 5.6b: Cross-Provider Session Search

When you run agents across 5+ tools (Claude Code, Pi, OpenCode, Gemini, OpenClaw), session history is scattered across different directories and formats.

The Problem

bash

~/.claude/sessions/*.jsonl      # Claude Code format
~/.pi/sessions/*.jsonl          # Pi format
~/.opencode/sessions/*          # OpenCode format
~/.gemini/sessions/*.jsonl      # Gemini format
mprocs-logs/*.log               # mprocs supervisor logs

Searching across all of them is impossible without a unified index.

The Solution

Reference implementation: Jeff Emanuel's coding_agent_session_search (783★)

Indexer:
  ├── Watches all session directories
  ├── Normalizes into canonical format
  ├── Full-text indexes prompts, responses, tool calls
  └── Stores in SQLite with FTS5

Search CLI:
  ├── Search across ALL providers from one command
  ├── Filter by: provider, date, model, tool, token count
  ├── Replay any session from any provider
  └── Export sessions as markdown or JSON

Why This Matters for Production

Debugging: Find what went wrong across all agents in one search
Audit: Full history of every agent action, unified format
Learning: Search past solutions instead of re-solving problems
Cost analysis: Aggregate costs across all providers

Integration with Our Stack

bash

# Add to mprocs-teams.yaml as a sidecar service
session-indexer:
  cmd: ["python", "session-indexer.py", "--watch-dirs",
        "~/.claude/sessions/", "~/.pi/sessions/",
        "~/.opencode/sessions/", "logs/"]

# Search across everything
session-search "deployment error" --providers claude,pi,opencode
# Returns results from all providers in unified format

Lesson 5.7: Deployment Modes

From Paperclip's deployment model:

Mode	Auth	Exposure	Use Case
Local trusted	None	localhost only	Single dev machine
Authenticated private	Login required	LAN/Tailscale/VPN	Team on private network
Authenticated public	Login required	Internet (behind reverse proxy)	Production cloud deployment

Reachability

Bind	What It Means
`loopback`	localhost only (default)
`lan`	All interfaces (0.0.0.0)
`tailnet`	Tailscale IP only
`custom`	Specific host/IP

Lesson 5.8: Cost Control

Budget Architecture

Company budget → Agent budgets → Session budgets → Per-call tracking

Budget Policies

yaml

budget_policies:
  - metric: "monthly_cost_cents"
    scope: "agent:backend-dev"
    amount: 50000  # $500/month
    warn_at: 80%   # warn at $400
    hard_stop: true # kill at $500
  - metric: "session_cost_cents"
    scope: "global"
    amount: 200     # $2/session max
    hard_stop: true

Warning vs Hard Stop

Warning (80%) — Notify operator, agent keeps running
Hard Stop (100%) — Agent paused, new tasks queued, running task cancelled

Lab 5.9: Set Up Agent Observability

Objective: Trace every tool call + LLM completion to a local SQLite database.

Starter: course/labs/L5-observability/starter/

Lab 5.10: CI/CD Pipeline

Objective: Create a golden dataset and automated regression gate.

Starter: course/labs/L5-cicd/starter/

Module 5: Production Patterns ​

Lesson 5.1: What Production Means for Agents ​

Production Readiness Checklist ​

Lesson 5.2: CI/CD for Agents ​

Golden Dataset ​

Pipeline ​

Lesson 5.2b: Case Study — The 5-Tool Production Stack ​

The Stack ​

Tool Roles and Selection Logic ​

How They Work Together (Real Session Flow) ​

Key Production Patterns ​

Lesson 5.2c: The Agent Manager Role ​

Responsibilities ​

90-Day Setup Playbook ​

Lesson 5.3: Shadow Deployments ​

How It Works ​

When to Shadow Deploy ​

Lesson 5.4: Rollback Strategies ​

What Rollback Means for Agents ​

Implementation ​

Lesson 5.5: Observability & Monitoring ​

What to Trace (Every Single Turn) ​

Decision Tracing ​

Key Metrics ​

Lesson 5.6: Alerting on Agent-Specific Signals ​

What to Alert On ​

Alert Routing ​

Lesson 5.6b: Cross-Provider Session Search ​

The Problem ​

The Solution ​

Why This Matters for Production ​

Integration with Our Stack ​

Lesson 5.7: Deployment Modes ​

Reachability ​

Lesson 5.8: Cost Control ​

Budget Architecture ​

Budget Policies ​

Warning vs Hard Stop ​

Lab 5.9: Set Up Agent Observability ​

Lab 5.10: CI/CD Pipeline ​