# Module 5: Production Patterns

## Lesson 5.1: What Production Means for Agents

Production for agents is fundamentally different from traditional software:

| Traditional Software | Agent Systems |
|---------------------|---------------|
| Deterministic output | Non-deterministic behavior |
| Fixed cost per operation | Variable cost per session |
| Error = known exception | Error = unexpected behavior |
| Rollback = revert code | Rollback = revert prompt + pin model |
| Monitoring = latency + errors | Monitoring = token usage + loop depth |
| Testing = unit + integration | Testing = golden datasets + evals |

### Production Readiness Checklist

```
[ ] Version-locked prompts (hashed, diffed, reviewed)
[ ] Model pinning (not "latest", specific version)
[ ] Cost budgets per session/task/user
[ ] Iteration limits (max tool calls per session)
[ ] Observability (every tool call logged)
[ ] Shadow deployment capability
[ ] Rollback plan (prompt + model + env)
[ ] Security review (L3+ minimum)
```

---

## Lesson 5.2: CI/CD for Agents

### Golden Dataset

A curated set of input/output/behavior pairs that define correct agent behavior:

```json
[
  {
    "input": "Find all users created in the last 24 hours",
    "expected_tools": ["query_database"],
    "expected_tool_params": {"query": "SELECT * FROM users WHERE created_at > now() - interval '24 hours'"},
    "expected_output_contains": ["users", "24 hours"]
  }
]
```

### Pipeline

```
1. Agent runs against golden dataset
2. Compare: tool selections match expected?
            params match expected?
            output contains expected strings?
3. Calculate pass@k
4. If pass@k < threshold (e.g., 80%), FAIL build
5. If pass, deploy new prompt/config
```

---

## Lesson 5.2b: Case Study — The 5-Tool Production Stack

A real production multi-agent deployment uses multiple agent tools together, each for its strength. See `TOOL-REFERENCE.md` for full command references.

### The Stack

```
agent-mux (Tauri UI) —— Meta-agent control plane
    │   └── cc agent SDK, pi-coding-agent SDK, opencode SDK via sidecar
    ▼
mprocs (process monitor) —— Launches all agents
    │   mprocs -c ~/mprocs-teams.yaml
    │
    ├── claude-lead (Claude Code)
    │     └── psmux → tmux.exe → --teammate-mode → split panes
    │           └── Each teammate = separate Claude session
    │
    ├── pi-agent (Pi Coding Agent)
    │     └── Extensions: damage-control, tilldone, coms
    │
    ├── opencode (OpenCode CLI)
    │     └── Model: opencode-go/deepseek-v4-flash (via proxy)
    │
    ├── hermes (Hermes Agent)
    │     └── TypeScript-native, MCP-first workflows
    │
    ├── openclaw (OpenClaw daemon)
    │     └── Always-on employee, heartbeat-driven
    │
    ├── gemini (Gemini fallback)
    │     └── Fast/cheap tasks, cascade routing
    │
    ├── qwen (Qwen specialist)
    │     └── Chinese + structured tasks
    │
    └── sidecar (agent-mux IPC proxy)
          └── RPC bridge between Tauri UI and agent processes
          │
          ▼
    psmux (tmux session manager)
    │   tmux.exe at ~/.cargo/bin/tmux
    │   Requires: start agent-teams first, then dmux inside it
    │
    ▼
    dmux (git worktree isolation)
        Each task gets an isolated worktree
        Rollback = delete worktree
```

### Tool Roles and Selection Logic

| Tool | Role | When to Use | Stack Position |
|------|------|-------------|----------------|
| **Claude Code** | Primary coding agent | Complex multi-step tasks, general development | `claude-lead` in mprocs |
| **Pi Agent** | Customizable harness | Custom workflows, safety-critical ops, P2P | Side agent with extensions |
| **OpenCode** | OSS alternative | Budget tasks, CI/CD, when license matters | Backup in mprocs |
| **Hermes** | TypeScript pipelines | MCP-native workflows, structured output | Specialist in mprocs |
| **OpenClaw** | Always-on employee | Scheduled tasks, heartbeats, recurring | Daemon (always running) |
| **Gemini** | Fast/cheap fallback | High-volume simple tasks | Cascade routing tier 1 |
| **Qwen** | Specialist model | Chinese content, structured generation | Cascade routing tier 2 |

### How They Work Together (Real Session Flow)

```
1. Human opens agent-mux Tauri UI
2. mprocs launches all agents from mprocs-teams.yaml
3. Claude Code (lead) runs in tmux via psmux
     └── --teammate-mode creates split panes:
           ├── pane 1: lead (primary coder)
           ├── pane 2: worker (sub-tasks)
           ├── pane 3: reviewer (code review)
           └── pane 4: verifier (read-only checks)
4. OpenCode runs alongside as budget-aware backup
5. OpenClaw daemon handles scheduled background tasks
6. dmux isolates each task in its own git worktree
7. agent-mux sidecar collects status from all agents
8. Human monitors via Tauri UI, intervenes when needed
```

### Key Production Patterns

1. **Model heterogeneity** — Different models for different roles. Cascade routing in practice (M6).
2. **Tool heterogeneity** — Five CLIs, each with different strengths. No single point of failure.
3. **Process management** — mprocs supervises. If one agent crashes, the stack keeps running.
4. **Session isolation** — psmux (terminal sessions) + dmux (git worktrees) = two layers.
5. **Meta-control plane** — agent-mux Tauri UI. Human watches and intervenes, not drives.
6. **Defense in depth** — tool-level (damage-control), session-level (psmux), filesystem-level (dmux).

---

## Lesson 5.2c: The Agent Manager Role

In enterprise deployments, someone owns the agent harness. This is the **Agent Manager** (or DevEx Lead for AI).

### Responsibilities

```
Agent Manager
├── Harness design (CLAUDE.md, skills, hooks, MCPs)
├── Tool selection (which agent CLIs, which models)
├── Security policy (damage-control rules, access levels)
├── Cost management (budgets per agent/task, optimization)
├── Quality gates (golden datasets, regression testing)
├── Update cadence (prompt versioning, model pinning)
└── Incident response (tool loops, cost spikes, failures)
```

### 90-Day Setup Playbook

**Month 1**: Foundation
- Set up agent CLI (Claude Code, Pi, or OpenCode)
- Create CLAUDE.md with project context
- Install damage-control with 3 access levels
- Set up basic observability (tool call logging)

**Month 2**: Scale
- Add multi-agent teams (lead + workers)
- Create golden dataset (10+ test cases)
- Implement CI/CD gate
- Set up cost tracking and budgets

**Month 3**: Production
- Shadow deployment pipeline
- Rollback procedures documented
- Monitoring dashboard live
- Team trained on agent interaction patterns

---

## Lesson 5.3: Shadow Deployments

### How It Works

```
Production agent: serves user traffic
Shadow agent: runs IDENTICAL inputs, but outputs are NOT served
Compare: did shadow make same decisions as production?
         did shadow cost more/less?
         did shadow hit any errors?
Decision: if shadow improves on all metrics, swap them
```

### When to Shadow Deploy

- New prompt version
- New model version
- New tool addition
- Agent architecture change

---

## Lesson 5.4: Rollback Strategies

### What Rollback Means for Agents

You can't just revert a Git commit. Agent behavior depends on:

1. **Prompt** — the text of the system prompt + tools
2. **Model** — which model version
3. **Parameters** — temperature, top_p, etc.
4. **Configuration** — tool list, iteration limits, budget

A proper rollback restores ALL four.

### Implementation

```yaml
# agent-config-v42.yaml
prompt_hash: "a1b2c3d4"
model: "claude-sonnet-4-20260501"  # pinned, not "latest"
temperature: 0.0
max_iterations: 25
tools: ["read", "write", "bash", "search"]
budget_per_session: 0.50
```

Rollback = `cp agent-config-v41.yaml agent-config.yaml` + reload.

---

## Lesson 5.5: Observability & Monitoring

### What to Trace (Every Single Turn)

1. **Input prompt** (full, including system prompt)
2. **LLM response** (including tool call choices)
3. **Tool calls** (name, params, timestamp)
4. **Tool results** (output, error status, duration)
5. **Token counts** (input, output, cached)
6. **Cost** (per-call and running total)
7. **Loop depth** (current turn number)

### Decision Tracing

Standard APM (Datadog, Grafana) captures latency and errors. Agents need **decision tracing** — the full chain of reasoning and actions:

```json
{
  "session_id": "sess_abc123",
  "turn": 5,
  "input_tokens": 12400,
  "output_tokens": 350,
  "tool_calls": [
    {"tool": "search_web", "params": {"query": "latest pricing"}, "duration_ms": 1200}
  ],
  "decision": "Found pricing page, will extract"
}
```

### Key Metrics

| Metric | Warning | Critical |
|--------|---------|----------|
| Tool calls per session | >20 | >50 |
| Cost per session | >$0.50 | >$2.00 |
| Loop depth | >15 | >30 |
| Same tool >5x in row | Investigate loop | Kill session |
| Context utilization | >80% | >95% |

---

## Lesson 5.6: Alerting on Agent-Specific Signals

### What to Alert On

1. **Tool loop detected** — same tool called 5+ times with same params
2. **Cost spike** — session cost > 3x average
3. **Context overflow imminent** — token count within 10% of limit
4. **Permission escalation** — agent attempting blocked operations
5. **Error cascade** — 3+ tool failures in a row
6. **Grinding detected** — identical code rerun without changes

### Alert Routing

```
P0 (immediate): Cost spike > $10, permission escalation, data exfil attempt
P1 (within 5 min): Tool loop, error cascade, grinding
P2 (within 1 hour): Context utilization high, cost trending up
P3 (daily report): Average session cost, success rate, failure modes
```

## Lesson 5.6b: Cross-Provider Session Search

When you run agents across 5+ tools (Claude Code, Pi, OpenCode, Gemini, OpenClaw), session history is scattered across different directories and formats.

### The Problem

```bash
~/.claude/sessions/*.jsonl      # Claude Code format
~/.pi/sessions/*.jsonl          # Pi format
~/.opencode/sessions/*          # OpenCode format
~/.gemini/sessions/*.jsonl      # Gemini format
mprocs-logs/*.log               # mprocs supervisor logs
```

Searching across all of them is impossible without a unified index.

### The Solution

**Reference implementation**: Jeff Emanuel's `coding_agent_session_search` (783★)

```
Indexer:
  ├── Watches all session directories
  ├── Normalizes into canonical format
  ├── Full-text indexes prompts, responses, tool calls
  └── Stores in SQLite with FTS5

Search CLI:
  ├── Search across ALL providers from one command
  ├── Filter by: provider, date, model, tool, token count
  ├── Replay any session from any provider
  └── Export sessions as markdown or JSON
```

### Why This Matters for Production

1. **Debugging**: Find what went wrong across all agents in one search
2. **Audit**: Full history of every agent action, unified format
3. **Learning**: Search past solutions instead of re-solving problems
4. **Cost analysis**: Aggregate costs across all providers

### Integration with Our Stack

```bash
# Add to mprocs-teams.yaml as a sidecar service
session-indexer:
  cmd: ["python", "session-indexer.py", "--watch-dirs",
        "~/.claude/sessions/", "~/.pi/sessions/",
        "~/.opencode/sessions/", "logs/"]

# Search across everything
session-search "deployment error" --providers claude,pi,opencode
# Returns results from all providers in unified format
```

---

## Lesson 5.7: Deployment Modes

From Paperclip's deployment model:

| Mode | Auth | Exposure | Use Case |
|------|------|----------|----------|
| Local trusted | None | localhost only | Single dev machine |
| Authenticated private | Login required | LAN/Tailscale/VPN | Team on private network |
| Authenticated public | Login required | Internet (behind reverse proxy) | Production cloud deployment |

### Reachability

| Bind | What It Means |
|------|---------------|
| `loopback` | localhost only (default) |
| `lan` | All interfaces (0.0.0.0) |
| `tailnet` | Tailscale IP only |
| `custom` | Specific host/IP |

---

## Lesson 5.8: Cost Control

### Budget Architecture

```
Company budget → Agent budgets → Session budgets → Per-call tracking
```

### Budget Policies

```yaml
budget_policies:
  - metric: "monthly_cost_cents"
    scope: "agent:backend-dev"
    amount: 50000  # $500/month
    warn_at: 80%   # warn at $400
    hard_stop: true # kill at $500
  - metric: "session_cost_cents"
    scope: "global"
    amount: 200     # $2/session max
    hard_stop: true
```

### Warning vs Hard Stop

- **Warning** (80%) — Notify operator, agent keeps running
- **Hard Stop** (100%) — Agent paused, new tasks queued, running task cancelled

---

## Lesson 5.8b: Dry-Run Workflow for Agent Actions

Before an agent executes a destructive action (write file, delete, deploy), you want a **preview mode** that shows what the agent WILL do without actually doing it.

### The Dry-Run Pattern

```
Agent proposes action → Preview output → Human reviews → Approve/Reject → Execute
```

```python
class DryRunContext:
    """Wrap tool execution in dry-run mode."""
    
    def __init__(self, dry_run=True):
        self.dry_run = dry_run
        self.proposed_actions = []
    
    def execute(self, tool_name, params):
        if self.dry_run:
            # Log what WOULD happen
            self.proposed_actions.append({
                "tool": tool_name,
                "params": params,
                "preview": self._generate_preview(tool_name, params),
            })
            return f"[DRY RUN] Would call {tool_name} with {params}"
        else:
            # Actually execute
            return real_execute(tool_name, params)
    
    def _generate_preview(self, tool_name, params):
        if tool_name == "write_file":
            return f"Would write {len(params.get('content',''))} chars to {params.get('path')}"
        elif tool_name == "exec_command":
            return f"Would run: {params.get('command','')[:100]}..."
        elif tool_name == "delete_file":
            return f"Would DELETE: {params.get('path')}"
        return f"Would call {tool_name}"
```

### Implementation Strategies

| Strategy | How It Works | Best For |
|----------|-------------|----------|
| **Flag-based** | `--dry-run` flag on agent start | Development, testing |
| **Hook-based** | Pre-tool hook logs intent, skips execution | Production agents |
| **UI-based** | Agent shows preview, human clicks Confirm | Interactive sessions |
| **Two-pass** | Agent plans first (dry), then executes (wet) | Complex multi-step tasks |

### Docker Dry-Run Example

From the dry-run workflow pattern — a Docker-based calculator that logs operations without running them:

```bash
# Build the dry-run sandbox
docker build -t dry-run-calc -f calculator/Dockerfile .

# Run in preview mode
docker run --rm -e DRY_RUN=true dry-run-calc add 5 3
# Output: [DRY RUN] Would add 5 + 3 = 8

# Run for real
docker run --rm -e DRY_RUN=false dry-run-calc add 5 3
# Output: 8
```

### When to Use Dry-Run

- **Always** for file writes, deletes, and deploys
- **Sometimes** for commands that modify state (DB migrations, config changes)
- **Never** for read-only operations (search, read file, list directory)

---

## Lesson 5.8c: Cross-Platform Agent Skills

Skills should work on any agent — Claude Code, Pi Agent, OpenCode, or Codex. The cross-platform format uses YAML frontmatter and tool-agnostic instructions:

```markdown
---
name: init-agents-md
description: Create or refresh AGENTS.md for coding agents.
  Works with Claude Code, Pi Agent, and Codex.
---

# Initialize AGENTS.md

Create a short, repo-specific AGENTS.md.

## Workflow

1. Check if AGENTS.md already exists — if so, stop and ask
2. Explore the repository structure
3. Draft AGENTS.md with project purpose, stack, and conventions
4. Mirror same context into CLAUDE.md if needed
```

### Key Principles

1. **Use `~~` or `---` frontmatter** — not agent-specific config
2. **Avoid CLI flags** — describe the desired outcome, not the command
3. **Include trigger patterns** — tell the agent when to invoke this skill
4. **One `SKILL.md` per skill** — no platform-specific variations

---

## Lab 5.9: Set Up Agent Observability

**Objective**: Trace every tool call + LLM completion to a local SQLite database.

**Starter**: `course/labs/L5-observability/starter/`

## Lab 5.10: CI/CD Pipeline

**Objective**: Create a golden dataset and automated regression gate.

**Starter**: `course/labs/L5-cicd/starter/`