529 lines
16 KiB
Markdown
529 lines
16 KiB
Markdown
# Module 5: Production Patterns
|
|
|
|
## Lesson 5.1: What Production Means for Agents
|
|
|
|
Production for agents is fundamentally different from traditional software:
|
|
|
|
| Traditional Software | Agent Systems |
|
|
|---------------------|---------------|
|
|
| Deterministic output | Non-deterministic behavior |
|
|
| Fixed cost per operation | Variable cost per session |
|
|
| Error = known exception | Error = unexpected behavior |
|
|
| Rollback = revert code | Rollback = revert prompt + pin model |
|
|
| Monitoring = latency + errors | Monitoring = token usage + loop depth |
|
|
| Testing = unit + integration | Testing = golden datasets + evals |
|
|
|
|
### Production Readiness Checklist
|
|
|
|
```
|
|
[ ] Version-locked prompts (hashed, diffed, reviewed)
|
|
[ ] Model pinning (not "latest", specific version)
|
|
[ ] Cost budgets per session/task/user
|
|
[ ] Iteration limits (max tool calls per session)
|
|
[ ] Observability (every tool call logged)
|
|
[ ] Shadow deployment capability
|
|
[ ] Rollback plan (prompt + model + env)
|
|
[ ] Security review (L3+ minimum)
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 5.2: CI/CD for Agents
|
|
|
|
### Golden Dataset
|
|
|
|
A curated set of input/output/behavior pairs that define correct agent behavior:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"input": "Find all users created in the last 24 hours",
|
|
"expected_tools": ["query_database"],
|
|
"expected_tool_params": {"query": "SELECT * FROM users WHERE created_at > now() - interval '24 hours'"},
|
|
"expected_output_contains": ["users", "24 hours"]
|
|
}
|
|
]
|
|
```
|
|
|
|
### Pipeline
|
|
|
|
```
|
|
1. Agent runs against golden dataset
|
|
2. Compare: tool selections match expected?
|
|
params match expected?
|
|
output contains expected strings?
|
|
3. Calculate pass@k
|
|
4. If pass@k < threshold (e.g., 80%), FAIL build
|
|
5. If pass, deploy new prompt/config
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 5.2b: Case Study — The 5-Tool Production Stack
|
|
|
|
A real production multi-agent deployment uses multiple agent tools together, each for its strength. See `TOOL-REFERENCE.md` for full command references.
|
|
|
|
### The Stack
|
|
|
|
```
|
|
agent-mux (Tauri UI) —— Meta-agent control plane
|
|
│ └── cc agent SDK, pi-coding-agent SDK, opencode SDK via sidecar
|
|
▼
|
|
mprocs (process monitor) —— Launches all agents
|
|
│ mprocs -c ~/mprocs-teams.yaml
|
|
│
|
|
├── claude-lead (Claude Code)
|
|
│ └── psmux → tmux.exe → --teammate-mode → split panes
|
|
│ └── Each teammate = separate Claude session
|
|
│
|
|
├── pi-agent (Pi Coding Agent)
|
|
│ └── Extensions: damage-control, tilldone, coms
|
|
│
|
|
├── opencode (OpenCode CLI)
|
|
│ └── Model: opencode-go/deepseek-v4-flash (via proxy)
|
|
│
|
|
├── hermes (Hermes Agent)
|
|
│ └── TypeScript-native, MCP-first workflows
|
|
│
|
|
├── openclaw (OpenClaw daemon)
|
|
│ └── Always-on employee, heartbeat-driven
|
|
│
|
|
├── gemini (Gemini fallback)
|
|
│ └── Fast/cheap tasks, cascade routing
|
|
│
|
|
├── qwen (Qwen specialist)
|
|
│ └── Chinese + structured tasks
|
|
│
|
|
└── sidecar (agent-mux IPC proxy)
|
|
└── RPC bridge between Tauri UI and agent processes
|
|
│
|
|
▼
|
|
psmux (tmux session manager)
|
|
│ tmux.exe at ~/.cargo/bin/tmux
|
|
│ Requires: start agent-teams first, then dmux inside it
|
|
│
|
|
▼
|
|
dmux (git worktree isolation)
|
|
Each task gets an isolated worktree
|
|
Rollback = delete worktree
|
|
```
|
|
|
|
### Tool Roles and Selection Logic
|
|
|
|
| Tool | Role | When to Use | Stack Position |
|
|
|------|------|-------------|----------------|
|
|
| **Claude Code** | Primary coding agent | Complex multi-step tasks, general development | `claude-lead` in mprocs |
|
|
| **Pi Agent** | Customizable harness | Custom workflows, safety-critical ops, P2P | Side agent with extensions |
|
|
| **OpenCode** | OSS alternative | Budget tasks, CI/CD, when license matters | Backup in mprocs |
|
|
| **Hermes** | TypeScript pipelines | MCP-native workflows, structured output | Specialist in mprocs |
|
|
| **OpenClaw** | Always-on employee | Scheduled tasks, heartbeats, recurring | Daemon (always running) |
|
|
| **Gemini** | Fast/cheap fallback | High-volume simple tasks | Cascade routing tier 1 |
|
|
| **Qwen** | Specialist model | Chinese content, structured generation | Cascade routing tier 2 |
|
|
|
|
### How They Work Together (Real Session Flow)
|
|
|
|
```
|
|
1. Human opens agent-mux Tauri UI
|
|
2. mprocs launches all agents from mprocs-teams.yaml
|
|
3. Claude Code (lead) runs in tmux via psmux
|
|
└── --teammate-mode creates split panes:
|
|
├── pane 1: lead (primary coder)
|
|
├── pane 2: worker (sub-tasks)
|
|
├── pane 3: reviewer (code review)
|
|
└── pane 4: verifier (read-only checks)
|
|
4. OpenCode runs alongside as budget-aware backup
|
|
5. OpenClaw daemon handles scheduled background tasks
|
|
6. dmux isolates each task in its own git worktree
|
|
7. agent-mux sidecar collects status from all agents
|
|
8. Human monitors via Tauri UI, intervenes when needed
|
|
```
|
|
|
|
### Key Production Patterns
|
|
|
|
1. **Model heterogeneity** — Different models for different roles. Cascade routing in practice (M6).
|
|
2. **Tool heterogeneity** — Five CLIs, each with different strengths. No single point of failure.
|
|
3. **Process management** — mprocs supervises. If one agent crashes, the stack keeps running.
|
|
4. **Session isolation** — psmux (terminal sessions) + dmux (git worktrees) = two layers.
|
|
5. **Meta-control plane** — agent-mux Tauri UI. Human watches and intervenes, not drives.
|
|
6. **Defense in depth** — tool-level (damage-control), session-level (psmux), filesystem-level (dmux).
|
|
|
|
---
|
|
|
|
## Lesson 5.2c: The Agent Manager Role
|
|
|
|
In enterprise deployments, someone owns the agent harness. This is the **Agent Manager** (or DevEx Lead for AI).
|
|
|
|
### Responsibilities
|
|
|
|
```
|
|
Agent Manager
|
|
├── Harness design (CLAUDE.md, skills, hooks, MCPs)
|
|
├── Tool selection (which agent CLIs, which models)
|
|
├── Security policy (damage-control rules, access levels)
|
|
├── Cost management (budgets per agent/task, optimization)
|
|
├── Quality gates (golden datasets, regression testing)
|
|
├── Update cadence (prompt versioning, model pinning)
|
|
└── Incident response (tool loops, cost spikes, failures)
|
|
```
|
|
|
|
### 90-Day Setup Playbook
|
|
|
|
**Month 1**: Foundation
|
|
- Set up agent CLI (Claude Code, Pi, or OpenCode)
|
|
- Create CLAUDE.md with project context
|
|
- Install damage-control with 3 access levels
|
|
- Set up basic observability (tool call logging)
|
|
|
|
**Month 2**: Scale
|
|
- Add multi-agent teams (lead + workers)
|
|
- Create golden dataset (10+ test cases)
|
|
- Implement CI/CD gate
|
|
- Set up cost tracking and budgets
|
|
|
|
**Month 3**: Production
|
|
- Shadow deployment pipeline
|
|
- Rollback procedures documented
|
|
- Monitoring dashboard live
|
|
- Team trained on agent interaction patterns
|
|
|
|
---
|
|
|
|
## Lesson 5.3: Shadow Deployments
|
|
|
|
### How It Works
|
|
|
|
```
|
|
Production agent: serves user traffic
|
|
Shadow agent: runs IDENTICAL inputs, but outputs are NOT served
|
|
Compare: did shadow make same decisions as production?
|
|
did shadow cost more/less?
|
|
did shadow hit any errors?
|
|
Decision: if shadow improves on all metrics, swap them
|
|
```
|
|
|
|
### When to Shadow Deploy
|
|
|
|
- New prompt version
|
|
- New model version
|
|
- New tool addition
|
|
- Agent architecture change
|
|
|
|
---
|
|
|
|
## Lesson 5.4: Rollback Strategies
|
|
|
|
### What Rollback Means for Agents
|
|
|
|
You can't just revert a Git commit. Agent behavior depends on:
|
|
|
|
1. **Prompt** — the text of the system prompt + tools
|
|
2. **Model** — which model version
|
|
3. **Parameters** — temperature, top_p, etc.
|
|
4. **Configuration** — tool list, iteration limits, budget
|
|
|
|
A proper rollback restores ALL four.
|
|
|
|
### Implementation
|
|
|
|
```yaml
|
|
# agent-config-v42.yaml
|
|
prompt_hash: "a1b2c3d4"
|
|
model: "claude-sonnet-4-20260501" # pinned, not "latest"
|
|
temperature: 0.0
|
|
max_iterations: 25
|
|
tools: ["read", "write", "bash", "search"]
|
|
budget_per_session: 0.50
|
|
```
|
|
|
|
Rollback = `cp agent-config-v41.yaml agent-config.yaml` + reload.
|
|
|
|
---
|
|
|
|
## Lesson 5.5: Observability & Monitoring
|
|
|
|
### What to Trace (Every Single Turn)
|
|
|
|
1. **Input prompt** (full, including system prompt)
|
|
2. **LLM response** (including tool call choices)
|
|
3. **Tool calls** (name, params, timestamp)
|
|
4. **Tool results** (output, error status, duration)
|
|
5. **Token counts** (input, output, cached)
|
|
6. **Cost** (per-call and running total)
|
|
7. **Loop depth** (current turn number)
|
|
|
|
### Decision Tracing
|
|
|
|
Standard APM (Datadog, Grafana) captures latency and errors. Agents need **decision tracing** — the full chain of reasoning and actions:
|
|
|
|
```json
|
|
{
|
|
"session_id": "sess_abc123",
|
|
"turn": 5,
|
|
"input_tokens": 12400,
|
|
"output_tokens": 350,
|
|
"tool_calls": [
|
|
{"tool": "search_web", "params": {"query": "latest pricing"}, "duration_ms": 1200}
|
|
],
|
|
"decision": "Found pricing page, will extract"
|
|
}
|
|
```
|
|
|
|
### Key Metrics
|
|
|
|
| Metric | Warning | Critical |
|
|
|--------|---------|----------|
|
|
| Tool calls per session | >20 | >50 |
|
|
| Cost per session | >$0.50 | >$2.00 |
|
|
| Loop depth | >15 | >30 |
|
|
| Same tool >5x in row | Investigate loop | Kill session |
|
|
| Context utilization | >80% | >95% |
|
|
|
|
---
|
|
|
|
## Lesson 5.6: Alerting on Agent-Specific Signals
|
|
|
|
### What to Alert On
|
|
|
|
1. **Tool loop detected** — same tool called 5+ times with same params
|
|
2. **Cost spike** — session cost > 3x average
|
|
3. **Context overflow imminent** — token count within 10% of limit
|
|
4. **Permission escalation** — agent attempting blocked operations
|
|
5. **Error cascade** — 3+ tool failures in a row
|
|
6. **Grinding detected** — identical code rerun without changes
|
|
|
|
### Alert Routing
|
|
|
|
```
|
|
P0 (immediate): Cost spike > $10, permission escalation, data exfil attempt
|
|
P1 (within 5 min): Tool loop, error cascade, grinding
|
|
P2 (within 1 hour): Context utilization high, cost trending up
|
|
P3 (daily report): Average session cost, success rate, failure modes
|
|
```
|
|
|
|
## Lesson 5.6b: Cross-Provider Session Search
|
|
|
|
When you run agents across 5+ tools (Claude Code, Pi, OpenCode, Gemini, OpenClaw), session history is scattered across different directories and formats.
|
|
|
|
### The Problem
|
|
|
|
```bash
|
|
~/.claude/sessions/*.jsonl # Claude Code format
|
|
~/.pi/sessions/*.jsonl # Pi format
|
|
~/.opencode/sessions/* # OpenCode format
|
|
~/.gemini/sessions/*.jsonl # Gemini format
|
|
mprocs-logs/*.log # mprocs supervisor logs
|
|
```
|
|
|
|
Searching across all of them is impossible without a unified index.
|
|
|
|
### The Solution
|
|
|
|
**Reference implementation**: Jeff Emanuel's `coding_agent_session_search` (783★)
|
|
|
|
```
|
|
Indexer:
|
|
├── Watches all session directories
|
|
├── Normalizes into canonical format
|
|
├── Full-text indexes prompts, responses, tool calls
|
|
└── Stores in SQLite with FTS5
|
|
|
|
Search CLI:
|
|
├── Search across ALL providers from one command
|
|
├── Filter by: provider, date, model, tool, token count
|
|
├── Replay any session from any provider
|
|
└── Export sessions as markdown or JSON
|
|
```
|
|
|
|
### Why This Matters for Production
|
|
|
|
1. **Debugging**: Find what went wrong across all agents in one search
|
|
2. **Audit**: Full history of every agent action, unified format
|
|
3. **Learning**: Search past solutions instead of re-solving problems
|
|
4. **Cost analysis**: Aggregate costs across all providers
|
|
|
|
### Integration with Our Stack
|
|
|
|
```bash
|
|
# Add to mprocs-teams.yaml as a sidecar service
|
|
session-indexer:
|
|
cmd: ["python", "session-indexer.py", "--watch-dirs",
|
|
"~/.claude/sessions/", "~/.pi/sessions/",
|
|
"~/.opencode/sessions/", "logs/"]
|
|
|
|
# Search across everything
|
|
session-search "deployment error" --providers claude,pi,opencode
|
|
# Returns results from all providers in unified format
|
|
```
|
|
|
|
---
|
|
|
|
## Lesson 5.7: Deployment Modes
|
|
|
|
From Paperclip's deployment model:
|
|
|
|
| Mode | Auth | Exposure | Use Case |
|
|
|------|------|----------|----------|
|
|
| Local trusted | None | localhost only | Single dev machine |
|
|
| Authenticated private | Login required | LAN/Tailscale/VPN | Team on private network |
|
|
| Authenticated public | Login required | Internet (behind reverse proxy) | Production cloud deployment |
|
|
|
|
### Reachability
|
|
|
|
| Bind | What It Means |
|
|
|------|---------------|
|
|
| `loopback` | localhost only (default) |
|
|
| `lan` | All interfaces (0.0.0.0) |
|
|
| `tailnet` | Tailscale IP only |
|
|
| `custom` | Specific host/IP |
|
|
|
|
---
|
|
|
|
## Lesson 5.8: Cost Control
|
|
|
|
### Budget Architecture
|
|
|
|
```
|
|
Company budget → Agent budgets → Session budgets → Per-call tracking
|
|
```
|
|
|
|
### Budget Policies
|
|
|
|
```yaml
|
|
budget_policies:
|
|
- metric: "monthly_cost_cents"
|
|
scope: "agent:backend-dev"
|
|
amount: 50000 # $500/month
|
|
warn_at: 80% # warn at $400
|
|
hard_stop: true # kill at $500
|
|
- metric: "session_cost_cents"
|
|
scope: "global"
|
|
amount: 200 # $2/session max
|
|
hard_stop: true
|
|
```
|
|
|
|
### Warning vs Hard Stop
|
|
|
|
- **Warning** (80%) — Notify operator, agent keeps running
|
|
- **Hard Stop** (100%) — Agent paused, new tasks queued, running task cancelled
|
|
|
|
---
|
|
|
|
## Lesson 5.8b: Dry-Run Workflow for Agent Actions
|
|
|
|
Before an agent executes a destructive action (write file, delete, deploy), you want a **preview mode** that shows what the agent WILL do without actually doing it.
|
|
|
|
### The Dry-Run Pattern
|
|
|
|
```
|
|
Agent proposes action → Preview output → Human reviews → Approve/Reject → Execute
|
|
```
|
|
|
|
```python
|
|
class DryRunContext:
|
|
"""Wrap tool execution in dry-run mode."""
|
|
|
|
def __init__(self, dry_run=True):
|
|
self.dry_run = dry_run
|
|
self.proposed_actions = []
|
|
|
|
def execute(self, tool_name, params):
|
|
if self.dry_run:
|
|
# Log what WOULD happen
|
|
self.proposed_actions.append({
|
|
"tool": tool_name,
|
|
"params": params,
|
|
"preview": self._generate_preview(tool_name, params),
|
|
})
|
|
return f"[DRY RUN] Would call {tool_name} with {params}"
|
|
else:
|
|
# Actually execute
|
|
return real_execute(tool_name, params)
|
|
|
|
def _generate_preview(self, tool_name, params):
|
|
if tool_name == "write_file":
|
|
return f"Would write {len(params.get('content',''))} chars to {params.get('path')}"
|
|
elif tool_name == "exec_command":
|
|
return f"Would run: {params.get('command','')[:100]}..."
|
|
elif tool_name == "delete_file":
|
|
return f"Would DELETE: {params.get('path')}"
|
|
return f"Would call {tool_name}"
|
|
```
|
|
|
|
### Implementation Strategies
|
|
|
|
| Strategy | How It Works | Best For |
|
|
|----------|-------------|----------|
|
|
| **Flag-based** | `--dry-run` flag on agent start | Development, testing |
|
|
| **Hook-based** | Pre-tool hook logs intent, skips execution | Production agents |
|
|
| **UI-based** | Agent shows preview, human clicks Confirm | Interactive sessions |
|
|
| **Two-pass** | Agent plans first (dry), then executes (wet) | Complex multi-step tasks |
|
|
|
|
### Docker Dry-Run Example
|
|
|
|
From the dry-run workflow pattern — a Docker-based calculator that logs operations without running them:
|
|
|
|
```bash
|
|
# Build the dry-run sandbox
|
|
docker build -t dry-run-calc -f calculator/Dockerfile .
|
|
|
|
# Run in preview mode
|
|
docker run --rm -e DRY_RUN=true dry-run-calc add 5 3
|
|
# Output: [DRY RUN] Would add 5 + 3 = 8
|
|
|
|
# Run for real
|
|
docker run --rm -e DRY_RUN=false dry-run-calc add 5 3
|
|
# Output: 8
|
|
```
|
|
|
|
### When to Use Dry-Run
|
|
|
|
- **Always** for file writes, deletes, and deploys
|
|
- **Sometimes** for commands that modify state (DB migrations, config changes)
|
|
- **Never** for read-only operations (search, read file, list directory)
|
|
|
|
---
|
|
|
|
## Lesson 5.8c: Cross-Platform Agent Skills
|
|
|
|
Skills should work on any agent — Claude Code, Pi Agent, OpenCode, or Codex. The cross-platform format uses YAML frontmatter and tool-agnostic instructions:
|
|
|
|
```markdown
|
|
---
|
|
name: init-agents-md
|
|
description: Create or refresh AGENTS.md for coding agents.
|
|
Works with Claude Code, Pi Agent, and Codex.
|
|
---
|
|
|
|
# Initialize AGENTS.md
|
|
|
|
Create a short, repo-specific AGENTS.md.
|
|
|
|
## Workflow
|
|
|
|
1. Check if AGENTS.md already exists — if so, stop and ask
|
|
2. Explore the repository structure
|
|
3. Draft AGENTS.md with project purpose, stack, and conventions
|
|
4. Mirror same context into CLAUDE.md if needed
|
|
```
|
|
|
|
### Key Principles
|
|
|
|
1. **Use `~~` or `---` frontmatter** — not agent-specific config
|
|
2. **Avoid CLI flags** — describe the desired outcome, not the command
|
|
3. **Include trigger patterns** — tell the agent when to invoke this skill
|
|
4. **One `SKILL.md` per skill** — no platform-specific variations
|
|
|
|
---
|
|
|
|
## Lab 5.9: Set Up Agent Observability
|
|
|
|
**Objective**: Trace every tool call + LLM completion to a local SQLite database.
|
|
|
|
**Starter**: `course/labs/L5-observability/starter/`
|
|
|
|
## Lab 5.10: CI/CD Pipeline
|
|
|
|
**Objective**: Create a golden dataset and automated regression gate.
|
|
|
|
**Starter**: `course/labs/L5-cicd/starter/`
|