agentic-ai-engineering/site/modules/reference-stack.md

# Reference Architecture: Production Multi-Agent Stack

This document documents a real production multi-agent deployment. Use it as a reference for Module 5 (Production Patterns) and Module 4 (Multi-Agent Orchestration).

## Architecture Overview

```
┌──────────────────────────────────────────────────────────────────┐
│                    agent-mux (Tauri UI)                           │
│              Meta-agent control plane + dashboard                 │
│         Spawns, monitors, kills agents. Human interface.          │
└─────────────────────────┬────────────────────────────────────────┘
                          │ orchestrates via sidecar
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                    mprocs (process monitor)                       │
│                    mprocs -c ~/mprocs-teams.yaml                  │
│          Launches and supervises all agent processes              │
├──────────────┬──────────────┬──────────────┬─────────────────────┤
│ claude-lead  │   gemini     │   opencode   │  sidecar            │
│ (primary)    │  (backup)    │  (OSS alt)   │  (agent-mux proxy)  │
└──────────────┴──────────────┴──────────────┴─────────────────────┘
                          │ each agent runs inside
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                    psmux (tmux session manager)                   │
│              tmux.exe at ~/.cargo/bin/tmux                        │
│          Claude Code --teammate-mode tmux                         │
├─────────┬─────────┬─────────┬─────────┬─────────┬───────────────┤
│  pane 1 │  pane 2 │  pane 3 │  pane 4 │  pane 5 │  ...          │
│  (lead) │ (worker)│(review) │(verify) │ (sidecar)│               │
└─────────┴─────────┴─────────┴─────────┴─────────┴───────────────┘
                          │ each task isolated by
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                    dmux (git worktree isolation)                  │
│           Each agent task gets its own isolated worktree          │
│         dmux runs inside a tmux session with agent-teams          │
└──────────────────────────────────────────────────────────────────┘
```

## Layer-by-Layer Breakdown

### Layer 1: Meta-Control Plane (agent-mux)

- **What**: Tauri desktop application
- **Function**: UI for spawning, monitoring, and killing agents
- **Communication**: Sidecar process (RPC over IPC)
- **Key features**:
  - Start/stop individual agents
  - View live output from all agents
  - Cost tracking per session
  - Session history and replay

### Layer 2: Process Orchestrator (mprocs)

- **Config**: `~/mprocs-teams.yaml`
- **Function**: Launches all agent processes, supervises them
- **Processes**:
  - `claude-lead`: Claude Code (primary agent)
  - `gemini`: Gemini CLI (backup/alternative model)
  - `opencode`: OpenCode CLI (open-source alternative)
  - `qwen`: Qwen model (specialist/fast tasks)
  - `sidecar`: agent-mux IPC proxy
  - `agent-mux`: Tauri UI process

### Layer 3: Session Management (psmux)

- **Binary**: `tmux.exe` at `~/.cargo/bin/tmux`
- **Function**: Terminal multiplexer for agent sessions
- **Key feature**: `Claude Code --teammate-mode tmux` — Claude's native multi-agent split-pane mode
- **Team spawns**: Each `--teammate-mode` invocation creates split panes, each running a separate Claude session

### Layer 4: Work Isolation (dmux)

- **Function**: Git worktree isolation per task
- **Requirement**: Must run inside a tmux session (start agent-teams first, then dmux)
- **Benefit**: Each task gets an isolated filesystem. No cross-task contamination. Easy rollback (delete worktree).

## Tool-Specific Configurations

### Claude Code

```bash
# Launch with teammate mode (multi-agent)
claude --teammate-mode tmux

# Headless mode (for CI/CD)
claude -p "prompt" --allowedTools "Read Write Edit Bash" --print

# With custom hooks
claude --hooks .claude/hooks/
```

### Pi Coding Agent

```bash
# Launch with extensions
pi -e extensions/damage-control.ts \
   -e extensions/tilldone.ts \
   -e extensions/minimal.ts

# RPC mode (programmatic control)
pi --mode rpc

# With custom skills
pi --skills-dir .pi/skills/
```

### OpenCode

```bash
# Using Go model (via proxy)
opencode --model opencode-go/deepseek-v4-flash

# Headless execution
opencode run "prompt"

# With config from shared ~/.grok/config.toml
opencode --config ~/.grok/config.toml
```

### OpenClaw

```bash
# Always-on employee mode
claw start --daemon

# One-shot task execution
claw do "task description"

# Heartbeat schedule
claw schedule --cron "0 */6 * * *" --task "daily-report"
```

## Cost Analysis

| Tool | Best For | Est. Cost/Task | Autonomy Level |
|------|----------|---------------|----------------|
| Claude Code | Complex multi-step tasks | $0.05-$0.30 | High (with hooks) |
| Pi Agent | Custom workflows, safety-critical | $0.02-$0.15 | Very high (extensible) |
| OpenCode | OSS-compatible, budget tasks | $0.01-$0.05 | Medium |
| Gemini | High-volume, simple tasks | $0.002-$0.01 | Low |
| OpenClaw | Scheduled, always-on tasks | $0.01-$0.10 | Autonomous |

## Key Production Patterns

1. **Model heterogeneity**: Different models for different roles. Claude for complex reasoning, Gemini for fast/cheap tasks, Qwen as specialist.

2. **Tool heterogeneity**: Not one agent CLI, but five. Each has different strengths. The stack uses each where it excels.

3. **Defense in depth**: psmux isolates sessions. dmux isolates files. damage-control restricts commands. mprocs restarts failed processes.

4. **Observability**: agent-mux shows live status. mprocs logs output. Session history enables replay debugging.

5. **No single point of failure**: If Claude Code fails, Pi or OpenCode can take over. The mprocs supervisor restarts crashed processes.

## What This Stack Proves

This architecture demonstrates every concept taught in Modules 1-7:

| Concept | Where It Appears |
|---------|-----------------|
| Agent loop | Every tool follows think→act→observe→repeat |
| Tool design | Each tool provides different tools (read, write, bash, search) |
| Security | damage-control, psmux isolation, dmux isolation |
| Multi-agent | teammate-mode, mprocs launching 5 agents |
| Production | mprocs supervision, cost tracking, worktree isolation |
| Economics | Cascade routing across 5 tools by task type |
| Advanced | agent-mux as meta-agent controlling other agents |