Skip to content

# Reference Architecture: Production Multi-Agent Stack ​

This document documents a real production multi-agent deployment. Use it as a reference for Module 5 (Production Patterns) and Module 4 (Multi-Agent Orchestration).

## Architecture Overview ​

    ┌──────────────────────────────────────────────────────────────────┐
    │                    agent-mux (Tauri UI)                           │
    │              Meta-agent control plane + dashboard                 │
    │         Spawns, monitors, kills agents. Human interface.          │
    └─────────────────────────┬────────────────────────────────────────┘
                              │ orchestrates via sidecar
                              ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │                    mprocs (process monitor)                       │
    │                    mprocs -c ~/mprocs-teams.yaml                  │
    │          Launches and supervises all agent processes              │
    ├──────────────┬──────────────┬──────────────┬─────────────────────┤
    │ claude-lead  │   gemini     │   opencode   │  sidecar            │
    │ (primary)    │  (backup)    │  (OSS alt)   │  (agent-mux proxy)  │
    └──────────────┴──────────────┴──────────────┴─────────────────────┘
                              │ each agent runs inside
                              ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │                    psmux (tmux session manager)                   │
    │              tmux.exe at ~/.cargo/bin/tmux                        │
    │          Claude Code --teammate-mode tmux                         │
    ├─────────┬─────────┬─────────┬─────────┬─────────┬───────────────┤
    │  pane 1 │  pane 2 │  pane 3 │  pane 4 │  pane 5 │  ...          │
    │  (lead) │ (worker)│(review) │(verify) │ (sidecar)│               │
    └─────────┴─────────┴─────────┴─────────┴─────────┴───────────────┘
                              │ each task isolated by
                              ▼
    ┌──────────────────────────────────────────────────────────────────┐
    │                    dmux (git worktree isolation)                  │
    │           Each agent task gets its own isolated worktree          │
    │         dmux runs inside a tmux session with agent-teams          │
    └──────────────────────────────────────────────────────────────────┘

## Layer-by-Layer Breakdown ​

### Layer 1: Meta-Control Plane (agent-mux) ​

- What: Tauri desktop application
- Function: UI for spawning, monitoring, and killing agents
- Communication: Sidecar process (RPC over IPC)
- Key features:
  - Start/stop individual agents
  - View live output from all agents
  - Cost tracking per session
  - Session history and replay

### Layer 2: Process Orchestrator (mprocs) ​

- Config: `~/mprocs-teams.yaml`
- Function: Launches all agent processes, supervises them
- Processes:
  - `claude-lead`: Claude Code (primary agent)
  - `gemini`: Gemini CLI (backup/alternative model)
  - `opencode`: OpenCode CLI (open-source alternative)
  - `qwen`: Qwen model (specialist/fast tasks)
  - `sidecar`: agent-mux IPC proxy
  - `agent-mux`: Tauri UI process

### Layer 3: Session Management (psmux) ​

- Binary: `tmux.exe` at `~/.cargo/bin/tmux`
- Function: Terminal multiplexer for agent sessions
- Key feature: `Claude Code --teammate-mode tmux` — Claude's native multi-agent split-pane mode
- Team spawns: Each `--teammate-mode` invocation creates split panes, each running a separate Claude session

### Layer 4: Work Isolation (dmux) ​

- Function: Git worktree isolation per task
- Requirement: Must run inside a tmux session (start agent-teams first, then dmux)
- Benefit: Each task gets an isolated filesystem. No cross-task contamination. Easy rollback (delete worktree).

## Tool-Specific Configurations ​

### Claude Code ​

bash

    # Launch with teammate mode (multi-agent)
    claude --teammate-mode tmux

    # Headless mode (for CI/CD)
    claude -p "prompt" --allowedTools "Read Write Edit Bash" --print

    # With custom hooks
    claude --hooks .claude/hooks/

### Pi Coding Agent ​

bash

    # Launch with extensions
    pi -e extensions/damage-control.ts \
       -e extensions/tilldone.ts \
       -e extensions/minimal.ts

    # RPC mode (programmatic control)
    pi --mode rpc

    # With custom skills
    pi --skills-dir .pi/skills/

### OpenCode ​

bash

    # Using Go model (via proxy)
    opencode --model opencode-go/deepseek-v4-flash

    # Headless execution
    opencode run "prompt"

    # With config from shared ~/.grok/config.toml
    opencode --config ~/.grok/config.toml

### OpenClaw ​

bash

    # Always-on employee mode
    claw start --daemon

    # One-shot task execution
    claw do "task description"

    # Heartbeat schedule
    claw schedule --cron "0 */6 * * *" --task "daily-report"

## Cost Analysis ​

Tool | Best For | Est. Cost/Task | Autonomy Level
--- | --- | --- | ---
Claude Code | Complex multi-step tasks | $0.05-$0.30 | High (with hooks)
Pi Agent | Custom workflows, safety-critical | $0.02-$0.15 | Very high (extensible)
OpenCode | OSS-compatible, budget tasks | $0.01-$0.05 | Medium
Gemini | High-volume, simple tasks | $0.002-$0.01 | Low
OpenClaw | Scheduled, always-on tasks | $0.01-$0.10 | Autonomous

## Key Production Patterns ​

1. Model heterogeneity: Different models for different roles. Claude for complex reasoning, Gemini for fast/cheap tasks, Qwen as specialist.

2. Tool heterogeneity: Not one agent CLI, but five. Each has different strengths. The stack uses each where it excels.

3. Defense in depth: psmux isolates sessions. dmux isolates files. damage-control restricts commands. mprocs restarts failed processes.

4. Observability: agent-mux shows live status. mprocs logs output. Session history enables replay debugging.

5. No single point of failure: If Claude Code fails, Pi or OpenCode can take over. The mprocs supervisor restarts crashed processes.

## What This Stack Proves ​

This architecture demonstrates every concept taught in Modules 1-7:

Concept | Where It Appears
--- | ---
Agent loop | Every tool follows think→act→observe→repeat
Tool design | Each tool provides different tools (read, write, bash, search)
Security | damage-control, psmux isolation, dmux isolation
Multi-agent | teammate-mode, mprocs launching 5 agents
Production | mprocs supervision, cost tracking, worktree isolation
Economics | Cascade routing across 5 tools by task type
Advanced | agent-mux as meta-agent controlling other agents