8.4 KiB

Raw Blame History

Reference Architecture: Production Multi-Agent Stack

This document documents a real production multi-agent deployment. Use it as a reference for Module 5 (Production Patterns) and Module 4 (Multi-Agent Orchestration).

Architecture Overview

┌──────────────────────────────────────────────────────────────────┐
│                    agent-mux (Tauri UI)                           │
│              Meta-agent control plane + dashboard                 │
│         Spawns, monitors, kills agents. Human interface.          │
└─────────────────────────┬────────────────────────────────────────┘
                          │ orchestrates via sidecar
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                    mprocs (process monitor)                       │
│                    mprocs -c ~/mprocs-teams.yaml                  │
│          Launches and supervises all agent processes              │
├──────────────┬──────────────┬──────────────┬─────────────────────┤
│ claude-lead  │   gemini     │   opencode   │  sidecar            │
│ (primary)    │  (backup)    │  (OSS alt)   │  (agent-mux proxy)  │
└──────────────┴──────────────┴──────────────┴─────────────────────┘
                          │ each agent runs inside
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                    psmux (tmux session manager)                   │
│              tmux.exe at ~/.cargo/bin/tmux                        │
│          Claude Code --teammate-mode tmux                         │
├─────────┬─────────┬─────────┬─────────┬─────────┬───────────────┤
│  pane 1 │  pane 2 │  pane 3 │  pane 4 │  pane 5 │  ...          │
│  (lead) │ (worker)│(review) │(verify) │ (sidecar)│               │
└─────────┴─────────┴─────────┴─────────┴─────────┴───────────────┘
                          │ each task isolated by
                          ▼
┌──────────────────────────────────────────────────────────────────┐
│                    dmux (git worktree isolation)                  │
│           Each agent task gets its own isolated worktree          │
│         dmux runs inside a tmux session with agent-teams          │
└──────────────────────────────────────────────────────────────────┘

Layer-by-Layer Breakdown

Layer 1: Meta-Control Plane (agent-mux)

What: Tauri desktop application
Function: UI for spawning, monitoring, and killing agents
Communication: Sidecar process (RPC over IPC)
Key features:
- Start/stop individual agents
- View live output from all agents
- Cost tracking per session
- Session history and replay

Layer 2: Process Orchestrator (mprocs)

Config: ~/mprocs-teams.yaml
Function: Launches all agent processes, supervises them
Processes:
- claude-lead: Claude Code (primary agent)
- gemini: Gemini CLI (backup/alternative model)
- opencode: OpenCode CLI (open-source alternative)
- qwen: Qwen model (specialist/fast tasks)
- sidecar: agent-mux IPC proxy
- agent-mux: Tauri UI process

Layer 3: Session Management (psmux)

Binary: tmux.exe at ~/.cargo/bin/tmux
Function: Terminal multiplexer for agent sessions
Key feature: Claude Code --teammate-mode tmux — Claude's native multi-agent split-pane mode
Team spawns: Each --teammate-mode invocation creates split panes, each running a separate Claude session

Layer 3b: Factory Orchestration (Paperclip)

Paperclip sits above the agent layer as the factory control plane:

CEO Board (creates issues)
    │ Plan
    ▼
Paperclip Heartbeat (assigns tasks)
    │ Build
    ▼
Coding Agent (Claude Code / Pi / OpenCode)
    │ Test
    ▼
CI/CD Pipeline (golden dataset)
    │ Verify
    ▼
Verifier Agent (read-only checks)
    │ Ship
    ▼
Paperclip (marks done, logs cost)

All authenticated via JWT (Tier 1: local short-lived, Tier 2: CLI API key, Tier 3: agent self-registration). See SOFTWARE-FACTORY.md for the full pipeline.

Layer 4: Work Isolation (dmux)

Function: Git worktree isolation per task
Requirement: Must run inside a tmux session (start agent-teams first, then dmux)
Benefit: Each task gets an isolated filesystem. No cross-task contamination. Easy rollback (delete worktree).

Tool-Specific Configurations

Claude Code

# Launch with teammate mode (multi-agent)
claude --teammate-mode tmux

# Headless mode (for CI/CD)
claude -p "prompt" --allowedTools "Read Write Edit Bash" --print

# With custom hooks
claude --hooks .claude/hooks/

Pi Coding Agent

# Launch with extensions
pi -e extensions/damage-control.ts \
   -e extensions/tilldone.ts \
   -e extensions/minimal.ts

# RPC mode (programmatic control)
pi --mode rpc

# With custom skills
pi --skills-dir .pi/skills/

OpenCode

# Using Go model (via proxy)
opencode --model opencode-go/deepseek-v4-flash

# Headless execution
opencode run "prompt"

# With config from shared ~/.grok/config.toml
opencode --config ~/.grok/config.toml

OpenClaw

# Always-on employee mode
claw start --daemon

# One-shot task execution
claw do "task description"

# Heartbeat schedule
claw schedule --cron "0 */6 * * *" --task "daily-report"

Cost Analysis

Tool	Best For	Est. Cost/Task	Autonomy Level
Claude Code	Complex multi-step tasks	$0.05-$0.30	High (with hooks)
Pi Agent	Custom workflows, safety-critical	$0.02-$0.15	Very high (extensible)
OpenCode	OSS-compatible, budget tasks	$0.01-$0.05	Medium
Gemini	High-volume, simple tasks	$0.002-$0.01	Low
OpenClaw	Scheduled, always-on tasks	$0.01-$0.10	Autonomous

Key Production Patterns

Model heterogeneity: Different models for different roles. Claude for complex reasoning, Gemini for fast/cheap tasks, Qwen as specialist.
Tool heterogeneity: Not one agent CLI, but five. Each has different strengths. The stack uses each where it excels.
Defense in depth: psmux isolates sessions. dmux isolates files. damage-control restricts commands. mprocs restarts failed processes.
Observability: agent-mux shows live status. mprocs logs output. Session history enables replay debugging.
No single point of failure: If Claude Code fails, Pi or OpenCode can take over. The mprocs supervisor restarts crashed processes.

What This Stack Proves

This architecture demonstrates every concept taught in Modules 1-7:

Concept	Where It Appears
Agent loop	Every tool follows think→act→observe→repeat
Tool design	Each tool provides different tools (read, write, bash, search)
Security	damage-control, psmux isolation, dmux isolation
Multi-agent	teammate-mode, mprocs launching 5 agents
Production	mprocs supervision, cost tracking, worktree isolation
Economics	Cascade routing across 5 tools by task type
Advanced	agent-mux as meta-agent controlling other agents

8.4 KiB Raw Blame History