16 KiB

Raw Blame History

Module 4: Multi-Agent Orchestration

Lesson 4.1: Why One Agent Is Not Enough

A single agent, no matter how smart, hits three ceilings:

Context ceiling — One agent doing everything means one context window holding everything. File contents, database schemas, business logic, deployment configs — it all competes for space.
Capability ceiling — A generalist agent is mediocre at everything. A specialized agent (backend dev, security reviewer, UI designer) outperforms the generalist at its domain.
Reliability ceiling — One agent failing = everything fails. Multi-agent systems degrade gracefully — one team's failure doesn't take down the whole system.

The Stacking Insight

Single smart model  <  Two specialized agents  <  Orchestrator + 3 teams

Benchmarks measure single models. Production wins with multi-agent orchestration.

When to Go Multi-Agent

You need multiple agents when any of these are true:

Different expertise required: Your task needs a planner, a coder, a reviewer, and a security auditor. One agent can't be great at all four.
Parallel work possible: Multiple independent sub-tasks can run simultaneously. A single agent serializes them.
Failure cost is high: If the agent makes a mistake, you want a second agent to catch it before it ships.
Context exceeds one window: The codebase, docs, and requirements together exceed ~100K tokens. Split across specialized agents.

If none of these are true, a single well-configured agent is simpler and cheaper.

Lesson 4.2: Orchestration Patterns

Pattern 1: Dispatcher (Hub-and-Spoke)

User → Orchestrator → dispatches to:
                       ├── Specialist A (parallel)
                       ├── Specialist B (parallel)
                       └── Specialist C (parallel)
              ← synthesizes results

Best for: Tasks that need multiple perspectives or parallel work. The orchestrator doesn't work — it delegates, then synthesizes.

Implementation: agent-team extension. Teams defined in teams.yaml. Orchestrator uses dispatch_agent tool.

Pattern 2: Pipeline (Sequential)

Step 1: Planner → Step 2: Builder (gets $INPUT=plan) → Step 3: Reviewer (gets $INPUT=code)

Best for: Workflows with clear stages where each depends on the previous.

Implementation: agent-chain extension. Pipelines in agent-chain.yaml. $INPUT carries forward, $ORIGINAL preserves user prompt.

Pattern 3: Peer-to-Peer (Flat)

Agent A ←→ Agent B ←→ Agent C
   (no orchestrator, all peers)

Best for: Cross-device work, heterogeneous model teams, flat information flow.

Implementation: coms / coms-net extensions. 4 tools: list, send, get, await.

How to Choose

Your situation	Use
One person, one agent	Single agent. Don't over-engineer.
One person, many tasks	P-Threads. Run parallel agents, pick best.
Team, defined workflow	Pipeline. Plan → Build → Review.
Team, complex project	Dispatcher. Orchestrator assigns work.
Cross-device, different models	P2P. Agents talk directly, no hierarchy.

Lesson 4.2b: P-Threads — Parallel Agent Execution

Concept: Run N agents simultaneously on the same task. Each agent works independently. Pick the best result.

Pattern

User prompt
     │
     ├── Agent 1 (Claude Opus)  ──►  output_1
     ├── Agent 2 (Gemini Pro)   ──►  output_2
     ├── Agent 3 (DeepSeek V3)  ──►  output_3
     └── Agent 4 (Qwen)         ──►  output_4
          │
          ▼
     Judge (or human) picks best

When to Use P-Threads

Creative work (UI generation, content writing, architecture design)
Benchmarking (compare models side-by-side on same task)
High-stakes decisions (multiple perspectives reduce blind spots)
Exploration (try N approaches, pick the winner)

Implementation with mprocs

# mprocs-teams.yaml
procs:
  claude-lead:
    cmd: ["claude", "-p", "{{PROMPT}}"]
  gemini:
    cmd: ["gemini", "-p", "{{PROMPT}}"]
  opencode:
    cmd: ["opencode", "--model", "opencode-go/deepseek-v4-flash", "-p", "{{PROMPT}}"]

Run all three simultaneously, collect outputs, pick best.

Implementation with agent-mux

Your Tauri UI (agent-mux) is a P-thread controller: it spawns agents into panes, monitors progress, and presents results for comparison.

Key Design Rules

Each agent gets the same prompt (fair comparison)
Agents are isolated (no cross-contamination)
Results are collected and compared by a judge or human
Cost = N × single-agent cost (budget accordingly)

Lesson 4.2c: F-Threads — Fusion (N Agents, One Winner)

Concept: N agents produce outputs. A synthetic judge (or rubric) selects the winner. More sophisticated than P-threads — includes evaluation.

Pattern

Agent 1 ──► Output 1 ──┐
Agent 2 ──► Output 2 ──┤
Agent 3 ──► Output 3 ──┤──► Judge → Winner
Agent 4 ──► Output 4 ──┤
Agent 5 ──► Output 5 ──┘

When to Use F-Threads

Code generation (generate N variants, pick best compiled/tested)
UI generation (your UI Agents system uses this — 3 Vue generators, 1 winner)
Architecture decisions (N proposals, judge evaluates against criteria)
Bug fixes (N approaches, pick the one that passes tests)

Your UI Agents System as F-Thread Example

Your ui-agents project uses F-threads natively:

3 UI Generation agents (Sonnet + open source variants)
Validation team screenshots + checks each
Lead picks the best result
Loser outputs are discarded (or logged for learning)

Judge Criteria Template

{
  "criteria": [
    {"name": "correctness", "weight": 0.4},
    {"name": "efficiency", "weight": 0.2},
    {"name": "maintainability", "weight": 0.2},
    {"name": "completeness", "weight": 0.2}
  ]
}

Lesson 4.3: Depth-2 Delegation

The production-proven hierarchy from lead-agents and ui-agents:

User
  │
  v
Orchestrator (Opus-level, thinking only)
  │
  ├── Planning Team Lead (synthesizes, delegates)
  │     ├── Product Manager (domain worker)
  │     └── UX Researcher (domain worker)
  │
  ├── Engineering Team Lead (synthesizes, delegates)
  │     ├── Frontend Dev (domain worker)
  │     └── Backend Dev (domain worker)
  │
  └── Validation Team Lead (synthesizes, delegates)
        ├── QA Engineer (domain worker)
        └── Security Reviewer (domain worker)

The Rule

Leads and orchestrators are thinkers, planners, and managers. They are not doers. They delegate to workers who write files and make changes. Workers don't make strategic decisions.

This rule exists because the most common multi-agent failure is the orchestrator doing the work instead of delegating. When the orchestrator gets involved in implementation, it loses visibility into the big picture. Tasks slip, context windows fill with irrelevant details, and the system collapses into a single-agent system with extra latency.

Enforce this rule with domain permissions: Give the orchestrator read-only access to the codebase and zero write tools. The only tool it should have is delegate. Workers get the write tools. The orchestrator gets the delegate tool.

Lesson 4.4: Agent Experts That Remember

Mental Models

Every agent maintains its own expertise file:

# engineering-lead-mental-model.yaml
observations:
  - type: architectural_pattern
    observation: "We use vertical slice architecture for new features"
    evidence: "PR #142, PR #156"
  - type: common_failure
    observation: "WebSocket reconnection logic needs retry with backoff"
    evidence: "Incident log 2026-03-15"

Self-Improve Commands

Agents periodically run self-improve commands that:

Read the mental model
Compare against actual codebase (grep, read files)
Update stale entries
Add new findings

Compounding Knowledge

Session 1: Agent learns project structure Session 2: Agent learns common patterns Session 3: Agent learns failure modes Session N: Agent operates at senior engineer level for this codebase

Lesson 4.5: Domain Locking

Every agent has explicit domain permissions:

# Engineering Lead domain
domain:
  - path: .pi/multi-team/
    read: true
    upsert: true
    delete: false
  - path: src/
    read: true
    upsert: true
    delete: false
  - path: .
    read: true
    upsert: false
    delete: false  # Can't delete anything at root

Three levels:

read: false — can't even see the files
upsert: false — can read, can't write
delete: false — can read+write, can't delete

Lesson 4.6: TillDone Task Discipline

The Pattern

TillDone forces the agent to define tasks before doing work. No "just start coding" — the task list must exist first.

Orchestrator creates task list before delegating
Tasks are assigned to specific agents
Agents mark tasks: idle → inprogress → done
Footer shows live progress: "TillDone: Build auth [3/5]"
If turn ends with incomplete tasks, agent is nudged to continue

Why It Matters

Prevents the #1 multi-agent failure mode: an orchestrator that delegates, gets results, and then wanders off without completing the plan. The task list is the forcing function. Without it, agents drift — they start strong on task A, get distracted by task B, partially complete both, and the session ends with nothing shipped.

TillDone ensures each session produces a concrete outcome. When the task list is empty, the work is done. No ambiguity.

Lesson 4.7: Agent Chains

YAML-Defined Pipelines

Agent chains let you define multi-step workflows in a single YAML file. Each step feeds its output to the next step via $INPUT.

steps:
  - agent: planner
    prompt: "Create a detailed plan for: $INPUT"
  - agent: builder
    prompt: "Implement the following plan: $INPUT"
  - agent: reviewer
    prompt: "Review this code: $INPUT"

Variables

$INPUT — Output of the previous step
$ORIGINAL — The user's original prompt (always accessible)

This creates a sequential pipeline where each agent receives the output of the previous agent. The planner produces a plan, the builder implements it, and the reviewer checks it. Any step can fail independently, and the pipeline stops at the failure point instead of continuing with bad input.

Lesson 4.8: Pi-to-Pi Communication

The Shift: Hierarchy → Flat

Traditional agent communication is top-down:

Orchestrator → Team Lead → Worker → (result goes back up)

Peer-to-peer flips this. Agents are equals:

Agent A ←→ Agent B
   ↕          ↕
Agent C ←→ Agent D

Reference Implementation: MCP Agent Mail (Jeff Emanuel, 1,955★)

Jeff's MCP Agent Mail provides a more formal coordination layer:

Agent identities: Each agent has a registered identity
Inboxes: Messages are delivered to agent inboxes
Searchable threads: Full thread history with search
Advisory file leases: Agents can reserve files to prevent conflicts

This complements our simpler P2P approach. Use direct coms for speed, MCP Agent Mail for audit trails.

Four Tools

Tool	What It Does
`*_list`	List peer agents (name, model, context usage)
`*_send`	Send a prompt; returns msg_id
`*_get`	Non-blocking poll on msg_id
`*_await`	Block until reply or timeout

Safety

Max 5 hops (prevents A→B→A→B loops)
Audit log (msg_id, sender, hops — never prompt bodies)
Self-healing (stale sockets pruned, heartbeats every 10s)

Lesson 4.9: Conversation Awareness

Every agent reads a shared JSONL conversation log before responding:

.pi/multi-team/sessions/<session-id>/conversation.jsonl

This gives every agent full context of what has been discussed. The orchestrator knows what each team has reported. Team leads know what other leads have discovered. Workers know what their lead has already decided.

Why It Matters

Without conversation awareness, agents repeat work. The planner creates a plan, the builder implements it, and the reviewer asks for changes the planner already rejected in the previous round. The conversation log prevents this by making every decision visible to every agent.

It also creates a complete audit trail. Every prompt, every response, every tool call is recorded. You can replay any session to debug failures or understand why a particular decision was made.

Lesson 4.9b: Service Connectors

Agents need to interact with external services: Twitter, GitHub, Linear, Slack, email.

The Pattern

Each service gets an MCP connector:

Agent → MCP Client → Service Connector → External API
  ├── twitter-connector: post, search, DM, timeline
  ├── github-connector: PRs, issues, actions, code search
  ├── linear-connector: issues, comments, projects, cycles
  ├── slack-connector: messages, channels, threads, search
  ├── email-connector: send, read, search, folders
  └── custom: build your own via connector SDK

Design Rules

One connector per service — composable, not monolithic
MCP protocol — standard interface for all connectors
Scoped credentials — each connector has its own auth (no shared tokens)
Rate-limited — connectors enforce service rate limits
Audit-logged — every external action is logged

Reference Implementation

Jeff Emanuel's flywheel_connectors (79★) provides a mesh-native protocol for this. His approach:

Each connector is a standalone Rust binary
Connectors communicate via a shared mesh protocol
Agents discover connectors dynamically
Built-in connectors for Twitter, Linear, GitHub

Integration with Agent Teams

Orchestrator → delegates to agent
  Agent → calls service connector via MCP
    Connector → authenticates → calls API → returns result
  Agent → processes result → continues work
Orchestrator → synthesizes final output

Lesson 4.10: The CEO Board System

8 specialist agents for strategic decision-making:

Board Member	Focus	Time Horizon
Revenue	Cash flow, short-term wins	30-90 days
Compounder	Trust, long-term value	6-24 months
Contrarian	Assumptions, blind spots	3x weight on dissent
Technical Architect	System durability	Ongoing
Product Strategist	Problem selection	Quarterly
Customer Oracle	User behavior	Ongoing
Market Strategist	Positioning	Quarterly
Moonshot	Asymmetric upside	1-5 years

Flow

CEO frames the decision
Board debates (sources required, no "I think")
Verifier checks facts (2+ sources per claim)
Executor creates execution plan
Tracker logs for quarterly review

Lesson 4.11: UI Agents System

12 agents across 4 teams for brand-consistent UI generation:

Brand → Product → Tree → Branch → Leaf

Each level has its own brand.yaml with CSS custom properties. Zero hardcoded values. Agents generate Vue components that use only CSS custom properties from the brand config.

Lab 4.12: Build an Agent Chain

Objective: Create a plan→build→review pipeline in YAML.

Starter: course/labs/L4-agent-chain/starter.yaml Solution: course/labs/L4-agent-chain/solution.yaml

Checkpoints:

Planner produces structured plan with tasks and file paths
Builder creates code matching the plan
Reviewer identifies issues with severity (critical/major/minor)
Verifier confirms each claim with file:line evidence

Lab 4.13: Deploy a Multi-Team System

Objective: Set up orchestrator + 2 teams with domain locking.

Starter: course/labs/L4-multi-team/starter-config.yaml Solution: course/labs/L4-multi-team/solution-config.yaml

Checkpoints:

Orchestrator delegates, never executes
Each team has lead + members with distinct roles
Domain permissions restrict each agent to its scope
Mental model files exist for each agent

Next: Module 5: Production Patterns

16 KiB Raw Blame History Unescape Escape

Module 4: Multi-Agent Orchestration

Lesson 4.1: Why One Agent Is Not Enough

The Stacking Insight

When to Go Multi-Agent

Lesson 4.2: Orchestration Patterns

Pattern 1: Dispatcher (Hub-and-Spoke)

Pattern 2: Pipeline (Sequential)

Pattern 3: Peer-to-Peer (Flat)

How to Choose

Lesson 4.2b: P-Threads — Parallel Agent Execution

Pattern

When to Use P-Threads

Implementation with mprocs

Implementation with agent-mux

Key Design Rules

Lesson 4.2c: F-Threads — Fusion (N Agents, One Winner)

Pattern

When to Use F-Threads

Your UI Agents System as F-Thread Example

Judge Criteria Template

Lesson 4.3: Depth-2 Delegation

The Rule

Lesson 4.4: Agent Experts That Remember

Mental Models

Self-Improve Commands

Compounding Knowledge

Lesson 4.5: Domain Locking

Lesson 4.6: TillDone Task Discipline

The Pattern

Why It Matters

Lesson 4.7: Agent Chains

YAML-Defined Pipelines

Variables

Lesson 4.8: Pi-to-Pi Communication

The Shift: Hierarchy → Flat

Reference Implementation: MCP Agent Mail (Jeff Emanuel, 1,955★)

Four Tools

Safety

Lesson 4.9: Conversation Awareness

Why It Matters

Lesson 4.9b: Service Connectors

The Pattern

Design Rules

Reference Implementation

Integration with Agent Teams

Lesson 4.10: The CEO Board System

Flow

Lesson 4.11: UI Agents System

Lab 4.12: Build an Agent Chain

Lab 4.13: Deploy a Multi-Team System

16 KiB

Raw Blame History