Skip to content

# Module 8: Capstone — Production Multi-Agent System ​

## Overview ​

Build, deploy, and evaluate a complete production-grade multi-agent system. Choose from three project options, or propose your own.

## Project Options ​

### Option 1: Brand Monitor (Intermediate) ​

Build a multi-agent system that monitors how LLMs mention a brand across multiple AI services.

Agents:

- Scanner Agent — queries ChatGPT, Gemini, Grok, Perplexity for brand mentions
- Analyzer Agent — classifies mentions as positive/negative/neutral
- Reporter Agent — generates daily summary reports

Requirements:

- Each agent has domain-locked permissions
- Each agent maintains a mental model
- Scanner uses cascade routing (Flash for scrape, Sonnet for analysis)
- Daily automated runs via heartbeat
- Cost tracking per scan

### Option 2: Code Review Pipeline (Intermediate-Advanced) ​

A plan → build → review → verify chain with CI/CD integration.

Agents:

- Planner — reads spec, creates implementation plan
- Builder — writes code per plan
- Reviewer — reviews code for bugs, style, security
- Verifier — read-only agent that checks all claims

Requirements:

- YAML-defined agent chain pipeline
- Golden dataset for regression testing
- Verifier agent with confidence ladder
- GitHub PR integration
- Cost analysis per review cycle

### Option 3: Strategic Decision Board (Advanced) ​

An 8-agent CEO board with adversarial debate, fact verification, and execution planning.

Agents:

- CEO (orchestrator)
- 6 board members with distinct perspectives
- Verifier (fact-checker)
- Tracker (quarterly review)

Requirements:

- Each board member has distinct persona + mental model
- Fact verification (2+ sources per claim)
- Contrarian position tracked at 3x weight
- Structured memo output
- Audio summary (TTS)
- Cost tracking per deliberation

## Capstone Phases ​

### Phase 1: System Design (60-90 min) ​

Deliverable: One-page architecture document including:

- System diagram (agents + data flow + tool surface)
- Agent roster (names, personas, domain permissions)
- Data model (what state is shared, how)
- Security architecture (what L-level, what hooks)
- Cost estimate per session

Template:

markdown

    # System Design: [Project Name]

    ## Architecture
    [ASCII diagram or mermaid]

    ## Agent Roster
    | Agent | Model | Tools | Domain | Memory |
    |-------|-------|-------|--------|--------|

    ## Data Flow
    [How agents communicate, what state they share]

    ## Security
    [L-level, hooks, verifier, damage-control rules]

    ## Cost Estimate
    [Per-session estimate with model pricing]

### Phase 2: Agent Implementation (2-4 hours) ​

Build each agent:

- System prompt with persona + behavioral rules
- Tool definitions with reasoning parameter
- Mental model file (agent-owned expertise)
- Skills for composable behavior

### Phase 3: Integration (1-2 hours) ​

Wire agents together:

- Chain (if pipeline pattern)
- Team (if dispatcher pattern)
- P2P (if flat pattern)
- Shared context (conversation log)

### Phase 4: Security Hardening (1-2 hours) ​

Apply security layer:

- Minimum L3 (blacklist hook)
- Target L4 (whitelist hook)
- Optional L5 (no bash)
- Damage-control rules
- Verifier agent (read-only checks)

### Phase 5: Testing & Evaluation (1-2 hours) ​

- Create golden dataset (10+ test cases)
- Run pass@k evaluation (k=3, k=5)
- Calculate cost per task
- Identify optimization opportunities

### Phase 6: Deployment (1-2 hours) ​

- CI/CD pipeline with golden dataset gate
- Shadow deployment vs baseline
- Observability: every tool call + cost logged
- Monitoring dashboard

### Phase 7: Review & Iterate (1 hour) ​

- Red-team your own system (try to break it)
- Document lessons learned
- Create retrospective

## Deliverables Checklist ​

    [ ] Architecture document (one page, with diagram)
    [ ] All agent source code with system prompts
    [ ] Agent configuration files (YAML/JSON)
    [ ] Tool definitions with reasoning parameters
    [ ] Security hooks configuration (L3+)
    [ ] Golden dataset (10+ test cases with expected outputs)
    [ ] Test script showing all agents respond correctly
    [ ] Cost analysis per session
    [ ] README with setup instructions
    [ ] Self-assessment against rubric

    ---

    ## Production Readiness Checklist

    Before submitting, verify your system meets these production standards:

    ### Reliability
    - [ ] All tools include error handling (FileNotFoundError, API timeout, etc.)
    - [ ] MAX_ITERATIONS set on every agent loop (no infinite loops)
    - [ ] Retry logic with exponential backoff for API calls
    - [ ] Agent gracefully handles tool failures instead of crashing

    ### Security
    - [ ] Minimum L3 blacklist hook installed on bash access
    - [ ] Domain locking prevents agents from accessing files outside their scope
    - [ ] Verifier agent has read-only tools only (no edit/bash)
    - [ ] Damage-control rules prevent rm -rf, DROP TABLE, etc.

    ### Observability
    - [ ] Every tool call logged to file (JSONL format)
    - [ ] Cost tracked per session
    - [ ] Agent loop depth monitored
    - [ ] Errors logged with stack traces

    ### Economics
    - [ ] Cost estimate per session documented
    - [ ] Cascade routing implemented where appropriate
    - [ ] Iteration limits set to minimize token waste
    - [ ] Model selection justified (why this model for this agent?)

    ---

    ## Graduation Criteria

    Your capstone passes if it meets ALL of these:

    1. **Three or more agents** that communicate and coordinate
    2. **Security at L3+** (blacklist or whitelist hook)
    3. **Working test script** that demonstrates all agents responding
    4. **Documented cost estimate** per session
    5. **Observability** — at least one trace log from a complete run
    6. **Self-assessment** against the rubric with honest notes on what you'd improve

    The capstone is not about perfection. It's about demonstrating you understand the full stack — from tool design to security to economics. A passing capstone with honest self-assessment is worth more than a perfect one with no reflection.
    [ ] Mental model files for each agent
    [ ] Security configuration (damage-control rules)
    [ ] Verifier agent (if applicable)
    [ ] Golden dataset (10+ test cases)
    [ ] Test results (pass@k at k=1, k=3, k=5)
    [ ] Cost analysis ($/task, optimization opportunities)
    [ ] Security audit (which L-level, what gaps remain)
    [ ] Retrospective (max 1 page)

## Pass Criteria ​

Criterion | Minimum | Target
--- | --- | ---
System runs without manual intervention | ✓ | ✓
All agents have domain-locked permissions | ✓ | ✓
Each agent has mental model file | ✓ | ✓
pass@k (k=3) on golden dataset | >60% | >80%
Cost analysis within 2x of optimal | ✓ | ✓
Security audit identifies ≥2 improvements | ✓ | ✓
Observability captures all tool calls | ✓ | ✓
Architecture document submitted | ✓ | ✓

## Grading Rubric ​

Area | Weight | Poor (0) | Good (1) | Excellent (2)
--- | --- | --- | --- | ---
Architecture | 20% | No diagram, unclear design | Diagram present, mostly clear | Clear diagram, justified choices
Implementation | 25% | Agents don't work | Agents work on happy path | Agents handle errors gracefully
Security | 20% | L1 only | L3+ with damage-control | L4+ with verifier
Testing | 15% | No eval | pass@k computed | pass@k + cost analysis + grind detection
Documentation | 10% | Minimal | Architecture + setup | Architecture + setup + retrospective
Cost Optimization | 10% | Single model | Cascade routing | Cascade + verified savings