agentic-ai-engineering/site/modules/m8-capstone.md

Module 8: Capstone — Production Multi-Agent System

Overview

Build, deploy, and evaluate a complete production-grade multi-agent system. Choose from three project options, or propose your own.

---

Project Options

Option 1: Brand Monitor (Intermediate)

Build a multi-agent system that monitors how LLMs mention a brand across multiple AI services.

Agents:

• Scanner Agent — queries ChatGPT, Gemini, Grok, Perplexity for brand mentions
• Analyzer Agent — classifies mentions as positive/negative/neutral
• Reporter Agent — generates daily summary reports

Requirements:

• Each agent has domain-locked permissions
• Each agent maintains a mental model
• Scanner uses cascade routing (Flash for scrape, Sonnet for analysis)
• Daily automated runs via heartbeat
• Cost tracking per scan

Option 2: Code Review Pipeline (Intermediate-Advanced)

A plan → build → review → verify chain with CI/CD integration.

Agents:

• Planner — reads spec, creates implementation plan
• Builder — writes code per plan
• Reviewer — reviews code for bugs, style, security
• Verifier — read-only agent that checks all claims

Requirements:

• YAML-defined agent chain pipeline
• Golden dataset for regression testing
• Verifier agent with confidence ladder
• GitHub PR integration
• Cost analysis per review cycle

Option 3: Strategic Decision Board (Advanced)

An 8-agent CEO board with adversarial debate, fact verification, and execution planning.

Agents:

• CEO (orchestrator)
• 6 board members with distinct perspectives
• Verifier (fact-checker)
• Tracker (quarterly review)

Requirements:

• Each board member has distinct persona + mental model
• Fact verification (2+ sources per claim)
• Contrarian position tracked at 3x weight
• Structured memo output
• Audio summary (TTS)
• Cost tracking per deliberation

---

Capstone Phases

Phase 1: System Design (60-90 min)

Deliverable: One-page architecture document including:

• System diagram (agents + data flow + tool surface)
• Agent roster (names, personas, domain permissions)
• Data model (what state is shared, how)
• Security architecture (what L-level, what hooks)
• Cost estimate per session

Template:

# System Design: [Project Name]

## Architecture
[ASCII diagram or mermaid]

## Agent Roster
| Agent | Model | Tools | Domain | Memory |
|-------|-------|-------|--------|--------|

## Data Flow
[How agents communicate, what state they share]

## Security
[L-level, hooks, verifier, damage-control rules]

## Cost Estimate
[Per-session estimate with model pricing]

Phase 2: Agent Implementation (2-4 hours)

Build each agent:

• System prompt with persona + behavioral rules
• Tool definitions with reasoning parameter
• Mental model file (agent-owned expertise)
• Skills for composable behavior

Phase 3: Integration (1-2 hours)

Wire agents together:

• Chain (if pipeline pattern)
• Team (if dispatcher pattern)
• P2P (if flat pattern)
• Shared context (conversation log)

Phase 4: Security Hardening (1-2 hours)

Apply security layer:

• Minimum L3 (blacklist hook)
• Target L4 (whitelist hook)
• Optional L5 (no bash)
• Damage-control rules
• Verifier agent (read-only checks)

Phase 5: Testing & Evaluation (1-2 hours)

• Create golden dataset (10+ test cases)
• Run pass@k evaluation (k=3, k=5)
• Calculate cost per task
• Identify optimization opportunities

Phase 6: Deployment (1-2 hours)

• CI/CD pipeline with golden dataset gate
• Shadow deployment vs baseline
• Observability: every tool call + cost logged
• Monitoring dashboard

Phase 7: Review & Iterate (1 hour)

• Red-team your own system (try to break it)
• Document lessons learned
• Create retrospective

---

Deliverables Checklist

[ ] Architecture document (one page, with diagram)
[ ] All agent source code with system prompts
[ ] Agent configuration files (YAML/JSON)
[ ] Tool definitions with reasoning parameters
[ ] Security hooks configuration (L3+)
[ ] Golden dataset (10+ test cases with expected outputs)
[ ] Test script showing all agents respond correctly
[ ] Cost analysis per session
[ ] README with setup instructions
[ ] Self-assessment against rubric

---

## Production Readiness Checklist

Before submitting, verify your system meets these production standards:

### Reliability
- [ ] All tools include error handling (FileNotFoundError, API timeout, etc.)
- [ ] MAX_ITERATIONS set on every agent loop (no infinite loops)
- [ ] Retry logic with exponential backoff for API calls
- [ ] Agent gracefully handles tool failures instead of crashing

### Security
- [ ] Minimum L3 blacklist hook installed on bash access
- [ ] Domain locking prevents agents from accessing files outside their scope
- [ ] Verifier agent has read-only tools only (no edit/bash)
- [ ] Damage-control rules prevent rm -rf, DROP TABLE, etc.

### Observability
- [ ] Every tool call logged to file (JSONL format)
- [ ] Cost tracked per session
- [ ] Agent loop depth monitored
- [ ] Errors logged with stack traces

### Economics
- [ ] Cost estimate per session documented
- [ ] Cascade routing implemented where appropriate
- [ ] Iteration limits set to minimize token waste
- [ ] Model selection justified (why this model for this agent?)

---

## Graduation Criteria

Your capstone passes if it meets ALL of these:

1. **Three or more agents** that communicate and coordinate
2. **Security at L3+** (blacklist or whitelist hook)
3. **Working test script** that demonstrates all agents responding
4. **Documented cost estimate** per session
5. **Observability** — at least one trace log from a complete run
6. **Self-assessment** against the rubric with honest notes on what you'd improve

The capstone is not about perfection. It's about demonstrating you understand the full stack — from tool design to security to economics. A passing capstone with honest self-assessment is worth more than a perfect one with no reflection.
[ ] Mental model files for each agent
[ ] Security configuration (damage-control rules)
[ ] Verifier agent (if applicable)
[ ] Golden dataset (10+ test cases)
[ ] Test results (pass@k at k=1, k=3, k=5)
[ ] Cost analysis ($/task, optimization opportunities)
[ ] Security audit (which L-level, what gaps remain)
[ ] Retrospective (max 1 page)

---

Pass Criteria

Criterion	Minimum	Target
System runs without manual intervention	✓	✓
All agents have domain-locked permissions	✓	✓
Each agent has mental model file	✓	✓
pass@k (k=3) on golden dataset	>60%	>80%
Cost analysis within 2x of optimal	✓	✓
Security audit identifies ≥2 improvements	✓	✓
Observability captures all tool calls	✓	✓
Architecture document submitted	✓	✓

---

Grading Rubric

Area	Weight	Poor (0)	Good (1)	Excellent (2)
Architecture	20%	No diagram, unclear design	Diagram present, mostly clear	Clear diagram, justified choices
Implementation	25%	Agents don't work	Agents work on happy path	Agents handle errors gracefully
Security	20%	L1 only	L3+ with damage-control	L4+ with verifier
Testing	15%	No eval	pass@k computed	pass@k + cost analysis + grind detection
Documentation	10%	Minimal	Architecture + setup	Architecture + setup + retrospective
Cost Optimization	10%	Single model	Cascade routing	Cascade + verified savings