agentic-ai-engineering/site/modules/m8-capstone.md

# Module 8: Capstone — Production Multi-Agent System

## Overview

Build, deploy, and evaluate a complete production-grade multi-agent system. Choose from three project options, or propose your own.

---

## Project Options

### Option 1: Brand Monitor (Intermediate)

Build a multi-agent system that monitors how LLMs mention a brand across multiple AI services.

**Agents**:
- **Scanner Agent** — queries ChatGPT, Gemini, Grok, Perplexity for brand mentions
- **Analyzer Agent** — classifies mentions as positive/negative/neutral
- **Reporter Agent** — generates daily summary reports

**Requirements**:
- Each agent has domain-locked permissions
- Each agent maintains a mental model
- Scanner uses cascade routing (Flash for scrape, Sonnet for analysis)
- Daily automated runs via heartbeat
- Cost tracking per scan

### Option 2: Code Review Pipeline (Intermediate-Advanced)

A plan → build → review → verify chain with CI/CD integration.

**Agents**:
- **Planner** — reads spec, creates implementation plan
- **Builder** — writes code per plan
- **Reviewer** — reviews code for bugs, style, security
- **Verifier** — read-only agent that checks all claims

**Requirements**:
- YAML-defined agent chain pipeline
- Golden dataset for regression testing
- Verifier agent with confidence ladder
- GitHub PR integration
- Cost analysis per review cycle

### Option 3: Strategic Decision Board (Advanced)

An 8-agent CEO board with adversarial debate, fact verification, and execution planning.

**Agents**:
- CEO (orchestrator)
- 6 board members with distinct perspectives
- Verifier (fact-checker)
- Tracker (quarterly review)

**Requirements**:
- Each board member has distinct persona + mental model
- Fact verification (2+ sources per claim)
- Contrarian position tracked at 3x weight
- Structured memo output
- Audio summary (TTS)
- Cost tracking per deliberation

---

## Capstone Phases

### Phase 1: System Design (60-90 min)

Deliverable: One-page architecture document including:
- System diagram (agents + data flow + tool surface)
- Agent roster (names, personas, domain permissions)
- Data model (what state is shared, how)
- Security architecture (what L-level, what hooks)
- Cost estimate per session

**Template**:
```markdown
# System Design: [Project Name]

## Architecture
[ASCII diagram or mermaid]

## Agent Roster
| Agent | Model | Tools | Domain | Memory |
|-------|-------|-------|--------|--------|

## Data Flow
[How agents communicate, what state they share]

## Security
[L-level, hooks, verifier, damage-control rules]

## Cost Estimate
[Per-session estimate with model pricing]
```

### Phase 2: Agent Implementation (2-4 hours)

Build each agent:
- System prompt with persona + behavioral rules
- Tool definitions with reasoning parameter
- Mental model file (agent-owned expertise)
- Skills for composable behavior

### Phase 3: Integration (1-2 hours)

Wire agents together:
- Chain (if pipeline pattern)
- Team (if dispatcher pattern)
- P2P (if flat pattern)
- Shared context (conversation log)

### Phase 4: Security Hardening (1-2 hours)

Apply security layer:
- Minimum L3 (blacklist hook)
- Target L4 (whitelist hook)
- Optional L5 (no bash)
- Damage-control rules
- Verifier agent (read-only checks)

### Phase 5: Testing & Evaluation (1-2 hours)

- Create golden dataset (10+ test cases)
- Run pass@k evaluation (k=3, k=5)
- Calculate cost per task
- Identify optimization opportunities

### Phase 6: Deployment (1-2 hours)

- CI/CD pipeline with golden dataset gate
- Shadow deployment vs baseline
- Observability: every tool call + cost logged
- Monitoring dashboard

### Phase 7: Review & Iterate (1 hour)

- Red-team your own system (try to break it)
- Document lessons learned
- Create retrospective

---

## Deliverables Checklist

```
[ ] Architecture document (one page, with diagram)
[ ] All agent source code with system prompts
[ ] Agent configuration files (YAML/JSON)
[ ] Tool definitions with reasoning parameters
[ ] Security hooks configuration (L3+)
[ ] Golden dataset (10+ test cases with expected outputs)
[ ] Test script showing all agents respond correctly
[ ] Cost analysis per session
[ ] README with setup instructions
[ ] Self-assessment against rubric

---

## Production Readiness Checklist

Before submitting, verify your system meets these production standards:

### Reliability
- [ ] All tools include error handling (FileNotFoundError, API timeout, etc.)
- [ ] MAX_ITERATIONS set on every agent loop (no infinite loops)
- [ ] Retry logic with exponential backoff for API calls
- [ ] Agent gracefully handles tool failures instead of crashing

### Security
- [ ] Minimum L3 blacklist hook installed on bash access
- [ ] Domain locking prevents agents from accessing files outside their scope
- [ ] Verifier agent has read-only tools only (no edit/bash)
- [ ] Damage-control rules prevent rm -rf, DROP TABLE, etc.

### Observability
- [ ] Every tool call logged to file (JSONL format)
- [ ] Cost tracked per session
- [ ] Agent loop depth monitored
- [ ] Errors logged with stack traces

### Economics
- [ ] Cost estimate per session documented
- [ ] Cascade routing implemented where appropriate
- [ ] Iteration limits set to minimize token waste
- [ ] Model selection justified (why this model for this agent?)

---

## Graduation Criteria

Your capstone passes if it meets ALL of these:

1. **Three or more agents** that communicate and coordinate
2. **Security at L3+** (blacklist or whitelist hook)
3. **Working test script** that demonstrates all agents responding
4. **Documented cost estimate** per session
5. **Observability** — at least one trace log from a complete run
6. **Self-assessment** against the rubric with honest notes on what you'd improve

The capstone is not about perfection. It's about demonstrating you understand the full stack — from tool design to security to economics. A passing capstone with honest self-assessment is worth more than a perfect one with no reflection.
[ ] Mental model files for each agent
[ ] Security configuration (damage-control rules)
[ ] Verifier agent (if applicable)
[ ] Golden dataset (10+ test cases)
[ ] Test results (pass@k at k=1, k=3, k=5)
[ ] Cost analysis ($/task, optimization opportunities)
[ ] Security audit (which L-level, what gaps remain)
[ ] Retrospective (max 1 page)
```

---

## Pass Criteria

| Criterion | Minimum | Target |
|-----------|---------|--------|
| System runs without manual intervention | ✓ | ✓ |
| All agents have domain-locked permissions | ✓ | ✓ |
| Each agent has mental model file | ✓ | ✓ |
| pass@k (k=3) on golden dataset | >60% | >80% |
| Cost analysis within 2x of optimal | ✓ | ✓ |
| Security audit identifies ≥2 improvements | ✓ | ✓ |
| Observability captures all tool calls | ✓ | ✓ |
| Architecture document submitted | ✓ | ✓ |

---

## Grading Rubric

| Area | Weight | Poor (0) | Good (1) | Excellent (2) |
|------|--------|---------|----------|---------------|
| Architecture | 20% | No diagram, unclear design | Diagram present, mostly clear | Clear diagram, justified choices |
| Implementation | 25% | Agents don't work | Agents work on happy path | Agents handle errors gracefully |
| Security | 20% | L1 only | L3+ with damage-control | L4+ with verifier |
| Testing | 15% | No eval | pass@k computed | pass@k + cost analysis + grind detection |
| Documentation | 10% | Minimal | Architecture + setup | Architecture + setup + retrospective |
| Cost Optimization | 10% | Single model | Cascade routing | Cascade + verified savings |