19 KiB
19 KiB
Agentic Engineering: From Foundations to Production
Course Overview
A comprehensive, hands-on course teaching you to build, deploy, and maintain production-grade AI agent systems. Based entirely on original research and battle-tested production systems — not framework wrappers.
Format: Self-paced with scaffolded labs, quizzes, and 3 capstone projects
Prerequisites: Basic Python/TypeScript, CLI comfort, familiarity with LLM APIs
Estimated time: 52-72 hours
Difficulty: Intermediate to Advanced
Module Map
Module 1: Foundations ───────────────────┐
│
Module 2: Architecture ──────────────────┤
│
Module 3: Safety & Security ─────────────┤
(Agent Harness) │
├──► Module 8: Capstone
Module 4: Multi-Agent Orchestration ─────┤ (build production-grade
(Software Factory) │ multi-agent system)
│
Module 5: Production Patterns ──────────┤
(Agentic Access + Extensible) │
│
Module 6: Economics & Evaluation ────────┤
(Tokenomics) │
│
Module 7: Advanced Topics ───────────────┘
(Always-On Agents)
Mapping to IndyDevDan's 5 Pillars
| His Pillar | Our Module | Key Lessons |
|---|---|---|
| Agent Harness | M1-M3 | Harness vs Model, Security, Verifier |
| Software Factory | M4 | Agent chains, teams, orchestration |
| Extensible Software | M2 + M5 | Tool design, Beyond MCP, CI/CD |
| Always-On Agents | M7 | Autoresearch, always-on, heartbeat |
| Agentic Access | M2 + M5 | Tool surface, deployment, API design |
| Tokenomics | M6 | 3-level framework, cascade routing |
Module 1: Foundations ───────────────────┐
│
Module 2: Agent Architecture ────────────┤
│
Module 3: Safety & Security ─────────────┤
├──► Module 8: Capstone
Module 4: Multi-Agent Orchestration ─────┤ (build production-grade
│ multi-agent system)
Module 5: Production Patterns ──────────┤
│
Module 6: Economics & Evaluation ────────┤
│
Module 7: Advanced Topics ───────────────┘
Module 1: Foundations (4-6 hours)
Goal: Understand what agents are, when to use them, and the mental models that separate agentic engineering from vibe coding.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 1.1 | What Makes an Agent? — LLM + Tools + Loop | Concept | 30 min |
| 1.2 | The Harness vs The Model — 5 subsystems + 4 dimensions of control | Concept | 30 min |
| 1.3 | Decision Framework: "Should I use an agent for this?" | Framework | 30 min |
| 1.4 | The Repository IS the Spec — single source of truth for agents | Concept | 20 min |
| 1.5 | The Agent Loop: Think → Act → Observe → Repeat | Concept | 25 min |
| 1.6 | Tool Calling Deep Dive: function schemas, forced calls, reasoning params | Technical | 45 min |
| 1.7 | Vibe Coding vs Agentic Engineering — the five hard rules | Concept | 20 min |
| 1.8 | Do You Trust Your Agents? — the central thesis of agentic engineering | Concept | 15 min |
| 1.9 | Lab: Your First Agent — single-tool agent from scratch | Lab | 60 min |
| 1.10 | Quiz: Module 1 Checkpoint | Assessment | 15 min |
Key sources: single-file-agents, mythos-learnings.md, agenticengineer.com (4 dimensions), walkinglabs.github.io (5 subsystems, repo IS the spec), Karpathy Sequoia Ascent 2026 (verifiability thesis, Software 3.0)
Module 2: Agent Architecture (6-8 hours)
Goal: Master the four components every agent system has — tools, loops, context, memory — and the architectural patterns that scale.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 2.1 | The Four Pillars + 5 Harness Subsystems | Concept | 25 min |
| 2.2 | Tool Design Patterns: single-purpose, composed, MCP, CLI, script | Technical | 40 min |
| 2.3 | Agent Loop Variants: simple → tool-use → task-completion → sub-agent | Technical | 45 min |
| 2.4 | Skills System Deep Dive: path-scoped, subdirectory rules, plugin distribution | Technical | 35 min |
| 2.5 | Agent-Readable Workspace: init phase, feature lists, clean state | Technical | 30 min |
| 2.6 | Context Window Management: sliding window, summarization, structured outputs | Technical | 40 min |
| 2.7 | Memory Patterns: mental models, expertise files, scratch pads, session logs | Technical | 35 min |
| 2.8 | The Reasoning Parameter — forced chain-of-thought on every tool call | Technical | 20 min |
| 2.9 | Codebase Architectures for Agents: atomic, layered, pipeline, vertical slice | Architecture | 45 min |
| 2.10 | Lab: Multi-Tool Agent — add search + file ops to your agent | Lab | 75 min |
| 2.11 | Lab: Context-Aware Agent — implement sliding window + summarization | Lab | 60 min |
| 2.12 | Quiz: Module 2 Checkpoint | Assessment | 15 min |
Key sources: single-file-agents, walkinglabs.github.io (init phase, feature lists, clean state), claudefa.st (skills, LSP MCP, plugin distribution)
Module 3: Safety & Security (5-7 hours)
Goal: Secure your agents against the five levels of bash risk, implement defense-in-depth, and build verifier agents that catch failures before they hit production.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 3.1 | Beyond Bash: Prompt Injection + Why Bash Is the Single Point of Failure | Concept | 25 min |
| 3.2 | The 5-Level Security Ladder: L0 ACIP → L5 no-bash | Technical | 45 min |
| 3.3 | The L3 Marque Break: agent writes cleanup.py → runs python cleanup.py | Technical | 30 min |
| 3.4 | Damage Control: three access levels (zero-access, read-only, no-delete) | Technical | 35 min |
| 3.5 | Hook Architecture: 13 lifecycle events for security enforcement | Technical | 40 min |
| 3.6 | The Verifier Pattern: builder + verifier over Unix socket, confidence ladder | Architecture | 45 min |
| 3.7 | Defense-in-Depth: stacking L1-L5, read-only verification, permission gating | Architecture | 25 min |
| 3.8 | Lab: Implement L4 Whitelist Hook — block all but 10 safelisted commands | Lab | 60 min |
| 3.9 | Lab: Build a Verifier Agent — read-only agent that checks the builder's work | Lab | 75 min |
| 3.10 | Quiz: Module 3 Checkpoint | Assessment | 15 min |
Key sources: bash-damage-from-within/ (all 5 levels + 6 attacks), the-verifier-agent/, pi-vs-claude-code damage-control, claude-code-hooks-mastery
Module 4: Multi-Agent Orchestration (7-9 hours)
Goal: Design and build multi-agent systems — dispatchers, pipelines, peer-to-peer, and full depth-3 delegation hierarchies.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 4.1 | Why One Agent Is Not Enough — specialization, context isolation, reliability | Concept | 20 min |
| 4.2 | Orchestration Patterns: dispatcher (hub-and-spoke) vs pipeline (sequential) vs P2P (flat) | Architecture | 40 min |
| 4.3 | P-Threads: Run N agents in parallel, pick the best result | Architecture | 25 min |
| 4.4 | F-Threads: Fusion — N agents produce, judge picks winner | Architecture | 25 min |
| 4.5 | Depth-2 Delegation: orchestrator → team leads → workers | Architecture | 35 min |
| 4.4 | Agent Experts That Remember: mental models, self-improve commands, expertise YAML | Technical | 40 min |
| 4.5 | Domain Locking: permissions per agent, read/write/delete scope | Technical | 25 min |
| 4.6 | TillDone Task Discipline: task list gating, live progress, nudge on incomplete | Technical | 25 min |
| 4.7 | Agent Chains: YAML-defined pipelines with $INPUT/$ORIGINAL flow | Technical | 35 min |
| 4.8 | Pi-to-Pi Communication: peer-to-peer agents over Unix sockets and HTTP/SSE | Technical | 45 min |
| 4.9 | Conversation Awareness: shared JSONL logs, cross-agent context | Technical | 20 min |
| 4.10 | The CEO Board System: 8 specialist agents, adversarial debate, verification | Architecture | 40 min |
| 4.11 | The UI Agents System: 12 agents across 4 teams, brand→product→tree→branch→leaf | Architecture | 35 min |
| 4.12 | Lab: Build an Agent Chain — plan→build→review pipeline in YAML | Lab | 60 min |
| 4.13 | Lab: Deploy a Multi-Team System — orchestrator + 2 teams + workers | Lab | 90 min |
| 4.14 | Quiz: Module 4 Checkpoint | Assessment | 15 min |
Key sources: lead-agents/, ui-agents/, ceo-agents/, pi-vs-claude-code (agent-team, agent-chain, coms), claude-code-hooks-mastery (subagents, teams)
Module 5: Production Patterns (5-7 hours)
Goal: Deploy agents that don't burn money, break silently, or require constant babysitting.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 5.1 | What Production Means for Agents — different from traditional software | Concept | 20 min |
| 5.2 | CI/CD for Agents: golden Q&A datasets, regression testing, prompt versioning | Technical | 40 min |
| 5.3 | Case Study: The 5-Tool Production Stack — Claude Code, Pi, OpenCode, Hermes, OpenClaw | Architecture | 35 min |
| 5.4 | The Agent Manager Role — owning the harness in enterprise deployments | Concept | 20 min |
| 5.5 | Shadow Deployments: run new agent alongside prod, compare decisions before switching | Technical | 30 min |
| 5.4 | Rollback Strategies: prompt version pinning, model freezing, canary rollback | Technical | 25 min |
| 5.5 | Observability & Monitoring: decision tracing, tool call logging, session replay | Technical | 45 min |
| 5.6 | Alerting on Agent-Specific Signals: token consumption, loop detection, cost spikes | Technical | 25 min |
| 5.7 | Deployment Modes: local trusted, authenticated, private network, internet-facing | Technical | 30 min |
| 5.8 | Cost Control: budgets per agent/session/task, hard stops, warning thresholds | Technical | 25 min |
| 5.9 | Lab: Set Up Agent Observability — trace every tool call + LLM completion | Lab | 60 min |
| 5.10 | Lab: CI/CD Pipeline — golden test suite + automated regression gate | Lab | 60 min |
| 5.11 | Quiz: Module 5 Checkpoint | Assessment | 15 min |
Key sources: paperclip/ (deployment modes, execution semantics, budget system), claude-code-hooks-multi-agent-observability/, aiproxy/benchmarks
Module 6: Economics & Evaluation (4-6 hours)
Goal: Understand model pricing, optimize costs, evaluate agent performance systematically.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 6.1 | The Compute Advantage Equation — measure your engineering leverage | Framework | 25 min |
| 6.2 | LLM Pricing Landscape 2026: Claude, GPT, Gemini, DeepSeek comparison | Reference | 30 min |
| 6.2 | Cascade Routing: cheap model for retrieval/simple steps, expensive for complex reasoning | Technical | 30 min |
| 6.3 | Cost Per Session Math: output tokens dominate (70%), retry multiplier (1.2-2x), the 3x rule | Technical | 30 min |
| 6.4 | Agent Evaluation Metrics: pass@k, tool call accuracy, task completion rate, cost per task | Technical | 35 min |
| 6.5 | Automated Evaluation: golden datasets, VCR-style replay, regression detection | Technical | 35 min |
| 6.6 | A/B Testing Agents: canary deployments, LangSmith experiments, traffic splitting | Technical | 30 min |
| 6.7 | Human Evaluation: spot-check sampling, red-teaming, what automated evals miss | Technical | 20 min |
| 6.8 | Lab: Build an Eval Harness — golden Q&A pairs + automated pass/fail scoring | Lab | 60 min |
| 6.9 | Lab: Cost Optimization — profile a session, identify savings, cascade-route it | Lab | 45 min |
| 6.10 | Quiz: Module 6 Checkpoint | Assessment | 15 min |
Key sources: aiproxy/ (provider benchmarks, test results), benchy/, agentic-coding-tool-eval/, mythos-learnings.md (grinding detection)
Module 7: Advanced Topics (5-7 hours)
Goal: Build self-improving agents, meta-agents that build agents, and understand the frontier.
| Lesson | Topic | Type | Est. Time |
|---|---|---|---|
| 7.1 | Autoresearch: agents that experiment, measure, and improve their own performance | Architecture | 40 min |
| 7.2 | The Experiment Loop: run → measure → log → decide (keep or discard) | Technical | 30 min |
| 7.3 | Integrity Guards: code hashing, grind detection, median-over-best measurement | Technical | 25 min |
| 7.4 | Meta-Agents: agents that build agents — parallel research experts, agent forge | Architecture | 35 min |
| 7.5 | Agent Frameworks Compared: paperclip (company OS), multica (team collab), cabinet (AI startup OS) | Architecture | 45 min |
| 7.6 | Beyond MCP: MCP vs CLI vs Script vs Skills — the context cost trade-off | Architecture | 30 min |
| 7.7 | The Mac Mini Agent: physical sandbox, steer (GUI) + drive (tmux) + listen (job server) | Architecture | 35 min |
| 7.8 | Always-On Agents: voice→CLI bridge, 24/7 job servers, heartbeat execution | Architecture | 25 min |
| 7.9 | Lab: Build an Autoresearch Loop — agent runs experiment, measures result, logs it | Lab | 75 min |
| 7.10 | Lab: Meta-Agent — agent that generates a new agent persona from documentation | Lab | 60 min |
| 7.11 | Quiz: Module 7 Checkpoint | Assessment | 15 min |
Key sources: mythos-learnings.md (integrity guards), brand-monitor/autoresearch, pi-vs-claude-code/pi-pi, beyond-mcp/, mac-mini-agent/, always-on-ai-assistant/
Module 8: Capstone — Production Multi-Agent System (8-12 hours)
Goal: Build, deploy, and evaluate a complete production-grade multi-agent system.
Capstone Project Options
| Project | Difficulty | Description |
|---|---|---|
| Brand Monitor | Intermediate | Multi-agent system that monitors how LLMs mention your brand across ChatGPT, Gemini, Grok, Perplexity. Includes scanner agents, analysis agent, reporting agent. |
| Code Review Pipeline | Intermediate-Advanced | Plan → Build → Review → Verify chain with CI/CD integration, GitHub PR comments, auto-fix suggestions |
| Strategic Decision Board | Advanced | 8-agent CEO board with adversarial debate, fact verification, execution planning, quarterly tracking |
Capstone Phases
| Phase | Deliverable | Est. Time |
|---|---|---|
| 8.1 | System Design: architecture diagram, agent roster, tool list, data model | 60-90 min |
| 8.2 | Agent Implementation: build each agent with system prompts, tools, mental models | 2-4 hours |
| 8.3 | Integration: wire agents together (chain, team, or P2P), shared context | 1-2 hours |
| 8.4 | Security Hardening: apply L3-L5 security, add damage-control, verifier | 1-2 hours |
| 8.5 | Testing & Evaluation: golden dataset, regression tests, cost analysis | 1-2 hours |
| 8.6 | Deployment: CI/CD pipeline, observability, monitoring, alerting | 1-2 hours |
| 8.7 | Review & Iterate: red-team your own system, fix gaps, document lessons | 1 hour |
Capstone Deliverables
- Architecture document (one page)
- Running agent system with source code
- Test suite with pass@k results
- Cost analysis ($ per task, optimization opportunities)
- Security audit (which L-level, what gaps remain)
- Retrospective (what broke, what surprised you)
Assessment Strategy
| Type | Frequency | Weight | Passing |
|---|---|---|---|
| Module quizzes (7) | End of each module | 35% | 80% |
| Lab submissions (12) | Integrated in modules | 35% | Pass/fail |
| Capstone project | End of course | 30% | Rubric-based |
Capstone Pass Criteria
- System runs without manual intervention
- All agents have domain-locked permissions
- Each agent has a mental model file
- Test suite achieves >70% pass@k on golden dataset
- Cost analysis within 2x of optimal estimate
- Security audit identifies at least 2 improvement areas
Module Dependency Graph
M1: Foundations ─────────────────────────────────────────────────
│
├──► M2: Architecture ──► M4: Orchestration ──► M8: Capstone
│ │ │
│ └──► M3: Safety ────┘
│
├──► M5: Production ───────────────────────────► M8: Capstone
│
└──► M6: Economics ────► M7: Advanced ──────────► M8: Capstone
Modules 1 and 2 are prerequisites for everything. Modules 3-7 can be taken in any order after M1+M2. Module 8 requires all previous modules.
Resource Map
Every lesson references specific files from the tac/ repository:
| Module | Primary Sources |
|---|---|
| M1 | single-file-agents/sfa_poc.py, mythos-learnings.md |
| M2 | single-file-agents/codebase-architectures/, 20 sfa_*.py files |
| M3 | bash-damage-from-within/, the-verifier-agent/, damage-control.ts, Jeff's DCG (1,055★), ACIP (330★), Karpathy verifiability thesis |
| M4 | lead-agents/, ui-agents/, ceo-agents/, pi-vs-claude-code/, Jeff's MCP Agent Mail (1,955★), CASS Memory (366★), Agent Farm (837★) |
| M5 | paperclip/doc/DEPLOYMENT-MODES.md, hooks-multi-agent-observability/, Jeff's Session Search (783★), NTM (319★), Vibe Cockpit (22★), Flywheel Setup (1,487★) |
| M6 | aiproxy/benchmarks, benchy/, agentic-coding-tool-eval/ |
| M7 | mythos-learnings.md, brand-monitor/autoresearch, beyond-mcp/ |
| M8 | brand-monitor/, the-verifier-agent/, ceo-agents/ |
| Reference | REFERENCE-STACK.md (production architecture), TOOL-REFERENCE.md (5-tool comparison) |
Estimated Totals
| Item | Count |
|---|---|
| Concept lessons | 28 |
| Technical lessons | 22 |
| Architecture lessons | 15 |
| Labs | 15 |
| Quizzes | 7 |
| Capstone projects | 3 options |
| Non-technical decision frameworks | 5 |
| Reference architecture documents | 4 (REFERENCE-STACK, TOOL-REFERENCE, COMPETITIVE-ANALYSIS, JEFF-INTEGRATION) |
| Total lessons | ~65 |
| Total estimated time | 56-78 hours |