264 lines
8.8 KiB
Markdown
264 lines
8.8 KiB
Markdown
# Module 6: Economics & Evaluation
|
||
|
||
## Lesson 6.0: The Compute Advantage Equation
|
||
|
||
**Core insight**: Your value as an agentic engineer scales with the amount of compute you can harness effectively.
|
||
|
||
### Tokenomics: The 3 Levels
|
||
|
||
Before we get into the math, understand the **tokenomics** framework. There are three levels to using tokens effectively:
|
||
|
||
**Level 1: Use more tokens** — Scale agents to do more work. This is the easy part. Anyone can spin up more agents and burn more tokens.
|
||
|
||
**Level 2: Make tokens useful** — Not all tokens are equal. A token spent on a hallucinated tool call is wasted. A token spent on verifying a critical production change is invaluable. The verifier pattern (M3), quality gates (M5), and golden datasets (M6) are how you make tokens useful.
|
||
|
||
**Level 3: Capture revenue from valuable tokens** — This is the arbitrage: buy a token for a dollar, run it through your business, sell the output for two — then scale it. A rising API bill becomes a productivity KPI.
|
||
|
||
This course teaches all three levels. Most engineers never get past Level 1.
|
||
|
||
### The Equation
|
||
|
||
```
|
||
Compute Advantage = (Compute Scaling × Autonomy) ÷ (Time + Effort + Monetary Cost)
|
||
```
|
||
|
||
| Variable | What It Means | How to Improve It |
|
||
|----------|---------------|-------------------|
|
||
| **Compute Scaling** | How much AI compute you can throw at problems | More agents, better models, larger context windows |
|
||
| **Autonomy** | How much the agent does without your intervention | Better prompts, better tools, verification layers |
|
||
| **Time** | How long it takes to get results | Faster models, parallel execution, fewer iterations |
|
||
| **Effort** | How much you have to craft prompts/instructions | Reusable skills, templates, mental models |
|
||
| **Monetary Cost** | What you pay for API calls | Cascade routing, cheaper models for simple steps |
|
||
|
||
### What It Tells You
|
||
|
||
- **High Compute Advantage** = You get more output for less input. You're leveraging agents effectively.
|
||
- **Low Compute Advantage** = You're spending too much time/cost for too little gain. Fix the bottleneck variable.
|
||
|
||
### Real-World Application (Your Stack)
|
||
|
||
| Tool | Compute Scaling | Autonomy | Time | Effort | Cost | Advantage |
|
||
|------|:---:|:---:|:---:|:---:|:---:|:---:|
|
||
| Claude Code (lead) | 8 | 7 | 6 | 5 | 4 | (8×7)÷(6+5+4)=3.7 |
|
||
| Pi Agent (custom) | 7 | 8 | 7 | 6 | 6 | (7×8)÷(7+6+6)=2.9 |
|
||
| OpenCode (OSS) | 5 | 5 | 7 | 7 | 9 | (5×5)÷(7+7+9)=1.1 |
|
||
| Gemini Flash (fast) | 4 | 3 | 9 | 8 | 9 | (4×3)÷(9+8+9)=0.5 |
|
||
|
||
Higher score = more output per unit of investment. Use this to decide which tool for which task.
|
||
|
||
### How to Optimize
|
||
|
||
1. **Improve numerator**: Run more agents in parallel (P-threads), increase autonomy with verification
|
||
2. **Reduce denominator**: Use cascade routing (cheap model for simple steps), reuse skills/mental models
|
||
3. **Track over time**: Your Compute Advantage should increase as you build better harnesses and mental models
|
||
|
||
---
|
||
|
||
## Lesson 6.1: LLM Pricing Landscape 2026
|
||
|
||
### Per-Million-Token Pricing
|
||
|
||
| Model | Input ($/M) | Output ($/M) | Best For |
|
||
|-------|-------------|--------------|----------|
|
||
| Gemini 2.5 Flash | $0.15 | $0.60 | High-volume, simple tool calls |
|
||
| DeepSeek V3 | $0.27 | $1.10 | Structured tasks, batch processing |
|
||
| DeepSeek R1 | $0.55 | $2.19 | Reasoning-heavy single steps |
|
||
| Gemini 2.5 Pro | $1.25 | $5.00 | Long-context sessions (1M+ tokens) |
|
||
| GPT-4o | $2.50 | $10.00 | Balanced cost/quality |
|
||
| Claude Sonnet 4 | $3.00 | $15.00 | General agentic reasoning |
|
||
| GPT-5 | $10.00 | $40.00 | Frontier research, complex plans |
|
||
| Claude Opus 4 | $15.00 | $75.00 | Complex multi-step agent orchestration |
|
||
|
||
### Price Range: 100x Difference
|
||
|
||
Cheapest (Gemini Flash) to most expensive (Claude Opus) is a **100x multiplier**. Choosing the right model for each step is your highest-leverage cost optimization.
|
||
|
||
---
|
||
|
||
## Lesson 6.2: Cascade Routing
|
||
|
||
### The Pattern
|
||
|
||
Don't use one model for everything. Route different steps to different models:
|
||
|
||
```
|
||
Step 1: Retrieve context → Gemini Flash ($0.15/M input)
|
||
Step 2: Analyze → Claude Sonnet ($3/$15)
|
||
Step 3: Make decision → Claude Opus ($15/$75)
|
||
Step 4: Format output → Gemini Flash ($0.15/$0.60)
|
||
```
|
||
|
||
### Savings Profile
|
||
|
||
| Pattern | Cost/Task | Savings |
|
||
|---------|-----------|---------|
|
||
| All Opus | $2.50 | Baseline |
|
||
| Cascade (Flash → Sonnet → Opus → Flash) | $0.85 | 66% savings |
|
||
| All Sonnet | $0.50 | 80% savings (but quality loss on complex steps) |
|
||
|
||
### Implementation
|
||
|
||
```python
|
||
def route_task(task_complexity: str) -> str:
|
||
if task_complexity == "retrieval":
|
||
return "gemini-2.5-flash"
|
||
elif task_complexity == "analysis":
|
||
return "claude-sonnet-4"
|
||
elif task_complexity == "decision":
|
||
return "claude-opus-4"
|
||
elif task_complexity == "formatting":
|
||
return "gemini-2.5-flash"
|
||
```
|
||
|
||
---
|
||
|
||
## Lesson 6.3: Cost Per Session Math
|
||
|
||
### Where Costs Come From
|
||
|
||
| Component | Share | Notes |
|
||
|-----------|-------|-------|
|
||
| Output tokens | ~70% | Model generation is most expensive |
|
||
| Input tokens | ~20% | Context + tool results |
|
||
| Cached tokens | ~10% | Can be zero if not configured |
|
||
|
||
### The Multipliers
|
||
|
||
| Factor | Multiplier | Why |
|
||
|--------|-----------|-----|
|
||
| Retry rate | 1.2-2.0x | Failed tool calls retry |
|
||
| Tool overhead | 3-5x per tool call | Planning + execution + error recovery + result parsing |
|
||
| Context growth | 1.5x per 10 turns | Every turn adds tokens to context |
|
||
|
||
### The 3x Rule
|
||
|
||
**Production agent costs 3x your prototype estimate.**
|
||
|
||
1x = ideal path (everything works first time)
|
||
2x = retries + edge cases
|
||
3x = monitoring + error handling + observability overhead
|
||
|
||
### Quick Estimation
|
||
|
||
```python
|
||
def estimate_cost(turns: int, avg_tokens_per_turn: int, model_price_per_m: float):
|
||
"""Very rough estimate."""
|
||
base = turns * avg_tokens_per_turn * model_price_per_m / 1_000_000
|
||
retry = base * 1.5
|
||
overhead = base * 3.0
|
||
return {"base": base, "with_retries": retry, "production": overhead}
|
||
```
|
||
|
||
---
|
||
|
||
## Lesson 6.4: Agent Evaluation Metrics
|
||
|
||
### The Key Metrics
|
||
|
||
| Metric | What It Measures | Target |
|
||
|--------|-----------------|--------|
|
||
| **pass@k** | % of k attempts where at least one succeeds | >80% |
|
||
| **pass^k** | % where ALL k attempts succeed (consistency) | >60% |
|
||
| **Tool Call Accuracy** | Correct tool + correct params | >90% |
|
||
| **Task Completion Rate** | End-to-end success | >70% |
|
||
| **Cost Per Task** | Total API cost per unit | Varies |
|
||
| **Loop Efficiency** | Steps taken vs optimal | <2x overhead |
|
||
| **Grind Rate** | % of attempts that re-run identical code | <5% |
|
||
|
||
### pass@k Explained
|
||
|
||
```
|
||
pass@k = probability that at least one of k attempts succeeds
|
||
|
||
k=1: 60% pass rate (single attempt)
|
||
k=3: 1 - (0.4)^3 = 93.6% (best of 3)
|
||
k=5: 1 - (0.4)^5 = 98.9% (best of 5)
|
||
```
|
||
|
||
Higher k = higher reliability but higher cost. The trade-off is the core optimization problem.
|
||
|
||
---
|
||
|
||
## Lesson 6.5: Automated Evaluation
|
||
|
||
### Golden Dataset Approach
|
||
|
||
```python
|
||
eval_cases = [
|
||
{
|
||
"input": "Find the user with email john@example.com",
|
||
"expected_tool": "query_database",
|
||
"expected_params": {"query": "SELECT * FROM users WHERE email = 'john@example.com'"},
|
||
"expected_output_contains": ["john@example.com"],
|
||
"weight": 1.0
|
||
}
|
||
]
|
||
```
|
||
|
||
### VCR-Style Recording
|
||
|
||
For non-deterministic tool results (search, API calls), record the response once, then replay it deterministically:
|
||
|
||
```python
|
||
# Record mode: capture real responses
|
||
# Replay mode: use recorded responses
|
||
# Test: verify agent makes correct decisions with known data
|
||
```
|
||
|
||
---
|
||
|
||
## Lesson 6.6: A/B Testing Agents
|
||
|
||
### Canary Deployment
|
||
|
||
1. Route 5% of traffic to new agent config
|
||
2. Compare against 95% on current config
|
||
3. Metrics: success rate, cost, latency, loop depth
|
||
4. If new config wins on all metrics → roll out to 100%
|
||
5. If new config loses → rollback, investigate
|
||
|
||
### What to A/B Test
|
||
|
||
- System prompt wording
|
||
- Model selection
|
||
- Temperature settings
|
||
- Tool descriptions
|
||
- Iteration limits
|
||
|
||
---
|
||
|
||
## Lesson 6.7: Human Evaluation
|
||
|
||
### What Automated Evals Miss
|
||
|
||
1. **Quality of reasoning** — agent made right decision for wrong reasons?
|
||
2. **Tone and style** — output technically correct but poorly written?
|
||
3. **Edge cases** — agent handled happy path but not real-world variation?
|
||
4. **Hallucination cascades** — plausible-looking but wrong intermediate steps?
|
||
|
||
### Spot-Check Sampling
|
||
|
||
Review 5-10% of all agent sessions manually. Focus on:
|
||
- Sessions with high cost (>2x average)
|
||
- Sessions with tool loops
|
||
- Sessions where automated eval scored low
|
||
- Random sample for baseline
|
||
|
||
---
|
||
|
||
## Lab 6.8: Build an Eval Harness
|
||
|
||
**Objective**: Create golden Q&A pairs + automated pass/fail scoring.
|
||
|
||
**Starter**: `course/labs/L6-eval-harness/starter.py`
|
||
|
||
## Lab 6.9: Cost Optimization
|
||
|
||
**Objective**: Profile a session, identify savings, implement cascade routing.
|
||
|
||
**Starter**: `course/labs/L6-cost-optimization/starter.py`
|
||
|
||
---
|
||
|
||
**Next**: [Module 7: Advanced Topics](M7-ADVANCED.md)
|