231 lines
7.9 KiB
Markdown
231 lines
7.9 KiB
Markdown
# Non-Technical Track: Decision Frameworks for AI Agents
|
||
|
||
## For Managers, PMs, and Decision-Makers
|
||
|
||
---
|
||
|
||
## Framework 1: "Should We Use an Agent for This?"
|
||
|
||
### The 4-Question Filter (for non-technical stakeholders)
|
||
|
||
**Q1: Can a human do this task with clear step-by-step instructions?**
|
||
- Yes → Could be agentizable
|
||
- No → Humans struggle too; don't agentize
|
||
|
||
**Q2: Does the task have clear success criteria?**
|
||
- Yes → We'll know if the agent succeeded
|
||
- No → You can't evaluate an agent if you can't define "done"
|
||
|
||
**Q3: What's the cost of failure?**
|
||
- Low (wrong suggestion, formatting error) → Full autonomy possible
|
||
- Medium (wrong code, wrong data) → Human review required
|
||
- High (financial loss, safety, legal) → Human-in-the-loop mandatory
|
||
|
||
**Q4: How many times does this task need to be done?**
|
||
- Once → Just do it manually
|
||
- 10x → Build a script
|
||
- 100x → Build an agent
|
||
- 1000x → Build a multi-agent system
|
||
|
||
### Decision Matrix
|
||
|
||
```
|
||
Low Failure Cost High Failure Cost
|
||
──────────────┼─────────────────────────────────────
|
||
High Volume │ Build agent │ Agent + HITL
|
||
Low Volume │ Do it manually │ Do it manually
|
||
```
|
||
|
||
---
|
||
|
||
## Framework 2: The 4 Agent Design Patterns (Andrew Ng)
|
||
|
||
### 1. Reflection
|
||
The agent checks its own work before presenting it.
|
||
|
||
```
|
||
Agent writes code → reviews own code → fixes issues → presents final
|
||
```
|
||
|
||
**Business value**: Reduces human review time by 40-60%. Catches obvious errors before they reach you.
|
||
|
||
### 2. Tool Use
|
||
The agent uses external tools to gather information or take actions.
|
||
|
||
```
|
||
Agent reason → call search API → read results → call database → compose answer
|
||
```
|
||
|
||
**Business value**: Agents can access real-time data, not just training data. Turns a static LLM into a live system.
|
||
|
||
### 3. Planning
|
||
The agent decomposes a complex request into sub-steps.
|
||
|
||
```
|
||
"Build a landing page" → plan steps (1. Design, 2. Code, 3. Test, 4. Deploy)
|
||
→ execute each step → verify results
|
||
```
|
||
|
||
**Business value**: Complex tasks get done without step-by-step human instruction. The agent figures out the steps.
|
||
|
||
### 4. Multi-Agent Collaboration
|
||
Multiple specialized agents work together on different parts of a task.
|
||
|
||
```
|
||
PM agent plans → Designer creates mockup → Frontend builds → QA tests
|
||
```
|
||
|
||
**Business value**: Each agent is an expert in its domain. Together they outperform one generalist agent at a fraction of the cost.
|
||
|
||
---
|
||
|
||
## Framework 3: Cost Estimation for Agent Projects
|
||
|
||
### The 3x Rule
|
||
|
||
A production agent costs 3x your prototype estimate. Plan accordingly.
|
||
|
||
| Phase | Cost Multiplier | What's Included |
|
||
|-------|----------------|-----------------|
|
||
| Prototype (works on happy path) | 1x | Single agent, one model, no safety |
|
||
| Production (handles edge cases) | 2x | Retry logic, error handling, testing |
|
||
| Production+ (operational) | 3x | Monitoring, security, CI/CD, budgets |
|
||
|
||
### Quick Cost Calculator
|
||
|
||
```
|
||
Per-task cost = (input_tokens × input_price + output_tokens × output_price) × retry_rate × overhead
|
||
|
||
Where:
|
||
retry_rate = 1.5 (typical for agent systems)
|
||
overhead = 3.0 (production multiplier)
|
||
|
||
Example:
|
||
Agent session: 4000 input tokens, 1000 output tokens
|
||
Model: Claude Sonnet ($3/$15 per M tokens)
|
||
|
||
Base cost: (4000 × $3/M + 1000 × $15/M) = $0.012 + $0.015 = $0.027
|
||
With retries: $0.027 × 1.5 = $0.041
|
||
Production cost: $0.041 × 3.0 = $0.123 per task
|
||
|
||
At 1000 tasks/month: $123/month
|
||
At 10000 tasks/month: $1230/month
|
||
```
|
||
|
||
### Cost by Autonomy Level
|
||
|
||
| Level | Cost/Task | Human Time/Task | Best For |
|
||
|-------|-----------|----------------|----------|
|
||
| No agent | $0 | 30 min | One-off tasks |
|
||
| AI-assisted | $0.01-0.05 | 10 min | Human does the work, AI helps |
|
||
| Agent with review | $0.05-0.30 | 5 min | Agent works, human reviews |
|
||
| Full autonomy | $0.10-1.00 | 0 min | Agent works, human audits |
|
||
| Multi-agent | $0.50-5.00 | 2 min | Complex workflows, human supervises |
|
||
|
||
---
|
||
|
||
## Framework 4: Autonomy vs Risk Matrix
|
||
|
||
```
|
||
Low Risk Medium Risk High Risk
|
||
(suggestions, (code gen, data (financial, medical,
|
||
content gen) processing) infrastructure)
|
||
─────────────┼─────────────────────────────────────────────────────────────
|
||
Tier 1: │ Full autonomy │ Agent + review │ Human only
|
||
Simple │ │ │
|
||
│ │ │
|
||
Tier 2: │ Agent + review │ Agent + verifier + │ Human + agent
|
||
Complex │ │ human spot-check │ as tool
|
||
│ │ │
|
||
Tier 3: │ Multi-agent + audit │ Multi-agent + │ Not recommended
|
||
Strategic │ │ verifier + human │ for agents
|
||
```
|
||
|
||
---
|
||
|
||
## Framework 5: The "Build vs Buy" Decision
|
||
|
||
### Build an Agent When:
|
||
- Your task is unique to your business
|
||
- You need to keep data in-house
|
||
- No existing tool solves the problem
|
||
- You have the engineering capacity
|
||
|
||
### Buy/Rent When:
|
||
- It's a common task (customer support, code review, monitoring)
|
||
- A SaaS product already exists
|
||
- You don't have AI engineering expertise
|
||
- Speed to market matters more than customization
|
||
|
||
### Hybrid When:
|
||
- Core task is unique, but supporting tasks are common
|
||
- You can build a small agent that calls existing SaaS tools
|
||
|
||
---
|
||
|
||
## Framework 6: Measuring Success
|
||
|
||
### Leading Indicators (week 1-4)
|
||
- Task completion rate (is the agent finishing its work?)
|
||
- Human review time reduction
|
||
- Cost per task vs cost of human doing it
|
||
- Error rate (how often does the agent need correction?)
|
||
|
||
### Lagging Indicators (month 2-3+)
|
||
- Time saved per week
|
||
- Quality compared to human baseline
|
||
- Number of tasks automated vs manual
|
||
- Maintenance burden (how often does the agent break?)
|
||
|
||
### Red Flags
|
||
- Agent costs more than the human it replaces
|
||
- Agent requires more oversight than doing it yourself
|
||
- Agent breaks every time the model updates
|
||
- Team doesn't trust the agent's output
|
||
|
||
---
|
||
|
||
## Framework 7: Governance & Compliance Quick Reference
|
||
|
||
### Regulatory Requirements by Region
|
||
|
||
| Requirement | EU (AI Act) | California (SB 1047) | UK/Japan |
|
||
|-------------|-------------|---------------------|----------|
|
||
| Human oversight | Mandatory for high-risk | Mandatory kill-switch | Voluntary |
|
||
| Audit trails | Required | Required | Recommended |
|
||
| Incident reporting | Required | Required | Voluntary |
|
||
| Disclosure (AI-generated) | Required | Required | Recommended |
|
||
| Data privacy (GDPR) | Full compliance | Similar | Different standard |
|
||
|
||
### Minimum Viable Governance
|
||
|
||
For teams not in regulated industries, start with:
|
||
1. **Every agent action is logged** with timestamp, agent, input, output
|
||
2. **Human review gate** for destructive operations (deletes, writes, spending money)
|
||
3. **Monthly cost report** per agent per task type
|
||
4. **Quarterly security review** — has the agent's behavior changed?
|
||
|
||
---
|
||
|
||
## Summary: Decision Flow
|
||
|
||
```
|
||
1. Can this task be automated?
|
||
├── No → Do it manually
|
||
└── Yes →
|
||
2. What's the failure cost?
|
||
├── High → Human-in-the-loop mandatory
|
||
└── Low →
|
||
3. What's the volume?
|
||
├── <10/month → Do it manually
|
||
├── 10-100/month → Build a script
|
||
└── >100/month → Build an agent
|
||
4. How many specialists needed?
|
||
├── One → Single agent
|
||
└── Multiple → Multi-agent system
|
||
5. What's the budget?
|
||
├── <$50/month → Use cheapest model (Gemini Flash)
|
||
├── $50-500/month → Cascade routing (mix models)
|
||
└── >$500/month → Multi-agent with verification
|
||
```
|