87 lines
2.3 KiB
Markdown
87 lines
2.3 KiB
Markdown
# Observability Kit
|
|
|
|
See what your agents are actually doing. Every tool call, every LLM completion, every cost.
|
|
|
|
## Included Skills
|
|
|
|
1. **Tool Call Tracer** — Log every tool call to SQLite
|
|
2. **Cost Tracker** — Per-session and per-agent costs in real-time
|
|
3. **Session Replay** — Replay agent sessions for debugging
|
|
4. **Loop Detector** — Alert when same tool called 5+ times
|
|
5. **Observability Dashboard** — Vue dashboard for live activity
|
|
|
|
## Data Model
|
|
|
|
```sql
|
|
-- Every tool call
|
|
CREATE TABLE tool_calls (
|
|
id INTEGER PRIMARY KEY,
|
|
session_id TEXT NOT NULL,
|
|
turn_number INTEGER NOT NULL,
|
|
tool_name TEXT NOT NULL,
|
|
tool_params TEXT, -- JSON
|
|
tool_result TEXT, -- truncated to 500 chars
|
|
duration_ms INTEGER,
|
|
timestamp TEXT NOT NULL
|
|
);
|
|
|
|
-- Every LLM completion
|
|
CREATE TABLE llm_completions (
|
|
id INTEGER PRIMARY KEY,
|
|
session_id TEXT NOT NULL,
|
|
turn_number INTEGER NOT NULL,
|
|
input_tokens INTEGER,
|
|
output_tokens INTEGER,
|
|
cached_tokens INTEGER,
|
|
cost_cents REAL,
|
|
model TEXT,
|
|
timestamp TEXT NOT NULL
|
|
);
|
|
|
|
-- Session summary
|
|
CREATE TABLE sessions (
|
|
session_id TEXT PRIMARY KEY,
|
|
start_time TEXT NOT NULL,
|
|
end_time TEXT,
|
|
total_tool_calls INTEGER DEFAULT 0,
|
|
total_tokens INTEGER DEFAULT 0,
|
|
total_cost_cents REAL DEFAULT 0,
|
|
status TEXT DEFAULT 'active'
|
|
);
|
|
```
|
|
|
|
## Key Metrics
|
|
|
|
| Metric | Warning | Critical |
|
|
|--------|---------|----------|
|
|
| Tool calls per session | >20 | >50 |
|
|
| Cost per session | >$0.50 | >$2.00 |
|
|
| Loop depth | >15 | >30 |
|
|
| Same tool >5x in row | Investigate | Kill session |
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from observability import ObservabilityTracker
|
|
|
|
tracker = ObservabilityTracker()
|
|
tracker.log_tool_call(turn=1, tool_name="read_file", params={"path": "test.txt"}, result="...", duration_ms=150)
|
|
tracker.log_llm_completion(turn=1, input_tokens=500, output_tokens=200, cost_cents=0.003)
|
|
summary = tracker.get_session_summary()
|
|
print(f"Cost: ${summary['total_cost_cents']/100:.4f}")
|
|
```
|
|
|
|
## Dashboard
|
|
|
|
The Vue dashboard shows:
|
|
- Live agent activity feed
|
|
- Per-session cost breakdown
|
|
- Tool call frequency chart
|
|
- Loop detection alerts
|
|
- Session replay controls
|
|
|
|
## Related Course Material
|
|
|
|
- M5 Production Patterns — Observability lesson + lab
|
|
- `claude-code-hooks-multi-agent-observability/` — full observability system
|