agentic-ai-engineering/skills/kits/observability/SKILL.md

Observability Kit

See what your agents are actually doing. Every tool call, every LLM completion, every cost.

Included Skills

1. Tool Call Tracer — Log every tool call to SQLite
2. Cost Tracker — Per-session and per-agent costs in real-time
3. Session Replay — Replay agent sessions for debugging
4. Loop Detector — Alert when same tool called 5+ times
5. Observability Dashboard — Vue dashboard for live activity

Data Model

-- Every tool call
CREATE TABLE tool_calls (
    id INTEGER PRIMARY KEY,
    session_id TEXT NOT NULL,
    turn_number INTEGER NOT NULL,
    tool_name TEXT NOT NULL,
    tool_params TEXT,           -- JSON
    tool_result TEXT,           -- truncated to 500 chars
    duration_ms INTEGER,
    timestamp TEXT NOT NULL
);

-- Every LLM completion
CREATE TABLE llm_completions (
    id INTEGER PRIMARY KEY,
    session_id TEXT NOT NULL,
    turn_number INTEGER NOT NULL,
    input_tokens INTEGER,
    output_tokens INTEGER,
    cached_tokens INTEGER,
    cost_cents REAL,
    model TEXT,
    timestamp TEXT NOT NULL
);

-- Session summary
CREATE TABLE sessions (
    session_id TEXT PRIMARY KEY,
    start_time TEXT NOT NULL,
    end_time TEXT,
    total_tool_calls INTEGER DEFAULT 0,
    total_tokens INTEGER DEFAULT 0,
    total_cost_cents REAL DEFAULT 0,
    status TEXT DEFAULT 'active'
);

Key Metrics

Metric	Warning	Critical
Tool calls per session	>20	>50
Cost per session	>$0.50	>$2.00
Loop depth	>15	>30
Same tool >5x in row	Investigate	Kill session

Usage

from observability import ObservabilityTracker

tracker = ObservabilityTracker()
tracker.log_tool_call(turn=1, tool_name="read_file", params={"path": "test.txt"}, result="...", duration_ms=150)
tracker.log_llm_completion(turn=1, input_tokens=500, output_tokens=200, cost_cents=0.003)
summary = tracker.get_session_summary()
print(f"Cost: ${summary['total_cost_cents']/100:.4f}")

Dashboard

The Vue dashboard shows:

• Live agent activity feed
• Per-session cost breakdown
• Tool call frequency chart
• Loop detection alerts
• Session replay controls

Related Course Material

• M5 Production Patterns — Observability lesson + lab
• claude-code-hooks-multi-agent-observability/ — full observability system