142 lines
4.6 KiB
Markdown
142 lines
4.6 KiB
Markdown
# Context Window Management for AI Agents
|
|
|
|
**June 18, 2026**
|
|
|
|
Your agent's context window is its working memory. Fill it with the wrong things, and the agent makes bad decisions. Fill it with too much, and you're burning $0.15 per call on irrelevant tokens.
|
|
|
|
---
|
|
|
|
## The Context Budget
|
|
|
|
Every token in the context window has a cost — literally (API pricing) and figuratively (attention dilution). The key insight: **not all tokens are equal**.
|
|
|
|
```
|
|
High-Value Tokens (always include):
|
|
├── System prompt (persona, rules, constraints)
|
|
├── Tool definitions (name, description, schema)
|
|
├── Current user request
|
|
└── Most recent tool results
|
|
|
|
Medium-Value Tokens (include if relevant):
|
|
├── Conversation history (last 3-5 exchanges)
|
|
├── Mental model files (agent-specific expertise)
|
|
└── Reference documents (API docs, style guides)
|
|
|
|
Low-Value Tokens (exclude in production):
|
|
├── Entire conversation history (use summaries instead)
|
|
├── Large reference files (link instead of inline)
|
|
├── Previous tool results that are no longer relevant
|
|
└── System messages older than the last 10 turns
|
|
```
|
|
|
|
---
|
|
|
|
## Sliding Window Strategy
|
|
|
|
The most common production approach. Keep the window focused on recent + important:
|
|
|
|
```python
|
|
class SlidingWindow:
|
|
def __init__(self, max_tokens=32000):
|
|
self.max_tokens = max_tokens
|
|
self.system_prompt = "" # always retained
|
|
self.messages = [] # conversation history
|
|
self.token_count = 0
|
|
|
|
def add_message(self, msg):
|
|
self.messages.append(msg)
|
|
self.token_count += count_tokens(msg)
|
|
|
|
# Trim to fit budget — remove oldest tool results first
|
|
while self.token_count > self.max_tokens and len(self.messages) > 3:
|
|
removed = self.messages.pop(1) # keep system + last user
|
|
self.token_count -= count_tokens(removed)
|
|
```
|
|
|
|
**Best for**: Chat-style agents, interactive coding assistants, support bots.
|
|
|
|
**Limitation**: Loses context from early in the conversation. If the user mentions something important 20 turns ago, the agent won't remember it.
|
|
|
|
---
|
|
|
|
## Summarization Strategy
|
|
|
|
Periodically summarize the conversation into condensed context:
|
|
|
|
```python
|
|
def summarize_context(messages):
|
|
prompt = f"Summarize this conversation in 3-5 sentences, \
|
|
preserving key decisions and user preferences: \
|
|
{messages[-20:]}"
|
|
summary = llm.call(prompt)
|
|
return {"role": "system", "content": f"[CONTEXT: {summary}]"}
|
|
```
|
|
|
|
**Pattern**:
|
|
```
|
|
Turns 1-10: full messages
|
|
Turn 11: summarize turns 1-10 → insert as system message
|
|
Turns 11-20: full messages (with summary in system prompt)
|
|
Turn 21: summarize turns 11-20 → update system summary
|
|
... repeat
|
|
```
|
|
|
|
**Best for**: Long-running research agents, complex multi-step tasks, customer support.
|
|
|
|
**Cost**: Each summarization costs ~50-100 tokens. That's $0.0003-0.0015 per summary — essentially free.
|
|
|
|
---
|
|
|
|
## Mental Model Strategy (Advanced)
|
|
|
|
Give the agent its own persistent memory that it reads and writes:
|
|
|
|
```python
|
|
# Agent reads this file at the start of every session
|
|
MENTAL_MODEL = "agent-expertise.md"
|
|
|
|
def load_mental_model():
|
|
if os.path.exists(MENTAL_MODEL):
|
|
return open(MENTAL_MODEL).read()
|
|
return ""
|
|
|
|
def save_observation(key, value):
|
|
with open(MENTAL_MODEL, "a") as f:
|
|
f.write(f"\n- {key}: {value}")
|
|
```
|
|
|
|
The agent builds expertise over time. This is the most token-efficient strategy because the agent curates what it remembers — it doesn't keep everything.
|
|
|
|
**Best for**: Specialized agents (code reviewers, security auditors, data analysts) that work on multiple disjoint tasks.
|
|
|
|
---
|
|
|
|
## Token Budget Allocation Formula
|
|
|
|
```
|
|
Budget = SystemPrompt(15%) + Tools(15%) + Conversation(40%) + ToolResults(30%)
|
|
|
|
If budget is tight:
|
|
1. Shorten tool descriptions (remove examples)
|
|
2. Summarize conversation history (keep last 3-5 full, summarize the rest)
|
|
3. Trim tool results to only the relevant sections
|
|
4. Remind the agent about mental models instead of re-reading them
|
|
```
|
|
|
|
---
|
|
|
|
## The 80/20 Rule of Context Management
|
|
|
|
80% of context problems come from 20% of the causes:
|
|
|
|
1. **Too much conversation history** — keep last 5 exchanges, summarize the rest
|
|
2. **Too many tool results** — only keep results that were directly used
|
|
3. **Redundant system instructions** — don't repeat the same rules in every system message
|
|
4. **Large reference files** — cite them, don't include them inline
|
|
|
|
Fix these four, and you solve most context window issues.
|
|
|
|
---
|
|
|
|
*This is adapted from Module 2: Architecture of the [Agentic Engineering the Hard Way](/free-preview) course. Full course includes 65 lessons, 13 labs, and 20 skill kits.*
|