4.6 KiB

Raw Permalink Blame History

Context Window Management for AI Agents

June 18, 2026

Your agent's context window is its working memory. Fill it with the wrong things, and the agent makes bad decisions. Fill it with too much, and you're burning $0.15 per call on irrelevant tokens.

The Context Budget

Every token in the context window has a cost — literally (API pricing) and figuratively (attention dilution). The key insight: not all tokens are equal.

High-Value Tokens (always include):
├── System prompt (persona, rules, constraints)
├── Tool definitions (name, description, schema)
├── Current user request
└── Most recent tool results

Medium-Value Tokens (include if relevant):
├── Conversation history (last 3-5 exchanges)
├── Mental model files (agent-specific expertise)
└── Reference documents (API docs, style guides)

Low-Value Tokens (exclude in production):
├── Entire conversation history (use summaries instead)
├── Large reference files (link instead of inline)
├── Previous tool results that are no longer relevant
└── System messages older than the last 10 turns

Sliding Window Strategy

The most common production approach. Keep the window focused on recent + important:

class SlidingWindow:
    def __init__(self, max_tokens=32000):
        self.max_tokens = max_tokens
        self.system_prompt = ""  # always retained
        self.messages = []       # conversation history
        self.token_count = 0

    def add_message(self, msg):
        self.messages.append(msg)
        self.token_count += count_tokens(msg)

        # Trim to fit budget — remove oldest tool results first
        while self.token_count > self.max_tokens and len(self.messages) > 3:
            removed = self.messages.pop(1)  # keep system + last user
            self.token_count -= count_tokens(removed)

Best for: Chat-style agents, interactive coding assistants, support bots.

Limitation: Loses context from early in the conversation. If the user mentions something important 20 turns ago, the agent won't remember it.

Summarization Strategy

Periodically summarize the conversation into condensed context:

def summarize_context(messages):
    prompt = f"Summarize this conversation in 3-5 sentences, \
               preserving key decisions and user preferences: \
               {messages[-20:]}"
    summary = llm.call(prompt)
    return {"role": "system", "content": f"[CONTEXT: {summary}]"}

Pattern:

Turns 1-10: full messages
Turn 11: summarize turns 1-10 → insert as system message
Turns 11-20: full messages (with summary in system prompt)
Turn 21: summarize turns 11-20 → update system summary
... repeat

Best for: Long-running research agents, complex multi-step tasks, customer support.

Cost: Each summarization costs ~50-100 tokens. That's $0.0003-0.0015 per summary — essentially free.

Mental Model Strategy (Advanced)

Give the agent its own persistent memory that it reads and writes:

# Agent reads this file at the start of every session
MENTAL_MODEL = "agent-expertise.md"

def load_mental_model():
    if os.path.exists(MENTAL_MODEL):
        return open(MENTAL_MODEL).read()
    return ""

def save_observation(key, value):
    with open(MENTAL_MODEL, "a") as f:
        f.write(f"\n- {key}: {value}")

The agent builds expertise over time. This is the most token-efficient strategy because the agent curates what it remembers — it doesn't keep everything.

Best for: Specialized agents (code reviewers, security auditors, data analysts) that work on multiple disjoint tasks.

Token Budget Allocation Formula

Budget = SystemPrompt(15%) + Tools(15%) + Conversation(40%) + ToolResults(30%)

If budget is tight:
  1. Shorten tool descriptions (remove examples)
  2. Summarize conversation history (keep last 3-5 full, summarize the rest)
  3. Trim tool results to only the relevant sections
  4. Remind the agent about mental models instead of re-reading them

The 80/20 Rule of Context Management

80% of context problems come from 20% of the causes:

Too much conversation history — keep last 5 exchanges, summarize the rest
Too many tool results — only keep results that were directly used
Redundant system instructions — don't repeat the same rules in every system message
Large reference files — cite them, don't include them inline

Fix these four, and you solve most context window issues.

This is adapted from Module 2: Architecture of the Agentic Engineering the Hard Way course. Full course includes 65 lessons, 13 labs, and 20 skill kits.

4.6 KiB Raw Permalink Blame History