Understanding LLM Context: The Hidden Challenge of AI Development

Context as a Shared Conversation

Imagine you're having dinner with a friend at a restaurant. When you say "pass the salt," your friend doesn't need you to specify which salt, from which table, in which restaurant. The context is clear from your shared environment and conversation history.

Now imagine if every time you spoke, your friend forgot everything: the restaurant, your previous conversations, even why you're there. You'd have to explain everything from scratch each time. This is what working with an LLM would be like without context.

Context in LLMs works like your friend's memory of the entire dinner conversation. Every message you send isn't processed in isolation. It includes everything that came before it, creating a continuous narrative thread.

What Happens Behind the Scenes

When you type a message into Claude Code or any LLM interface, here's what actually happens:

The Context Assembly Process

Think of context like a rolling transcript of a meeting. Every time you speak (send a message), the AI doesn't just hear your latest words. It reviews the entire meeting transcript first:

// What gets assembled for EVERY single request
const contextSentToLLM = {
  // Fixed instructions (stays constant ~2,000 tokens)
  systemPrompt: "You are Claude Code, an AI assistant...",

  // THIS BECOMES MASSIVE! (grows with every message)
  conversationHistory: [
    { role: "user", content: "Help me debug this function" },
    { role: "assistant", content: "I'll analyse your function..." },
    { role: "user", content: "It's still not working" },
    { role: "assistant", content: "Let me check the error..." },
    // ... 50 more messages later ...
    { role: "user", content: "npm test\n[500 lines of output]" },
    { role: "assistant", content: "[2000 token response]" },
    { role: "user", content: "git diff\n[300 lines of changes]" },
    // ... another 30 messages ...
    { role: "user", content: "Can you read these 5 files?" },
    { role: "assistant", content: "[10,000 tokens of file content]" },
    // By now: 50,000+ tokens of conversation history
  ],

  // Your innocent new message (but processed with ALL the above)
  currentMessage: { role: "user", content: "What about line 42?" }
}

This entire package, meaning system instructions, the ENTIRE conversation history from message #1, and your new message, gets sent to the LLM's servers as one massive input. After 100 messages, you might be sending 100,000+ tokens with every single request! The model then generates a response based on everything in this increasingly bloated context.

How Fast Context Grows

The token count sent with every request grows with the conversation, not just with the latest message:

• Message #1: ~100 tokens sent
• Message #10: ~5,000 tokens sent
• Message #50: ~30,000 tokens sent
• Message #100: ~80,000 tokens sent
• Message #150: ~150,000 tokens sent (approaching limits)

Every message includes everything that came before it, all the way back to the first one.

The Context Window: Your Conversation's Memory Limit

Every LLM has a "context window", the maximum amount of information it can process at once. Think of it like RAM in a computer or the number of items you can juggle simultaneously.

Current Context Window Sizes (2025)

The context window arms race has led to impressive numbers. The figures below are a snapshot from around August 2025, when this article was first written; token limits move quickly, so treat them as illustrative rather than current by the time you're reading this:

• Google Gemini 2.5 Pro: 1 million tokens (expanding to 2 million in Q3 2025)
• Claude Sonnet 4: 1 million tokens (public beta) / 200,000 tokens (standard)
• GPT-4.1: 1 million tokens (with performance degradation)
• GPT-4o: 128,000 tokens

To put this in perspective: 1 million tokens ≈ 2,500 pages of text, roughly equivalent to reading all seven Harry Potter books in a single conversation!

When Context Becomes Contamination

Irrelevant information mixed into your context has a real cost: it makes it harder for the model to find what's actually relevant to your current task, in the same way clutter buried among your working files slows you down even when the file you need is technically still there.

Too Many Conversations at Once

Context bloat is like trying to have a focused conversation in an increasingly noisy room. At first, with just a few people talking, you can easily focus. But as more conversations start around you, some relevant, some not, it becomes harder to maintain clarity.

Common Context Polluters

• Debug Output Dumps: Pasting entire log files when only specific errors matter
• Repetitive Information: Running the same commands multiple times without clearing results
• Task Switching Residue: Moving from debugging to feature development without context reset
• Contradictory Instructions: Conflicting requirements from different phases of work
• Verbose Explorations: Extensive file searching and reading that's no longer relevant

# Example of context pollution
$ npm test
... 500 lines of test output ...
$ npm test  # Running again
... another 500 lines ...
$ npm test --verbose  # Even more detail
... 2000 lines of verbose output ...
# Now the context has 3000+ lines of similar test results!

# Impact: Next request gets confused response
"Fix the failing test"
# AI struggles to identify which of the 3000 lines matters

The Hidden Costs of Bloated Context

Performance Degradation

Model accuracy measurably degrades as context approaches its limit, even when the context window technically has room left. It's like asking someone to remember a phone number after reading an entire encyclopedia: the important information gets lost in the noise.

Attention Dilution

LLMs use attention mechanisms to focus on relevant parts of the context. When the context is small, this works well. As it grows, the model has to spread that same attention capacity across more content, and less of it lands on what's actually relevant to your current message.

Confusion and Hallucination

When context contains contradictory information, LLMs may blend incompatible instructions or fabricate responses to reconcile conflicts:

// Early in conversation: Setting up a React project
"Use React hooks and functional components"

// After debugging session: Working on build issues
"This is a vanilla HTML/CSS project, no frameworks"

// LLM confusion result:
"Let's use React hooks in your HTML file with useEffect()"
// Nonsensical mixture of contradictory contexts

Recognising Context Problems

Context Red Flags

Watch for these warning signs that your context has become problematic:

• Generic responses: AI gives vague advice instead of specific solutions
• Forgotten instructions: Suggestions ignore recent clarifications or requirements
• Mixed terminology: Blending concepts from different parts of the conversation
• Declining quality: Responses become less helpful over time
• Contradictory advice: AI suggests conflicting approaches in the same response
• Lost context: "I don't see that in the code" when it was just discussed

When you notice these signs, it's time to apply context management strategies.

Essential Context Management Techniques

1. Manual Context Hygiene

Unlike browser tabs that persist, AI conversations require explicit clearing. Here's how to actually reset your context:

How to Clear Context in Different Tools

$ /clear
✓ Conversation history cleared

$ /compact
✓ Conversation compressed to key points

$ exit
# Close and reopen to fully reset

Other tools:

• ChatGPT/Claude web: start a new chat/conversation
• VS Code Copilot: close and reopen the chat panel

Learn more about Claude Code slash commands

The Phase Transition Clear, Step by Step

STEP 1: Complete the current task
  "We've fixed the authentication bug successfully"

STEP 2: Save important info, if needed
  Copy any critical findings or solutions

STEP 3: Clear context
  Type: /clear
  Or: close the Claude Code window

---- CONTEXT BOUNDARY ----

STEP 4: Start fresh
  Open a new Claude Code session
  "I need to add user profile features to my Express app"

STEP 5: New, clean context
  No debugging history polluting the conversation
  AI focuses entirely on the new task

Bridging Old and New Context With a Summary

# OLD CONTEXT (before clearing)
$ Please summarise what we discovered and fixed, and save it to DEBUG_SUMMARY.md
I'll create a summary of our debugging session...
✓ Created DEBUG_SUMMARY.md

$ /clear
✓ Conversation history cleared

# ---- CONTEXT BOUNDARY ----

# NEW CONTEXT (completely fresh)
$ Read DEBUG_SUMMARY.md to understand previous work
I'll read the summary from the previous session...
I can see you fixed an async race condition in the auth module by...

$ Now let's implement the user profile features, building on the auth system we fixed
Perfect! Based on the summary, I understand the auth system is now working. Let's build the profile features...

Important: The summary is NOT automatically included after clearing. You must either:

• Save it to a file and read it in the new session
• Manually copy and paste relevant parts
• Reference it as a document in your project

2. Plan Documents as Context Anchors

Plan documents act as persistent memory across context resets: they survive even when the conversation itself is wiped.

PHASE 1: Planning session

  STEP 1: Discuss feature
    "I need to add user authentication to my app"

  STEP 2: Iterate on requirements
    Back-and-forth refining the approach

  STEP 3: Create plan document
    "Write a detailed plan to IMPLEMENTATION_PLAN.md"

  STEP 4: Review and refine
    "Update the plan to include rate limiting"

---- CLEAR CONTEXT ----

PHASE 2: Execution session (fresh context)

  STEP 5: Start new session
    Open a fresh Claude Code session

  STEP 6: Load the plan
    "Read IMPLEMENTATION_PLAN.md"

  STEP 7: Confirm understanding
    AI: "I understand we're implementing JWT auth with..."

  STEP 8: Execute step 1
    "Let's implement step 1 from the plan"

---- CLEAR CONTEXT ----

PHASE 3: Continue next day (fresh context)

  STEP 9: Load plan and progress
    "Read IMPLEMENTATION_PLAN.md - we completed step 1"

  STEP 10: Execute step 2
    "Now implement step 2 from the plan"

Example Plan Document

# IMPLEMENTATION_PLAN.md

## Objective

Implement user authentication system

## Requirements

- JWT-based authentication
- PostgreSQL user storage
- Rate limiting on login attempts

## Steps

1. [x] Create user database schema
2. [ ] Implement registration endpoint with Express.js
3. [ ] Add login with JWT generation
4. [ ] Setup middleware for protected routes

## Technical decisions

- bcrypt for password hashing (rounds: 10)
- 15-minute JWT expiry with refresh tokens
- Redis for rate limiting state

## Progress log

- 2025-08-20: Completed database schema (step 1)
- 2025-08-21: Starting registration endpoint (step 2)

Key benefits:

• Plan survives all context resets
• Each execution starts clean but informed
• Progress tracking across sessions
• No confusion from old debugging attempts

Advanced Delegation Strategies

Sub-Agent Delegation in Claude Code

Claude Code's sub-agents do the messy exploratory work in an isolated context and return only the essential findings to your main conversation:

# WITHOUT a sub-agent (pollutes main context)
$ Search the entire codebase for all uses of the deprecated API
Searching for deprecated API usage...
Found in: src/auth/login.js:42
Found in: src/users/profile.js:156
[... 500 more lines of search results ...]
# Main context now contains 500+ lines of search output

# ---------------------------------

# WITH a sub-agent (keeps main context clean)
$ Use a sub-agent to audit deprecated API usage and report back a summary
Delegating to sub-agent for comprehensive search...
✓ Sub-agent completed analysis

Summary: Found 23 instances of deprecated API across 8 files
- Authentication: 5 instances (needs urgent update)
- User profiles: 8 instances (low priority)
- Data processing: 10 instances (can be batch updated)

# Main context stays clean - only 5 lines instead of 500!

Sub-agents are perfect for:

• QA Operations: Running comprehensive tests and returning just the failures
• Code Analysis: Scanning large codebases with ripgrep for patterns
• Research Tasks: Web searches with Google and documentation review
• Exploration: Finding files, understanding project structure

The key advantage: sub-agents work in isolated contexts. Their explorations don't contaminate your main conversation, keeping it focused and efficient.

Context-Aware Communication

Structure your messages to minimise context pollution:

Inefficient - adds noise:
"Let me check something... run this... okay try this...
hmm not that... what about... oh wait I found it!"

Efficient - direct and focused:
"Check if the auth middleware is applied to the /api/users route"

The Paradox of Large Context Windows

Bigger Isn't Always Better

Having a 1-million-token context window is like having a 10,000-page notebook. Yes, you can write everything down, but finding specific information becomes increasingly difficult. The cognitive load on the model increases, potentially leading to:

• Lost Instructions: Early directives buried under thousands of tokens
• Conflicting Context: Contradictions between different parts of the conversation
• Attention Scatter: Model struggles to identify what's currently relevant
• Slower Processing: More context means more computation time

The ideal context size is one that's enough to maintain continuity and necessary information, but not so much that it becomes unwieldy. For most development tasks, 10,000-50,000 tokens of well-curated context will probably outperform 200,000 tokens of chaotic conversation history.

Advanced Context Strategies

Checkpointing Your Progress

Like saving your game progress, create context checkpoints at major milestones:

## Checkpoint: Authentication system complete

- Implemented: JWT auth, user registration, login endpoints
- Database: Users table with bcrypt passwords
- Middleware: requireAuth() for protected routes
- Tests: 24 passing, 100% coverage
- Next: Build user profile management

Budgeting Your Context Tokens

Treat context like a budget and allocate tokens to different purposes:

## Context budget allocation

- System instructions: 2,000 tokens (fixed overhead)
- Active code files: 5,000 tokens (current work)
- Recent conversation: 10,000 tokens (working memory)
- Reference documents: 3,000 tokens (plans, requirements)
- Safety buffer: 5,000 tokens (unexpected expansion)
- Total target: 25,000 tokens (well below limits)

Layering Context by Relevance

Structure context in semantic layers, from most to least relevant:

1. Immediate Context: Current task and recent exchanges
2. Working Context: Active files and recent changes
3. Reference Context: Project structure and conventions
4. Historical Context: Summaries of completed work

Context Management Best Practices

Do's

• ✅ Start fresh contexts for distinctly different tasks
• ✅ Create plan documents before complex implementations
• ✅ Use sub-agents for exploratory or research tasks
• ✅ Summarise before context resets
• ✅ Be explicit about what information is currently relevant
• ✅ Prune verbose output before continuing

Don'ts

• ❌ Paste entire log files without filtering
• ❌ Repeat the same operations multiple times
• ❌ Mix unrelated tasks in the same conversation
• ❌ Assume the model remembers early instructions in long contexts
• ❌ Include conflicting requirements without clarification

Quick Context Health Check

Before your next message, ask yourself:

• Is this conversation focused on one clear objective?
• Have I included conflicting information?
• Could I explain the current state in 2-3 sentences?
• Am I about to paste more than 50 lines of output?
• Would starting fresh be more efficient?

If you answered "no" to the first question or "yes" to any others, it's time to manage your context.

The Future of Context Management

As we move toward even larger context windows, the challenge shifts from capacity to curation. The winners in AI development probably won't be those with the largest contexts, but those who manage context most intelligently.

Emerging Patterns

• Hierarchical Context: Multi-level context systems with different retention policies
• Semantic Compression: Automatic summarisation of older context
• Context Routing: Different sub-contexts for different aspects of work
• Persistent Memory: Long-term storage separate from working context

Building Context Management Into Your Workflow

Working effectively with LLMs like Claude Code means curating context deliberately rather than just accumulating it. Two habits do most of the work: clearing context at natural task boundaries, backed by a plan document that survives the clear, and delegating exploratory work to sub-agents so it never pollutes your main conversation in the first place. Everything else in this article is detail layered on top of those two moves.

Treat context the way you'd treat any other resource with a cost attached to it: prune it, budget it, and reset it deliberately rather than letting it accumulate by default.