Context Engineering: The Hidden Moat Behind AI Product Success
Prompt & Circuit Editorial
May 30, 2026 · 12 Min Read

Summary
- Swapping to a newer model is the most visible lever in an LLM product, but every competitor can pull it too.
- What competitors cannot easily copy is the context layer: which information reaches the model, in what form, in what order, and at what cost.
- Five disciplines cover most of that work: selection, compression, structure, cache discipline and evaluation.
- The assets these disciplines produce (curated examples, retrieval logic, tool descriptions, stable prompt prefixes, eval sets) accumulate over time, which is what makes them defensible.
What context engineering is
The term took off in June 2025, when Andrej Karpathy endorsed "context engineering" over "prompt engineering" on X. His argument was that people hear "prompt" and think of a short task description, while production LLM applications are really about filling the context window with the right information for the next step.
Anthropic's engineering team later described it as "the set of strategies for curating and maintaining the optimal set of tokens" during inference, and as "the natural progression of prompt engineering." Its guiding principle is to find "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome."
A useful working definition: context engineering is the deliberate design of what information enters a model's context window, how it is formatted, when it is retrieved, and what it costs.
To make this concrete, consider a hypothetical food-logging assistant receiving the message "I had what I had yesterday." To answer well, the model needs far more than that sentence:
- System prompt: role, goals, safety constraints.
- Tool definitions: schemas for the logging functions.
- Few-shot examples: a handful of curated edge cases.
- User profile: goals and preferences from a database.
- Personal vocabulary: what "a bowl" means for this user.
- Conversation history: recent turns, or a summary of them.
- Retrieved knowledge: nutrition facts from a retrieval index.
- The current message.
The instruction text that people usually call "the prompt" is only a slice of this. The rest is where products differ.
Why the context layer is a moat
Model advantages are short-lived
Frontier models are released frequently, and every team with an API key gets access on the same day. "We use the newest model" is not a durable differentiator. That is not an argument against upgrading; it is an argument against relying on upgrades alone.
Context assets are specific to you
| Asset | Why it is hard to copy |
|---|---|
| Corrections and failure cases from real users | They encode what "right" looks like in your domain, and only your traffic produces them. |
| Per-user vocabularies and preferences | They grow with usage. |
| A curated few-shot pool | Examples are chosen against your own quality bar. |
| Tool names and descriptions | They are refined through observed tool-calling failures. |
| Stable, cache-friendly prompt structure | It lowers latency and cost and reflects many small design decisions. |
Replication takes time
A competitor can match your model in an afternoon. Rebuilding hundreds of curated examples, retrieval rules and an evaluation set that reflects real user behaviour takes months of iteration on real traffic.
The five disciplines
1. Selection: what to include
The counter-intuitive rule is to leave things out.
- Liu et al. ("Lost in the Middle", 2023) found that models use information best when it sits at the beginning or end of a long input, and noticeably worse when it sits in the middle, even for models built for long contexts.
- Anthropic describes a related effect it calls context rot: as context grows, the model's ability to recall information from it declines.
- In practice: send the smallest set of facts that lets the model do the task. For "what I had yesterday", retrieve yesterday's entries, not the whole week.
Tool selection counts too. Anthropic calls bloated tool sets "one of the most common failure modes" and suggests a simple test: if a human engineer cannot say which tool fits a situation, the agent will not do better.
2. Compression: what to do with everything else
Three patterns appear repeatedly in published guidance:
- Compaction. When a conversation nears the context limit, summarise it and continue from the summary (Anthropic).
- External memory. Keep durable state in a database or file system and fetch it on demand. Manus treats the file system as context that is "unlimited in size, persistent by nature", and keeps compression restorable: a web page can be dropped from context as long as its URL is kept.
- Recitation. Manus has its agent maintain a
todo.mdfile and rewrite it as it works, which keeps the current objectives at the end of the context, where the model attends most, and reduces goal drift on long tasks.
Raw history is not the same as necessary history.
3. Structure: ordering and formatting
- Split prompts into clearly delimited sections. Anthropic recommends XML tags or Markdown headers for this.
- Put stable material first and variable material last. This helps both attention (see "Lost in the Middle") and caching (below).
- Prefer the provider's native tool-calling interface over tool descriptions pasted into prose.
- Use few-shot examples, but choose a small set of "diverse, canonical examples" (Anthropic) rather than a long list of rules. Manus adds a warning for agents: uniform examples cause the model to imitate the pattern and drift. In its words, "The more uniform your context, the more brittle your agent becomes."
4. Cache discipline
Manus calls the KV-cache hit rate "the single most important metric for a production-stage AI agent", because it drives both latency and cost. The pricing gap is large. On Anthropic's API, cache reads cost 0.1x the base input price on most models (cache writes cost more than base), and OpenAI applies prompt caching automatically on supported models.
Both providers cache by exact prefix match. Anthropic's documentation says cache hits require "100 percent identical prompt segments"; OpenAI's says reuse "requires the entire rendered prefix to match." That has direct design consequences:
- Do not put a timestamp or per-user greeting at the top of the system prompt. Manus calls the timestamp a "common mistake"; OpenAI advises moving timestamps and user-specific text to the end.
- Keep context append-only rather than editing earlier turns.
- Serialise deterministically. Manus notes that JSON key ordering is not always stable across libraries.
- Changing tool definitions invalidates the whole cache on Anthropic's API, so avoid adding and removing tools mid-session.
Treat the prompt prefix like a database schema: stable, version-controlled and changed deliberately.
5. Evaluation: close the loop
- Gate every context change behind an automated evaluation run, just as you would gate code changes behind tests.
- Build the evaluation set from real user queries and real failures, and track quality alongside latency, cost and cache hit rate.
- Small improvements only count if you can show there was no regression elsewhere.
Why many teams under-invest
- It is invisible. Context improvements rarely produce a UI change to demo.
- Ownership is unclear. Deciding what the model needs to know is a product question; implementing it is an engineering question. Without a named owner, it drifts.
- The payoff compounds slowly. Early gains look small, which makes the work hard to prioritise, and that is exactly why it becomes a lead that is hard to close.
Working principles
| Principle | In practice |
|---|---|
| Context is a product decision | Someone owns the list of what goes into the model, not just the UI. |
| Fix context before swapping models | Try retrieval, ordering and examples before paying for a migration. |
| Cache-friendly by default | Review every dynamic string in the prefix for cache impact. |
| Iterate in small steps | Review traces regularly, add failures to the example pool and eval set, and watch cache metrics. |
Getting started checklist
- Map every component of the context for your main user flow.
- Measure token usage per component and question the largest ones.
- Retrieve fewer, more relevant results and rank them carefully.
- Delimit sections with XML tags or Markdown headers; put stable content first.
- Monitor cache hit rate, latency and cost per request.
- Run an automated evaluation on every change that touches context.
- Keep the prompt prefix in version control.
FAQ
Does this matter for single-turn applications? Yes. Even one-shot requests benefit from a clean system prompt, a few well-chosen examples and clear tool descriptions.
Do larger context windows make selection irrelevant? No. Long-context research and Anthropic's "context rot" guidance both show recall degrading as context grows, and every extra token costs money and latency.
Is caching only an OpenAI feature? No. Anthropic and OpenAI both offer prompt caching on their APIs, and self-hosted inference servers commonly reuse KV-cache for shared prefixes. The exact-prefix principle applies broadly.
Conclusion
The model you call matters, but everyone can call it. What you feed it, and how, is what accumulates into an advantage. Own the context layer.
Sources
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (arXiv, 2023)
- Anthropic Engineering: Effective context engineering for AI agents
- Manus: Context Engineering for AI Agents, Lessons from Building Manus
- Anthropic documentation: Prompt caching
- OpenAI documentation: Prompt caching
- Drew Breunig, "Prompts vs. Context" (quotes and links Karpathy's post)
Responses (0)
Sign in to join the conversation and share your thoughts.

Prompt & Circuit Editorial
Contributing Editor
Articles on AI infrastructure for cloud architects, written from official vendor sources and checked claim by claim before publishing.
Continue Reading

Demystifying LangChain Models: From LLMs to Embeddings in Plain English
A plain-English guide to LangChain's model layer: chat models, legacy text-completion LLMs, embedding models, invoke(), key parameters, and how embeddings power semantic search, with code for the current LangChain 1.x APIs.

The Statistics of Token Selection: Logits, Temperature, and Top-P Explained
How an LLM turns raw logits into the next token: softmax, temperature scaling, top-k and nucleus (top-p) sampling, with worked numbers and runnable Python.

From "Just a Chatbot" to a Full AI System: The Layers Explained
Modern AI products are not a single model behind a text box. A layer-by-layer guide to the stack: LLMs, retrieval-augmented generation, tools and agents, the Model Context Protocol, and the runtime that keeps it all reliable.