Context Engineering Is Not Prompt Engineering

On June 19, 2025, Shopify CEO Tobi Lütke posted that he preferred the term “context engineering” over “prompt engineering,” defining it as “the art of providing all the context for the task to be plausibly solvable by the LLM.” Six days later, Andrej Karpathy sharpened the framing: “context engineering is the delicate art and science of filling the context window with just the right information for the next step.” By September, Box CEO Aaron Levie had declared it “increasingly the most critical component for building effective AI agents in the enterprise right now” and predicted it would be “the long pole in the tent for AI agents adoption in most organizations.”

In the year since, the term has moved from influential tweets into Gartner briefings, engineering hiring posts, and platform team roadmaps. Most organizations have taken this as an invitation to rename their prompt engineers and continue doing what they were doing.

That is a costly interpretation.

MIT’s Project NANDA published “The GenAI Divide” in August 2025, drawing on 150 leadership interviews, 350 employee surveys, and an analysis of 300 public AI deployments. Its central finding was that 95% of organizations were seeing no measurable return to the income statement from their generative AI pilots. The failure mode the researchers identified was not model capability, not data volume, and not organizational resistance. It was what they described as the “learning gap”: organizations that had not developed the discipline of structuring and delivering the right information to their AI systems before expecting reliable results. They were treating a system design problem as a prompting problem.

What prompt engineering actually is

Prompt engineering was the right discipline for a specific moment: when AI features were single-turn or lightly multi-turn interactions, when the task was bounded and well-specified, and when the question was genuinely “how do I phrase this instruction to get the output I want?”

Its domain is the instruction layer. Given that you have decided what information the model has access to, prompt engineering asks how to frame the instruction that directs what the model does with it. The craft is real: instruction phrasing, few-shot examples, output format directives, and chain-of-thought scaffolding all affect model behavior in ways that matter. Prompt engineering did not become obsolete; it became insufficient.

The insufficiency arrives at exactly the moment when the AI feature moves from a bounded demo into a production system: when it needs to draw on current data the model was not trained on, when it operates across multiple turns with accumulating state, when it must respect organizational policies that vary by user or context, and when several agents coordinate across a workflow. At that point, the question “how do I phrase this?” is downstream of a question the team has not yet asked: “what does the model need to know, where does that information come from, and how does it arrive reliably in the context window at the right moment?”

That is the context engineering question. It is not a prompting question. It is a system design question.

What the distinction means in practice

Prompt engineering operates on a fixed information set. It shapes the instruction, but it does not decide what the model knows. Context engineering designs the information set itself: what gets into the context window, from which sources, at what point in the workflow, in what format, and subject to what update policy.

The practical scope of context engineering includes decisions that prompt engineering never touched: whether stable domain knowledge belongs in the system prompt or in a retrieval index; how to budget context tokens across competing information sources when the window is not infinite; how to manage conversation state across a multi-turn session so that stale references from two exchanges ago do not shadow the current instruction; how to structure tool outputs so the model can parse and reason about them consistently; and how to validate that the context the model received at inference time was actually what the deployment configuration intended.

These are not prompt-writing decisions. They are architecture decisions with the same properties as any other architecture decision: they have trade-offs, they interact with one another, they compound over time, and they fail in predictable ways when made badly.

Three anti-patterns that follow from conflating them

When teams treat context engineering as prompt engineering with a new name, three failure modes are predictable enough to name.

Context dumping is the most common. The reasoning is intuitive: if the model has more information, it will do better. The system prompt accumulates everything: full API documentation, complete product catalogs, all organizational policies, every behavioral constraint, and the last six months of relevant domain knowledge. Context budgets fill with information the model will never use for the current task. When the window overflows, the model silently truncates. When the information is stale, the model answers from outdated facts. When the context contains conflicting signals, the model makes an arbitrary choice between them. Teams that have experienced this often conclude that the model is unreliable, when the actual problem is that they designed a context that guarantees unreliable behavior.

Context coupling is the failure mode that surfaces in the second half of the year, after the initial deployment looks fine. The AI feature is wired to a specific context format: a particular JSON schema from a retrieval step, a specific field order in a tool output, a hard-coded section ordering in the system prompt. The format changes for any of the ordinary reasons formats change: the retrieval pipeline is updated, the tool author changes the response schema, the system prompt is edited to add a new policy. The model’s behavior changes in ways that are difficult to trace, because the coupling between context format and model behavior was never documented. Context coupling is what happens when context engineering decisions are made implicitly, without an owner, and without versioning.

Context drift is the failure mode in long multi-turn sessions or agents that operate across many steps. Conversation history accumulates without a pruning policy. The model’s current instruction must compete for attention with three turns of reasoning about a subtask that has already been resolved. A working hypothesis from step four is still in context at step twelve and now contradicts the new information the tool call at step nine returned. The model hedges, produces inconsistent outputs, or quietly acts on the superseded hypothesis rather than the current one. Context drift is what happens when the session state layer has no write-and-read policy: when information goes in but nothing governs what comes out or when.

The four-layer context placement matrix

The practical tool for context engineering is a placement decision: for each type of information the AI feature needs, which layer of the context should own it? The failure modes above all stem from placing information in the wrong layer or from having no deliberate policy at all.

Information type	Belongs in	Why this layer	Failure mode in the wrong layer
Role definition, behavioral constraints, output format rules, organizational policies	System prompt (static layer)	The model needs these on every request; they change slowly; they must be consistent across all users and sessions.	In retrieval: intermittent retrieval misses produce inconsistent behavior; the model ignores constraints some of the time. In conversation history: restatement on every turn wastes tokens and creates competing signals.
Domain knowledge, product information, code conventions, current documentation	Retrieval / RAG (dynamic injection)	This content is too large to fit in the system prompt and changes more frequently than deployments allow. Retrieve only what is relevant to the current request.	In system prompt: stale facts, context budget consumed by low-relevance content, scale limit hit quickly. In tool output: slower and more expensive than vector retrieval; not designed for knowledge lookup.
Real-time data: user session state, live records, current operational status	Tool calls (runtime resolution)	This data changes faster than any context layer can be kept current; it is user-specific and cannot be shared across sessions; it requires authorization the model should not bypass.	In system prompt: stale within minutes; security boundary violation for user-specific data. In retrieval: indexing latency makes it stale; lacks authorization enforcement.
Active task goal, in-progress decisions, current working hypothesis, tool outputs from the current session	Conversation history (managed state)	This is working memory: what the agent is thinking through right now. It must be available for reasoning but cleared or summarized at session boundaries.	Unbounded: context exhaustion; old contradictory information shadows current state. In retrieval: indexing introduces latency; past sessions leak into current reasoning without a clear boundary.

The table is not a formula. The placement decision for any given information type involves judgment about retrieval latency, context budget, update frequency, and access control requirements. The point is that the decision should be explicit. When teams leave it implicit, they typically end up with everything in the system prompt or in an undifferentiated retrieval index, which produces context dumping and context coupling by default.

The layer boundaries also need ownership. The system prompt layer should have a clear owner, a versioning discipline, and a review process for changes (the same care a team would apply to a configuration change that affects every request to a production service). The retrieval layer needs a pipeline owner, a chunking and embedding policy, and a relevance threshold below which results are not injected. The conversation state layer needs an explicit pruning or summarization strategy for long sessions. None of these are prompt engineering tasks; they are software engineering tasks that happen to involve AI.

Making context engineering a team discipline

The framing shift matters for how teams organize around AI features. Prompt engineering lives naturally in a single person’s workflow: a developer writes instructions, iterates in a playground, ships the result. Context engineering requires collaboration across disciplines, because the layers it governs cross team boundaries.

The retrieval layer is a data engineering problem. The tool output format is an API contract problem. The system prompt version is a configuration management problem. The conversation state policy is an architecture problem. Coordinating these requires the same kind of cross-functional ownership model that production data pipelines or security review processes require: a named owner per layer, an audit trail for changes, and observability instrumentation that makes context quality visible.

Observability is the discipline that is most often missing. Across the AI features we work with that have unexplained reliability degradation, the most common finding is that nobody can answer the question “what context was actually in the window when the model produced that response?” If the team cannot reconstruct the exact context for a given inference, debugging production failures requires inference and guesswork. Logging the assembled context (subject to appropriate data handling constraints) for a sample of production requests, with a structured diff that surfaces changes from the previous version of the system prompt, is the minimum viable observability for a production AI feature.

The Datadog State of AI Engineering report for 2026 found that 69% of companies now run three or more AI models in production. Multi-model deployments compound the context engineering problem, because context format decisions made for one model’s inference API may not transfer cleanly to another’s. A system prompt structure optimized for one model’s behavior can degrade another model’s performance on the same task. Context engineering decisions need to be model-aware, not just task-aware.

Aaron Levie’s prediction that context engineering will be “the long pole in the tent for AI agents adoption in most organizations” is specific and testable: the bottleneck for most enterprise AI rollouts in the next two years will not be model capability, which is advancing faster than most organizations can absorb. It will be the ability to design, govern, and observe the information environment that surrounds those models. Organizations that treat that as a prompting problem will remain in the 95%.

Across the engineering and AI platform teams we work with, the ones extracting durable value from AI features are the ones that have built context engineering as a team discipline: explicit layer ownership, versioned system prompts, retrieval pipeline observability, and conversation state policies defined before the first production session. The ones that have not are the ones producing the MIT NANDA statistics.

If you are building production AI features, or if your current deployment demos reliably and fails unpredictably, the context architecture is usually where the diagnosis lives. We have helped engineering teams establish this discipline and are glad to think through your specific situation.