Presented by Claude

• A large context window genuinely beats human cognition at brute-force tasks — ingesting a 500-page contract in seconds, holding dozens of documents at once, and recalling exact text verbatim — but it is closer to a vast, temporary working memory than to genuine knowledge, understanding, or accumulated expertise.
• The thesis that “AI can excel human work merely by the size of the context window” is half-true and half-misleading: advertised windows (200K to 10M tokens) far exceed the effective window where accuracy holds, and every frontier model measurably degrades as input grows (“context rot”), while humans bring causal reasoning, judgment, metacognition, and years of persistent learning that a context window does not provide and resets every session.
• For knowledge workers, the practical winner is rarely “stuff everything into the biggest window.” It is disciplined context engineering — feeding the model the smallest set of high-signal information, keeping critical facts at the start or end, and using retrieval (RAG) — combined with human oversight for synthesis, judgment, and verification.

Key Findings

1. Context windows have exploded ~5,000x in five years — from GPT-3’s 2,048 tokens (2020) to today’s 1M–10M-token models — but the comparison to human memory is a category error: a context window resembles working memory, not long-term knowledge.
2. Advertised context ≠ usable context. Independent benchmarks (RULER, NoLiMa, Chroma’s “Context Rot”) consistently show models become unreliable far below their stated limits.
3. The “lost in the middle” effect is real but narrowing. Models attend best to the beginning and end of context; newer models have largely solved this for simple fact retrieval but not for reasoning or synthesis.
4. Human working memory is tiny but powerful. We hold only ~4 chunks at once, yet compress information through expertise and draw on effectively unlimited long-term memory.
5. Where each wins is now reasonably clear — and the smart move for businesses is to combine them.

Details

The technical reality: what a context window is, and how big they’ve become

A context window is the maximum span of text — measured in tokens, where one token is roughly ¾ of a word — that a model can “see” at once, including your prompt, any uploaded documents, the conversation history, and the model’s own reply. As a rule of thumb, 100 words ≈ 130 tokens, so a 200,000-token window holds roughly 150,000 words, about two novels or 500 pages. MultipleChatAmazon Web Services

The growth has been staggering. GPT-3 launched in 2020 with 2,048 tokens; GPT-4 arrived in March 2023 with 8K and 32K variants; GPT-4 Turbo reached 128K in November 2023; Claude reached 200K in late 2023. As of early-to-mid 2026, Anthropic’s Claude models offer a 200K standard window with 1M tokens available, OpenAI’s GPT-5 family offers around 400,000 tokens (with later variants advertising ~1M), Google’s Gemini 2.5/3 Pro offers ~1M tokens, and Meta’s Llama 4 Scout advertises an industry-leading 10M tokens. (Several of these top-line numbers are marketing-inflated — Llama 4 Scout, for instance, was trained at 256K and reaches 10M only through length-generalization techniques, with weak independent benchmark scores.) Medium + 3

The reason windows were limited in the first place — and why they remain costly — is the transformer’s self-attention mechanism, which compares every token to every other token. This scales quadratically: doubling the context roughly quadruples the compute and memory. At long sequence lengths, attention dominates the entire computation, which is why bigger windows mean higher cost and latency. Mem0

Documented limitations: the gap between advertised and effective context

The single most important nuance for any knowledge worker is that “supports” does not mean “uses well.” Three converging lines of research establish this:

• “Lost in the Middle” (Liu et al., Stanford, TACL 2024). Models retrieve information best when it sits at the beginning or end of the context and significantly worse from the middle — a U-shaped accuracy curve — and performance falls as context grows, “even for explicitly long-context models.”
• RULER (Hsieh et al., NVIDIA, 2024). This synthetic benchmark found that although models all claim 32K-plus token windows, “only half of them can maintain satisfactory performance at the length of 32K.” Nearly all models score near-perfectly on the simple needle-in-a-haystack test yet collapse on harder multi-hop and aggregation tasks as length grows. OpenReview
• NoLiMa (Modarressi et al., Adobe Research, ICML 2025). When the answer can’t be found by literal word-matching and requires inferring an association, performance craters. At 32K tokens, most models tested dropped below half their short-context baseline. Even GPT-4o, a top performer, fell from a near-perfect 99.3% at under 1K tokens to 69.7% at 32K.
• Chroma “Context Rot” (Hong, Troynikov & Huber, July 2025). Testing 18 frontier models including GPT-4.1, Claude 4, and Gemini 2.5, the team found “performance grows increasingly unreliable as input length grows,” with degradation appearing well before the window is full. (Chroma sells retrieval tooling, so it has a commercial interest in this conclusion — but the finding is corroborated by the independent academic benchmarks above.)

The honest counter-evidence: a November 2025 Google paper by Max McKinnon, “Retrieval Quality at Context Limit,” found that Gemini 2.5 Flash answered needle-in-a-haystack questions accurately regardless of position, even near the 1M-token limit — suggesting “lost in the middle” is fading for simple factoid retrieval. But McKinnon explicitly limits the claim: no paraphrased or ambiguous queries, no conflicting facts, no synthesis. The degradation narrative holds for everything harder than looking up a unique fact.

Human memory and attention: small but profound

Human cognition runs on a strikingly small active buffer. George Miller’s famous 1956 paper proposed a limit of “seven, plus or minus two” chunks — though Miller himself called it “a rhetorical device.” Nelson Cowan’s influential revision (Cowan, “The magical number 4 in short-term memory,” Behavioral and Brain Sciences 24(1):87–114, 2001) argued that prior estimates of seven were “meant more as a rough estimate and a rhetorical device,” with the real limit “only three to five chunks”; he proposed that focus-of-attention capacity “averages about four chunks in normal adults.” Unaided short-term memory also decays in roughly 15–30 seconds without rehearsal. PhilPapers + 3

But two features make this tiny buffer formidable. First, chunking through expertise: Chase and Simon’s classic chess studies showed masters recall board positions far better than novices not because they have bigger working memory, but because they encode the board as a few meaningful patterns drawn from thousands stored in long-term memory. Second, long-term memory is effectively unlimited in capacity and duration, and it is associative and reconstructive — we retrieve by meaning and relationship, integrating new information into rich schemas built over years. Working memory, in modern accounts (Baddeley; Cowan), is best understood as the activated portion of long-term memory under attentional control — not a passive store but an active workspace that manipulates information.

This is the crux of the analogy. A context window is the AI’s working memory — “what’s in front of it right now.” It is not the AI’s knowledge (that lives in the frozen model weights, analogous to learned long-term expertise), and crucially it does not persist. As one analysis puts it, “the context resets every session… Every session starts from scratch.” Humans accumulate expertise across years; an AI’s context evaporates when the conversation ends and produces no learning.

Where each genuinely wins

Large AI context clearly beats humans at:

• Speed and scale of ingestion. A human reading at 200–300 words per minute needs ~10 hours to read a 150,000-word book; a frontier model ingests the same text in seconds and can generate output far faster than anyone can read. Baseten
• Breadth. Holding dozens of documents simultaneously and cross-referencing them in one pass.
• Verbatim recall within the window. No paraphrase drift on exact quotes, clauses, or numbers it can currently see.
• Tirelessness and consistency across repetitive review at volume.

Humans (or the framing itself) still win at:

• Genuine comprehension and synthesis versus retrieval. A bigger window improves looking things up; it does little for judgment about what matters.
• Causal reasoning. Research finds LLMs perform mostly “shallow” causal reasoning and lack genuine human-like interventional reasoning.
• Metacognition. Hu et al., “Judgments of learning distinguish humans from large language models in predicting memory” (Nature Scientific Reports, 2025), found that “while human JOL reliably predicted actual memory performance, none of the tested LLMs (GPT-3.5-turbo, GPT-4-turbo, and GPT-4o) demonstrated comparable predictive accuracy… they struggle at the meta-level” — i.e., knowing what they know.
• Persistent, cumulative expertise across months and years, and transfer to genuinely novel situations.

The practical knowledge-work picture

The stakes are concrete. In legal work, the American Bar Association estimates that document review “accounts for more than 80 percent of total litigation spend, or $42.1 billion dollars a year,” and AI summarization can cut reviewer hours dramatically — but Stanford RegLab/HAI’s “Hallucination-Free?” study (Magesh et al., 2024, 202 queries) found GPT-4 hallucinated on 43% of legal queries, with general-purpose tools erring “as high as 82%” of the time when used for legal purposes. Even purpose-built RAG legal tools fared imperfectly: the same study found they “each hallucinate between 17% and 33% of the time” — 17% for LexisNexis Lexis+ AI and 33% for Thomson Reuters Westlaw AI-Assisted Research. The lesson generalizes to contract review, financial-document analysis, research synthesis, codebase analysis, and meeting transcripts: AI is a powerful first-pass engine, not a final authority.

This is also why retrieval-augmented generation (RAG) did not die when context windows ballooned. RAG feeds the model only the relevant chunks, which is dramatically cheaper and faster than stuffing everything into context (vendor and practitioner benchmarks put RAG at a fraction of full-context cost — one independent test reported RAG at roughly 4% of full-context cost with about half the latency), keeps the signal-to-noise ratio high, and — unlike a black-box long context — can cite its sources. The 2025–2026 industry consensus is captured in the new discipline of “context engineering.” As Anthropic’s engineering blog “Effective context engineering for AI agents” (published Sept 29, 2025, alongside Claude Sonnet 4.5) puts it, “good context engineering means finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome” — not maximizing tokens. SubstackAnthropic

Recommendations

1. Match the window to the task, then stop. Use large-context models when you genuinely need cross-document reasoning over a big corpus (e.g., reviewing a 500-page deal, analyzing an entire codebase). For most queries, a focused prompt or RAG pipeline is cheaper, faster, and more accurate. Threshold to change course: if your input routinely exceeds ~50% of a model’s advertised window, expect reliability to drop and switch to retrieval or chunking.
2. Engineer placement. Put the most important instructions and facts at the beginning or end of the context, where attention is strongest. Don’t bury the key clause in the middle of a 200-page dump.
3. Test recall before you trust it. On any high-stakes long-context task, spot-check the model’s recall of mid-document facts at your actual input length. If accuracy falls off, reduce context or retrieve.
4. Keep humans on synthesis and judgment. Use AI for ingestion, first-pass extraction, and verbatim recall; reserve causal judgment, relevance decisions, and final sign-off for people — especially in legal, financial, and medical contexts where hallucination carries real cost.
5. Build persistent memory deliberately. Because context resets each session, use external memory, structured notes, or memory features to carry knowledge across sessions rather than re-pasting it. Benchmark that would change the calculus: if independent benchmarks (not vendor claims) show a model holding accuracy uniformly across its full window on reasoning tasks — not just factoid retrieval — the case for naive large-context use strengthens considerably.

Caveats

• The field moves fast. Specific token counts and model names cited here reflect late-2025/early-2026 snapshots and will date quickly; the structural findings (effective < advertised context, context rot, the working-memory analogy) are more durable.
• Some sources have commercial incentives. Chroma (context rot) and RAG vendors benefit from emphasizing long-context limits; model providers benefit from emphasizing big windows and high recall. Where possible, weight independent academic benchmarks (RULER, NoLiMa, Lost-in-the-Middle) over vendor claims.
• Cognitive numbers are estimates, not constants. Miller’s 7±2 and Cowan’s ~4 chunks are central tendencies that vary by individual, modality, and expertise; the 15–30 second short-term duration is a textbook figure, and the precise mechanism of memory transfer remains debated.
• The working-memory analogy is functional, not mechanistic. Saying a context window is “like” working memory captures behavior (capacity limits, primacy/recency effects) — it does not imply the underlying processes are the same.