Did you know that LLMs can get “lost in the middle”, even when the information they need is right there in the prompt?

Picture this: you give a model a long prompt with all the documents it needs, the answer is sitting right there in the text, and it still gets it wrong. Not because the information is missing, but because of where it is in the prompt.

In this article, we’ll go over what the “Lost in the Middle” phenomenon is, why it happens, and what you can do about it, especially if you are building RAG systems.

What Is The “Lost In The Middle” Phenomenon?

In 2023, researchers from Stanford, UC Berkeley and Samaya AI published a paper called “Lost in the Middle: How Language Models Use Long Contexts”. They wanted to answer a simple question: when you give an LLM a long context, does it actually use all of it?

To test this, they gave models a question together with a set of documents (for example, 20 of them), where only one document contained the answer. Then they moved that one relevant document around. First position, somewhere in the middle, last position, and so on, measuring the accuracy each time.

If a model used its context equally well, the accuracy should stay roughly the same no matter where the document is placed. Well, that is not what happened.

The U-Shaped Curve

When you plot accuracy against the position of the relevant document, you get a U-shaped curve. Models performed best when the relevant information was at the very beginning of the context (primacy) or at the very end (recency). When the same information was buried in the middle, performance dropped noticeably.

The most surprising part? In some of their experiments, when the answer was in the middle of the context, GPT-3.5-Turbo did worse than when it was given no documents at all and had to answer from memory. Let that sink in for a second. Adding the right document in the wrong place made the answer worse.

Funny enough, this is not so different from us humans. Psychologists call it the serial position effect: we tend to remember the first and last items of a list much better than the ones in the middle.

Bigger Context Windows Don’t Fix It

You might be thinking, “Okay, but modern models have huge context windows, 128K or even 1M tokens. Surely that solves this, right?” Not quite.

The paper found that the extended-context versions of models were not any better at using their context than the standard versions. A bigger context window means the model can accept more tokens. It does not mean the model pays equal attention to all of them.

Newer models have gotten better at this, but it is still something you should test for in your own application, not something you can assume away.

Why Does This Happen?

There is no single, settled answer yet, and research on this is still ongoing. But a few factors are very likely to contribute, and they tie back nicely to how Transformer LLMs work:

1. Causal attention. In decoder LLMs like GPT, each token can only attend to the tokens that come before it. Tokens at the start of the prompt are seen by every single token after them, so they end up with a lot of influence. Models even learn to put large attention weights on the very first tokens, sometimes called “attention sinks”.
2. Positional encodings favour nearby tokens. Methods like RoPE (rotary position embeddings), used in models like Llama, have a built-in tendency for attention to weaken as the distance between tokens grows. Content at the end of the prompt sits closest to where the answer is being generated, so it gets a recency boost.
3. How models are trained. In most training and instruction-tuning data, the important parts sit at the start (the instructions) and at the end (the question). Models learn to lean heavily on those positions, and the middle ends up being treated as less important.

If you want to understand attention and positional encodings in more depth, check out my free course Transformer LLMs: The Basics, where we break these concepts down step by step.

Why This Matters For RAG Systems

This is where it gets really practical. In a typical RAG (Retrieval-Augmented Generation) pipeline, we retrieve the top k chunks from a vector database and stuff them into the prompt, usually in order of their similarity score.

The problem is that retrievers are not perfect. The chunk with the actual answer is not always ranked first. As you increase k to “give the model more context”, there is a good chance the one chunk that matters ends up somewhere in the middle, exactly where the model is least likely to use it.

The paper also found that retrieving more documents gives diminishing returns. Accuracy stopped improving long before the retriever stopped finding more relevant documents. So simply throwing more chunks at the model is not a reliable fix.

How To Mitigate It

Luckily, there are a few simple things you can do to work with this behaviour instead of against it.

1. Place Important Information Strategically

Put your most important documents at the beginning and the end of the context, and the least relevant ones in the middle. Here is a small Python function that does exactly that, given a list of documents sorted from most to least relevant:

def reorder_for_long_context(docs):
    """docs: a list sorted from most to least relevant."""
    front, back = [], []
    for i, doc in enumerate(docs):
        if i % 2 == 0:
            front.append(doc)
        else:
            back.append(doc)
    # Most relevant at the start, second most relevant at the end,
    # least relevant in the middle.
    return front + back[::-1]


docs = ["doc1", "doc2", "doc3", "doc4", "doc5"]  # most relevant first
print(reorder_for_long_context(docs))
# ['doc1', 'doc3', 'doc5', 'doc4', 'doc2']

If you use LangChain, the LongContextReorder document transformer does the same thing for you.

2. Use Rerankers

A reranker, usually a cross-encoder model, looks at the query and each retrieved chunk together and gives a much more accurate relevance score than plain embedding similarity. Retrieve a wider set of candidates, rerank them, and only keep the top few. This way, the chunks that actually matter are the ones you place in the strong positions.

3. Remove Unnecessary Information

Every extra chunk pushes something else towards the middle. Remove duplicate or near-duplicate chunks, drop low-scoring results, and keep your chunks focused. A shorter, cleaner context often beats a longer one.

4. Mind Where You Put The Question

Keep your instructions and question close to the parts of the prompt the model pays the most attention to. A simple trick is to state the question before the documents and repeat it after them. The paper found that placing the query both before and after the data helped on some of their tasks.

5. Test How Ordering Affects Answer Quality

Don’t guess, measure. Build a small evaluation set of questions where you know which chunk holds the answer, move that chunk to different positions in the prompt, and compare the results. Every model behaves a bit differently, so this is the only way to know how much your system is affected.

Key Takeaway

More context doesn’t always mean better answers. How you organize that context matters.

The next time your RAG system gives a wrong answer even though the right document was retrieved, don’t just look at what you passed to the model. Look at where you put it.

Research paper: Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)

That’s all I wanted us to go over in this article. If you found it useful, share it with someone building with LLMs, and see you in the next one.

Cheers!