Large Language Models: The Hard Parts

Chapter 4. From Data to Context

One home run is much better than two doubles.

Steve Jobs

We have already examined several methodologies and tools for measuring the performance of LLMBAs, models, and prompts. We focused on evaluating the results, or outputs. To put it another way, we focused on the textual answers generated by our LLMBAs.

In this chapter, we turn our attention away from the results and toward the data passed to those LLMBAs, specifically the unstructured textual input, which becomes context once it is passed to our LLMs. Context is a hugely important concept in the world of LLMs. It is what comes together with a prompt and a model to form an LLMBA, and it’s what allows our application to answer a question or accomplish a task while being grounded in truth.

While advances in long-context language models have expanded the amount of information LLMs can process (as we will discuss later in this chapter), significant challenges remain in managing and effectively utilizing extended data inputs:

• LLMs are sensitive to input formatting and structure, requiring careful data preparation to achieve optimal results.¹

• LLMs operate with knowledge cutoffs, providing potentially outdated information that may not reflect current reality and demonstrate problems with temporal knowledge accuracy. ²

• LLMs also exhibit a “lost-in-the-middle” effect,³ where information placed in the middle of long contexts is often ignored, and they struggle to recall less common but important ...