Generative AI with Python

5.4    Splitting Documents

Often, you need to split data into smaller chunks. Why is this step needed? A couple of reasons include the following:

• With splitting, you can get more granular results because smaller document chunks are more likely to contain focused and relevant information.

• You can work around limitations in embedding models. Don’t worry, we’ll cover this in more depth later in Section 5.5. Recall all embedding models have a limited context window of tokens to work with. We discuss tokens further in the next info box.

• Splitting can be more efficient in terms of database size and querying because you are working and storing smaller chunks.

No doubt you’ve encountered the word “token,” so let’s briefly define this concept. ...