Generative AI with Python

5.2    Data Ingestion Process

Let’s start with a high-level overview of how data is ingested into a vector database. You’ll encounter these steps whenever you want to add data to a vector database:

• Data loading

• Data chunking

• Embedding

• Storing

You always start with a data source. We touched upon the many data type options—generally, any kind of text-based source, like Markdown files, Word documents, or raw text files can be used. But you can also use more specific data types, like a YouTube video from which you want to extract a transcript, or a Wikipedia article. The options are nearly endless. Luckily, the community of users contributing specific data handlers is large as well, so that hundreds of different sources can be used out ...