Generative AI with Python

5.6    Storing Data

We are reaching our final state of the data ingestion pipeline. Let’s recap what we’ve achieved so far. First, we loaded some documents. Second, we created smaller chunks out of a large text corpus. Third, we learned how to create vector embeddings based on text chunks. So now, we can store them in a database.

You have multiple options to store the data. The main decision you must make is whether you want to host your own vector database locally or if you want to rely on web-based service providers. Both approaches have their pros and cons.

5.6.1    Selection of a Vector Database

The basic functionality of a vector database is handled similarly by different providers. But they all have their unique selling propositions; the ...