book

Vector Databases

Name: Vector Databases
Author: Nitin Borwankar
ISBN: 9781098177591

by Nitin Borwankar

April 2026

Intermediate to advanced

292 pages

6h 42m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Includes

Quizzes

Preface
What’s in This BookWho This Book Is ForHow to Use This BookSoftware, Environment, and Resource RequirementsConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Introduction to Vector Databases
Why Do You Need Vector Databases?A New Data Type: VectorSimilarity SearchWhat’s Different About the Vector Type?Where Do You Use Vector Databases?SQL Versus Vector DatabasesThe Foundation of Business Math: Accounting ArithmeticVector Representation in a Relational Database Management SystemThe Need for Vector-Specific CapabilitiesNoSQL Versus Vector DatabasesNoSQL Databases and Vector StorageLimitations of Vector Extensions in NoSQL DatabasesWhen to Choose NoSQL with Vector ExtensionsHybrid Approaches: Combining Structured and Vector DataThe Need for Both Vector Data and MetadataLimitations of Pure Vector StorageHybrid Database ArchitectureExample of a Hybrid QueryBenefits of the Hybrid ApproachConclusion
2. Embeddings
Understanding Vector Embeddings: Why We Need ThemWord2Vec: The Breakthrough That Changed EverythingDoc2Vec: From Words to DocumentsFrom Embeddings to Modern Language Models: The Transformer ConnectionEncoder-Only Transformers (BERT and Its Variants)Decoder-Only Transformers (GPT Family)Encoder-Decoder Transformers (T5, BART)Embedding Models: The Specialized Vector GeneratorsDistinction from Traditional ModelsRole in Modern LLM ApplicationsPractical Applications and Use CasesSimple RAG PipelineThe sentence-transformers Library: The Swiss Army Knife of Text EmbeddingsBest Practices for Using SentenceTransformers: A Detailed GuideThe Embedding Layer: The Gateway to Zero-Shot LearningAnatomy of Transformer EmbeddingsConnection to Zero-Shot LearningKey Characteristics That Enable Zero-Shot LearningLimitations and ConsiderationsLatest Developments and TrendsVector Arithmetic with Word2Vec: A Hands-On GuideStep 1: Setup and InstallationStep 2: Load Pretrained Word2Vec ModelStep 3: Implement Vector Arithmetic FunctionsStep 4: Classic King–Queen AnalogyStep 5: More Interesting AnalogiesStep 6: Interactive Exploration ToolFinal Words on Vector ArithmeticConclusion
3. Similarity Search with FAISS
FoundationsVector RepresentationsDistance MetricsSelection HeuristicsFAISS IndexesFlat Indexes (Brute Force)IVF-Based IndexesLSH-Based IndexesHNSW-Based IndexesOther Specialized IndexesComposite and Transformative IndexesChoosing the Right IndexQuantizationSQPQThe ANN ProblemThe ProblemAvoid Computational CostKey ANN Techniques in FAISSChoosing an Index in FAISSCode ExampleUnderstanding HNSW IndexesWhat Is HNSW?How HNSW WorksKey Parameters ExplainedPractical Example: Building a Similarity Search SystemPerformance CharacteristicsBest PracticesFAISS Architecture and ComponentsFoundationCore ConceptsKey ComponentsCommon WorkflowIllustrative ExampleKey TakeawaysFurther ExplorationConclusion
4. Semantic Search with SQLite3
Understanding the SQLite Vector Similarity Search ExtensionCore CapabilitiesArchitecture OverviewLimitationsSetting Up the Development EnvironmentInstalling DependenciesVerifying the InstallationOperational PragmasDesigning the Database SchemaSchema RequirementsTable DefinitionsSchema Design DecisionsConnecting to Reddit with the Python Reddit API WrapperCreating Reddit API CredentialsPRAW Client ImplementationUsage ExampleContent Extraction and PreprocessingText Cleaning PipelineQuality FilteringGenerating and Storing EmbeddingsEmbedding GeneratorDatabase StorageBatch Processing PipelineBuilding the Vector IndexUnderstanding VSS IndexingIndex ManagementImplementing Semantic SearchSearch Result ContainerSearch EnginePutting It All TogetherWorkflow ExampleExample OutputExtension: Incremental IndexingConclusion
5. Building an ArXiv Paper Search System with PostgreSQL pgvector
The Challenge of Searching Scientific LiteratureWhy ArXiv Makes an Ideal Data SourceReal-World Use CasesTechnology Stack RationaleArchitecture OverviewSystem ComponentsData FlowDesign PhilosophyEnvironment Setup and DependenciesPostgreSQL and pgvector InstallationPython Environment ConfigurationDirectory Structure and ConfigurationVerification and TestingDatabase Design for Scientific PapersSchema Design PrinciplesCore Tables StructureVector Storage StrategyIndexing StrategyArXiv Integration and PDF ManagementArXiv API Client ImplementationPDF Download PipelineBatch Processing SystemPDF Text Extraction and ProcessingPDF Extraction ChallengesIntelligent Text ChunkingEmbedding Generation and StorageEmbedding Model StrategyBatch Processing PipelineSimilarity Search ImplementationInteractive Application and UIDocker Packaging for Local DeploymentContainer ArchitectureDocker Compose ConfigurationDatabase Initialization ScriptsDevelopment WorkflowCloud-Ready DesignBasic Performance TuningIndex ConfigurationQuery PerformanceResource ManagementNext StepsCurrent LimitationsEnhancement IdeasWhat We DidSystem AchievementsTechnical Skills GainedPractical Research ToolFoundation for Advanced SystemsFuture PotentialConclusion
6. Building a Retrieval-Augmented Generation System with SQLite VSS and Ollama
System Architecture OverviewDatabase Foundation with Vector SupportSetting Up the Vector-Enabled DatabaseSchema Design for RAGCreating Search IndexesText Processing and Embedding GenerationEmbedding Model ManagementIntelligent Text ChunkingStoring Content with EmbeddingsHybrid Search ImplementationHybrid Search AlgorithmSemantic Search ComponentKeyword Search ComponentScore Fusion and RankingLLM Integration with OllamaOllama API ClientHealth Check FunctionThe RAG PipelineContext FormattingQuestion-Answering PipelineDemonstration and TestingSample Data LoadingMain Demonstration FunctionInteractive Q&A InterfaceQuick Testing UtilityNext Steps: Extending the SystemMissing Reddit Data FeaturesPerformance OptimizationsProduction ConsiderationsAdvanced RAG PatternsConclusion
7. Building a Scientific RAG System with PostgreSQL and pgvector
System Goals and CapabilitiesArchitecture OverviewDatabase Foundation with pgvectorDatabase Configuration and SetupSchema Design for Scientific PapersHigh-Performance Vector IndexesEmbedding Generation StrategyArXiv Integration and PDF ProcessingPaper Discovery with ArXiv APIIntelligent PDF Text ExtractionAdvanced Text ChunkingStorage Pipeline with EmbeddingsMultilevel Semantic SearchAbstract-Level SearchSection-Level SearchThe RAG Pipeline: Deep DiveLocal LLM Integration with OllamaHealth Check and Model DiscoveryIntelligent Context RetrievalScientific Prompt EngineeringComplete RAG Execution PipelineDemonstration and Interactive InterfaceMain Demonstration FlowSearch DemonstrationsRAG DemonstrationInteractive Search InterfaceEntry Point with Mode SelectionTechnical Note on HNSWHow to Evaluate Your ResultsNext Steps: Extending the Scientific RAG SystemConclusion
8. Building a Complete Conversation Search and RAG System
System Goals and CapabilitiesSystem Architecture OverviewWhat We’ll Build TogetherDatabase Foundation for Conversation StorageDesigning the Conversation SchemaThree-Table Architecture for Optimal PerformanceHigh-Performance Vector IndexingConversation Import and Data Processing PipelineRobust JSON Import with Error HandlingAtomic Transaction ProcessingTimestamp Handling and Data ValidationError Recovery and LoggingEfficient Embedding Generation and Batch ProcessingSingleton Pattern for Model ManagementIncremental Processing StrategyBatch Processing for Optimal PerformanceDatabase Insertion with Conflict HandlingContextual Search with Conversational UnderstandingSemantic Similarity SearchMultitable Joins for Rich ContextResult Formatting and StructureConversation Context RetrievalContext Window CalculationRAG Integration for Conversation HistoryStructured Context ManagementLocal LLM Integration with OllamaHealth Monitoring and Model DiscoveryContext Retrieval and AssemblyConversational Prompt EngineeringComplete RAG Pipeline with Performance MonitoringComplete Web API with FastAPIFastAPI Application StructureRequest Models with ValidationSearch Endpoint ImplementationRAG Question-Answering EndpointSystem Statistics and MonitoringServer Startup and ConfigurationDemonstration and Sample DataRealistic Sample Data GenerationMultitopic Sample CoverageSample Data Processing PipelineComprehensive System DemonstrationProgressive Feature DemonstrationRAG Demonstration with Conditional ExecutionProduction Import FunctionalityApplication Entry PointsConclusion: A Complete Personal Knowledge System
9. Vector Query Language
Core ConceptsData ModelBasic Syntax StructureVector OperationsSimilarity SearchHybrid SearchRange SearchBatch OperationsVector Functions and AggregationsVector FunctionsVector Aggregations

Index
About the Author

Content preview from Vector Databases

Chapter 7. Building a Scientific RAG System with PostgreSQL and pgvector

The explosion of scientific publications presents researchers with an overwhelming challenge: how to efficiently discover, understand, and synthesize relevant knowledge from millions of papers. ArXiv alone publishes over 15,000 papers monthly across physics, mathematics, computer science, and other fields. Traditional keyword search fails to capture the semantic richness of scientific discourse, where the same concept may be expressed in countless ways across different domains and research communities.

This chapter builds a RAG system specifically designed for scientific literature. Unlike general-purpose RAG systems, scientific RAG must handle unique challenges:

Technical terminology: Papers use precise, domain-specific language that requires semantic understanding beyond simple keywords.
Structured content: Scientific papers follow conventions (abstract, methodology, results, conclusions) that can inform retrieval strategies.
Citation networks: Papers exist in a web of references that provide additional context.
Mathematical notation: Formulas and equations carry meaning that is not captured by this implementation. Handling mathematical notation requires specialized tools like LaTeX OCR.
Evidence quality: Not all sources are equal—peer review, venue, and citation count matter.

System Goals and Capabilities

Our scientific RAG system will enable the following:

Semantic discovery: Find papers based on conceptual ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098177584Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Vector Databases

by Nitin Borwankar

Chapter 7. Building a Scientific RAG System with PostgreSQL and pgvector

System Goals and Capabilities

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.