If you've spent any time building AI applications in the past two years, you've encountered the term "vector database." You might have used one already without fully understanding what it is or why the architecture matters. This guide covers the fundamentals, the practical use cases, and the crowded market of options — without the hype.
The Problem Vector Databases Solve
Language models don't have persistent memory. Ask GPT-4 a question today, and it knows nothing about the conversation you had yesterday, the documents you showed it last week, or the internal knowledge base your company has built up over years. Every conversation starts fresh. This is a fundamental limitation for enterprise applications, where the AI needs to "know" your proprietary information.
The standard solution is Retrieval Augmented Generation (RAG): you store your documents in a vector database, convert user queries to vectors, find the most semantically similar document chunks, and inject them into the model's context window. The model appears to "know" your data because you give it the relevant information at query time. It's not actual memory, but it works remarkably well in practice.
What Is a Vector, Anyway?
An embedding model (like OpenAI's text-embedding-3-large) converts text into a list of numbers — typically 1,536 or 3,072 numbers — called a vector. These numbers encode the semantic meaning of the text, such that similar concepts map to vectors that are mathematically "close" to each other. The sentence "the cat sat on the mat" and "a feline rested on a rug" would have very similar vectors, even though they share almost no words.
A vector database is purpose-built to store millions of these vectors and answer the question "which of my stored vectors are most similar to this query vector?" very quickly — typically using approximate nearest neighbor algorithms like HNSW (Hierarchical Navigable Small World graphs).
The Main Options: A Practical Comparison
Pinecone: The managed vector database market leader. Fully hosted, scales automatically, strong developer experience. Best for: teams that want zero operational overhead and are willing to pay for it. Pricing starts at $0.096/hour for a pod; can get expensive at scale.
Weaviate: Open-source with managed hosting option. More feature-rich than Pinecone (built-in modules for text vectorization, hybrid search, knowledge graphs). Best for: teams that want flexibility and are comfortable with more configuration.
Chroma: Lightweight, open-source, designed for local development. Runs in-memory or with a local disk store. Best for: prototyping and small-scale applications where managed infrastructure isn't needed yet.
pgvector: A PostgreSQL extension that adds vector similarity search to your existing Postgres database. Best for: teams already running Postgres who want to avoid adding another managed service. Performance is good for datasets up to ~1 million vectors; at larger scale, purpose-built vector DBs pull ahead.
Qdrant: Rust-based, extremely fast, open-source. Supports payload filtering (combining vector similarity with metadata filters) better than most competitors. Best for: high-performance applications where latency is critical and teams are comfortable with self-hosting.
When You Don't Need a Vector Database
Not every AI application needs a vector database. If your application has a small, fixed knowledge base (under 100 documents), you can often stuff the entire knowledge base into the context window at query time — simpler and often adequate. If you're building a conversational agent that only needs short-term conversation memory, a simple key-value store or even an in-memory dict is sufficient. Reach for a vector database when your knowledge base is large (1,000+ chunks), frequently updated, or when retrieval latency matters.