Vector databases get treated as an interchangeable commodity — pick one, plug it into your RAG pipeline, done. In practice, indexing strategy and the specific database's filtering capabilities have a real, measurable effect on retrieval quality, and the "just use Pinecone" default advice skips decisions worth actually making deliberately.
Table of Contents
- What a Vector Database Actually Does
- Approximate Nearest Neighbor: The Core Tradeoff
- pgvector vs Dedicated Vector Databases
- Filtering: The Underrated Deciding Factor
- Comparison Table
- Indexing Strategy Choices
- Common Mistakes
- FAQs
- Key Takeaways
What a Vector Database Actually Does
At its core, a vector database stores high-dimensional embedding vectors and answers "which stored vectors are most similar to this query vector" quickly, at scale. The "at scale" part is what actually differentiates it from just computing cosine similarity in a loop — a naive brute-force comparison against every stored vector becomes too slow past a few thousand vectors, so vector databases use approximate nearest neighbor (ANN) indexing to make similarity search fast even across millions of vectors, at the cost of the "approximate" part: results aren't guaranteed to be the mathematically exact top-k, just a very close approximation in exchange for dramatically better speed.
Approximate Nearest Neighbor: The Core Tradeoff
Most vector databases use HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) indexing under the hood — both trade some recall (the chance of finding the true nearest neighbors) for large speed gains. The practical tunable is usually something like ef_search or a similar parameter controlling how exhaustively the index searches — higher values improve recall at the cost of latency. For most RAG use cases, the default settings are reasonable starting points, but if retrieval quality feels off, tuning this parameter before assuming the embedding model itself is the problem is worth trying first.
pgvector vs Dedicated Vector Databases
If you're already running Postgres, pgvector is a genuinely strong default — it avoids adding a new infrastructure dependency, supports HNSW indexing for reasonable ANN performance, and lets you join vector search results against your existing relational data in a single query, which dedicated vector databases typically can't do natively.
Dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus) earn their complexity at larger scale (tens of millions+ vectors), when you need features pgvector doesn't provide well (built-in hybrid search, more sophisticated metadata filtering at scale, multi-tenancy isolation), or when vector search is genuinely the primary workload rather than a feature bolted onto an existing relational database.
Filtering: The Underrated Deciding Factor
The question that matters more than raw similarity search speed for most real applications: how well does the database support filtering by metadata alongside vector similarity — e.g., "find similar documents, but only within this user's accessible documents" or "only from the last 30 days." Naive implementations do vector search first, then filter results afterward, which can return too few (or zero) results if the filter is restrictive and the initial similarity search didn't happen to surface matching candidates. Better implementations support filtering during the index traversal itself, ensuring you get the top-k results that satisfy the filter, not the top-k overall results that happen to pass a filter applied afterward. This distinction directly affects retrieval quality for any multi-tenant or permission-scoped system — see my RAG piece for why access-control-aware filtering at retrieval time specifically matters for security.
Comparison Table
| Database | Best Fit | Filtering Quality | Operational Overhead |
|---|---|---|---|
| pgvector | Already on Postgres, moderate scale, need relational joins | Good (native SQL WHERE alongside vector search) | Low — no new infrastructure |
| Pinecone | Managed, minimal ops burden, large scale | Good, metadata filtering built in | Very low (fully managed) |
| Weaviate | Need hybrid search and filtering as first-class features | Strong, purpose-built for this | Moderate (self-hosted or managed) |
| Qdrant | Performance-focused, open-source preference | Strong | Moderate |
Indexing Strategy Choices
Beyond choosing a database, the index type itself matters: HNSW generally offers better recall/speed tradeoffs than IVF for most workloads and is the default in most modern vector databases, but requires more memory. For very large datasets where memory is a genuine constraint, IVF or a hybrid approach (IVF with product quantization to compress vectors) trades some recall for meaningfully lower memory footprint — a real consideration once you're indexing tens of millions of vectors, less relevant at smaller scale.
Common Mistakes
- Choosing a dedicated vector database before actually needing one, adding infrastructure complexity that pgvector would have handled fine at the team's actual scale.
- Filtering after vector search instead of during it, silently returning fewer or lower-quality results than expected for filtered queries — a common, hard-to-notice retrieval quality bug.
- Never re-indexing after embedding model changes. Switching embedding models means all existing vectors were generated by a different model and are not directly comparable to new queries embedded with the new model — a full re-embed and re-index is required, not optional.
- Assuming higher-dimensional embeddings are always better. Higher dimensions increase storage and search cost without guaranteed retrieval quality improvement for a given task — benchmark against your actual data rather than assuming dimension count is a quality proxy.
FAQs
Do I need a vector database for a small RAG project? For a genuinely small document set (a few thousand chunks), even a simple in-memory or file-based vector store can work adequately for early development — pgvector is a reasonable next step once you need persistence and better performance without adding new infrastructure.
Is Pinecone worth its cost compared to self-hosted alternatives? Pinecone's value is in removing operational burden entirely — worth it if you don't want to manage infrastructure. Self-hosted Weaviate or Qdrant can be more cost-effective at scale if you have the operational capacity to run them.
Can I switch vector databases later without much pain? Generally yes for the vector storage/search layer itself, since the surrounding embedding and chunking logic is database-agnostic — the actual migration work is mostly re-indexing existing vectors into the new store, not rearchitecting your application logic.
How much does embedding dimensionality affect cost? Directly — storage and compute cost scale with vector dimensionality, so a 1536-dimension embedding model costs meaningfully more to store and search at scale than a 384-dimension one, which matters when comparing embedding model options for large datasets.
Key Takeaways
- Vector databases trade exact nearest-neighbor accuracy for speed via approximate nearest neighbor indexing — this tradeoff is tunable, not fixed.
- pgvector is a strong default if you're already on Postgres; dedicated vector databases earn their complexity at larger scale or when you need advanced filtering/hybrid search as first-class features.
- Filtering during index traversal (not after) is the deciding factor for retrieval quality in any permission-scoped or multi-tenant system.
Related Reading
- RAG Explained: When Retrieval Actually Helps
- Multi-Agent Systems with LangGraph: A Practical Introduction
- Prompt Engineering Patterns That Hold Up in Production
Next Steps
Choosing a vector database or debugging retrieval quality in an existing RAG system? Let's talk about what fits your actual scale and requirements.