"We need a vector database" is one of the most common things we hear from a prospective client before we've actually looked at their data. Sometimes it's true. Often, a spreadsheet they already own does the job, and the vector database would have been unnecessary infrastructure to build and maintain.
What a vector database is actually for
Retrieval-augmented generation (RAG) needs a way to find the most relevant chunk of text out of a much larger body of source material, fast, based on meaning rather than exact keyword matches. A vector database — Pinecone, in our stack — stores content as embeddings (numerical representations of meaning) and can retrieve the closest matches to a query almost instantly, even across a large, constantly-changing knowledge base.
That's exactly the shape of the problem in our retrieval-augmented support email agent: a brand's FAQ and policy library, embedded with Cohere and stored in Pinecone, gets searched fresh for every incoming email so the model's reply is grounded in real source material rather than guessed. The knowledge base there is large enough and general-purpose enough that "search by meaning" earns its keep.
What a vector database is not for
A vector database solves unstructured search — finding the right paragraph in a pile of documents. It does not solve structured lookup — finding the right row in a table you already know the shape of. If your question is "how many students enrolled this month" or "what's this customer's plan tier," that's a query against structured data, and a vector store is the wrong tool: slower, harder to get exactly right, and adding infrastructure for a problem a spreadsheet already solves.
That's the exact design decision behind the analytics chat app we built for Salsa te Gusta. Their business questions were structured — season-over-season enrollment pacing, try-out-to-course conversion — and the data already lived in Google Sheets. Rather than moving it into a database "because AI," we built the app to call typed, purpose-built analytics tools that read directly from those sheets. The model orchestrates and explains; it never freehand-calculates from raw rows, and every number it states traces back to a real spreadsheet cell. No vector database, no new infrastructure — because the actual problem was never "find the right paragraph," it was "get the right number, provably."
The decision, in practice
Ask what kind of question the system needs to answer:
- "Find the passage that's relevant to this query" — unstructured, semantic search. A vector database is the right tool.
- "Compute or look up a specific fact from data you already have structured" — structured lookup. A spreadsheet, a proper database with typed queries, or a tool-calling pattern like the one above is usually simpler, cheaper, and more trustworthy, because every answer can point back to its exact source.
The mistake to avoid isn't picking the wrong database — it's reaching for a vector store as a reflex because "AI" and "database" sound like they go together, when the actual bottleneck was never search quality. Start from the question you're trying to answer, not the architecture that sounds most impressive.