Modern Databases: Polyglot Persistence and Vector DBs
The right database for each job beats one database for everything
The era when a single relational database was sufficient to handle all of an organisation's needs is over. Modern architectures use Polyglot Persistence: employing different database types (SQL, NoSQL, Graph, Vector) for the specific purposes each is best suited for.
Applications using Generative AI often need vector indexing to search semantic information efficiently. Whether a dedicated Vector DB or an extension (e.g. pgvector) fits depends on scale, latency, and operating model.
Anti-Patterns: The Database Monolith
- One Size Fits All: Trying to force unstructured data, time series, and vectors into a classic SQL database leads to poor performance and high complexity.
- Lack of scalability: Relational systems frequently hit their limits under massive parallel write loads (see Cloud Native).
- No semantic search: Classic databases search by exact terms (keywords) and, without extensions (e.g. pgvector), do not understand the semantic context of a query.
Specialised Storage
- Relational (SQL): Remains the standard for structured transactional data (orders, finance, master data), e.g. PostgreSQL.
- Document Store (NoSQL): For unstructured, rapidly changing data formats (e.g. MongoDB).
- Vector Databases: Store mathematical representations (Embeddings) of text, images, or audio for semantic similarity search (e.g. Weaviate, Pinecone, Milvus, or pgvector).
- Time Series Databases: Optimised for capturing massive volumes of telemetry data and logs (e.g. InfluxDB, Prometheus).
- Graph Databases: For highly connected data such as social networks or fraud detection (e.g. Neo4j).
The Focus: API-based Persistence
In a modern architecture, microservices do not access the databases of other services directly. Each service manages its own optimal persistence layer and exposes data via API.
FAQ
Doesn't Polyglot Persistence massively increase our maintenance overhead?
Yes, it requires more expertise. That is why we consistently rely on Managed Database Services from cloud providers. The gains in performance and feature coverage for new applications (such as AI) far outweigh the administrative overhead.
Can't I just use a vector plugin for my SQL database?
Yes, for many SME use cases that is an excellent starting point (e.g. pgvector for PostgreSQL). Dedicated Vector DBs only become strictly necessary at very large data volumes or with complex AI workflows.
References
- Martin Kleppmann Designing Data-Intensive Applications. Foundations of modern database paradigms and data systems. (2017). dataintensive.net
- Pinecone Vector Database Fundamentals. A guide for beginners to vector databases. (2023). www.pinecone.io/learn/vector-database/
- pgvector contributors pgvector: Open-source vector similarity search for Postgres. Vector search as an extension for PostgreSQL. (2023). github.com/pgvector/pgvector