Modern Databases: Polyglot Persistence and Vector DBs
The era when a single relational database was sufficient to handle all of an organisation's needs is over. Modern architectures use Polyglot Persistence — employing different database types (SQL, NoSQL, Graph, Vector) for the specific purposes each is best suited for.
For applications using Generative AI in particular, Vector DBs have become an indispensable building block for efficiently storing and searching semantic information.
Anti-Patterns: The Database Monolith
- One Size Fits All: Trying to force unstructured data, time series, and vectors into a classic SQL database leads to poor performance and high complexity.
- Lack of scalability: Relational systems frequently hit their limits under massive parallel write loads (see Cloud Native).
- No semantic search: Classic databases only find exact terms (keywords) — they do not understand the context or meaning of a query.
Specialised Storage
- Relational (SQL): Remains the standard for structured transactional data (orders, finance, master data) — e.g. PostgreSQL.
- Document Store (NoSQL): For unstructured, rapidly changing data formats (e.g. MongoDB).
- Vector Databases: Store mathematical representations (Embeddings) of text, images, or audio for semantic similarity search (e.g. Weaviate, Pinecone, Milvus, or pgvector).
- Time Series Databases: Optimised for capturing massive volumes of telemetry data and logs (e.g. InfluxDB, Prometheus).
- Graph Databases: For highly connected data such as social networks or fraud detection (e.g. Neo4j).
The Focus: API-based Persistence
In a modern architecture, microservices do not access the databases of other services directly. Each service manages its own optimal persistence layer and exposes data via API.
FAQ
Doesn't Polyglot Persistence massively increase our maintenance overhead?
Yes, it requires more expertise. That is why we consistently rely on Managed Database Services from cloud providers. The gains in performance and feature coverage for new applications (such as AI) far outweigh the administrative overhead.
Can't I just use a vector plugin for my SQL database?
Yes — for many SME use cases that is an excellent starting point (e.g. pgvector for PostgreSQL). Dedicated Vector DBs only become strictly necessary at very large data volumes or with complex AI workflows.
Reference Guide
- Designing Data-Intensive Applications: Martin Kleppmann on database paradigms. O'Reilly
- Vector Database Fundamentals: A guide for beginners. Pinecone Learning
- PostgreSQL pgvector: Vector search in the most popular open-source DB. GitHub