Modern Distributed Database Architectures - Part 2

Beyond Foundational Distributed Database Architectures

In the first part of our exploration into modern distributed database architectures, we learned that a distributed database is a data management system that spreads data across multiple servers or nodes using two key mechanisms, sharding and replication, to achieve horizontal scalability and fault tolerance.

Previously, we examined two foundational designs: the primary-replica model and the shared-nothing architecture. We discussed how the primary-replica approach offers simplicity and read scalability but struggles with write-heavy workloads due to its lack of sharding. On the other hand, shared-nothing architectures excel in horizontal scalability and fault tolerance but come with challenges like metadata management and uneven data distribution. These architectures highlight the tradeoffs between scalability, consistency, and operational complexity that define distributed systems.

Today, we’ll dive into more trending and innovative architectures that address the evolving demands of modern data management. We’ll analyze their unique approaches to balancing scalability, consistency, and fault tolerance, as well as the tradeoffs they introduce. Whether you’re building a global-scale application or evaluating database solutions, understanding these architectures will help you make informed decisions. Let’s continue our journey!

Layered Architecture

The layered architecture is a modern approach to distributed databases that separates the system into distinct, specialized roles or layers, each responsible for a specific function and coordinating with other layers. This design often includes the following major roles:

• Metadata Manager (also known as Control Plane, Meta Server, Placement Driver, etc.): This component stores metadata about the cluster topology and the data distribution across storage nodes. It acts as the brain of the system, ensuring that queries are routed correctly and that the cluster operates efficiently. In the meantime, if there’s a topology change (i.e., adding a storage node to the cluster), the metadata manager also stores that and instructs the fellow storage nodes to perform data migration.
• Query Layer: The query layer is stateless, elastic, and horizontally scalable. It accepts incoming queries, authenticates them, translates them into an internal format, and generates a distributed execution plan. This layer acts as the interface between the user and the database, abstracting the complexity of the underlying distributed architecture.
• Storage Layer: The storage layer is responsible for persisting data. It typically implements a consensus mechanism (e.g., Raft or Paxos) to ensure data consistency across nodes and supports a simplified API, often key-value based, with distributed transactional capabilities.

The layered architecture is gaining popularity because it combines the benefits of centralized coordination with the scalability of distributed systems. However, there’s no silver bullet. With this comprehensive and sophisticated architecture, there are still pros and cons.

Advantages of a Layered Architecture

• Centralized Metadata Management: By centralizing metadata in the control plane, the system can efficiently manage data distribution, topology changes, and query routing. This simplifies coordination and reduces the overhead of decentralized metadata management.
• Layered Architecture: The separation of concerns into distinct layers allows each layer to scale independently. For example, the query layer can scale to handle more concurrent queries, while the storage layer can scale to accommodate larger datasets. Because of this, this type of database can normally observe much higher throughput.
• Flexibility and Modularity: The modular design makes it easier to upgrade or replace individual components without disrupting the entire system. This also allows for flexibility in choosing technologies for each layer.
• Distributed Transaction Support: The storage layer’s support for consensus and distributed transactions ensures strong data consistency, making this architecture suitable for applications requiring ACID guarantees even in a distributed environment.

Disadvantages of a Layered Architecture

• Increased Latency: Due to the layered architecture and the need for consensus mechanisms, users may observe slightly higher latency for individual requests compared to standalone systems. However, this tradeoff is offset by the system’s ability to handle much larger datasets and higher query workloads, drastically improving overall throughput and scalability.
• Increased Complexity: The presence of multiple components (metadata manager, query layer, storage layer) makes the system harder to develop, deploy, and maintain. Proper tooling and expertise are required to manage the complexity effectively.
• Potential Bottlenecks: While the metadata manager simplifies coordination, it can become a bottleneck if not designed to scale horizontally. High availability and fault tolerance for the control plane are critical to avoid single points of failure.
• Operational Overhead: Managing and monitoring multiple layers can introduce operational overhead, especially in large-scale deployments. Updated knowledge of this type of database, automated deployment and upgrade workflows, and robust monitoring tools are essential to mitigate this challenge.

Google Spanner, YugabyteDB, and TiDB are several great examples of this layered and orchestrated architecture. For instance, TiDB separates the query layer (TiDB server), storage layer (TiKV and TiFlash), and metadata management (Placement Driver) to provide horizontal scalability and strong consistency. This architectural division allows each layer to scale independently, enabling horizontal scalability to handle growing workloads.

TiDB Layered & Modularized Architecture (as of TiDB v8.5)

CockroachDB also features a multi-layered architecture as well, encompassing a SQL layer for query processing, a transactional layer ensuring ACID properties, a distribution layer for metadata and data partitioning, a replication layer for data consensus, and a storage layer for data persistence. Although its architecture is layered, CockroachDB operates more as a peer-to-peer system, distributing data algorithmically among the nodes in the cluster.

By separating concerns into distinct layers, this architecture promotes flexibility, modularity, and strong consistency, making it well-suited for modern applications demanding scalability, reliability, and geographically distributed data. While some consider it the ultimate architecture for distributed databases, the inherent tradeoffs—including increased latency, system complexity, and operational overhead—require careful management to fully realize its potential.

Shared-Storage Architecture

The shared-storage architecture is a modern database design that completely separates compute and storage layers, delegating the storage layer to highly scalable and durable object storage systems like Amazon Simple Storage Service (S3), Google Cloud Storage, and Azure Blob Storage. By doing so, the database system can focus on compute-intensive tasks (e.g., query execution and transaction management) while relying on object storage for infinite scalability and durability. With the recent launch of S3 Tables, it is becoming an even more suitable storage backend for distributed databases.

Advantages of a Shared-Storage Architecture

• Simplified Development: By offloading storage to systems like S3, database developers can focus on optimizing the compute layer or write-ahead logging (WAL) layer. This reduces the complexity of building and maintaining a distributed storage system. Obviously, this is a comparison to the layered architecture. Developing a database is still inherently hard compared to many other software sectors.
• Infinite Storage Scalability: Object storage systems like S3 provide virtually unlimited storage capacity, eliminating the need for manual scaling of the storage layer.
• Elastic Compute Layer: The compute layer can scale independently of the storage layer, allowing for fully elastic scaling based on workload demands. This is particularly useful for bursty or unpredictable workloads and ad-hoc complex analytical queries.
• Durability and Reliability: Object storage systems are designed for high durability (e.g., 99.999999999% durability for S3) and reliability, reducing the need for custom replication and backup mechanisms.
• Separation of Concerns: The clear separation between compute and storage simplifies system design, making it easier to upgrade or replace individual components without affecting the entire system, especially when the compute layer is stateless.

Disadvantages of a Shared-Storage Architecture

• Increased Latency: While object storage systems like S3 offer reasonable latency, they are still remote storage solutions over the network. This introduces slightly higher latency compared to local or dedicated storage systems.
• Limited Transactional Support: Object storage systems are not originally designed for database and transactional workloads. Databases using shared storage may need to implement their own mechanisms for consistency, concurrency control, and transactional guarantees.
• Network Dependency: The performance of the database is heavily dependent on network connectivity and the bandwidth between the compute layer and the remote storage. Network issues or latency spikes can impact overall system performance.
• Vendor Lock-In: Relying on a specific object storage provider can lead to vendor lock-in, making it harder to migrate to other cloud providers or on-premise platforms in the future. However, object storage services are often part of the private cloud bundle. And open-source alternatives to S3, such as MinIO, are also available.

The shared-storage architecture is exemplified by systems like Snowflake, Delta Lake, Databend, Greptime, and Neon. Snowflake and Delta Lake are pioneers in leveraging object storage for scalable data warehousing and transactional big data workloads. Greptime, a distributed time-series database, and Databend, a cloud-native data warehouse, combine elements of layered and shared-storage architectures. They use a control plane for metadata, cluster, and security management while relying on object storage for data persistence.

Databend Architecture (as of Databend v1.2)

On the other hand, Neon takes a unique approach by blending shared-nothing and shared-storage principles. Its magic lies in the WAL (write-ahead log) stream and storage layer, which is decoupled from compute nodes, enabling elastic scaling and cost efficiency for PostgreSQL workloads.

The shared-storage architecture is gaining traction because it aligns with the principles of cloud-native design: scalability, elasticity, and cost efficiency. By leveraging object storage, databases can offload the complexities of distributed storage management and focus on delivering high-performance compute capabilities. However, the tradeoffs—such as increased latency and network dependency—must be carefully considered when choosing this architecture for specific use cases.

A Shift to Hybrid Architectures

Modern databases are increasingly adopting hybrid models to balance flexibility and performance. OceanBase, for instance, supports both shared-nothing and shared-storage modes, adapting to diverse deployment environments.

OceanBase Architecture - Shared Storage (as of OceanBase v4.3.5)

Similarly, TiDB has evolved into a hybrid transactional-analytical processing (HTAP) system with TiFlash, enabling real-time analytics on transactional data without external ETL tools. TiDB’s serverless offering also leverages S3 for storage, merging scalability with cost efficiency.

These innovations reflect a broader shift: distributed databases now blend architectures (shared-nothing, layered, orchestrated, shared-storage, HTAP) and embrace object storage to optimize scalability, cost, and workload versatility. The future lies in modular designs that unify transactional, analytical, and cloud-native capabilities seamlessly.

Dragonfly’s Architectural Approach

Before we conclude, let’s revisit where Dragonfly sits in this architectural spectrum. At its core, a standalone Dragonfly instance is a multi-threaded, shared-nothing system—where "shared-nothing" specifically refers to threads avoiding heavy locking mechanisms to achieve millions of queries per second (QPS). For Dragonfly Cluster, we adopt an orchestrated and layered approach, employing a centralized, highly available control plane to manage cluster topology. As an in-memory data store, Dragonfly prioritizes throughput and low latency. While consensus across shards is avoided to minimize overhead, high availability is ensured through a primary-replica setup for each shard.

Dragonfly Cluster Architecture

Throughout this blog series, we’ve explored the tradeoffs of modern distributed database architectures. Each architecture addresses specific challenges in scalability, consistency, and fault tolerance. As distributed systems continue to evolve, the key takeaway is clear: understanding these tradeoffs empowers software developers and system architects to choose the right tools for their unique workloads. Whether you’re optimizing for latency, scalability, or cost efficiency, the right architecture can make all the difference.