Dragonfly: The Future-Proof Alternative to Redis

Redis has been a cornerstone in the world of in-memory data stores for over 15 years, providing the backbone for countless applications. However, as the technological landscape evolves, Redis's limitations have become increasingly apparent. Modern applications demand more than what Redis's single-threaded architecture can offer, struggling to scale efficiently under the pressure of extremely heavy workloads. This architecture bottleneck not only affects performance but also results in higher costs and increased complexity in management.

Moreover, in the world of open-source software, trust is paramount. Redis's licensing changes earlier this year have sparked concerns within the developer community, shaking the confidence that many had in this technology. As a result, these changes have introduced uncertainty and hesitancy, prompting many to seek alternatives that offer stability and a clear path forward.

As the community scrambles for alternatives, it's essential to consider two simple but crucial factors: better technology and permissive licensing. In this blog post, we'll explore why Dragonfly stands out as an ideal, future-proof alternative to Redis.

A Modern Take: Redis Limitations and Dragonfly Advancements

Redis Limitations

Redis, originally designed 15 years ago, was revolutionary for its time, but many of its design decisions no longer align with the demands of modern infrastructure. One significant limitation is its single-threaded architecture. Although Redis has introduced child threads for supportive tasks such as network multiplexing, RDB snapshotting, AOF (Append-Only File) rewriting, etc., the core functionality of in-memory data manipulation remains single-threaded to ensure atomic operations. This architectural choice poses significant challenges in today's environment, where in-memory data platforms must respond to requests with sub-millisecond latency at millions of ops/sec. Implementing a locking mechanism for multi-threading to avoid contention in such a high-performance context is not a trivial task. Dragonfly solves this problem with an innovative approach, and we will discuss more on this later.

Another critical area where Redis falls short is its snapshotting algorithm. Redis relies on the system fork() call for snapshotting, which uses a copy-on-write mechanism. While effective in theory, this approach can lead to significant memory spikes and necessitate over-provisioning in write-heavy instances. This inefficiency can be costly and challenging to manage. Dragonfly offers a more efficient snapshotting mechanism that avoids these pitfalls, ensuring stable and predictable performance even under intense write operations.

Dragonfly's Technical Advancements

Dragonfly is a drop-in Redis replacement designed to meet the performance and efficiency requirements of modern cloud applications. Through the implementation of modern data structures and a cutting-edge multi-threaded shared-nothing architecture, Dragonfly delivers 25x the performance of Redis at up to 80% lower infrastructure costs.

• Multi-Threading: Dragonfly's multi-threading capabilities set it apart from Redis. By utilizing multiple threads and an elegant lightweight locking mechanism, Dragonfly can handle more operations concurrently and fully utilize the underneath multi-core hardware. This makes it an ideal choice for applications with high demand for performance and scalability.

• Dashtable: Dragonfly's dashtable data structure is a significant advancement. It allows for more efficient data retrieval and storage, improving overall performance. Dashtable avoids the pitfalls of traditional hash tables, such as expensive rehashing, and is designed to handle large volumes of data with minimal impact on speed and extremely low per-entry overhead.

• B+ Tree-based Sorted Sets: By using B+ trees for sorted sets, Dragonfly offers better performance and memory efficiency. B+ trees are optimized for read and write operations with built-in ordering. The per-set-member overhead is also significantly reduced, so that the sorted set memory usage can be lowered by up to 40%.

• Snapshotting Algorithm: Dragonfly's snapshotting algorithm ensures reliable data persistence. The algorithm is designed to take snapshots of the data with minimal impact on performance, ensuring that backups are quick and efficient without spiking memory usage during the process even under heavy write loads.

• Cluster Mode: Dragonfly's cluster mode provides seamless scalability. While a single Dragonfly instance scales vertically to fully use a single multi-core server, clustering allows Dragonfly to scale horizontally by sharding the in-memory data across multiple nodes. This ensures that the system can handle increasing loads without compromising performance.

Beyond these technical advancements, Dragonfly also provides rich data structures and features, such as JSON and Search, out-of-the-box without additional extensions, which ensures a top-tier developer experience. Teams adopting Dragonfly will benefit from:

• Better Performance: Thanks to multi-threading and optimized data structures, Dragonfly outperforms Redis in many scenarios. The multi-threading capability alone makes Dragonfly a powerhouse, efficiently handling 6.43 million ops/sec and terabyte-level in-memory data on a single instance.

• Cost Efficiency: Dragonfly's efficient use of resources translates to cost savings, particularly in large-scale deployments. Its ability to handle more operations with fewer resources means lower operational costs and better utilization of hardware. For common scenarios, we can observe a 40% reduction in cost compared to AWS ElastiCache.

• Scalability, Reliability and Simplicity: With Dragonfly's high compatibility with the Redis ecosystem and robust architecture, Dragonfly ensures scalability and high availability. The multi-threaded shared-nothing nature ensures vertical scalability. The snapshotting algorithm minimizes performance impacts during backups. And if your workloads require more than 1TB of in-memory data, the cluster mode ensures further horizontal scalability. Unlike Redis, you never have to manage a cluster on a single multi-core machine, which leads to simpler operations and infrastructure management.

Licensing Hurdles: Redis's Shift and Dragonfly's Balanced Approach

Redis License Changes

On March 20, 2024, Redis announced that all future versions of Redis would be released under source-available licenses. Under the new terms, cloud service providers hosting Redis offerings will no longer be permitted to use Redis source code free of charge.

Redis's decision to alter its licensing model caused an uproar within the community, with many feeling a deep sense of betrayal. Developers and contributors who had invested significant time and effort into the project now faced uncertainty. This licensing change led to widespread chaos and concern, with many developers feeling that their trust had been breached, resulting in a scramble for alternatives.

In response, Valkey, a relatively young fork of Redis, and KeyDB, an attempt at multi-threading based on Redis, quickly emerged as potential substitutes. An article that went viral and a subsequent HackerNews thread emphasized licensing issues but overlooked critical technical advancements. While licensing is undoubtedly a crucial aspect, it should not overshadow the technical merits of an alternative. Neither Valkey nor KeyDB fundamentally changed the underlying architecture of Redis. Moreover, development around KeyDB has not been very active since its acquisition by Snap.

Dragonfly's Approach to Licensing

Dragonfly's licensing strategy offers a key advantage for developers. Dragonfly operates under the Dragonfly Business Source License (BSL), which, although not classified as open-source by the Open Source Initiative (OSI), is intentionally designed to be very permissive. Despite the word "business" in its name, the BSL allows developers to use and modify Dragonfly freely, provided they do not offer it as a managed service. Moreover, each new version of Dragonfly will eventually be converted to the Apache 2.0 license, ensuring it remains fully open-source in the future.

This licensing model was chosen from day one to create a robust, open-minded, source-available technology backed by a sustainable business model. It provides developers with a reliable and transparent roadmap without any surprises. By balancing permissiveness with sustainability, Dragonfly joins many other projects and organizations in exploring a new path for the future of open software, offering a stable and forward-thinking licensing strategy.

The Future of In-Memory Data Infrastructure

The Redis license change has created a critical need for a reliable and future-proof alternative. While the initial reactions to its license change have been driven by a sense of trust-brokenness, it's crucial to consider both licensing and technical merits in selecting an alternative. Dragonfly is a choice that combines trustworthiness with technological brilliance.

For developers, database administrators, DevOps, and platform engineers seeking a robust, high-performance, and reliable in-memory data store, Dragonfly is the clear path forward. Its innovative data structures and robust modern architecture make it a standout choice. Furthermore, its liberal licensing approach ensures that it will continue to be a valuable asset to the OSS community.

Try Dragonfly today and experience the difference in performance, efficiency, and reliability. Visit the Dragonfly documentation page to get started, or explore Dragonfly Cloud for an easy and managed setup. Embrace the future of in-memory data with Dragonfly and ensure your infrastructure is ready for the challenges of today and tomorrow.