Announcing Dragonfly Swarm: Scaling In-Memory Data Without Limits

For over 15 years, Redis (and now its fork, Valkey) has been the backbone of in-memory workloads, with Redis Cluster mode serving as the default approach for horizontal scaling. But as applications continue to grow in scale and complexity, Redis Cluster has increasingly become a source of operational challenges, performance bottlenecks, and rising infrastructure costs. Managing a large Redis deployment often means navigating intricate shard balancing, excessive metadata overhead, and balance over-provisioning just to maintain stability.

Until now, there has been no viable alternative.

Today, we’re changing that with the launch of Dragonfly Swarm.

Goodbye, Redis Cluster. Hello, Dragonfly Swarm!

Dragonfly was built to redefine in-memory data performance, eliminating the pain points of traditional single-threaded architectures by maximizing modern multi-core hardware. With the launch of Dragonfly Swarm, we expand the capabilities of Dragonfly beyond the boundaries of a single server.

Available immediately in all AWS, GCP, and Azure regions supported by Dragonfly Cloud, Dragonfly Swarm is the first in-memory data solution that seamlessly scales both vertically and horizontally to deliver terabyte-scale in-memory real-time data. It maintains the simplicity, efficiency, and raw performance that Dragonfly is known for.

Why Dragonfly Swarm is a Game Changer

The Future of Scaling: Vertical + Horizontal = Infinite Possibilities

With Dragonfly Swarm, we combine the best of both worlds—vertical and horizontal scaling. Unlike Redis, which requires horizontal scaling at relatively small sizes^[1], a single Dragonfly shard can fully utilize all CPU cores within a single instance before it needs to scale out. Vertical scaling on Dragonfly can handle terabyte-size workloads with over one million requests per second. Dragonfly Swarm enables frictionless horizontal expansion, allowing you to scale your workloads beyond 100TB of memory and 100M requests per second (RPS).

Radical Simplicity

Dragonfly Swarm minimizes cluster overhead, ensuring optimal resource efficiency. It reimplemented the slot migration algorithm, utilizing multiple parallel TCP connections for this process, leading to significantly faster scaling operations than Redis Cluster. Additionally, by centralizing cluster metadata management in a control plane and abandoning the gossip protocol approach, Dragonfly Swarm accelerates node discovery during cluster changes.

Finally, vertical scaling of a Dragonfly shard reduces the number of shards required for the same workload and reduces the hardware footprint of the cluster. 

Unparalleled Scale & Performance Designed for Cloud

Smaller cloud server instances have weaker networking capabilities and often suffer from noisy neighbor effects. Dragonfly Swarm eliminates the need for premature horizontal scaling by allowing it to grow vertically and use more powerful server instances first before scaling out horizontally. Each Dragonfly shard can fully utilize all its CPUs and networking capabilities because of Dragonfly’s multi-threaded shared-nothing design. Vertical scaling dramatically improves your cluster’s reliability and makes it more robust against traffic spikes and sporadic overloads.

Dragonfly Swarm enhances cost efficiency by using larger nodes, which decreases the volatility of memory and CPU loads. This allows for optimizing provisioning margins in each cluster shard without sacrificing reliability.

"We saw a 30X speed-up compared to managed Redis and a 10X reduction in cost. Only took us about a day to integrate. No-brainer for anyone currently using Redis."
— Raine Scott, Co-Founder & CPO at Verisoul

Who Should Use Dragonfly Swarm?

Dragonfly Swarm is suitable for workloads that require gradual updates during cluster resize operations or maintenance updates or if datastore capacity has reached its single-node limits. Dragonfly Swarm is designed for teams that require a scalable, high-performance in-memory data store solution, specifically those that:

• Manage large Redis Clusters (>200GB memory or hundreds of thousands of RPS)
• Face high infrastructure costs with their in-memory workloads
• Struggle with performance bottlenecks and scaling limitations
• Spend too much time on cluster management
• Need a frictionless path to terabyte-scale in-memory data

Try Dragonfly Swarm Today

Experience an in-memory data store without limits with Dragonfly Swarm. Try Dragonfly Swarm today with a free trial of Dragonfly Cloud, or request a demo to see how we can help your organization unlock effortless scalability.

Welcome to limitless scale.

• [1] In this blog post, Redis states that their shards are limited to 25GB.