Amazon ElastiCache for Redis has gained a lot of traction in recent years as the default caching infrastructure in AWS. Users will often choose ElastiCache because it integrates well with other AWS services and because of the lack of other great managed options.
However, ElastiCache has some serious drawbacks, namely high costs and operational complexity as you scale. This is largely due to the limitations of the underlying Redis engine. One of the alternatives to ElastiCache for Redis is Dragonfly Cloud, a new fully-managed caching service with a multi-threading model.
In this article, we'll break down how Dragonfly Cloud compares to ElastiCache across key dimensions like pricing, performance, scalability, and feature set.
What is AWS ElastiCache for Redis?
ElastiCache is a managed in-memory caching service provided by AWS, first launched in 2011.
ElastiCache allows users to create clusters with either Redis (ElastiCache for Redis) or Memcached (ElastiCache for Memcached) as the in-memory data store engine. It provides a fully managed Redis or Memcached experience that automates common tasks such as software patching, setup, configuration, monitoring, and backups.
ElastiCache for Redis supports Redis 7 and is fully compatible with existing Redis deployments. This means that code, applications, drivers, and tools used with standalone Redis will continue to work with ElastiCache for Redis without requiring any code changes.
Some key features of ElastiCache for Redis include:
- Scalability: You can handle increased workloads on ElastiCache by easily adding or removing cache nodes as needed.
- High Availability: With support for Multi-AZ deployments, automatic failover, and automatic backups, ElastiCache ensures that your cache data remains highly available.
- Security: ElastiCache operates within the Amazon Virtual Private Cloud (VPC), isolating your cache instances from the public internet.
- Monitoring: Integration with Amazon CloudWatch provides real-time monitoring and performance metrics.
- Integration: ElastiCache plays well with the broader AWS ecosystem, like Amazon RDS, DynamoDB, and EC2.
ElastiCache Drawbacks
While AWS has made significant improvements to ElastiCache for Redis with features like dynamic network processing, it's important to understand that Redis itself is still fundamentally single-threaded regarding data operations. Redis commands such as GET, SET, and INCR are executed on a single thread, even if the underlying hardware has multiple cores available. This single-threaded nature can become a bottleneck in high-throughput scenarios, leading to increased latency and reduced throughput when the Redis process becomes CPU-bound.
To work around this limitation, ElastiCache users often need to scale their clusters horizontally by adding more nodes, even if individual nodes have spare CPU capacity. This approach leads to increased costs and complexity in managing the cluster.
What is Dragonfly Cloud?
Dragonfly is a modern open-source in-memory data store designed for high performance and efficiency. It is fully compatible with Redis and Memcached APIs, making it easy to integrate with existing applications. Dragonfly utilizes a multi-threaded architecture and memory-optimized data structures to deliver superior performance compared to traditional in-memory data stores.
Dragonfly Cloud is a managed in-memory data store service built on top of the Dragonfly open-source project. It offers a simple and cost-efficient model for running Dragonfly in the cloud, enabling developers to leverage the power of Dragonfly without the burden of infrastructure management. With Dragonfly Cloud, you simply define the amount of storage you need, and the service takes care of the rest. This includes automatically provisioning the necessary resources, managing the underlying infrastructure, and ensuring configurable high availability and automatic backups of your data.
While it's possible to deploy the open-source Dragonfly yourself, this blog post will focus on a comparison between ElastiCache and Dragonfly Cloud for a more direct comparison of the two managed services.
Dragonfly Cloud Pricing
One of the biggest advantages of Dragonfly Cloud is its transparent and affordable pricing model. With Dragonfly Cloud, customers only pay for the memory resources they provision, calculated per GB. Prices vary by region and volume but can be as low as $6.47 per GB of memory per month in the GCP Asia South (Mumbai) region. This often results in cost savings of over 40% compared to ElastiCache.
To illustrate this, let's compare the costs of ElastiCache and Dragonfly Cloud in various scenarios. The ElastiCache cost estimates are sourced from the AWS pricing calculator as of July 2024, based on the US East (N. Virginia) region.
| Scenario | ElastiCache (Monthly Cost) | Dragonfly Cloud (Monthly Cost) |
|---|---|---|
| 50GB, 10k QPS | $599.33 (1x cache.r6g.2xlarge) | $450 |
| 200GB, 100k QPS | $2,397.32 (1x cache.r6g.8xlarge) | $1,800 |
| 200GB, 100k QPS | $2,996.65 (5x cache.r6g.2xlarge) | $1,800 |
| 400GB, 200k QPS | $4,793.91 (1x cache.r6g.16xlarge) | $3,600 |
| 400GB, 200k QPS | $5,993.30 (5x cache.r6g.4xlarge) | $3,600 |
As you can see, Dragonfly Cloud can be 40% cheaper than ElastiCache,
and this doesn't even take into account the fact that the amount of usable capacity on an ElastiCache node is often misleading.
Let's take the cache.r6g.large ElastiCache instance as an example.
While AWS lists the maximum capacity as 13.7 GB, the actual usable capacity is only around 9.8 GB.
The price for this instance in the US East (N. Virginia) region is $150 per month, which equates to over $15 per usable GB.
In contrast, Dragonfly Cloud is priced at $9 per usable GB in the same region.
Dragonfly Cloud Makes More Efficient Use of Hardware
As previously mentioned, ElastiCache for Redis is single-threaded, which often necessitates clustering and over-provisioning to handle high-throughput scenarios and accommodate its snapshotting mechanism. Dragonfly Cloud, on the other hand, makes full use of all hardware resources via its multi-threaded processing capabilities. Under the hood, Dragonfly's shared-nothing architecture allows it to partition data across multiple cores, utilizing the full CPU capacity of the node. This means that Dragonfly Cloud can serve the same workloads on far less hardware.
For instance, to handle a 180GB workload on ElastiCache, you might need to reserve around ten 25GB instances due to Redis's limitations. With Dragonfly Cloud, a single 200GB instance would suffice, demonstrating its high efficiency and resource utilization.
Dragonfly Cloud Scalability & Performance
Beyond cost, Dragonfly Cloud offers major performance and scalability benefits compared to ElastiCache.
Dragonfly Cloud Scalability
ElastiCache's horizontal scaling model can be complex and expensive to manage. To scale an ElastiCache cluster, users need to add more nodes, which requires careful capacity planning and can result in increased network traffic and latency.
Dragonfly Cloud, on the other hand, abstracts away the need to manage instances and clusters. Instead, you simply specify the memory capacity, and Dragonfly Cloud automatically scales up or down to handle your traffic patterns. This approach simplifies management and ensures optimal performance without the need for complex planning and manual intervention.
Dragonfly Cloud Latency
One of the most frequently cited reasons that users choose ElastiCache is the belief that co-locating their caching layer within AWS will decrease latency. While this is true, Dragonfly Cloud offers an equally compelling alternative. Dragonfly Cloud is a multi-cloud offering, providing the flexibility to choose AWS and your preferred region to co-locate Dragonfly instances with your services. This ensures optimal performance and reduced latency, similar to ElastiCache, but with the added advantage of being able to operate seamlessly across multiple cloud environments. This multi-cloud capability allows you to leverage the best of AWS while maintaining the versatility and resilience of a cross-cloud infrastructure.
Dragonfly Cloud allows AWS customers to establish a VPC peering connection. With VPC peering, you can achieve secure and efficient private network communication between your application and Dragonfly Cloud. This setup not only enhances security by keeping data within the AWS network but also ensures network latency comparable to what you would experience with ElastiCache, as traffic between peered VPCs benefits from direct, low-latency connections within the AWS infrastructure.
But what really sets Dragonfly Cloud apart is its superior server-side performance and reduced latency, thanks to Dragonfly's fundamentally different architecture. Let's take a look at the latency graph of a customer who migrated from ElastiCache to Dragonfly Cloud.
Prior to switching, P95 latency on ElastiCache was very high (i.e., ~800 ms). Latency was also extremely volatile, with occasional spikes to over 3 seconds. After cutting over to Dragonfly Cloud, P95 latency dropped dramatically and remained stable.
The customer described Dragonfly Cloud as "50x faster for us." How is this possible? By intelligently spreading the workload across multiple CPUs, Dragonfly makes much more efficient use of compute resources. This greatly reduces the chances of the server becoming unresponsive under load. Achieving this level of performance with ElastiCache would require extensive manual sharding and complex operational oversight, whereas Dragonfly Cloud delivers it out of the box.
Migrating from ElastiCache to Dragonfly Cloud
Dragonfly Cloud is fully compatible with the Redis and Memcached APIs, allowing you to switch from ElastiCache with minimal code changes—typically just updating the connection string. Dragonfly currently supports over 250 Redis commands, ensuring comprehensive compatibility. We have previously blogged about various techniques for performing migrations with minimal to zero downtime. For more details, check out our blog posts (#1, #2) and documentation.
Expanded Feature Set
Dragonfly Cloud also provides powerful features not available in ElastiCache. A great example is Dragonfly Search, Dragonfly's native indexing and querying engine. Dragonfly Search enables both vector search and faceted search use cases directly within Dragonfly, eliminating the need to manage a separate search engine. This allows developers to build RAG applications without leaving the data store. In addition to Search, Dragonfly natively supports queryable JSON documents, probabilistic data structures such as Bloom filters, and much more, offering a comprehensive and versatile solution for modern application needs.
Deployment Flexibility
Even when using the managed Dragonfly Cloud service, customers can select their preferred cloud provider and region, including AWS, GCP, and others. Dragonfly Cloud provisions the service with a local endpoint in the customer's environment, minimizing data transfer costs and latency. Unlike ElastiCache, which is exclusive to AWS, Dragonfly Cloud offers multi-cloud support, providing greater flexibility and optimization for your infrastructure.
Conclusion
While ElastiCache remains a popular choice for in-memory data storage on AWS, it faces significant limitations and challenges at scale. If you're seeking to improve the performance and cost efficiency of your caching layer or other in-memory workloads, Dragonfly Cloud is worth considering. Architects and developers can easily evaluate Dragonfly Cloud by provisioning an instance in minutes and directing a subset of staging or test traffic to it. By monitoring the performance and costs compared to your current ElastiCache setup, you can make an informed decision about adopting a more efficient and powerful solution.
