Provisioning a Large Cache on AWS with Dragonfly Cloud and Terraform

Modern applications increasingly rely on caching to improve response times and reduce backend load. As your application scales, the size of your cache grows significantly, making the choice of caching solution really important. This is especially true for high-throughput, low-latency use cases like real-time bidding systems, online gaming, high-frequency trading platforms, e-commerce flash sales, and social media applications.

In this blog post, I’ll show you how to seamlessly provision a large cache (200GB) on AWS, one that delivers 25x more throughput and 12x lower snapshotting latency compared to Redis or Valkey-based solutions!

What is Dragonfly?

Dragonfly is a modern, high-performance, drop-in replacement for Redis. It is designed from the ground up to handle heavy data workloads on modern cloud infrastructure, offering substantial performance improvements over traditional in-memory data stores like Redis. How much better performance, you may ask?

• 25x Higher Throughput: Dragonfly delivers significantly better performance compared to Redis when tested on the same hardware.
• 12x Lower Snapshotting Latency: Faster snapshots minimize potential disruptions, ensuring smoother operations and enhanced reliability without requiring significant downtime.
• 80% Lower Infrastructure Costs: Dragonfly is more compute and memory-efficient, which can significantly reduce your infrastructure expenses.

If you’re curious about the benchmarks (vs. Redis, vs. Valkey), I recommend checking out the detailed performance reports to see the results for yourself!

Dragonfly Cloud Makes Managed Caching Easy

One of the most common use cases for in-memory data stores like Dragonfly is caching, where its better performance really starts to make a big difference. Modern applications depend on caching to accelerate response times, reduce database load, and efficiently handle high request volumes. For example, Meesho, an e-commerce giant serving 50 million daily active users, relies on Dragonfly-powered caches to deliver a seamless experience to its customers.

While Dragonfly offers a forever free community edition for self-managed instances, many teams prefer to avoid the hassle of managing their own cache. Dragonfly Cloud takes care of provisioning and managing your cache, offering additional functionality out of the box.

We recently launched the Dragonfly Cloud Terraform Provider, which lets you programmatically manage resources on the platform. Let’s explore how to use it to provision a cache on AWS.

Provisioning a Cache on AWS Using Terraform

There are some prerequisites before we start provisioning our cache:

• You need to have Terraform installed on your machine.
• You need to create a Dragonfly Cloud account (free 100 USD starting credits!) and then generate your API key.

Once you’ve completed these steps, it’s time to set up your cache.

What is a Data Store in Dragonfly Cloud?

A data store in Dragonfly Cloud acts as an endpoint for storing and accessing in-memory data. It supports both Redis and Memcached protocols, making it a great choice for a cache. Behind the scenes, the data store is a fully managed service powered by Dragonfly server instances.

Let’s walk through a sample Terraform configuration to provision a cache on AWS:

terraform {
  required_providers {
    dfcloud = {
      source = "registry.terraform.io/dragonflydb/dfcloud"
    }
  }
}

provider "dfcloud" {
	api_key = "<YOUR_API_KEY>"
}

resource "dfcloud_datastore" "cache" {
  name = "frontend-cache"

  location = {
    region   = "us-east-1"
    provider = "aws"
    availability_zones = ["us-west-1"]
  }
 
  tier = {
    max_memory_bytes = 200000000000
    performance_tier = "enhanced"
    replicas         = 1
  }
}

Understanding the Configuration

Provider Initialization

• We initialize the Dragonfly Cloud provider and specify its source in the Terraform registry.
• The api_key is passed to authenticate with Dragonfly Cloud. Alternatively, you can leave this empty and set the API key as an environment variable, namely DFCLOUD_API_KEY, when running terraform apply.
• This is all very standard stuff when it comes to using Terraform. If you’ve never used Terraform before, I’d recommend checking out this getting started guide.

Data Store Resource

• name: The name you want to give your data store.
• location: Specifies the cloud provider (e.g., AWS) and region (e.g., us-east-1) where the Dragonfly server instances will be created. Note that this cannot be changed after creation. If you specify a list of regions in the availability_zones key, then the one specified in region is considered your main zone, and the others are used for any replicas you may choose to provision.
• tier: Defines the memory size and CPU-to-memory ratio for the data store. In this example, we provision a 200GB cache. There are three main tiers that Dragonfly offers, depending on what CPU-to-memory ratio you want. You can read more about them in detail here. You can also optionally specify the number of replicas. Each replica would be placed in the list of zones you specified in the location key in a round-robin fashion. Replicas help you achieve high availability by ensuring that if the Dragonfly server in your main zone fails for any reason, your data is still not lost, and your service doesn’t fail.

These three fields are mandatory to be able to create your cache. They correspond to the following sections in the Dragonfly Cloud console:

Once you apply this Terraform configuration using terraform apply, you’ll see your newly created cache on the console:

While you can get the connection URI from the Dragonfly Cloud console, we can also update our Terraform file to show the URI as an output when it’s done creating the store. Add the following code to the same file:

output "redis-endpoint" {
  sensitive = true
  value     = "redis://default:${dfcloud_datastore.cache.password}@${dfcloud_datastore.cache.addr}"
}

Now, when you run terraform apply again, you should see the following output:

Outputs:

redis-endpoint = <sensitive>

This is expected because we have set sensitive to true (like you should!), which ensures that Terraform will redact the value of the output in the terminal to prevent accidental exposure of sensitive data. To view your endpoint, you can run terraform output redis-endpoint, and you should see something like this:

"redis://default:689tjruXXXXX@fv50hgvlv.dragonflydb.cloud:6385"

We can now test this out using the redis-cli or any other method you prefer:

$> redis-cli -u "redis://default:689tjruXXXXX@fv50hgvlv.dragonflydb.cloud:6385" PING
PONG

Advanced Configuration Options

In addition to the required fields we discussed above, you can optionally configure the following.

The dragonfly Nested Attributes

• This nested schema enables you to configure Dragonfly-specific features, such as cache mode, specializations, ACL rules, and TLS.
• When cache mode is enabled, the data store behaves as a cache, automatically evicting items to free up space when it reaches its limit. Without cache mode, the data store returns out-of-memory errors when full.
• Dragonfly Cloud offers specialized configurations to optimize performance for specific workloads. These specializations ensure your data store is tailored to your application’s needs. Currently, we support the following specializations: BullMQ, Memcached, and Sidekiq.
• You can also specify ACL rules with Dragonfly Cloud, allowing you to define fine-grained access control rules for your data store.
• By default, the data store is configured with a public endpoint and TLS. You can optionally disable TLS, though we strongly recommend keeping it enabled for public endpoints.

The network_id Field

• Allows you to associate the data store with a private network created in Dragonfly Cloud.

Here’s an example Terraform configuration that leverages these advanced features:

resource "dfcloud_datastore" "cache" {
  name = "frontend-cache"

  location = {
    region   = "us-east-1"
    provider = "aws"
  }
 
  tier = {
    max_memory_bytes = 200000000000
    performance_tier = "enhanced"
  }

  dragonfly = {
    cache_mode = true
    bullmq     = true  
    sidekiq    = true
    acl_rules = [
    "USER app_user ON >strongpassword ~app:* +@readwrite"
	  ]
  }
  
  # Get from your Dragonfly Cloud account after creating a private network and a peering connection.
  network_id = "vpc-079e8d6b5c760da96"
}

I want to specifically call attention to the cache_mode configuration here. Turning this on is what enables our data store to behave like a cache effectively. With cache mode enabled, when you perform write actions, the data store will automatically evict items if it’s full. This is in contrast to the default behavior of throwing out-of-memory errors when full. You can read more about Dragonfly’s cache design in this blog post!

Try Out the Terraform Provider Yourself!

If your application requires a large, high-performance cache, you need a modern caching solution built for cloud-native environments. Dragonfly is designed to leverage modern infrastructure, outperforming traditional in-memory data stores like Redis to offer better performance and faster backups.

To get started, you can try provisioning your first cache on Dragonfly Cloud using the Terraform configuration in this article. You can also explore the full documentation and see samples for the Dragonfly Terraform provider here.

Happy hacking!