Live Migration from AWS ElastiCache to Dragonfly Cloud: A Hands-on Demo

In this blog post, we’ll demonstrate how to perform a live migration from an AWS ElastiCache cluster to Dragonfly Cloud using a sample application. We’ll focus on achieving a zero-downtime migration while maintaining data consistency throughout the process.

Why Migrate to Dragonfly Cloud?

Migrating from AWS ElastiCache to Dragonfly Cloud offers several advantages. Dragonfly Cloud, which uses the Dragonfly core project without any gated features, provides significant cost savings compared to ElastiCache, especially for larger deployments. Built on modern multi-threaded shared-nothing architecture, Dragonfly delivers higher throughput and lower latency compared to traditional Redis or Valkey-based solutions.

Migration Approach

Our migration strategy focuses on three key principles:

• Zero Downtime: Applications continue running without interruption during migration.
• Data Consistency: Ensure no data loss and maintain consistency throughout the process.
• Verification: Real-time visualization and metrics to verify successful migration.

This approach works particularly well for production environments requiring 24/7 availability, applications with strict data consistency requirements, and teams looking to validate migration success in real time.

In this blog post, we will continue to use the RedisShake migration tool, which was introduced in one of our previous blog posts. It would be helpful to have a basic understanding of RedisShake, such as different modes (PSYNC, SCAN) and its configuration file in TOML format. Overall, it is a powerful tool designed to facilitate the migration of Redis data across different environments with easiness, configurability, and zero-downtime migration as design goals.

The Sample Application

Our demo application is a Python-based Redis client that helps visualize the migration process. The demo app continuously writes data to Redis in real time, provides a grid-based visualization of the data distribution, shows real-time metrics like memory usage and key counts, and includes an HTTP endpoint to dynamically update Redis connection details.

The diagram below shows how the application and migration components interact. The demo application initially writes to ElastiCache RedisShake, then reads from ElastiCache and writes to Dragonfly Cloud. The process includes a live cutover where the application switches to Dragonfly Cloud, with real-time monitoring through the visualization dashboard. The application is already deployed and running against your AWS ElastiCache cluster. You can monitor the migration progress through its visualization output.

ElastiCache to Dragonfly Cloud: Live Migration with RedisShake

Migration Prerequisites

Before starting the migration, ensure you have:

• A running AWS ElastiCache cluster.
• A Dragonfly Cloud private network peered with your application’s VPC.
• A Dragonfly Cloud data store created in the private network.
• Access to both ElastiCache and Dragonfly Cloud endpoints.

Migration Process

1. Setting Up the Migration Instance

First, we’ll create an EC2 instance that can access both ElastiCache and Dragonfly Cloud:

# Create security group for migration instance
REDISSHAKE_SG_ID=$(aws ec2 create-security-group \
  --group-name "migration-instance-sg" \
  --description "Security group for migration EC2 instance" \
  --vpc-id $APP_VPC_ID \
  --query 'GroupId' \
  --output text)

# Allow outbound access to both Redis clusters
aws ec2 authorize-security-group-ingress \
  --group-id $ELASTICACHE_SG_ID \
  --protocol tcp \
  --port 6379 \
  --source-group $REDISSHAKE_SG_ID

2. Configuring RedisShake

Next, on the migration instance, create the RedisShake configuration for cluster mode as shown below. Note that in this case, the ElastiCache setup is a cluster, whereas the Dragonfly data store is a standalone instance running in emulated cluster mode. Yes, a standalone Dragonfly can handle large amounts of data, which can often replace a clustered Redis of Valkey.

# A RedisShake configuration file in TOML.
[sync_reader]
cluster = true
address = "${ELASTICACHE_ENDPOINT}"
tls = true

[redis_writer]
cluster = true
address = "${DRAGONFLY_CLOUD_URL}:6379"
username = "default"
password = "${DRAGONFLY_CLOUD_PASSWORD:-}"
tls = true

3. Starting the Migration

Launch RedisShake to begin the migration:

docker run -v "$(pwd)/redis-shake-config.toml:/redis-shake-config.toml" \
  --entrypoint ./redis-shake \
  ghcr.io/tair-opensource/redisshake:latest \
  /redis-shake-config.toml

During this phase, you can observe the migration progress in real time. The output shows RedisShake performs the initial data sync while the application continues writing to ElastiCache. You can monitor progress through RedisShake logs, and the visualization output shows ongoing operations.

## The demo application writes data to ElastiCache.
## Then it switches to Dragonfly Cloud.

#=> Memory Usage: 322.99 MB (Informational)
#=> Total Keys Inserted (Real): 300000 / 300000
#=> [██████████████████████████████████████████████████] 100.0%

## RedisShake migrates data from ElastiCache to Dragonfly Cloud
## while the demo application is running.

#=> INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], syncing aof, diff=[0]
#=> INF read_count=[0], read_ops=[0.00], write_count=[0], write_ops=[0.00], syncing aof, diff=[0]
#=> INF read_count=[42640], read_ops=[8526.93], write_count=[42640], write_ops=[8526.93], syncing aof, diff=[98408]
#=> INF read_count=[135706], read_ops=[18614.50], write_count=[135706], write_ops=[18614.50], syncing aof, diff=[2]
#=> INF read_count=[230000], read_ops=[18853.82], write_count=[230000], write_ops=[18853.82], syncing aof, diff=[0]
#=> INF read_count=[300000], read_ops=[14002.85], write_count=[300000], write_ops=[14002.85], syncing aof, diff=[0]
#=> INF read_count=[300000], read_ops=[0.00], write_count=[300000], write_ops=[0.00], syncing aof, diff=[0]
#=> INF read_count=[300000], read_ops=[0.00], write_count=[300000], write_ops=[0.00], syncing aof, diff=[0]

4. Verifying Data Consistency

Once the migration is complete, you can verify the data consistency by checking the key size:

# Check key size of ElastiCache.
redis-cli -h $ELASTICACHE_ENDPOINT dbsize

# Check key size of Dragonfly Cloud.
redis-cli -h $DRAGONFLY_CLOUD_URL dbsize

When the key counts match between ElastiCache and Dragonfly Cloud, this confirms successful data synchronization. You can also check the RedisShake logs—when diff=[0] appears, it signals that the migration is fully caught up. Note that if your dataset includes TTL-based keys, minor discrepancies in key counts may occur due to the expiration algorithm randomness. As long as these variances are within expected bounds and don’t impact application functionality, the migration can be considered complete. More details can be found in the RedisShake documentation in this regard.

5. Live Cutover

When RedisShake enters real-time replication mode, we can perform the live cutover. While a live cutover is not possible for many applications, this would be the same as updating the Redis connection string in the application and performing a deploy/restart of the application.

# Update the demo application's Redis connection, assuming it's also running in the EC2 instance.
curl -X POST http://localhost:8080/update_redis \
  -H "Content-Type: application/json" \
  -d '{
    "host": "your-dragonfly-cloud-url",
    "port": 6379,
    "password": "your-dragonfly-cloud-password"
  }'

A Successful Zero-Downtime Migration

This blog post shows that you can migrate from AWS ElastiCache to Dragonfly Cloud without downtime or data loss. The visualization output from our demo application and real-time metrics from RedisShake help ensure a smooth transition by providing immediate feedback throughout the process. This approach can be adapted for production workloads, ensuring smooth transitions while maintaining business continuity.

For more detailed information about migrating from AWS ElastiCache to Dragonfly Cloud, refer to our official migration guide.