Sidekiq
Note: Dragonfly v1.13+ is required for this integration.
Introduction
Sidekiq is a background job processing system that is built for Ruby and originally used Redis as its backend store. Sidekiq clients push jobs into one or more Redis lists, and the Sidekiq server pops jobs from those lists and executes them. The unit of parallelism for Sidekiq is the list (referenced as a queue in Sidekiq lingo), and the way to scale Sidekiq involves creating multiple queues.
Dragonfly is highly compatible with Redis, and Sidekiq use cases map nicely to the Dragonfly's multi-threaded architecture. By replacing Redis with Dragonfly, you can achieve superior performance and scalability for your Sidekiq applications.
Running Sidekiq with Dragonfly
Integration with Sidekiq is seamless. Run Dragonfly and Sidekiq, and they will work together out of the box. However, to utilize Dragonfly's multi-threaded architecture, you need to distribute queues equally among the number of Dragonfly shard threads available.
1. Dragonfly Initialization
There are several options available to get Dragonfly up and running quickly. Assuming you have a local Dragonfly binary, you can run Dragonfly with the following flags:
$> ./dragonfly --bind localhost --port 6379 --shard_round_robin_prefix={QUEUE_PREFIX}
2. Sidekiq Configuration
Detailed instructions on how to configure Sidekiq with its backend store can be found here.
Since Dragonfly is highly compatible with Redis, just make sure to use the Dragonfly server address and port (i.e., localhost:6379
for the local Dragonfly instance above) in the Sidekiq configuration.
3. Parallelization & Load Balancing
To increase parallelism and load balance in your application, you need to change the default way Dragonfly distributes its keys among the available shard threads.
It's therefore advised to configure Dragonfly with the shard_round_robin_prefix
flag.
With this configuration, Dragonfly equally spreads the Sidekiq queues (lists) among the available shard threads.
Replace QUEUE_PREFIX
with the prefix name of the Sidekiq queues.
For example, if your queues are named my_queue:1
, my_queue:2
, my_queue:3
, etc., then replace QUEUE_PREFIX
with my_queue
.
Useful Resources & Benchmarks
- Sidekiq Homepage, GitHub, and Documentation.
- Sidekiq also has a dedicated documentation page on Using Dragonfly.
- Read our blog post Running and Optimizing Sidekiq Workloads with Dragonfly with internal implementation details and performance benchmarks.