Intro
When it comes to application monitoring, real-time data processing is critical. Inspector, a code execution monitoring platform that allows developers to quickly identify and troubleshoot performance bottlenecks within their apps, needs to be able to efficiently manage high-volume data ingestion. When their existing queue management system began to struggle with scale, CTO Valerio Barbera knew it was time for a change. That's when he discovered Dragonfly, a modern drop-in Redis replacement designed to address the performance and scalability challenges they were facing.
The Challenge: Scaling Queue Management
Just one CPU was at 100%, and it never scaled across multiple CPUs.
Inspector's system ingests large amounts of data from monitored applications worldwide. This data is processed in the background using a queue system, which acts as a buffer between data ingestion and processing. As Inspector's customer base grew to over 1,000 monitored applications, their existing Redis-based queue system began to show its limitations. "We deployed a machine with four CPUs, thinking that more CPUs would give us more room to scale the queue," Barbera explained. "But it wasn't the case. Just one CPU was at 100%, and it never scaled across multiple CPUs."
Redis Doesn't Scale Across Multiple CPUs
This bottleneck caused serious problems. The queue system would slow down during peak traffic, causing delays in data ingestion. This meant that Inspector's customers saw gaps on their dashboards where no new data was displayed—a significant problem for a real-time monitoring service.
The Search for a Solution
Initially, Inspector turned to Amazon SQS to tackle their scaling issues. The scalability problem was solved, but costs quickly became a concern. "SQS is an absolutely great system with infinite scalability and never a downtime," Barbera explained. "But it practically became the first billing item in our cloud costs—$800 per month." As Inspector continued to grow, Barbera knew they needed a solution that could not only solve their current issues but also provide room for future expansion. That's when he stumbled upon Dragonfly in a YouTube video discussing its use with Sidekiq, a popular background job processor for Ruby on Rails.
Dragonfly: A Perfect Fit for Laravel
It was very, very fluent.
Recognizing Dragonfly's potential to boost performance and cut costs, Barbera saw it as an ideal solution for Inspector's Laravel-based system. "Laravel's queue system is basically a porting from the Ruby on Rails Sidekiq," he explained. "Knowing about Dragonfly's success with Sidekiq, I thought that 99% it could be a solution for me too." The evaluation process was smooth and straightforward. Dragonfly's Redis compatibility meant Inspector could migrate their existing queue system with minimal code changes. "I just had a technical call with Roman from Dragonfly to be sure to use the keys for the cache, queue names, etc., in the right way," Barbera recalled. "It was very, very fluent."
Impressive Results
Inspector saw immediate benefits from switching to Dragonfly:
- Improved Performance: Dragonfly's ability to make full use of modern cloud hardware solved the scaling bottleneck Inspector experienced with Redis and allowed them to meet the data demands of their growing solution.
- Cost Savings: Inspector significantly reduced costs compared to their previous SQS solution. "We saved about $650 per month," Barbera said.
- Simplified Management: With Dragonfly Cloud, Inspector no longer had to manage their own infrastructure, freeing up valuable time and resources.
A Scalable Future with Dragonfly
Great technology, great team.
By removing the limits of traditional in-memory data stores, Dragonfly enables Inspector to scale as needed. "It allows us to use the hardware efficiently," Barbera explained. "It's not a panacea for infinite scaling, but with the cloud, we can instance a machine with 5, 6, or 10 CPUs. If you use this kind of system in the right way, you have a lot of room to grow." Barbera had high praise for the technology, his experience, and the team behind Dragonfly. "Great technology, great team," he summarized. "The approach by the team was very, very tuned in with the market problem."
