Crypto News Aggregation with Dragonfly: A High-Performance Solution for Traders and Developers - Part 1

Disclaimer: This post focuses on the technical implementation of a high-performance crypto news aggregation system using Dragonfly, applicable to other news aggregation use cases as well. It does not provide investment advice on cryptocurrencies or any other assets.

In the rapidly evolving world of cryptocurrency trading, staying ahead requires close monitoring of overall market trends. Among the key elements driving these trends are news updates, which play a significant role in influencing market movements. This information is vital for making informed decisions, whether you’re focusing on short-term trades or planning long-term investments. Additionally, it proves highly beneficial whether you rely on automated tools or take a more hands-on approach to managing your trades.

Learn how to build an efficient solution for traders in under five minutes using Dragonfly’s JSON and SEARCH APIs. This guide will walk you through how Dragonfly addresses the challenges of aggregating crypto news, providing a high-performance and user-friendly tool designed for today’s fast-paced crypto market.

Challenges of Aggregating and Analyzing Crypto News

Aggregating and analyzing crypto news presents several challenges due to the massive volume of information spread across various platforms. News comes from diverse sources, such as CoinDesk, CryptoSlate, Decrypt, and social media platforms. This data is often provided in JSON format and needs to be processed efficiently for actionable insights.

For trading, it’s crucial to extract and analyze information such as:

• Most Mentioned Cryptocurrencies: Identify which cryptocurrencies have been mentioned most frequently in the last 10 days.
• Trend Analysis: Track how often a specific cryptocurrency was mentioned over the past three months.
• Recent Mentions: Retrieve the latest mentions of a cryptocurrency to react quickly to emerging developments.
• Sentiment Analysis: Gauge market sentiment by determining whether the mentions are positive, negative, or neutral.

Dragonfly makes it all simpler. It’s fast, easy to use, and quick to set up, offering a powerful toolkit with features like JSON and SEARCH APIs to help traders efficiently process and analyze crypto news for better decision-making.

Crypto News Aggregation: A Step-by-Step Guide

This guide will walk you through setting up a crypto news aggregation system using Dragonfly and the redis-py client library. Our system will ingest news data, create indexes, and enable aggregation by key parameters like frequency or sentiment. We’ll also implement an advanced semantic search to find similar articles based on their embeddings.

Preparing and Storing the Data in Dragonfly

Before we dive into indexing and queries, we need to gather our data. Let’s assume you have a function that collects news from various sources and normalizes each article to a consistent format, for example:

{
  "id": 1245186,
  "title": "Bitfinex Decrease BTC Withdrawal Fees By 25%",
  "published_at": 1737350000,
  "url": "https://cryptopanic.com/news/1245186/Bitfinex-Decrease-BTC-Withdrawal-Fees-By-25",
  "source": {
    "domain": "reddit.com",
    "title": "r/Bitcoin Reddit",
    "path": "r/bitcoin"
  },
  "votes": {
    "negative": 0,
    "positive": 11,
    "important": 6
  },
  "currencies": [
    {
      "code": "BTC",
      "title": "Bitcoin",
      "slug": "bitcoin"
    }
  ]
}

Now, let’s prepare a Dragonfly instance to experiment with the data. One easy way to get started with Dragonfly is to use our cloud offering—Dragonfly Cloud. It takes only a few steps to sign up and have a data store running in the cloud of your preferred provider and region. Alternatively, getting started locally with Docker can be quick as well. Either way, you should have a connection URI ready to use in just a few minutes. Note that the redis-py client library supports both normal and asynchronous connections, as you can choose the appropriate one for your application.

import redis

# Replace CONNECTION_URI with:
# - The actual Dragonfly Cloud connection URI, or
# - "localhost:6379" if running Dragonfly locally with Docker and default settings.
client = redis.Redis.from_url("CONNECTION_URI")
client.ping()

Once the data is collected and normalized, it needs to be stored in Dragonfly in a structured way that allows efficient retrieval and indexing.

news_data = fetch_and_normalize_news_data()

# Store the news data in Dragonfly.
for article in news_data:
    key = get_key(article)
    value = json.dumps(article)
    client.execute_command("JSON.SET", key, "$", value)

In practice, one process will asynchronously fetch and store new articles in Dragonfly, while another will handle querying and aggregation for efficient lookups and semantic search.

Creating a Basic Search Index

With the news data in place, the next step is to index the JSON documents so queries and aggregations become efficient. For instance, you can create an index news_idx that indexes the publication date, votes, and currencies:

client.execute_command(
    "FT.CREATE", "news_idx", "ON", "JSON",
    "PREFIX", 1, "article:", 
    "SCHEMA",
    "$.id", "AS", "id", "NUMERIC",
    "$.published_at", "AS", "published_at", "NUMERIC", "SORTABLE",
    "$.votes.negative", "AS", "votes_negative", "NUMERIC",
    "$.votes.positive", "AS", "votes_positive", "NUMERIC",
    "$.votes.important", "AS", "votes_important", "NUMERIC",
    "$.currencies[*].code", "AS", "currency_code", "TAG",
    "$.weight", "AS", "weight", "NUMERIC"
)

Identifying the Most Mentioned Cryptocurrencies Over a Period

This section of the system focuses on finding the cryptocurrencies that appear most frequently in recent news. All you need to do is specify the timestamp from which mentions should be counted. Under the hood, it uses a Dragonfly FT.AGGREGATE query to group articles by currency_code and then returns the top results based on the number of mentions.

This is especially helpful for quickly spotting which coins or tokens are trending.

def get_most_mentioned_cryptocurrencies(client, from_timestamp, top=10):
    return client.execute_command(
        "FT.AGGREGATE", "news_idx",
        f"@published_at:[{from_timestamp} +inf]",
        "GROUPBY", 1, "@currency_code",
        "REDUCE", "COUNT", 0, "AS", "mentions_count",
        "SORTBY", 2, "@mentions_count", "DESC",
        "LIMIT", 0, top,
    )

Tracking Recent Mentions of a Specific Cryptocurrency

If you want to see exactly which articles are talking about a particular cryptocurrency rather than just a count, you can run a targeted search. This function uses a Dragonfly FT.SEARCH query to look for any articles containing a given currency code published after a specified timestamp. It then sorts those articles by publication date, giving you the most recent coverage first.

def get_cryptocurrency_recent_mentions(client, currency, from_timestamp, limit=10):
    return client.execute_command(
        "FT.SEARCH", "news_idx",
        f"@currency_code:{{{currency}}} @published_at:[{from_timestamp} +inf]",
        "SORTBY", "published_at", "DESC",
        "LIMIT", 0, limit,
    )

Evaluating the Sentiment for a Specific Cryptocurrency

In addition to frequency, you may also want to measure how positively or negatively a particular cryptocurrency is being received. This is where sentiment scores come in. Using a Dragonfly FT.AGGREGATE query, this function sums up all positive, negative, and important votes for a given currency, giving you an instant snapshot of overall sentiment.

def get_cryptocurrency_votes(client, currency, from_timestamp):
    return client.execute_command(
        "FT.AGGREGATE", "news_idx",
        f"@currency_code:{{{currency}}} @published_at:[{from_timestamp} +inf]",
        "GROUPBY", 1, "@currency_code",
        "REDUCE", "SUM", 1, "@votes_positive", "AS", "total_positive_votes",
        "REDUCE", "SUM", 1, "@votes_negative", "AS", "total_negative_votes",
        "REDUCE", "SUM", 1, "@votes_important", "AS", "total_important_votes",
    )

Building Your Foundation with Dragonfly

In the first part of our two-part blog series, we explained how to build a robust foundation for crypto news aggregation using Dragonfly. We covered essential building blocks such as storing the news data, creating efficient search indexes, tracking cryptocurrency mentions, and analyzing sentiment through vote aggregation. These fundamentals set the stage for more advanced vector search based functionality, which we’ll explore in the next part of the series. Stay tuned.

Ready to implement this solution? Get started with Dragonfly today!