Cache-Aside Pattern with Redis and Fingerprinting to Offload MongoDB

The Problem

At Telefónica, we used Azure IoT Hub on a telemetry project to receive millions of events daily. The system was well-engineered, utilizing Project Reactor and Kotlin, allowing it to process incoming traffic efficiently and asynchronously.

MongoDB became the primary infrastructure bottleneck due to the sheer volume of read operations.

The application checked whether certain values of an incoming event differed from the ones already saved in the database. If they did, it updated them and triggered a chain of other microservices and processes; if not, the event was discarded.

Although functionally the workflow was correct, operationally it meant that any event received ended up burdening MongoDB. At scale, MongoDB became one of the most heavily utilized components in the system.

Applying some kind of cache solution was the way to go.

Constraints

A traditional cache was not particularly useful for this use case.

MongoDB’s documents held significantly more information than what was needed to decide whether an update should be triggered. Storing the whole document was unnecessary and would probably have made the cost of Redis too high to even consider as a solution. Saving only the values the business logic defined as “update worthy” was the only way. But we could do better.

The Solution: Fingerprinting the Relevant Values

Following the strategy outlined by the architecture team, I collaborated with a senior engineer to implement the fingerprinting mechanism through close pair programming.

As previously stated, this was a high-throughput reactive system so speed was key. This is why we turned to fingerprinting.

Instead of saving the values that could trigger an update, we saved a hash composed of those values.

This way, the workflow became:

1. Receive an event.
2. Generate the hash fingerprint from relevant fields.
3. Compare the fingerprint against the value stored in Redis.
4. If the fingerprint differs, perform the update workflow.
5. If the fingerprint matches, skip the database interaction entirely.

This way, Redis became a lightweight change-detection layer sitting in front of our MongoDB, reducing the costly reads that previously burdened our database.

For hashing, we choose xxHash due to its speed and low collision rate.

The Result

MongoDB greatly reduced the number of reads received, improving its speed as it was no longer under such a high load.

As for Redis, as only compact hash values were stored, its footprint remained a tiny fraction of what full document snapshots would require, optimizing memory efficiency.

Fingerprinting was fast, efficient, and after calculations, the theoretical risk of hash collisions was considered negligible for this use case and was outweighed by the performance benefits.

In the end, our solution was highly efficient: Redis size was small, read times vastly improved compared to the previous implementation and a significantly lower overall MongoDB resource utilization.

Key Takeaways & Personal Growth

• Architectural knowledge gained: This project taught me to look for optimization opportunities in the system’s workflow, not just in the database indexes.
• Technical deep dive: It was an amazing opportunity to improve my expertise in Kotlin and headfirst into reactive programming.
• Teamwork: I always enjoy pair programming, and working surrounded by people with such technical skill and vision, gained from more than a decade of experience, was amazing. I look back on this project with immense pride thanks to the amazing team that helped me grow as a professional.