Application caching: good, but not great

Adding a nice cache to speed things up is a classic move, as we all know. And at first, it seems to work: instant responses, fewer database calls, less network traffic… everyone’s happy. Then traffic picks up, responses start slowing down, the product data doesn’t add up and a nervous customer reports information that “doesn’t look right”. Worse still, in a state of sheer panic, we find ourselves having to restart an application that has already collapsed. And suddenly, the cache that promised so many benefits becomes the main reason we’re awake at three in the morning, staring at logs and cursing the day we added it.

In this first instalment of Learning from mistakes, we look at five common mistakes in application caching. We’ll describe each bad practice, its symptoms and how to fix it.

We always start with the best intentions. The usual mindset is, “Oh, it’s fine as it is. I’ll invalidate the cache manually when I need to.” The result: a user keeps seeing a price that no longer exists for weeks, or a profile with data that’s months out of date. By then, we’re busy with other projects, and only get our lovely surprise when our friendly boss comes asking for an explanation, slamming an email from a furious customer onto the desk.

The solution is simple: whenever application data can change and keeping it fresh matters, we need to set a TTL for it in the cache. We can choose its duration according to how much stale data the specific business context can tolerate.

This one is sneakier because nobody in particular is to blame. A cache key for a popular product or item expires, and hundreds of concurrent requests arrive during a traffic spike. Finding nothing in the cache, all of them try to recompute the same value, perhaps by running an expensive query or calling an API that, unfortunately, was never designed to handle hundreds of requests in parallel. This is known as a cache stampede.

How do we prevent it? Let’s coordinate the cache rebuild! Within a single process, we can use the singleflight pattern; across multiple instances, we can use a distributed lock. The idea is that one request updates the cache while the others wait, up to a time limit, for the value to become available. Alongside this, we use a slightly randomised TTL (jitter) to spread expiry across different keys.

One rebuild, several requests

Enlarge diagram

Choosing the right cache key seems like a minor detail. Then someone knocks at the door and voilà: “Hi! My name is… inconsistency!” A key that’s too generic, and suddenly an Italian user sees the interface in English or, worse, sees other users’ data. A key that’s too complex, and we’ll watch our cache grow endlessly, with cache hits becoming rarer than a blue moon.

How do we fix it? There’s nothing much to invent here; we just need to be rigorous: the key must contain all and only the data that uniquely identifies the cached item. If the key contains too little information or, at the other extreme, too much, it’s better not to implement a cache at all. Also, centralise key generation in one place in the application, rather than letting each module build keys its own way.

Brilliant idea! “I’ll just update the database. The cache will catch up on its own.” Er, no… that’s not how it works, but it’s amazing how often we fall into this trap. Meanwhile, half our users see the new value and the other half see the old one, depending on whether their request found anything in the cache. And if we’re talking about prices or stock availability, we’ll also get the delightful experience of someone shouting at us.

Don’t panic. Let’s fix it. There are two approaches. The first is write-through: write the new value to both the database and the cache. The second, simpler approach is to invalidate the key after the database commit: instead of updating the cache, delete the entry, and the next read will use the database to repopulate it.

Here too, be careful! Implement invalidation in one place, rather than reinventing it in ten different files, written by ten different developers with ten different ideas.

An external service returns a 500 or times out. To “optimise” things, our code caches the error using the same TTL as successful responses — let’s say 30 minutes. The external service recovers, perhaps after thirty seconds, but the application keeps returning the error. A thirty-second failure becomes a 30-minute disruption for our users.

Temporarily caching an error can help avoid bombarding a service that’s already struggling: this is negative caching. But the bad practice is indiscriminately caching errors with the same policy and TTL as successful responses.

How do we fix it? Define a dedicated policy: which errors to cache and for how long, distinguishing, for example, a missing resource from a transient failure. For transient failures, choose a short TTL based on how much recovery delay we can tolerate.

Expiry, rebuilds, keys, invalidation and cached errors: five things worth designing properly before our friendly bosses remind us about them.

In Part 2, we’ll cover five more bad practices in application caching.

See you in the next instalment of Learning from mistakes.