Using a central backplane in a multi-region scenario

[jmwnoble]

Hi and thank you so much for this great library. I have an approach I want to test out, and want to sense-check it here:

Our scenario:

• We operate a Saas platform where the API and cache are located in region and talk to a central SQL cluster.
• Customers are mostly region specific, so beyond a small amount of lookup data most of what is cached is region-specific
• We are planning to set up Redis clusters in each region, to minimise latency
• Cache invalidation is always triggered by a change in the central database, so we can centralise cache invalidation.

Our plan to manage cache validity:

• Specify a 'master' redis cluster as the backplane, ie N1 in US would have the local Redis as its 2nd Level and master Redis as backplane
• A central service watches for changes to the DB and sends the appropriate cache remove message to the master redis. This service would be set up to automatically deliver backplane messages, as per your documentation. My assumption here is even if the key does not exist in the master redis, it will send the message to N1 in the US to invalidate its cache.
• However this approach would not invalidate the cache on the US redis cluster so we would either need this cache invalidation service to explicitly remove the entry from the US cluster at the same time, or a separate service that achieved the same goal by subscribing to the backplane

The other option we are considering is a multi-region redis cluster, but that has cost and complexity implications that make it unattractive, especially given we have a read-through cache architecture (ie GetSet) where the data in each region is generally different.

Does this sound like a reasonable approach? Anything you would do differently? I know this is similar to other backplan discussions but in our case we absolutely what all three features (1st, 2nd, backplane), just need to manage it across regions.

Many thanks
Jeremy

[jodydonetti]

Hi @jmwnoble and thanks for using FusionCache, and for your words 🙂

NOTE: after more thoughts, I've update my answer to avoid future readers getting the wrong take, sorry for the first (incomplete/wrong) answer.

I have an approach I want to test out, and want to sense-check it here:

Our scenario:

• We operate a Saas platform where the API and cache are located in region and talk to a central SQL cluster.
• Customers are mostly region specific, so beyond a small amount of lookup data most of what is cached is region-specific
• We are planning to set up Redis clusters in each region, to minimise latency

Ok so if I got this right, each region has its own API instances running + a Redis instance, and all of them talk to a unique, central database.

• Cache invalidation is always triggered by a change in the central database, so we can centralise cache invalidation.

Our plan to manage cache validity:

• Specify a 'master' redis cluster as the backplane, ie N1 in US would have the local Redis as its 2nd Level and master Redis as backplane

Ok, good so far.

• A central service watches for changes to the DB and sends the appropriate cache remove message to the master redis. This service would be set up to automatically deliver backplane messages, as per your documentation.

Ok, still good.

My assumption here is even if the key does not exist in the master redis, it will send the message to N1 in the US to invalidate its cache.

Correct: even though you can use Redis as a distributed cache and Redis as the backplane (both as the same Redis instance or a different one), the 2 things are not connected whatsoever: the 2nd level cache uses the key/value store part, and the backplane uses the pub/sub part, and they are completely disconnected. The pub/sub part is simply a way to send messages, which are not related at all with cache entries or cache keys (the messages just happen to contain a field that specifies on which cache keys to act, but they are not strictly connected).

• However this approach would not invalidate the cache on the US redis cluster so we would either need this cache invalidation service to explicitly remove the entry from the US cluster at the same time, or a separate service that achieved the same goal by subscribing to the backplane

Ah, damn it. In this very particular scenario you are right!

I'd like to expand a little bit here just to write things down, and to be sure we are on the same page.
So, all the nodes will receive all the cache invalidation messages, and all FusionCache instances, when receiving those messages, will try to mark the related cache entries (in memory) as expired. This, in your scenario, means that all of them will try to expire cache entries that they do not have (because the cached data locality you described means that US data will only be cached in the US region), but that is not a problem per se, it's just some extra remove/expire calls to each node's memory cache, nothing more.

The problem comes when considering that each US (API) node will invalidate their own local memory cache but NOT the shared cache in the US Redis instance.

To be even more clear, here's a quick simplified flow of what would happen:

• a request for "foo" (meaning, cache key: "foo") arrives on the API on a US node
• FusionCache looks in the node's memory cache of that node: the data is not there
• FusionCache looks in the US Redis instance: the data is not there
• FusionCache executes the specified factory, which will call the centralized database, and get the data
• FusionCache saves the data in the US Redis instance
• FusionCache saves the data in the US nodes' memory cache
• FusionCache notifies the central Redis instance about the change for the cache key "foo"
• the central Redis instance notifies all the other nodes about this (both US nodes and other nodes outside of US)
• each node will expire that cache entry in their own local memory cache (and that cache entry will not be anywhere, except in the US nodes)
• a new request arrives for "foo" on the API on a US node
• FusionCache looks in the node's memory cache of that node: the data is not there (or is there but expired, per fail-safe, so it's ignored)
• FusionCache looks in the US Redis instance: the data IS there since nobody expired the data there
• the seemgly valid (but in reality stale) data is saved in the local memory cache, and returned

So yeah, you got a point here I think 🤔

The other option we are considering is a multi-region redis cluster, but that has cost and complexity implications that make it unattractive, especially given we have a read-through cache architecture (ie GetSet) where the data in each region is generally different.

Makes sense. The caching architecture you have in mind seems like an interesting one to explore.

Does this sound like a reasonable approach?

Yep, and I'll see if I find a way to make it work.
Like, for example, it's been some time now that I'm thinking about something like a "remote notification hub" or something like that, maybe usable for when you have a central place that can notify about changes in the cached data, even in slightly different situations.

I know this is similar to other backplan discussions but in our case we absolutely what all three features (1st, 2nd, backplane), just need to manage it across regions.

Give me a little bit of time to explore the scenario, will get back to you as soon as possible.

Many thanks Jeremy

Thanks to you for being involved in the community!

[jodydonetti]

Between a spaghetti alle vongole and a fritto misto (of course with a touch of white wine) I'm thinking about it...

[jmwnoble]

Wow thanks so much for putting some thought into this!

The other approach I thought about was having each region's nodes talk to their local redis as the backplane, and rely on the central redis cache to publish invalidation messages to the regional redis instances - but then I would need to ensure that the message passed from the local redis to fusioncache on the nodes was in the form expected.

[jodydonetti]

Hi @jmwnoble , one question: in you scenario, the users are routed to different regions/nodes (US, EU, etc) using the same public url + some underlying load balancer/gateway logic in a transparent way or are there different urls/domain/subdomains for each regions and maybe some redirects?

Or, let's put it another way: is there a way to make an http call specifically for each region?

[jmwnoble]

Hi @jodydonetti , it is the former, same url but cloudfare directs traffic to the appropriate region

[jodydonetti]

Also, about this passage:

A central service watches for changes to the DB and sends the appropriate cache remove message to the master redis. This service would be set up to automatically deliver backplane messages, as per your documentation. My assumption here is even if the key does not exist in the master redis, it will send the message to N1 in the US to invalidate its cache.

If this central service doesn't care about getting or setting data from the cache, but only sending notifications, I may be thinking about something... let me know if this is your case.

Thanks!

[jmwnoble]

Yes 100% just a notification which commands the regional redis to expire/remove the entry it has

[jodydonetti]

Got it, thanks.

I was asking that because I was thinking about a sort of remote backplane component that would use http requests to the different regions (1 req per region) to send notifications: having a way to send 1 request that would reach 1 (and only 1) node per region would mean that that node would receive the request and either delete the data from the cache (local + distributed) or anyway invalidated them, and in turn send a usual change notification to the other nodes via a Redis backplane that would use the region Redis instance.

This design btw would also be valid for another use case that another member of the community asked for: a separate backoffice app the would change some data and communicated with the frontend app(s), not necessarily using the same Redis backplane instance.

Anyway I'll keep thinking about a solution, will let you know.