Advanced caching scenario

[suchoss]

Hello, I wrote some caching scenario and I would like to verify, if I am correct.

We are using one instance of a web application (=> no backplane needed - is that correct?).

Now with the following settings I expect following scenario (correct me if I am wrong).

1. If cache is empty, then factory method runs and populate 1st level + 2nd level
2. When getting value under 1 minute it returns data from 1st level
3. When getting value after 1 minute, but under 7 days it returns data from 2nd level
4. When 2nd level is not responding and getting value after 1 minute, but under 3 minutes, it returns expired data from 1st level
5. When getting value after 7 days it returns data from factory if the data is returned under 300 ms
6. When getting value after 7 days it returns expired data from 2nd level if it takes factory over 300 ms to serve response
   b) On the background there is still running factory method which updates data after it is done
7. What does FactorySoftTimeout do here?

var redis = new RedisCache(new RedisCacheOptions() { Configuration = redisConnectionString });
var seroptions = new JsonSerializerOptions()
{
    PropertyNameCaseInsensitive = true,
    IncludeFields = true
};
var serializer = new FusionCacheSystemTextJsonSerializer(seroptions);

// INSTANTIATE FUSION CACHE
var cache = new FusionCache(new FusionCacheOptions()
{
    CacheName = cacheName,
    CacheKeyPrefix = cacheName,
    DefaultEntryOptions = new FusionCacheEntryOptions()
    {
        Duration = TimeSpan.FromMinutes(1), 
        IsFailSafeEnabled = true,
        FailSafeThrottleDuration = TimeSpan.FromSeconds(30), 
        FailSafeMaxDuration = TimeSpan.FromMinutes(3), 
        FactorySoftTimeout = TimeSpan.FromMilliseconds(150), 
        
        DistributedCacheDuration = TimeSpan.FromDays(7), 
        DistributedCacheFailSafeMaxDuration = TimeSpan.FromDays(365), 
        AllowBackgroundDistributedCacheOperations = true,
        DistributedCacheSoftTimeout = TimeSpan.FromMilliseconds(300)
    }
});
    
// SETUP THE DISTRIBUTED 2ND LEVEL
cache.SetupDistributedCache(redis, serializer); 

[suchoss]

I also tried to draw how I think it works

[jodydonetti]

Hi, this is almost correct, since DistributedCacheSoftTimeout is used when talking to the distributed cache and is the maximum time allowed for the distributed cache to respond with something. Basically it is used right inside the "Is in L2" block.

The thing is that L1 cannot take too much time to answer (it's in memory, always there, always reachable, etc) whereas any remote/distributed part (distributed cache, backplane, factory) can takes longer for whatever reason (network issues, db overloaded, etc think about the Fallacies of Distributed Computing).

Because of this, FusionCache allows you to specify a timeout to each of the remote/distributed parts, and actually allows you to specify 2 of them:

• soft timeout: when there IS a stale value to use as a fallback (in the worst scenario you'll be using an old value)
• hard timeout: when there IS NOT a stale value to use as a fallbaack (in the worst scenario you'll get an exception)

Since you may want to act and wait differently based on the worst case outcome (use old value VS exception) there are 2 different timeouts you can specify.

The rest seems correct.

Makes sense?

[jodydonetti]

Hi @suchoss , and thanks for using FusionCache!

We are using one instance of a web application (=> no backplane needed - is that correct?)

Correct! The backplane is useful in multi-node scenarios.

var redis = new RedisCache(new RedisCacheOptions() { Configuration = redisConnectionString });
var seroptions = new JsonSerializerOptions()
{
    PropertyNameCaseInsensitive = true,
    IncludeFields = true
};
var serializer = new FusionCacheSystemTextJsonSerializer(seroptions);

// INSTANTIATE FUSION CACHE
var cache = new FusionCache(new FusionCacheOptions()
{
    CacheName = cacheName,
    CacheKeyPrefix = cacheName,
    DefaultEntryOptions = new FusionCacheEntryOptions()
    {
        Duration = TimeSpan.FromMinutes(1), 
        IsFailSafeEnabled = true,
        FailSafeThrottleDuration = TimeSpan.FromSeconds(30), 
        FailSafeMaxDuration = TimeSpan.FromMinutes(3), 
        FactorySoftTimeout = TimeSpan.FromMilliseconds(150), 
        
        DistributedCacheDuration = TimeSpan.FromDays(7), 
        DistributedCacheFailSafeMaxDuration = TimeSpan.FromDays(365), 
        AllowBackgroundDistributedCacheOperations = true,
        DistributedCacheSoftTimeout = TimeSpan.FromMilliseconds(300)
    }
});
    
// SETUP THE DISTRIBUTED 2ND LEVEL
cache.SetupDistributedCache(redis, serializer); 

This code seems technically correct, below I'll try to explain more each option.
Consider I'll use for brevity L1 for 1st level and L2 for 2nd level.

Now with the following settings I expect following scenario (correct me if I am wrong).

1. If cache is empty, then factory method runs and populate 1st level + 2nd level

Correct.

2. When getting value under 1 minute it returns data from 1st level

Correct.

3. When getting value after 1 minute, but under 7 days it returns data from 2nd level

Correct, IF the data is in L2 (Redis may have restarted or else), and IF the communication with L2 takes less than 300 ms (because of DistributedCacheSoftTimeout).
Also, not only it returns data from L2, but it also updates L1 (to avoid going to L2 for every subsequent call).

So, in the "easy scenario" (without considering shifted updates from different nodes, Redis restarts, etc), L1 will be updated every 1 min from the L2 for up to 7 days.

Makes sense?

4. When 2nd level is not responding and getting value after 1 minute, but under 3 minutes, it returns expired data from 1st level

Not exactly: at every call FusionCache tries with L1 and, if no fresh data is there it tries with L2, and if not fresh data is there (or if talking with L2 took more than the configured DistributedCacheSoftTimeout of 300 ms in this case) it calls the factory.

If then the factory fails (eg: the database is down or similar), then fail-safe kicks in and temporarily re-use the stale value, by saving it again in the cache for some time (FailSafeThrottleDuration, in this case 30 sec).

After 30 sec, it will try again to do the entire dance (including chacking L2 in case some other node has updated it), and keeps calling the factory if nothing else worked, and if that fails too it will keep extending the stale data in L1 for another 30 sec at a time, up to a maximum of FailSafeMaxDuration (in this case 3 min).

Makes sense?

5. When getting value after 7 days it returns data from factory if the data is returned under 300 ms

Not exactly: DistributedCacheSoftTimeout is the max time allowed to read from the distributed cache IF there's a value from L1 to fall back to. DistributedCacheHardTimeout is the the same, BUT for when there's no value from L1 to fall back to.

Having said that, when calling L2 after 7 days it will call the factory (since the data there is stale) and, if the factory succeds, will update both L1 and L2. If the factory fails (db down, etc) it can still re-use data from L2 by extending it for another FailSafeThrottleDuration, update L1, and then the cycle goes on up to the maximum of DistributedCacheFailSafeMaxDuration (365 days in this case).

6. When getting value after 7 days it returns expired data from 2nd level if it takes factory over 300 ms to serve response

See above.

b) On the background there is still running factory method which updates data after it is done

This is always the case: if a factory is executed completely in the background (eg: Eager Refresh) or if it starts normally but is then is moved to finish execution in the background (eg: soft/hard timeouts), as soon as it finishes it will automatically update "the cache", meaning both L1 and L2 (if configured).

7. What does FactorySoftTimeout do here?

Think about it this way: in general a "soft timeout" is used when there IS a stale value to use as a fallback, and a "hard timeout" is used when there IS NOT one.

This is in general a way for you to say to FusionCache "if you have something stale to use, you can return that faster and finish inthe background, no big deal, but if you don't have something stale to return you can wait a little bit longer (or indefinitely)".

Timeouts then exist for both when executing the factory (FactorySoftTimeout and FactoryHardTimeout) and for when talking with the distributed cache (DistributedCacheSoftTimeout and DistributedCacheHardTimeout).

So, getting back to your question of "What does FactorySoftTimeout do here?" the answer is that it is the maximum time to wait for a factory execution to finish, IF fail-safe is enabled and IF there is a stale value to fall back to: if it takes longer than that, the stale value will be temporarily re-used and the factory will end execution in the background, automatically updating the cache right after finishing execution.

Hope this helps.

[jodydonetti]

One favour: if this answers your question please mark it as the answer, thanks!

[suchoss]

Hi @jodydonetti Many thanks for your answers. In some cases it is a bit hard to wrap my head around it.

I have some additional questions:

Q: When getting value after 1 minute, but under 7 days it returns data from 2nd level
A: Correct, IF the data is in L2 (Redis may have restarted or else), and IF the communication with L2 takes less than 300 ms (because of DistributedCacheSoftTimeout).
Also, not only it returns data from L1, but it also updates L1 (to avoid going to L2 for every subsequent call).

So, in the "easy scenario" (without considering shifted updates from different nodes, Redis restarts, etc), L1 will be updated every 1 min from the L2 for up to 7 days.

This is a bit unclear - is L1 always updated with new data from L2 when its data expires, or L1 will not get data from L2 until cache is asked for data? I am asking because our L2 is going to be several GB big and we want only a small part of it in L1 (for recent requests).

Q: If FailSafeThrottleDuration is used for L1 and L2, then I probably do not understand how it works. My assumption now is:

• After L1 Duration the FailSafeThrottleDuration shifts the expiration time until FailSafeMaxDuration
• After that L1 gets data from L2 and the new expiration (in L1) is set to Now + FailSafeThrottleDuration? Or is it set to Now + Duration?

Q: In my case if there was a DistributedCacheHardTimeout set to 5 minutes, then it would throw an exception if it took more than 5 minutes to get response from L2 (since there is no fallback value in L1, because FailSafeMaxDuration is set to three minutes)?

Q: In which case is L2 used after its expiration? In our case factory might take too long and we would like to serve stale L2 data while its being refreshed (without eager refresh).

Many thanks

[jodydonetti]

Hi @jodydonetti Many thanks for your answers. In some cases it is a bit hard to wrap my head around it.

I'm thinking about a different ways to spin the docs, maybe a different approach for somebody new to FusionCache.
Will update if I come up with something.

This is a bit unclear - is L1 always updated with new data from L2 when its data expires, or L1 will not get data from L2 until cache is asked for data? I am asking because our L2 is going to be several GB big and we want only a small part of it in L1 (for recent requests).

Good question: apart from the backplane, everything in FusionCahe is "lazy", meaning it only does something when asked for.

Concrete example: your Redis cache contains millions of cached values for a total of 100 GB of data, but on a certain node only 1 value is asked. That node will only contain that single value, end of story. Everything else is only on the distributed cache.

This is also true with multiple nodes: let's say you have 3 nodes (N1, N2, N3) and 1.000 requests come in for 10 different products, but all those 1.000 requests arrive only via node N1. Only N1 will have those 10 products in memory, while the other 9 nodes have their memory cache empty.

This allows a sort of adaptive load based on actual usage: only the data being used, and only when being used, will be in each memory cache.

Makes sense?

Q: If FailSafeThrottleDuration is used for L1 and L2, then I probably do not understand how it works. My assumption now is:

• After L1 Duration the FailSafeThrottleDuration shifts the expiration time until FailSafeMaxDuration
• After that L1 gets data from L2 and the new expiration (in L1) is set to Now + FailSafeThrottleDuration? Or is it set to Now + Duration?

It is now + duration, because there's no fail-safe here.

I think a good approach may be to think about it this way, starting with L1 only, going step by step:

• you put something in the cache for Duration (let's say 10 min)
• every call inside that Duration will just return the value from it
• the first GetOrSet call after Duration will run the factory and save the result in the cache

This is the happy path, now let's add some failures:

• if, while running a factory, the database is down (just an example), the call would normally throw
• but if you enabled fail-safe and the factory is a refresh call (meaning the value was in the cache but is now expired) then that value will be saved again in the cache, temporarily. But temporarily how much? Another 10 min (the original Duration)? Nope, that would be too much. So it will be saved again for FailSafeThrottleDuration (let's say 30 sec)
• after 30 sec, the first GetOrSet call that arrives will trigger a new refresh (meaning, it will run the factory). If it succeeds, everything starts from scratch again (new value will be saved for Duration etc). If the factory fails another time (eg: the db is still down), the expired value will be saved again for another 30 sec (FailSafeThrottleDuration) and it will go on like that for another try, and another, etc
• so will this go on forever, maybe serving stale data after a month? Nope, at most it will keep doing this for FailSafeMaxDuration, so that by setting it for example to 24 hours you can say "hey FusionCache, you can keep re-using an expired value in case the db is down, and please if I keep askign for it retry every 30 sec, but after 24 hours just stop because I prefer an exception which I will handle in my application code, because it would be too risky to use such an old value"

Another "failure" may happen NOT when the factory throws, BUT whe it takes too much time: in that case we can enable fail-safe and also specify a FactorySoftTimeout so that, after that timeout, it would be as if the factory had thrown (so re-save for throttle duration up to a maximum of max duration etc).

But there's a catch here: the factory is NOT actually discarded, but it will just finish executing in the background and then immediately update the cache, getting you the best of both worlds (always fast response + cache updated asap).

Also, why "soft" timeout? Because that is the timeout used IF there is an expired value usable as a fallback. And if there's NOT? That is what FailSafeHardTimeout is for, so you can prefer a fail fast approach, meaning "I prefer handling an error fast instead of waiting too long for the data". This may seem counter-intuitive (and is in fact rarer) but is usually used for secondary data that is not so fundamental, like related products in a product page.

Ok, failures are handled, but we are still only with one level (L1, memory only): let's add L2 (distributed) in the picture and be L1+L2.

Basically you should think about all of what I just described as... the same, except that:

• when reading data "from the cache", FusionCache will check L1 and, if nothing is there, will also check L2. If valid data is in L2, it will copy it into L1 and from there on everything is as if it found it directly in L1
• when writing data "to the cache" it will write it to L1 and to L2

Makes sense?

Now the last bit: memory vs distributed specific options, like Duration vs DistributedCacheDuration.
All of the above remains the same, but when writing in the memory cache FusionCache will use Duration, and when writing in the distributed cache it will use DistributedCacheDuration (or, if null, the normal Duration).
Same thing when needing a FailSafeMaxDuration in memory cache vs DistributedCacheFailSafeMaxDuration in distributed cache.

Q: In my case if there was a DistributedCacheHardTimeout set to 5 minutes, then it would throw an exception if it took more than 5 minutes to get response from L2 (since there is no fallback value in L1, because FailSafeMaxDuration is set to three minutes)?

Oh right, soft/hard timeouts but for the distributed cache (instead of for the factory).
It's kinda the same as the ones for the factory, but for when talking with the distributed cache (again "soft" if there's a fallback value from the memory cache and "hard" if there's not).

It is particularly important for when READING from the distributed cache, because a READ implies that you have to wait for the value. Also, watch out when setting a value that is too low, otherwise FusionCahe will never have enough time to read from it, making it basically useless.

Also, when WRITING to the distributed cache, FusionCache can execute the writes in the background (if AllowBackgroundDistributedCacheOperations is enabled) so that timeouts will not be considered (since the operation is now non-blocking) and delays would not matter and all would be speedy.

But wait, what if the write on the distributed cache fails? Wouldn't that create sync issues?
In theory yes, but in practice... auto-recovery will solve it all automatically 😬
So, my suggestion is to just allow background execution and forget about it.

Woah, this has been quite a ride. I hope this helped clarifying the thing.

ps: say hi to Michal 🤣

[suchoss]

Hi @jodydonetti

Thanks for your exhausting answers. It brought some clarification 😄.

BTW I was thinking about the documentation. Flowcharts and sequence diagrams might be a way to go (I like the documentation here https://www.pollydocs.org/strategies/retry.html). I will try to draw something and let you know.

ps: I said hi to Michal 🤣

[jodydonetti]

Good point for the flowchart, I'll take a look at them and see what I can do!