Conditional Refresh and Duration

[tjmoore]

I'm trying to implement Conditional Refresh based on a modified timestamp. For example on a file based database where if the file has been modified then I want the cache to update fairly quickly, otherwise return the last cached value.

However it seems the conditional refresh check only occurs when the Duration expires. Therefore if I set Duration to say 2 minutes, it won't check the database for modified until 2 minutes has expired.

I can set Duration to a shorter value like a second or two, and it will regularly run the conditional check, though I seem to be hitting performance issues under load. Noting that the database can change frequently (between every 10 to 30 seconds) and some of the cache factory operations can take a few seconds. I've got FailSafe on and FactoryTimeout at 200ms to handle long factory cache time and expecting anything longer to cache in the background and caller gets the last cached value. However I'm seeing my load tests back up and responses can increase greatly over time and I start getting thread pool starvation.

Perhaps because the Duration is short makes every caller wait for a fresh cache value instead of getting stale value when returning NotModified() ?

For example:

(where LastUpdate is a DateTimeOffset property, key is the cache key and addItemFactory is the factory method to call)

return cache.GetOrSet<T>(key,
    (ctx, ct) =>
    {
        if (ctx.HasLastModified && LastUpdate <= ctx.LastModified)
        {
            return ctx.NotModified();
        }

        return ctx.Modified(addItemFactory(), lastModified: LastUpdate);
    },
    options => 
        options
            .SetDuration(TimeSpan.FromSeconds(2))
            .SetFailSafe(true)
            .SetFactoryTimeouts(TimeSpan.FromMilliseconds(200))
    );

My tests are complicated to show, but they're simulating hundreds of concurrent requests on an API every few seconds, combined with background updates to the database.

Basically I need response times on the API to be as fast as possible and not delayed by cache entries being rebuild from factory, but cache to update in the background relatively frequently if database has known to be changed.

I've compared this vs LazyCache where I have a crude modified check before calling the cache method and I remove the key from the cache if known to be modified so it forces re-cache. This works with LazyCache and I get less pile-up on the load test, but response time average can increase for a while each time something invalidates and the factory methods have to run. The response times are a bit too slow for the API still.

i.e. LazyCache performs better but it's not ideal which is why I'm looking at FusionCache, but I'm not quite getting the performance I want.

The ideal is to sort out the slow factory methods in my code, but then with the background caching and maybe the cache stampede protection in FusionCache I'd expect it to be less of an issue and not impact on API response times.

[tjmoore]

I've ended up improving performance elsewhere in my code that has improved LazyCache performance to the point that I don't seem to need the factory timeout any more, so less need for Fusion Cache.

[jodydonetti]

I've ended up improving performance elsewhere in my code

Awesome, cache or no cache that is a good result!

that has improved LazyCache performance to the point that I don't seem to need the factory timeout any more, so less need for Fusion Cache.

No problem: if one day you'll need one of the other features of FusionCache you know where to find it 😬

Btw, by re-reading the description of your scenario I thought about a seemingly similar one I encountered a couple of times lately where I initially used a cache, but ended up not using a cache at all, because in the end it was the right thing to do.

I'm just wondering if this is the same case here... let me know if you want to discuss it further.

[jodydonetti]

Hi @tjmoore , and thanks for considering FusionCache!

However it seems the conditional refresh check only occurs when the Duration expires.
Therefore if I set Duration to say 2 minutes, it won't check the database for modified until 2 minutes has expired.

Yep: conditional refresh is still a refresh, and a refresh happens only when Duration expires, otherwise it would not be a cache.
What I mean is that if, for every call to GetOrSet the cache would do a refresh, with 100 GetOrSet calls you would end up having 100 database call... so effectively no cache.
Or am I missing something?

I can set Duration to a shorter value like a second or two

That is normally know as microcaching, and it usually works great.

and it will regularly run the conditional check, though I seem to be hitting performance issues under load. Noting that the database can change frequently (between every 10 to 30 seconds) and some of the cache factory operations can take a few seconds. I've got FailSafe on and FactoryTimeout at 200ms to handle long factory cache time and expecting anything longer to cache in the background and caller gets the last cached value. However I'm seeing my load tests back up and responses can increase greatly over time and I start getting thread pool starvation.

The situation as described is quite strange and not that clear to me honestly, I would need to understand more.

(where LastUpdate is a DateTimeOffset property, key is the cache key and addItemFactory is the factory method to call)

Is LastUpdate the last update of the file, somehow obtained externally from the cache, maybe via a watcher?

Basically I need response times on the API to be as fast as possible and not delayed by cache entries being rebuild from factory, but cache to update in the background relatively frequently if database has known to be changed.

How do you know that the database is changed?

I've compared this vs LazyCache where I have a crude modified check before calling the cache method and I remove the key from the cache if known to be modified so it forces re-cache.

Interesting, why the different approach with LazyCache?

This works with LazyCache and I get less pile-up on the load test

Because there's no soft timeout support, I suppose.

but response time average can increase for a while each time something invalidates and the factory methods have to run.

Same as above: because there's no soft timeout support.

i.e. LazyCache performs better

But as said with a different approach, right?

but it's not ideal which is why I'm looking at FusionCache, but I'm not quite getting the performance I want.

Honestly: if you're happy with LazyCache and don't need the extra features of FusionCache, by all means stay with LazyCache.
I won't try to sell you FusionCache or anything 🙂
If one day you'll decide to switch to FusionCache, happy to help.