IDistributedCache only option

[brianmat]

I like all of the functionality currently provided within FusionCache, but I would like the ability to turn off the memorycache option.

When using Azure Functions I would prefer to just go straight to Redis and not have the overhead of memorycache due to their stateless nature. I still want to have the fallback value capability in the event Redis is unavailable, but I just don't need the extra step of dealing with memorycache.

This would also help with longer-lived data which could change very infrequently but is accessed often. If I have a long TTL on a value I could have an instance where a load-balanced environment could be out of sync. Let's say I have a TTL of 90 minutes on data but due to a production problem I need to update the current value in Redis. I have no way of evicting the current memorycache values even though Redis has a more current (and correct) value.

This does touch a bit on your backplane idea, but in that case, I would still like to have a preferred cache type for a value. I think this could go in the FusionCacheEntryOptions as a PreferredCache property. This value could be an enum of Local or Distributed. This way I could determine which cache to prefer for a value.

So, this could be seen as 1 of 2 proposals to the FusionCacheEntryOptions:

public CacheLocation CacheLocation { get; set; } = CacheLocation.All;

public enum CacheLocation
{
	All,
	LocalOnly,
	RemoteOnly
}
	

public PreferredCache PreferredCache { get; set; } = PreferredCache.Local;

public enum PreferredCache
{
	Local,
	Remote
}

Looking at the GetOrSetEntryInteral I am not sure if one will be easier than the other. If I had to pick only one option I would say option 1 would do what I really want at the moment. Since I know I would be losing performance by going to Redis first then memorycache would have a smaller performance benefit since I can still use the fallback option.

What are your thoughts?

[jodydonetti]

Hi @brianmat thanks for your interest in FusionCache!

Your issue is very detailed and well thought out, and it covers a scenario I'm not currently using a lot (Azure Functions) so there's a lot of food for thought.

I'll take some time to think about it and see if I can come up with a reasonable path forward and a good design that makes sense.

I'll update you asap.

[arnoldsi-vii]

Hi @jodydonetti , Any updates on that?

[jodydonetti]

Hi @arnoldsi-vii , I've been quite busy with finishing the design and impl of the new backplane feature, which I finally released yesterday as a first alpha release 🎉

All of this took way more time and effort than expected to be able to support all the use cases that emerged from the community.

Anyway, I'm saying this because I hope to be able to finally release the next official version very soon (hopefully next week 🤞), and after that I will go over the backlog which includes this very issue, too.

Thanks.

[arnoldsi-vii]

I really like how this library progressing and In a future I probably will be able to add some PRs.
My team currently debating between EasyCaching Library and yours (I do like yours :))

Our platform Using mostly using Serverless where local cache is not really needed due serverless limits, but also EC2 machines where local cache with distributed do a great job

[jodydonetti]

I really like how this library progressing

Thanks, appreciate that!

My team currently debating between EasyCaching Library and yours

If it can be useful youcan find a comparison here.
Also, if they find some wrong informations please tell your colleagues to let me know so I'll update that.

(I do like yours :))

Ahah thanks 😬

Our platform Using mostly using Serverless where local cache is not really needed due serverless limits, but also EC2 machines where local cache with distributed do a great job

Yeah the serverless scenario is something I'd like to support more. I hope to be able to move on very soon, after the final release of the backplane.

[arnoldsi-vii]

@jodydonetti thank you.
I think for server less scenarios it’s better to save only on distributed cache and have fallback to memory cache for short periods.
Basically switch places.

few ideas for future:

• Circuit breaker - with polly it can be very easy implementation
• Only distributed cache option (not sure it worth developing, because people can use default one)

[jodydonetti]

I think for server less scenarios it’s better to save only on distributed cache and have fallback to memory cache for short periods. Basically switch places.

Ok, help me to understand more please.
If it saves only on distributed it means the memory cache would be empty, and if it is empty it means it cannot be used as a fallback. Or am I missing something?
And in case we are talking about both memory + distributed again, now that there is a backplane (which would synchronize all the memory caches) what would be the benefit of always getting data from the distributed first? It seems to me it would be always be slower.

few ideas for future:

• Circuit breaker - with polly it can be very easy implementation

I thought about that initially, but I preferred to avoid an extra dependency.
Do you think the current impl is lacking somehow?
If so, I may think about dropping my own impl and maybe just use Polly.

And thanks for this conversation, it's useful to me for collecting different experiences and opinions from the community, and it will help me to shape the future of FusionCache!

[arnoldsi-vii]

@jodydonetti i didn’t checked the current implementation of circuit breaker, I used Polly in our projects and it worked great (Polly has a lot of abilities as you know)

As for serverless scenario. Lambda (aws scenario) lives for a very short time and has memory limit that you can use, therefore occupying memory in lambda for cache seems not a best option, usually redis in cloud environment close to those lambdas (serverless functions).
I know it will be slower, not sure you will really feel the impact tho. Imagine you server million requests from different tenants on same lambda, the cache size can create timeouts or service failures due lambda limitations.
One solution is to limit memory cache, there is a default “SizeLimit” property of IMemoryCache, but it really works odd.

to summarize, in serverless scenario main cache is distributed and backup is memory

[arnoldsi-vii]

@jodydonetti I want to give another example of serverless and would like to hear you thoughts of how the library will operate.

Let's say we have lambda which will run for 500ms. user made first call to save the data. Memory save was 10ms. Then its started to save to distributed cache (in background) which from some reason started to take more than 500ms. This means the cache will never hit the distributed cache data and other nodes will never know about the change and might show wrong data.

One solution to this issue is to disable background factory in serverless scenario, but from what I saw, if it fail to save to distributed cache it only log the error. Maybe you can add fail event (not sure that FailSafeActivation is that event).

Other solution as I mentioned before is to make distribution cache as main cache, but I think it will lose the purpose of this awesome library.

Let me know that you think.

[jodydonetti]

@arnoldsi-vii thanks for the additional info, I'll think about it and let you know!

[brianmat]

@arnoldsi-vii I don't think making distributed cache the main cache would hamper the library that much. In my opinion, one of the strong points of FusionCache is the ability to handle the fallback values correctly. It's easy to implement a simple Redis cache, but there's a lot of thought put into the cache update logic and I think that's very valuable.

Serverless code does add a wrinkle and I think memorycache has a more limited value due to recycling of apps and management of state.

[jodydonetti]

Hi @brianmat and @arnoldsi-vii , I'm getting back to my backlog of things to consider, so here we go.

In general the main observation behind this proposal was that in some scenarios like serverless, the 1st layer (memory cache) can either 1) be avoided or 2) not be so prominent, since the instances where the code runs are short-lived.

A suggested approach may be to just use a distributed cache (IDistributedCache) directly without using FusionCache at all, but some of FusionCache features like fail-safe and soft/hard timeouts would be useful to have there too.

🗓️ The times they are a-changin'

Some time has passed, and in this time some features have been added and changes have been made: because of this I'd like to get back to some of your previous points to re-evaluate them.

The most important ones are the introduction of the backplane (which solves the synchronization problem) and/or the DistributedCacheDuration recently introduced (here and here).

NOTE: I may comment again on some of your points already commented before, but I want to take this opportunity to re-evaluate everything so please excuse me if I repeat myself.

But first, let's get back to the 2 proposals.

2️⃣ The 2 options

Basically 2 possible solutions have been suggested:

1. NO MEMORY CACHE: add the ability to skip/disable the memory cache, and use only the distributed cache
2. CHANGE ORDER: add the ability to change the order of use, to work first with the distributed cache and only then with the memory cache

Let's reason about them a little bit.

Option 1: no memory cache

One of the reasons against simply using a distributed cache instance (IDistributedCache) though is that it's useful to have fail-safe and the whole fallback behaviour. The problem though is that without a memory cache, there's nothing to fall back to in case there's a problem with the distributed cache, so I'm failing to see how that would work.

If it's true that some of the other features like soft/hard timeouts would still work, others would not, or at least not fully. For example the cache stampede prevention would only work partially: while the factory is running you would have cache stampede prevention (so not a lot of database calls) but the same cannot be said for the initial distributed cache calls, all of which will be made (since without a local memory cache there would be nowhere to locally save the data and let other callers use it 🤷).

Option 2: change order

In this case the idea would be to keep the memory cache (so memory will be allocated, etc) but we would first check the distributed cache, and only then we would check the memory cache in case of problems.

Here the problem would be the same as above for the cache stampede prevention related to the distributed cache, since all the calls to the distributed cache would need to be made.

Other than that the points being made were related to local memory caches out-of-sync, and I don't see much value in this approach since the introduction of the backplane and the DistributedCacheDuration mentioned above.

Now I'd like to comment on specific points made by you.

🙋‍♂️ Points made by @brianmat

When using Azure Functions I would prefer to just go straight to Redis and not have the overhead of memorycache due to their stateless nature. I still want to have the fallback value capability in the event Redis is unavailable, but I just don't need the extra step of dealing with memorycache.

As I said without a memory cache there would be nothing to fall back to, in case Redis would fail. How would you approach this in such scenario?

This would also help with longer-lived data which could change very infrequently but is accessed often. If I have a long TTL on a value I could have an instance where a load-balanced environment could be out of sync. Let's say I have a TTL of 90 minutes on data but due to a production problem I need to update the current value in Redis. I have no way of evicting the current memorycache values even though Redis has a more current (and correct) value.

One thought in general is that, after the beginning of this discussion, I've finally introduced the backplane which would really solve the synchronization problem between an updated distributed cache and a local memory cache.

On top of that the recent introduction of the DistributedCacheDuration mentioned above added ability to specify a different duration for the distributed cache so that you can have, say, a local duration of 1 min and a distributed cache duration of 1 hour.

Do you think one of these new approaches (either a backplane or different durations) would cover your scenario?

This does touch a bit on your backplane idea, but in that case, I would still like to have a preferred cache type for a value. I think this could go in the FusionCacheEntryOptions as a PreferredCache property. This value could be an enum of Local or Distributed. This way I could determine which cache to prefer for a value.

I'm open to this idea, but at the same time it may be a lot of work and introduce new behaviours and edge cases to handle, so I'd like to know if you are still of the same idea and exactly why you would still like to specify a preferred cache type with the other 2 options available.

In my opinion, one of the strong points of FusionCache is the ability to handle the fallback values correctly. It's easy to implement a simple Redis cache, but there's a lot of thought put into the cache update logic and I think that's very valuable.

Agree, but I think you would agree that at this point we are not talking about the option 1 (no memory cache) approach, since there would not be a fallback in case of Redis problems, but only about the option 2 (distributed-first-memory-second), am I right?
I'm trying to reduce the scope of the discussion to better analyze it.

Serverless code does add a wrinkle and I think memorycache has a more limited value due to recycling of apps and management of state.

Since I don't have a lot of real-world experience with a serverless approach, I'd like to better understand how much time (I mean the lifetime of each instance/pod/whatever) are we talking about here: seconds? minutes?

I'm asking because in a scenario where the hypothetical serverless endpoints are called frequently I fail to see the benefits of not having an instance that runs for some time, where a local cache can give you a lot of advantages even if we are talking about a relatively small amount of time (eg: think microcaching) including full cache stampede prevention etc.

And if instead these endpoints are not called that frequently, I think the 1st pass on the memory cache would not be that impactful on the overall run time (I think we are talking about microseconds). It would be like micro optimizing a foreach loop when there's a database query that is taking a way longer execution time.

But again, I'm not that experienced with a serverless approach, so if you can help me understand more I would appreciate that 🙏

🙋‍♂️ Points made by @arnoldsi-vii

I think for server less scenarios it’s better to save only on distributed cache and have fallback to memory cache for short periods.
Basically switch places.

To have the fallback on memory we should 1) not exclude memory and 2) also save on memory, otherwise there wouldn't be anything there to use as a fallback. And when you finally say "Basically switch places" that to me seems like a confirmation that you are not talking about option 1 (no memory cache) but instead about option 2 (distributed-first-memory-second), did I understood it correctly?

As for serverless scenario. Lambda (aws scenario) lives for a very short time and has memory limit that you can use, therefore occupying memory in lambda for cache seems not a best option, usually redis in cloud environment close to those lambdas (serverless functions).

Here instead you seem to suggest avoiding memory to consume less memory.

I know it will be slower, not sure you will really feel the impact tho. Imagine you server million requests from different tenants on same lambda, the cache size can create timeouts or service failures due lambda limitations.

If we are talking about millions of requests I'm not sure a serverless approach is the best use of resources, but again I'm not that experienced with the serverless world so if you can explain a bit more that would be really helpful 🙏

Also, in such a scenario the memory cache has a lot of advantages:

• is always there for sure (no network issues can make it "disappear" randomly, no Fallacies of distributed computing etc...)
• is incredibly fast compared to the distributed one
• has the best data locality in respect of where the data will be used (eg: it sits inside your app's memory)

to summarize, in serverless scenario main cache is distributed and backup is memory

Ok so to me what it's clear that you would like is distributed first, memory second (so, option 2).

I want to give another example of serverless and would like to hear you thoughts of how the library will operate.
Let's say we have lambda which will run for 500ms. user made first call to save the data. Memory save was 10ms. Then its started to save to distributed cache (in background) which from some reason started to take more than 500ms. This means the cache will never hit the distributed cache data and other nodes will never know about the change and might show wrong data.

Not really, if I got the example right: if you don't actively enable background processing of distributed cache operations it will not run in the background. Also if you use the backplane the changes will be automatically propagated as soon as possible, so that the other nodes will be automatically synchronized.

One solution to this issue is to disable background factory in serverless scenario

Watch out about the difference between background processing of the factory (which will happen only if you also enable timeouts, and the timeouts actually occur) and background processing of the distributed cache (which will happen only if you set AllowBackgroundDistributedCacheOperations to true, and by default is false).

In any other case the code will wait for everything to finish.

but from what I saw, if it fail to save to distributed cache it only log the error

This was true, but is not anymore: if you really want you can enable the new ReThrowDistributedCacheExceptions option, which is something I've recently added thanks to a community request.
This option is part of the FusionCacheEntryOptions, so it can be set per-call, in a very granular way.

While we are here though I'd like to point out that manually managing distributed cache errors may be a little too much to handle, so think about it.

🏁 Conclusions

If you withstood this giant wall of thext, thanks 😅, I really appreciate your involvement and your contributions!
Please let me know what you think so we can work something out.