Cache diverges across nodes

[markedwards]

I have a deployment in AWS which has two server nodes deployed via ECS, both connecting to a single Redis service. They both use the same keyPrefix, based on the Git commit hash. All caching is via the next unstable_cache, using tags.

If a cached item is deleted in the DB and revalidateTag is called, I'm seeing a cache divergence. One refresh will show the deleted value, the next it disappears, etc.

I'm using a standard redis-strings config following the docs. Just want to confirm that this approach should work as expected out of the box?

[markedwards]

After troubleshooting for some time, I've concluded that the issue in prod is that something is failing with the CacheHandler (debugging shows using filesystem cache handler). My assumption is the app containers cannot communicate with the Redis container, but I haven't sorted it out yet and it would be very helpful if debugging had any detail as to what is failing.

The docs could also be clearer about the fact that the cache-handler.mjs file needs to be included in the production build (its obvious in hindsight, but it would be good to call it out). And also the docs don't mention cacheMaxMemorySize: 0, although I frankly it's not clear whether that setting actually does anything.

Most importantly -- is it possible to configure things so that, if the custom CacheHandler fails for any reason, it is a terminal error? In no case do I want the in-memory cache or any other fallback, because that just means the application is broken in generally subtle ways. I would expect some detailed logging on what's wrong, followed by termination of the app.

Thanks for any pointers.

[markedwards]

Okay, various mysteries have been solved now, just following up here in case it helps anyone.

The confusing behavior turned out to be due to the next.config.js file not being present in our production docker build. Previously it was not needed at runtime, but with the new cacheHandler and cacheMaxMemorySize properties it is. This resulted in default caching behavior in production, which was very confusing and difficult to diagnose.

In short -- both the cache-handler.mjs AND next.config.js files must be present in a production build.

[better-salmon]

Hey @markedwards!

I'm sorry for not paying attention to this earlier. Thank you for providing valuable information about the files: cache-handler.mjs and next.config.js.

I want to highlight that you can debug using this environmental variable NEXT_PRIVATE_DEBUG_CACHE. If the cache handler is present, the logs will provide clues about how many handlers you've attached and which ones are currently in use.

$ npm run start

 ✓ Starting...
 ✓ Ready in 36ms
using custom cache handler @neshca/cache-handler is not configured yet
Connecting Redis client...
Redis client connected.
get from "redis-stack" /app/with-params/dynamic-true/200 false

get from "redis-stack" /app/with-params/dynamic-true/200 false — you can see that the redis-stack Handler active.

The logs will appear like this if there is no cache handler configuration:

$ npm run start

 ✓ Starting...
 ✓ Ready in 37ms
using filesystem cache handler
not using memory store for fetch cache
loadTagsManifest { version: 1, items: {} }
get /app/with-params/dynamic-true/200 undefined app false

using filesystem cache handler — this indicates that native Next.js cache is used.

Refer to the Troubleshooting docs section for more info.

[markedwards]

Thanks, yes I was using NEXT_PRIVATE_DEBUG_CACHE. The problem was, in my case, the missing next.config.js disabled everything, so I wasn't getting any useful debugging info. But indeed that is helpful if you have a configuration that is at least attempting to load.