Occasionally assets fail to load (404)

[MadLittleMods]

Occasionally our assets fail to load and a return a 404 (observed on https://archive.matrix.org/). It seems like this spawns from our origin server serving a 404 : Cannot GET /assets/styles-63d89b7d.css (Express error). So for some reason the express.static(...) call can't find our assets on disk.

https://github.com/matrix-org/matrix-public-archive/blob/5caf9dc1b8d5e98fdc99ab06ab9b9d09d2f546ed/server/routes/install-routes.js#L42-L43

Cloudflare will cache our 404 responses for 3 minutes. And things seem to come back after the cache expires.

This is a bit confusing though because Cloudflare should be caching our assets forever (for 1-year) after seeing a successful response. I even see the Cache-Control: max-age=31536000 header in place and responds with Cf-Cache-Status: HIT indicating that Cloudflare is using the cache.

Potential solutions

Find the root cause

What's causing Express to not be able to find the assets on disk sometimes?

Always serve the /assets from Nginx

We could maybe workaround the problem by serving the /assets directory directly from nginx instead of relying on Express. I'd rather find the root cause first in any case.

Why isn't Cloudflare serving the cached response?

To be determined

Or what's causing Cloudflare to bust it's own cache when our origin server starts spitting out a 404?