PHP 8 image - intermittent 502

[alexjcollins]

During the week of the 20th December Tam and I noticed a few of the projects that were using the new image on staging would be going down and start returning a 502 error. It seemed related to deployments (i.e. it would go down shortly after a cms deployment), but that might just be a coincidence. I think TOKA went down on the Monday (20th) without a deploy.

Restarting the project would then sort the problem.

I'll add uptime monitoring to the projects I know have been updated and let you know when they drop.

Becuase of this I've held back pushing anything Craft-based to production, in case it's not just something inherent to staging.

@Tam feel free to weigh in with anything you can remember from our conversation that week.

[jamesedmonston]

Interesting. Did you or @Tam happen to look inside docker-compose logs when it happened?

I can see there's been a subsequent update to 8.1.1 (we're running 8.1.0). I'll update the images to that, and then do some further eyeballs.

[alexjcollins]

I don't think Tam did, and I didn't - sorry I know that would have been useful!

Sounds like a plan 👍

[jamesedmonston]

They're updated now. I've rebuilt TOKA on staging. Could be an idea to rebuild any of the projects you were having issues with – you can either redeploy from Buddy or run make deploy in the directory.

Will then be worth us keeping an eye on them and looking inside the logs if/when they go down again.

[alexjcollins]

@jamesedmonston Okay cool - thanks! I'll give that a go now on the projects i know have updated.

[jamesedmonston]

@alexjcollins I had the same issue on PK when I deployed a change. I'm wondering if there's some kind of cache that doesn't get cleared properly when updating from 8.0 to 8.1, and it's only resolved once rebooting the containers. PK has been completely fine since.

[alexjcollins]

I feel like it has happened a couple of times on each project (at least the couple I've been making CMS changes on recently) - is it possible that lines up with what you're saying?

[jamesedmonston]

That's interesting. Do you know if it only happens when there are project config changes, or does it happen on any deployment?

[alexjcollins]

Good question. I couldn't say with certainty, as two changes (#1 & #2) were made not long after one another but I'm fairly sure it went down after the first one, which would have just been a composer change.

[jamesedmonston]

Leaving a paper trail: there seems to be an issue with having JIT mode enabled in 8.1.x. Disabling it appears to fix the issue.

Will keep an eye on patches to see if/when they fix it.