CSP errors loading fonts on stage deployments

[alexgibson]

Description

When viewing stage deployments directly (e.g. at https://bedrock-stage.gcp.moz.works/en-US/) something a bit strange is happening with web font loading. Fonts still seem to load eventually, but often the first request seems to get blocked due to a CSP error e.g.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://www.allizom.org/media/fonts/Inter-Regular.d55e957612a3.woff2. (Reason: CORS header 'Access-Control-Allow-Origin' missing).

Steps to reproduce

Visit: https://bedrock-stage.gcp.moz.works/en-US/

Expected result

Web fonts should load the same without error, as when viewing https://www.allizom.org/en-US/

Actual result

The first request for each font seems to get blocked.

Environment

Firefox Nightly

[janbrasna]

This happens all the time. I've made sure I don't have any Inter, Zilla et al. installed locally to avoid false positives, and no matter how I browse or refresh, all I get is a basic sans-serif.

No surprise here, font-src 'self' only and the stylesheets are not loaded "relatively" from gcp.moz.works. but from allizom.org thus the fonts relatively load from allizom.otg too.

Either the assets need to be linked just from / root without the server part, or the font-src CSP has to be augmented the same way style-src or default-src is today.

(Maybe when addressing #14082 or #11943 one day this could be fixed with it…)

[stevejalim]

I think i'm going to need to cover this as part of #14451 else we'll have the integration tests setting off alarm bells every time

[janbrasna]

@stevejalim I'll quickly open a PR trying to fix this for moz.works or run.app URLs, you can try to deploy it to some of the demos to check the new CSP policy live.

[stevejalim]

@janbrasna Thanks but that is maybe going to overlap with the work I'd started on this. Will take a look later and confirm either way

[janbrasna]

@stevejalim No worries, I'm avoiding any lines you're working on, and eventually the PR could merge with #14453 without conflicts — I'm only trying this as PoC to get around the violations in the most simple way possible to get the noise out of reporting, without being too obnoxious in production (for both mozorg & pocket that share the same code for merging the rules). You'll find some notes to consider when you get to it…

[stevejalim]

@janbrasna I don't think you can see the existence of our k8s config repo, but adding the extra -src hostnames there is our preferred approach.

That said, I don't yet know if we'll also still need bedrock-side change if font-src somehow does not draw from default-src. Will explore but feel free to throw up a draft PR. Cheers!

[janbrasna]

@stevejalim The explainer is there: #14466 — basically two things: font-src does override default-src (but doesn't join the values in settings code), and any demo/dev env links to *.css are absolute with FQDN so no matter what hostname you use, the styles are linked from *.allizom.org, so the font files relatively are also requested from allizom.org, therefore the self here doesn't suffice. The PR is the "cheapest" fix, even though probably not the best. A little repetitive/relaxed, but predictable and consistent with other rules. This would resolve the staging hosts, I'll look into the integration test logs to get the idea how it's run there, if it has any impact too.

[janbrasna]

@alexgibson This is now on staging https://bedrock-stage.gcp.moz.works/ (and coincidentally already in prod, and neither mozorg nor pocket imploded so that's a good sign;D) and the fonts there seem to be working consistently for me in nightly geckos & webkits — could you please confirm that the intermittent issues you've kept seeing are resolved now?

[alexgibson]

The CSP errors are gone but are now replaced by a bunch of CORS errors. So I agree this bug is fixed, but now presented by a new problem :)

[stevejalim]

Yeah, @pmac and I have been discussing the CORS issue and can likely fix this at the bucket end of things

(At least the absence of CSP warnings now means we can merge #14245 without it triggering a zillion false positives every time integration tests run

[janbrasna]

💩 Ah yea, I had overrides for missing origin allows and focused only on the CSP reporting — sorry I totally forgot about these. You probably know where in the dev deployment configs this could end up if/when truly needed, I only tackled django/app specifics to close the gaps bedrock-side. There's #13940 for staging CORS et al.

[janbrasna]

So I went on to take a better look at what hostnames are being used in tests and what that means for fetch/anon resources, and the issue is slightly bigger.

TL;DR it would help if "relative" assets could be linked without the explicit hostname (like it does when run locally; or if in non-prod mode the app would use the request hostname as its hostname it uses for constructing URLs).

Because even if you allow wildcard origins in dev (or mirror the one from request origin, if you feel like keeping a restricted list of origins to let mirror back in responses), you'd still get prod.bedrock.prod.webservices.mozgcp.net that thinks it's "mozilla.org" (not "allizom.org" or anything else) in integration tests, so it hotlinks all the assets from mozilla.org not *.mozgcp.net because they're output as ://hostname/path not just /path — so not only fonts but also some SVGs (depending on the context how they're loaded, sometimes they're anonymous/fetch) fail CORS too. You can probably relax origin allows for allizom.org, but that won't help the test instances loading assets from mozilla.org 🤷

BTW the weirdest results are for manual pushes to integration tests branch, that run from yet another env test.bedrock.nonprod.webservices.mozgcp.net which links assets from storage.googleapis.com/bedrock-nonprod-test-media (subdirectory) and most of the bedrock code doesn't expect there ever might be a prefix, so a bunch of things is loaded relatively from that, but pointing at root, as /path… trying to load assets from storage.googleapis.com/media/protocol/img/*… etc., so with CORS errors you also get a couple dozen 403s from Goog not liking where we're trying to load stuff from;D I don't know how much of that really matters, just an observation there's more assets broken under different hostnames.