[fetch] Introduce test generation tool & add more tests for metadata request headers

[jugglinmike]

This patch is the latest in a research project we've been running since the beginning of the year. The README.md file included in this patch summarizes the practical usage aspects and the high-level design goals.

If you're interested in learning more about the design process, you can check out:

• Fetch Metadata research report - an initial survey of the "Fetch Metadata request headers" proposal, including a review of interesting coverage areas
• Fetch Metadata Test Generation Report - a summary of some of the tool's evolution
• Prototype for fetch metadata test generation - the complete development and peer review process of the prototype

This patch replaces tests that were manually written with tests that have been procedurally generated by the new tool. I've verified that there are no regressions in coverage. To promote transparency on that detail, I've organized each replacement as a distinct commit in this pull request. If/when this is accepted for WPT, I recommend using a merge commit in order to preserve this information.

There are still more tests to be written (both to replace existing tests and to further improve coverage). I will be pushing additional commits to this feature branch over the next few days, though I do not expect to reach completeness. I'll be careful not to rewrite history to avoid disrupting the review process.

[jugglinmike]

Also CC @lweichselbaum (sorry Lukas: GitHub didn't allow me to designate you as a reviewer)

[mikewest]

I reviewed earlier iterations on this PR in detail at bocoup#13, and I'm quite supportive of landing it. Thank you for the hard work you've done to get this into reasonable shape, @jugglinmike! LGTM.

[jugglinmike]

The check named, "wpt-decision-task" failed on the latest commit, reporting:

subprocess.CalledProcessError: Command '[u'git', 'diff', '--name-only', '-z', u'0d794eb31a9920dd63065a1c4de2d14132b986f5..5e4ad738d4758f62d16410afae6485f839cf8f44']' returned non-zero exit status 128
[taskcluster 2020-08-28 18:39:39.241Z] === Task Finished ===
[taskcluster 2020-08-28 18:39:39.336Z] Unsuccessful task run with exit code: 1 completed in 45.22 seconds

It had a similar problem with the prior head of this branch, though it was successful when I initially opened this pull request. Both of the failed commands work locally, so this may be related to branch availability. This branch is located on a fork of WPT.

@Hexcles (hello!) are you familiar with the failing task?

[Hexcles]

I saw a similar failure on my PR just now but forgot to keep the link. Closing and reopening the PR seems to have fixed it.

[jugglinmike]

Thanks; I'll give that a try

[arturjanc]

Thanks for doing this, Mike!

I really like the readability of the config format in fetch/metadata/tools/fetch-metadata.conf.yml.

[jugglinmike]

I'm glad to hear that, @arturjanc. The configuration file is where the complexity of the generation logic is most apparent, so I really tried to make it as clear as possible. There's still a lot to take in, but it's encouraging to hear that someone other than me can make some sense of it.

[zcorpan]

From what I understand, this is reviewed and ready to land. The stability checks failed I believe because this introduces too many tests and so the stability checks time out.

@jugglinmike is currently working on a different project. Worst case if these tests somehow catch fire, we can revert.

Any objection to merging?

[mikewest]

Ha! I thought this landed months ago. Yes. Please merge it. :)

[foolip]

@jugglinmike I've rebased this branch to resolve conflicts, can you enable "Allow edits and access to secrets by maintainers" so I can push those? (@zcorpan asked me)

[foolip]

https://wpt.fyi/results/fetch/metadata?label=pr_head&max-count=1&pr=25247 show the results.

Everything looks OK except https://wpt.fyi/results/fetch/metadata/generated/appcache-manifest.https.sub.html?label=pr_head&max-count=1&pr=25247 where there's a harness error:

Test named "sec-fetch-site - Same origin" passed a function to async_test that returned a value. Consider using promise_test instead when using Promises or async/await.

This should be fixed by just not returning a promise, it's already chained correctly to pass or fail the async test.

Stability jobs are failing on /webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/no-cors.https.html because fetch/api/resources/redirect.py was updated, but the test does a regular GET request and I presume was already flaky. I'll admin merge to ignore this failure.

[foolip]

https://wpt.fyi/results/fetch/metadata/generated/appcache-manifest.https.sub.html?label=pr_head&max-count=1&pr=25247 looks better now, no longer a harness error.

[zcorpan]

Thank you @foolip !