Omit secondary pages from canonical BQ tables

[rviscomi]

Until we complete the BQ dataset migration in #15, BQ users shouldn't have to think about handling secondary pages.

We should continue to store primary page data in the canonical BQ datasets:

• summary_pages
• summary_requests
• pages
• requests
• response_bodies
• lighthouse
• technologies
• blink_features
• core_web_vitals
• latest

The rollout process of secondary pages will be:

• Measure secondary pages in the 2022_05_12 crawl (aliased to 05_01 on BQ) as of Add the ability to test secondary pages #12 and Rerun 2022_05_01 #44
• Continue writing summary tables to the experimental dataset
• Create the new table schemas described in Reorganize the BigQuery datasets to be more efficient #15
• Write home page-only data to non-experimental summary tables #81

[rviscomi]

Running queries to generate home page only tables for the 2022_05_01 crawl. Here are some examples:

CREATE OR REPLACE TABLE
  `httparchive.pages.2022_05_01_desktop`
AS
SELECT
  *
FROM
  `httparchive.pages.2022_05_12_desktop`
WHERE
  JSON_VALUE(payload, '$._metadata.crawl_depth') = '0'

CREATE OR REPLACE TABLE
  `httparchive.requests.2022_05_01_mobile`
AS
SELECT
  *
FROM
  `httparchive.requests.2022_05_12_mobile`
WHERE
  page IN (
    SELECT
      url
    FROM
      `httparchive.pages.2022_05_01_mobile`)

Progress on filtering the 2022_05_01 tables down to only home pages:

• pages desktop
• pages mobile
• requests desktop
• requests mobile
• lighthouse desktop
• lighthouse mobile
• technologies desktop
• technologies mobile
• response_bodies desktop
• response_bodies mobile

The 2022_05_12 crawl still contains all secondary crawl data.

[giancarloaf]

#51 is blocking #12