Code

Upgrade report data pipelines#30

Code

Merged

max-ostapenko merged 81 commits intomainHTTPArchive/dataform:mainfrom reportsHTTPArchive/dataform:reportsCopy head branch name to clipboardDec 9, 2024

+4,113-201

Commits

Commits on Nov 16, 2024

• demo report

  max-ostapenko

  committed
  48a3b35

• fix local package

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  f752d9f

• crawl reports tag triggered

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  172e120

Commits on Nov 17, 2024

• Merge branch 'reports' of https://github.com/HTTPArchive/dataform into reports

  max-ostapenko

  committed
  39ae950

• timeseries added

  max-ostapenko

  committed
  eb76476

• split tables

  max-ostapenko

  committed
  4a2d145

• lint

  max-ostapenko

  committed
  a8e2137

Commits on Nov 20, 2024

• tech report tables

  max-ostapenko

  committed
  4250677

Commits on Nov 21, 2024

• check tech report sql

  max-ostapenko

  committed
  607c6b2

• Merge branch 'main' into main

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  aba1af4

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  791c0e8

• missing declaration

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  3fff267

• formatting

  max-ostapenko

  committed
  3be0274

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  4c361a9

• preOps

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  b804540

Commits on Nov 23, 2024

• dataset change

  max-ostapenko

  committed
  cc04bb6

• cwv_tech_report tested

  max-ostapenko

  committed
  9c8e567

• Merge branch 'main' into main

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  45f1095

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  e38d4f0

Commits on Nov 25, 2024

• tech_reports moved

  max-ostapenko

  committed
  1185758

• exporter function draft

  max-ostapenko

  committed
  9591bf0

• fix depependencies

  max-ostapenko

  committed
  4acdf05

• rename

  max-ostapenko

  committed
  9a1f13e

• dataset renamed

  max-ostapenko

  committed
  b2cd6b4

• storage exp draft

  max-ostapenko

  committed
  02d1db7

• date column for histograms

  max-ostapenko

  committed
  e090ebc

• dev flag

  max-ostapenko

  committed
  256fd88

• gsc export tested

  max-ostapenko

  committed
  c3f75d2

Commits on Nov 26, 2024

• pubsub sink prepared

  max-ostapenko

  committed
  6ae6e72

• export fn deployed

  max-ostapenko

  committed
  537aa60

• order incompatible with partitions

  max-ostapenko

  committed
  b5b625b

• monitoring

  max-ostapenko

  committed
  486ec2e

• lint

  max-ostapenko

  committed
  3fbf2bb

• event parsing draft

  max-ostapenko

  committed
  c34c57a

• cleanup before inserts

  max-ostapenko

  committed
  1db9ff6

• event parsing

  max-ostapenko

  committed
  f8bc51a

• partitioned exports

  max-ostapenko

  committed
  08d9fa6

• exclude scripts

  max-ostapenko

  committed
  1a2188d

Commits on Nov 27, 2024

• firestore export draft

  max-ostapenko

  committed
  0e11edf

• Merge branch 'main' into reports

  max-ostapenko

  authored
  65d310d

Commits on Nov 28, 2024

• optional description

  max-ostapenko

  committed
  d46b68e

Commits on Dec 2, 2024

• single dataset

  max-ostapenko

  committed
  8d316ce

• move

  max-ostapenko

  committed
  941e157

• incremental operations

  max-ostapenko

  committed
  4b34849

• docs update

  max-ostapenko

  committed
  dd38945

Commits on Dec 3, 2024

• firestore dict tested

  max-ostapenko

  committed
  d64a316

• reports tested

  max-ostapenko

  committed
  73c3100

• full sql export

  max-ostapenko

  committed
  d786036

• trigger params

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  3d6657d

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  b4dd900

• hashed doc ids

  max-ostapenko

  committed
  76b3d5f

• more resources and timeout

  max-ostapenko

  committed
  0819a7b

• extend timeout

  max-ostapenko

  committed
  eee1311

Commits on Dec 5, 2024

• gzip

  max-ostapenko

  committed
  9b317ca

• event example

  max-ostapenko

  committed
  33df626

Commits on Dec 6, 2024

• esm

  max-ostapenko

  committed
  58b6c3c

• more parallelization improvements

  max-ostapenko

  committed
  da4718c

• Merge branch 'main' into main

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  8e042a3

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  1a1caa6

Commits on Dec 8, 2024

• tested batch reports

  max-ostapenko

  committed
  7baa4df

• Merge branch 'reports' into reports

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  687750a

• testing fast deletion

  max-ostapenko

  committed
  d61666b

• deletion tested

  max-ostapenko

  committed
  ab581df

• limit concurrency

  max-ostapenko

  committed
  2f5bed6

• retries

  max-ostapenko

  committed
  a78999b

• wait to resolve

  max-ostapenko

  committed
  85a5690

Commits on Dec 9, 2024

• tested deployed version

  max-ostapenko

  committed
  2d116dd

• cleanup for test merge

  max-ostapenko

  committed
  9fd868a

• cwv-tech-report to prod db

  max-ostapenko

  committed
  e859d29

• note to unwrap pubsub payloads

  max-ostapenko

  committed
  0c81fb2

• cleanup

  max-ostapenko

  committed
  dbe38a1

• lint

  max-ostapenko

  committed
  dc5732e

• Merge branch 'main' into reports

  max-ostapenko

  committed
  ae875d9

• Merge branch 'main' into reports

  max-ostapenko

  committed
  a4eba5a

• revisited template builder

  

  max-ostapenko

  authored and

  GCP Dataform

  committed
  963ebfa

• cleanup

  max-ostapenko

  committed
  91822e1

• tf 6.13

  max-ostapenko

  committed
  e0de181

• lint

  max-ostapenko

  committed
  87909ef

• renamed

  max-ostapenko

  committed
  24a9bac

• aligned timeout with prod

  max-ostapenko

  committed
  ade5867

• simplify tags

  max-ostapenko

  committed
  f2b56f0