Skip to content
This repository has been archived by the owner on Dec 18, 2024. It is now read-only.

Create a process to backfill all tables from HARs older than March 2022 #138

Closed
giancarloaf opened this issue Sep 1, 2022 · 0 comments · Fixed by HTTPArchive/dataform#10
Closed

Create a process to backfill all tables from HARs older than March 2022 #138

giancarloaf opened this issue Sep 1, 2022 · 0 comments · Fixed by HTTPArchive/dataform#10
Assignees
@giancarloaf
Milestone
M4: Improving dis...

Comments

@giancarloaf
Copy link
Collaborator

giancarloaf commented Sep 1, 2022 •
edited
Loading

There are some attributes necessary for the pipeline to function properly that are only available in recent HARs (i.e. March 2022 and on).

We will need a methodology to backfill older tables on request.

A suggestion from @rviscomi - this might include nulling out fields where the necessary data is missing from HARs, and supplementing from the historical tables.

Reference this method used previously for generating all.pages from legacy tables.
#15 (comment)

Originally posted by @rviscomi in #136 (comment)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@giancarloaf giancarloaf

Labels
None yet
Projects
None yet
Milestone
M4: Improving discoverability
Development

Successfully merging a pull request may close this issue.

Backfills & deprecating legacy tables HTTPArchive/dataform
2 participants
@rviscomi @giancarloaf