Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate EIA 861 2022 data #2780

Closed
13 tasks done
Tracked by #2699
aesharpe opened this issue Aug 10, 2023 · 0 comments · Fixed by #2782
Closed
13 tasks done
Tracked by #2699

Integrate EIA 861 2022 data #2780

aesharpe opened this issue Aug 10, 2023 · 0 comments · Fixed by #2782
Assignees
@aesharpe
Labels
eia861 Anything having to do with EIA Form 861 new-data Requests for integration of new data.

Comments

@aesharpe
Copy link
Member

aesharpe commented Aug 10, 2023 •
edited
Loading

  • Add the new Zenodo archive DOI values to pudl/workspace/datastore.py.
  • Run the datastore script to download the new year of data.
  • Add the new year to the working_partitions in pudl/metadata/sources.py
  • Add the new year to the years in etl_full.yml and etl_fast.yml.
  • Update the information in pudl/package_data:
    • file maps (add _Early_Release suffix!)
    • column maps (probably the same)
    • page maps (probably the same)
    • skip footer (probably the same)
    • skip rows (+1 for early release data)
  • Launch dagit and refresh the code location.
  • Materialize the raw_eia861 asset group. Look out for warnings in the logs about missing or extra columns. If they appear, check and update the package_data accordingly.
  • Materialize the clean_eia861 asset group. You'll probably see some errors related to encoding. Take a look at which column it's talking about and look at metadata/resources/eia.py to see which encoder in CODE_METADATA to tweak.

There are no tests for EIA861....so....yeah.
@aesharpe aesharpe mentioned this issue Aug 10, 2023
Closed
@aesharpe aesharpe changed the title Integrate EIA 861 2022 data (not currently released) Integrate EIA 861 2022 data Aug 10, 2023
@aesharpe aesharpe self-assigned this Aug 10, 2023
@aesharpe aesharpe added eia861 Anything having to do with EIA Form 861 new-data Requests for integration of new data. labels Aug 10, 2023
@aesharpe aesharpe moved this from New to In progress in Catalyst Megaproject Aug 10, 2023
@aesharpe aesharpe linked a pull request Aug 10, 2023 that will close this issue
Eia861 2022 #2782
Merged
@aesharpe aesharpe mentioned this issue Aug 10, 2023
Merged
@aesharpe aesharpe moved this from In progress to In review in Catalyst Megaproject Aug 14, 2023
@github-project-automation github-project-automation bot moved this from In review to Done in Catalyst Megaproject Aug 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aesharpe aesharpe

Labels
eia861 Anything having to do with EIA Form 861 new-data Requests for integration of new data.
Projects
Catalyst Megaproject
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Eia861 2022 catalyst-cooperative/pudl
2 participants
@zaneselvans @aesharpe