Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eia861 2022 #2782

Merged
merged 5 commits into from
Aug 14, 2023
Merged

Eia861 2022 #2782

merged 5 commits into from
Aug 14, 2023

Conversation

aesharpe
Copy link
Member

@aesharpe aesharpe commented Aug 10, 2023

This PR adds in EIA861 early release data.
The only quirks were that the directory name was different from the zipfile name, so the file_map includes a prefix of f8612022er/ for each of the files, and a change in skip rows for the reliability table in 2021. It also adds an encoding step for the reliability table to avoid foreign key failures.

The former might change for the regular release data. See issue #2780 (linked)

There are no tests for 861 so that's all we've got.

@aesharpe aesharpe added new-data Requests for integration of new data. eia861 Anything having to do with EIA Form 861 labels Aug 10, 2023
@aesharpe aesharpe self-assigned this Aug 10, 2023
@aesharpe aesharpe linked an issue Aug 10, 2023 that may be closed by this pull request
13 tasks
@e-belfer
Copy link
Member

I think you'll have to add reliability_eia861 to the exceptions list for the momentary_interruptions_eia table to fix this last error.

FAILED test/integration/etl_test.py::test_pudl_engine - pudl.io_managers.ForeignKeyErrors: Foreign key error for table: reliability_eia861 -- momentary_interruptions_eia (code) -- on rows [18738]

…add the encoding step so the code_fixes get applied and the foreign key error isn't tripped
Copy link
Member

@zaneselvans zaneselvans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good assuming the CI passes successfully -- we should look at some of the stuff that's downstream too, like the service territory tables.

@codecov
Copy link

codecov bot commented Aug 14, 2023

Codecov Report

Patch coverage: 100.0% and no project coverage change.

Comparison is base (cce89d1) 88.3% compared to head (b9d3228) 88.4%.
Report is 45 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #2782   +/-   ##
=====================================
  Coverage   88.3%   88.4%           
=====================================
  Files         89      90    +1     
  Lines      10206   10193   -13     
=====================================
- Hits        9021    9016    -5     
+ Misses      1185    1177    -8     
Files Changed Coverage Δ
src/pudl/metadata/codes.py 100.0% <ø> (ø)
src/pudl/metadata/sources.py 100.0% <ø> (ø)
src/pudl/workspace/datastore.py 75.6% <ø> (+1.8%) ⬆️
src/pudl/transform/eia861.py 97.4% <100.0%> (+<0.1%) ⬆️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@zaneselvans zaneselvans merged commit ef336db into dev Aug 14, 2023
8 checks passed
@zaneselvans zaneselvans deleted the eia861-2022 branch August 14, 2023 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
eia861 Anything having to do with EIA Form 861 new-data Requests for integration of new data.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Integrate EIA 861 2022 data
3 participants