Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract more data from FERC XBRLs and handle that new data in ETL #2821

Merged
merged 12 commits into from
Oct 6, 2023
Merged

Extract more data from FERC XBRLs and handle that new data in ETL #2821

merged 12 commits into from
Oct 6, 2023

Commits on Oct 6, 2023

  1. Update to use new version of ferc-xbrl-extractor

    @zschira @jdangerx
    zschira authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    66044a4 View commit details
    Browse the repository at this point in the history

  2. Fix issues arising from stricter typing used in pandas 2.1

    @zaneselvans @jdangerx
    zaneselvans authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    6f5a0af View commit details
    Browse the repository at this point in the history

  3. Use integer transmission circuits.

    @zaneselvans @jdangerx
    zaneselvans authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    8f0ed65 View commit details
    Browse the repository at this point in the history

  4. Use pd.to_numeric() to convert numeric strings to numeric dtypes.

    @zaneselvans @jdangerx
    zaneselvans authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    103cbd3 View commit details
    Browse the repository at this point in the history

  5. Remove obsolete references to ferc1_schema tests.

    @zaneselvans @jdangerx
    zaneselvans authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    534729d View commit details
    Browse the repository at this point in the history

  6. Get rid of an unnecessary temporary variable.

    @zaneselvans @jdangerx
    zaneselvans authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    6613941 View commit details
    Browse the repository at this point in the history

  7. Make new extractor compatible with 2021 data 

    The new extractor added some data to the 2021 XBRL archives. This caused some integration and validation test fails. I added some plants to the pudl_id mapping spreadsheet, all of which are considered totals. I.e., not real plants, but we're mapping them for the sake of giving them an ID (they are not connected to EIA records). Because this is how we treat other total records reported to FERC1.

This also updates the way that values were assigned to a slice of the ferc1_eia_train output spreadsheets. NA values were causing an issue, so I had to change how the values were being converted.

This also updates the test_minmax_rows test to reflect the new rows in the 2021 data.
    @aesharpe @jdangerx
    aesharpe authored and jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    4954a95 View commit details
    Browse the repository at this point in the history

  8. Remove migrations that were already part of dev

    @jdangerx
    jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    ed9bf32 View commit details
    Browse the repository at this point in the history

  9. Add a few plants to pudl_id_mapping 

    Totally new:

* 18012: pjm interconnection, llc / total
* 18013: new york state electric & gas corporation / see footnote
* 18014: southwest power pool, inc. / total
* 18015: public service company of colorado / community solar gardens
* 18016: the empire district electric company / n/a
  each & 73 units at 2.52 mw each)
* 18017: wisconsin electric power company / see footnote
* 18018: upper michigan energy resources company (pudl determined) / total
* 18019: new york transco, llc / total
* 18020: wilderness line holdings, llc / total
* 18021: mt. carmel public utility co / total

Mapped to existing PUDL ID:

* 8671: pacific gas & electric company, small hydroelectric generating plants
* 15000: idaho power company / hydro
* 15001: idaho power company / internal combustion
* 15068: public service company of colorado / conventional hydro
* 12926: midamerican energy company / ida grove ii wind farm (8 units at 2.3 mw
* 1287: alaska electric light and power company / salmon creek hyrdo

Note the misspelling of the plant name in 1287.

Changed:

* 15031: mt. carmel public utility co / not applicable -> ameren
  illinois company / not applicable

  This one had a mismatch between utility_id_ferc 222, which corresponds
  to Ameren, not Mt. Carmel (397).
    @jdangerx
    jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    01cb8e2 View commit details
    Browse the repository at this point in the history

  10. Update validation test expectations. 

    There are some missing data due to messy deduplication:
#2822

But we'll do the deduplication better in here:
#2899
    @jdangerx
    jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    6f37ca8 View commit details
    Browse the repository at this point in the history

  11. Add clarification on a confusing bit of code.

    @jdangerx
    jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    5a06b41 View commit details
    Browse the repository at this point in the history

  12. Filtering pudl_id_mapping.xlsx in LibreOffice changes the actual file… 

    … on disk???
    @jdangerx
    jdangerx committed Oct 6, 2023
    Configuration menu
    Copy the full SHA
    2abf505 View commit details
    Browse the repository at this point in the history