Extract more data from FERC XBRLs and handle that new data in ETL

[jdangerx]

Changes required to get the FERC 1 assets materializing properly, while pointed at the new XBRL extractor (for a more-complete xbrl2sqlite) and the new XBRL archives.

• change how we find the filings to run xbrl2sqlite on
• update how we call ferc-xbrl-extractor - to match what's on the api_compat branch
• change num_transmission_circuits from int to number - this value contains "0.0" which int doesn't like, but float can handle. Potentially we can do a pre-conversion from string to float, and then convert that to int when we're applying dtypes. This was breaking transmission_statistics_ferc1.
• drop end-of-previous-year values for the instant table that feeds utility_plant_summary_ferc1
• add total and other as utility type categories across the board. These were being turned into NAs because they were "unrecognizable" categories, which then led to spurious dupes down the line, and broke electric_plant_depreciation_changes_ferc1.

One big change we made in the extractor itself was to take multiple filings from one entity and merge them, treating later filings as updates to earlier filings.

TODO:

• remove "grab last filing only" logic - this should be obviated by the "take multiple filings and merge them" logic mentioned above.
  • we do still have to refactor to pass the report publish time into ferc-xbrl-extractor instead of relying on the ReportDate fact, but I think that can be a follow-up PR.
• fix unit tests that touch our extract.xbrl logic, since we changed the logic quite a lot
• release api-compat as ferc-xbrl-extractor 1.0, and point PUDL at that instead of this git ref we have now
• see if we can standardize on new-style or old-style archives - this code is currently written to handle "new-style" archives that have the taxonomies separately archived from the XBRL files, but all archives except for 10.5072/zenodo.1234455 are still old-style.

[jdangerx]

Mostly a TODO list for further review / massage.

[jdangerx]

OK, this is in the commit message, but I went ahead and committed some changes. @aesharpe let me know if these are reasonable:

Totally new:

• 18012: pjm interconnection, llc / total
• 18013: new york state electric & gas corporation / see footnote
• 18014: southwest power pool, inc. / total
• 18015: public service company of colorado / community solar gardens
• 18016: the empire district electric company / n/a
  each & 73 units at 2.52 mw each)
• 18017: wisconsin electric power company / see footnote
• 18018: upper michigan energy resources company (pudl determined) / total
• 18019: new york transco, llc / total
• 18020: wilderness line holdings, llc / total
• 18021: mt. carmel public utility co / total

Mapped to existing PUDL ID:

• 8671: pacific gas & electric company, small hydroelectric generating plants
• 15000: idaho power company / hydro
• 15001: idaho power company / internal combustion
• 15068: public service company of colorado / conventional hydro
• 12926: midamerican energy company / ida grove ii wind farm (8 units at 2.3 mw
• 1287: alaska electric light and power company / salmon creek hyrdo

Note the misspelling of the plant name in 1287.

Changed:

• 15031: mt. carmel public utility co / not applicable -> ameren
  illinois company / not applicable

  This one had a mismatch between utility_id_ferc 222, which corresponds
  to Ameren, not Mt. Carmel (397).

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (67822df) 88.6% compared to head (2abf505) 88.5%.
Report is 1 commits behind head on dev.

Additional details and impacted files

@@           Coverage Diff           @@
##             dev   #2821     +/-   ##
=======================================
- Coverage   88.6%   88.5%   -0.1%     
=======================================
  Files         90      90             
  Lines      10809   10795     -14     
=======================================
- Hits        9577    9563     -14     
  Misses      1232    1232             

Files	Coverage Δ
src/pudl/analysis/ferc1_eia_train.py	53.8% <100.0%> (+0.8%)	⬆️
src/pudl/extract/xbrl.py	95.5% <100.0%> (-1.6%)	⬇️
src/pudl/metadata/classes.py	86.4% <ø> (ø)
src/pudl/output/ferc714.py	96.2% <100.0%> (ø)
src/pudl/transform/classes.py	94.6% <100.0%> (+<0.1%)	⬆️
src/pudl/transform/ferc1.py	96.6% <100.0%> (+<0.1%)	⬆️
src/pudl/transform/params/ferc1.py	100.0% <ø> (ø)
src/pudl/workspace/datastore.py	77.1% <100.0%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zaneselvans]

The integration tests are passing so I assume this isn't an issue, but I was confused by the removal of the ferc_xbrl fixture. Did it get replaced by something previously, but not ripped out?

[zaneselvans]

Without this fixture, how are the FERC XBRL databases being generated for use in the ETL tests, and how are we doing integration testing to ensure that we're able to extract data from all the forms? Is this just cruft that's been replaced by other fixtures now?

[jdangerx]

These guys are being generated by the ferc_to_sqlite_xbrl_only fixture, now.