Integrate 2021+ FERC-714 XBRL data

[zaneselvans]

Overview

Thus far we have only integrated the 2006-2020 FERC-714 data, which was published as CSV dumps from VisualFoxPro DBF database files. Since 2021 this data has been transitioned to XBRL, but we haven't integrated it yet. To have the most up to date hourly demand and 10-year load forecast data available, we should get the FERC-714.

Logistics

• We are already archiving the post-2020 FERC-714 XBRL data and converting it to SQLite for convenient extraction.
• We would need to adapt / repurpose our existing XBRL extraction infrastructure to getting the FERC-714 data.
• Structurally the tables we're pulling from FERC-714 are simple, and only have a few columns each.
• Currently in PUDL we have out_ferc714__hourly_planning_area_demand.
• Transform FERC-714 load forecast table #3519 and Transform FERC-714 load forecast table #3670 will bring in the 10-year load forecast table.

What do you know about it so far?

• There's probably a bunch of new timezone / timestamp issues to clean up in the newer years of data for consistency with the old data in the hourly demand table.
• The newer years of data have even more freeform utility IDs -- there's the FERC-714 ID (which is different from the FERC-1 IDs) and then a choose-your-own-adventure ID, many of which are from EIA, but not all. So there will be some manual mapping, but there's only ~100 planning areas so this is tractable even if it's all manual.

[e-belfer]

@zaneselvans Was this issue addressed by #3809, and should it be closed? If not, it probably wants to be updated.

[zaneselvans]

Yeah I think this is good to close now! 🎉