Integrate EIA860 2023 Early Release Data

[aesharpe]

Annual Updates Docs: https://catalystcoop-pudl.readthedocs.io/en/latest/dev/existing_data_updates.html

Tasks

Beta Give feedback

1. Archive early release data
2. Map the Structure of the New Data
3. Test Data Extraction
4. Update Table & Column Transformations
5. Update the PUDL DB Schema
6. Run the ETL
7. Connect Datasets
8. Update the Output Routines and Run Full Tests
9. Run and Update Data Validations
10. Update the Documentation
11. Add the FK relationship to the new EIA860 storage direct support plant/generators #3695
    6 of 6
    data-update eia860 +2
    

Extraction Notes

• There are some new solar and storage columns. I tried to name them but would love input:
  • Most of the boolean columns in the storage tabs use the prefix served_ because EIA's descriptions of them indicate its a bool of whether or not a unit served x/y/z thing. but these new columns have descriptions that are more "indicates whether the storage device is....."
  • Support other units.
• There is a new proposed storage tab!

Transform Notes

• There were a small handful of null generator ids. In the ownership table transform, there were two different versions of dealing with the same plant with null generator ids. I condensed these.
• Question: Are the operational statuses of these solar/storage plants ending up anywhere (presumably in the annual harvested table)? Do we need to add them based on their tab? Answer: There are status columns in these og tabs and that is getting properly slurped into the harvested annual assets.

PUDL ID Mapping Notes

• I used a very small amount of notebook sleuthing to do a better/quicker/less laggy way to map the plant ID's. I will probably check this in as a little devtools notebook.

Tests & Validation Notes

• why are there some negative values here? my assumption is that w/ this new 860 ER we are replacing the last 860m from 2023. So we had a chunk of 860m generators, some of which could no longer be in the ER version. This makes sense so far bc it looks like this doesn't happen to the 860 tables that are not included in the 860m data. Will confirm by using the sql differ notebook.
  • okay yes I have confirmed my suspicion via investigating the differences between the old and new _out_eia__yearly_generators. All but one of the newly missing generators (the one with the "nan" generator_id that is now being removed in the transform) is missing from 2023. Weirdly most of them are retired. I spot checked the retired and other tabs in the ER sheets and it seems like these missing guys are truly missing.

Validation errors to fix

Beta Give feedback

1. eia_test[eia_raw-denorm_emissions_control_equipment_eia860-56616-56616-56616] - ValueError: denorm_emissions_control_equipment_eia860: found 62102 rows, expected 56616. Off by 9.690%
2. eia_test[eia_raw-boiler_cooling_assn_eia860-41282-41282-41282] - ValueError: boiler_cooling_assn_eia860: found 44185 rows, expected 41282. Off by 7.032%
3. eia_test[eia_raw-boiler_stack_flue_assn_eia860-41673-41673-41673] - ValueError: boiler_stack_flue_assn_eia860: found 44579 rows, expected 41673. Off by 6.973%
4. eia_test[eia_raw-utils_eia860-146776-146776-146776] - ValueError: utils_eia860: found 147877 rows, expected 146776. Off by 0.750%
5. eia_test[eia_raw-boiler_emissions_control_equipment_assn_eia860-77705-77705-77705] - ValueError: boiler_emissions_control_equipment_assn_eia860: found 83977 rows, expected 77705. Off by 8.072%
6. eia_test[eia_raw-emissions_control_equipment_eia860-56616-56616-56616] - ValueError: emissions_control_equipment_eia860: found 62102 rows, expected 56616. Off by 9.690%
7. eia_test[eia_raw-boil_eia860-86675-86675-86675] - ValueError: boil_eia860: found 88893 rows, expected 86675. Off by 2.559%
8. eia_test[eia_raw-plants_eia860-216056-216056-216056] - ValueError: plants_eia860: found 215877 rows, expected 216056. Off by -0.083%
9. eia_test[eia_raw-pu_eia860-215147-215147-215147] - ValueError: pu_eia860: found 214965 rows, expected 215147. Off by -0.085%
10. eia_test[eia_raw-bf_eia923-1626501-1626501-134619] - ValueError: bf_eia923: found 1626381 rows, expected 1626501. Off by -0.007%
11. eia_test[eia_raw-bga_eia860-142391-142391-142391] - ValueError: bga_eia860: found 151860 rows, expected 142391. Off by 6.650%
12. eia_test[eia_raw-gens_eia860-591509-591509-591509] - ValueError: gens_eia860: found 590839 rows, expected 591509. Off by -0.113%
13. eia_test[eia_raw-gf_eia923-2949903-2949903-251758] - ValueError: gf_eia923: found 2949885 rows, expected 2949903. Off by -0.001%
14. eia_test[eia_raw-own_eia860-89741-89741-89741] - ValueError: own_eia860: found 95104 rows, expected 89741. Off by 5.976%
15. eia_test[eia_annual-bf_eia923-1626501-1626501-134619] - ValueError: bf_eia923: found 134609 rows, expected 134619. Off by -0.007%
16. eia_test[eia_annual-boil_eia860-86675-86675-86675] - ValueError: boil_eia860: found 88893 rows, expected 86675. Off by 2.559%
17. eia_test[eia_annual-bga_eia860-142391-142391-142391] - ValueError: bga_eia860: found 151860 rows, expected 142391. Off by 6.650%
18. eia_test[eia_annual-gen_eia923-None-5179478-433327] - ValueError: gen_eia923: found 459691 rows, expected 433327. Off by 6.084%
19. eia_test[eia_annual-denorm_emissions_control_equipment_eia860-56616-56616-56616] - ValueError: denorm_emissions_control_equipment_eia860: found 62102 rows, expected 56616. Off by 9.690%
20. eia_test[eia_annual-boiler_emissions_control_equipment_assn_eia860-77705-77705-77705] - ValueError: boiler_emissions_control_equipment_assn_eia860: found 83977 rows, expected 77705. Off by 8.072%
21. eia_test[eia_annual-utils_eia860-146776-146776-146776] - ValueError: utils_eia860: found 147877 rows, expected 146776. Off by 0.750%
22. eia_test[eia_annual-boiler_stack_flue_assn_eia860-41673-41673-41673] - ValueError: boiler_stack_flue_assn_eia860: found 44579 rows, expected 41673. Off by 6.973%
23. eia_test[eia_annual-emissions_control_equipment_eia860-56616-56616-56616] - ValueError: emissions_control_equipment_eia860: found 62102 rows, expected 56616. Off by 9.690%
24. eia_test[eia_annual-plants_eia860-216056-216056-216056] - ValueError: plants_eia860: found 215877 rows, expected 216056. Off by -0.083%
25. eia_test[eia_annual-pu_eia860-215147-215147-215147] - ValueError: pu_eia860: found 214965 rows, expected 215147. Off by -0.085%
26. eia_test[eia_annual-gf_eia923-2949903-2949903-251758] - ValueError: gf_eia923: found 251752 rows, expected 251758. Off by -0.002%
27. eia_test[eia_annual-own_eia860-89741-89741-89741] - ValueError: own_eia860: found 95104 rows, expected 89741. Off by 5.976%
28. eia_test[eia_annual-gens_eia860-591509-591509-591509] - ValueError: gens_eia860: found 590839 rows, expected 591509. Off by -0.113%
29. eia_test[eia_annual-boiler_cooling_assn_eia860-41282-41282-41282] - ValueError: boiler_cooling_assn_eia860: found 44185 rows, expected 41282. Off by 7.032%
30. eia_test[eia_monthly-bga_eia860-142391-142391-142391] - ValueError: bga_eia860: found 151860 rows, expected 142391. Off by 6.650%
31. eia_test[eia_monthly-boil_eia860-86675-86675-86675] - ValueError: boil_eia860: found 88893 rows, expected 86675. Off by 2.559%
32. eia_test[eia_monthly-gens_eia860-591509-591509-591509] - ValueError: gens_eia860: found 590839 rows, expected 591509. Off by -0.113%
33. eia_test[eia_monthly-gf_eia923-2949903-2949903-251758] - ValueError: gf_eia923: found 2949885 rows, expected 2949903. Off by -0.001%
34. eia_test[eia_monthly-gen_eia923-None-5179478-433327] - ValueError: gen_eia923: found 5494698 rows, expected 5179478. Off by 6.086%
35. eia_test[eia_monthly-emissions_control_equipment_eia860-56616-56616-56616] - ValueError: emissions_control_equipment_eia860: found 62102 rows, expected 56616. Off by 9.690%
36. eia_test[eia_monthly-own_eia860-89741-89741-89741] - ValueError: own_eia860: found 95104 rows, expected 89741. Off by 5.976%
37. eia_test[eia_monthly-utils_eia860-146776-146776-146776] - ValueError: utils_eia860: found 147877 rows, expected 146776. Off by 0.750%
38. eia_test[eia_monthly-boiler_emissions_control_equipment_assn_eia860-77705-77705-77705] - ValueError: boiler_emissions_control_equipment_assn_eia860: found 83977 rows, expected 77705. Off by 8.072%
39. eia_test[eia_monthly-boiler_stack_flue_assn_eia860-41673-41673-41673] - ValueError: boiler_stack_flue_assn_eia860: found 44579 rows, expected 41673. Off by 6.973%
40. eia_test[eia_monthly-pu_eia860-215147-215147-215147] - ValueError: pu_eia860: found 214965 rows, expected 215147. Off by -0.085%
41. eia_test[eia_monthly-denorm_emissions_control_equipment_eia860-56616-56616-56616] - ValueError: denorm_emissions_control_equipment_eia860: found 62102 rows, expected 56616. Off by 9.690%
42. eia_test[eia_monthly-boiler_cooling_assn_eia860-41282-41282-41282] - ValueError: boiler_cooling_assn_eia860: found 44185 rows, expected 41282. Off by 7.032%
43. eia_test[eia_monthly-plants_eia860-216056-216056-216056] - ValueError: plants_eia860: found 215877 rows, expected 216056. Off by -0.083%
44. eia_test[eia_monthly-bf_eia923-1626501-1626501-134619] - ValueError: bf_eia923: found 1626381 rows, expected 1626501. Off by -0.007%
45. mcoe_test[eia_annual-mcoe-5179886-433361] - ValueError: mcoe: found 459726 rows, expected 433361. Off by 6.084%
46. mcoe_test[eia_annual-capacity_factor-5179478-433327] - ValueError: capacity_factor: found 459691 rows, expected 433327. Off by 6.084%
47. mcoe_test[eia_annual-fuel_cost-602580-50327] - ValueError: fuel_cost: found 52978 rows, expected 50327. Off by 5.268%
48. mcoe_test[eia_annual-hr_by_unit-389530-32569] - ValueError: hr_by_unit: found 33946 rows, expected 32569. Off by 4.228%
49. mcoe_test[eia_annual-hr_by_gen-602580-50327] - ValueError: hr_by_gen: found 52978 rows, expected 50327. Off by 5.268%
50. mcoe_test[eia_monthly-hr_by_unit-389530-32569] - ValueError: hr_by_unit: found 406067 rows, expected 389530. Off by 4.245%
51. mcoe_test[eia_monthly-hr_by_gen-602580-50327] - ValueError: hr_by_gen: found 634378 rows, expected 602580. Off by 5.277%
52. mcoe_test[eia_monthly-fuel_cost-602580-50327] - ValueError: fuel_cost: found 634378 rows, expected 602580. Off by 5.277%
53. mcoe_test[eia_monthly-capacity_factor-5179478-433327] - ValueError: capacity_factor: found 5494698 rows, expected 5179478. Off by 6.086%
54. mcoe_test[eia_monthly-mcoe-5179886-433361] - ValueError: mcoe: found 5495136 rows, expected 5179886. Off by 6.086%