PUDL 0.4.0

This is our first release in more than a year and a half, and it contains lots of new data and analyses (and breaking changes...) but it doesn't yet include 2020 datasets for FERC and EIA.

See the complete v0.4.0 release notes for details.

Merged Pull Requests

• Unified logic for excel extraction by @rousik in #566
• fuel cost output to ref 860 generators. by @cmgosnell in #574
• Ferc714 by @yashkumar1803 in #594
• Ei mcoe by @aesharpe in #592
• Transform function for distribution systems and other edits by @aesharpe in #643
• Add manually compiled balancing authority id fixes by @zaneselvans in #646
• Transform function for AMI EIA861 by @aesharpe in #647
• Transform function for EIA 861 Dynamic Pricing Table by @aesharpe in #649
• Normalize the Balancing Authority Table and add a BA Association Table by @zaneselvans in #651
• Transform func for Eia861 Green Pricing table by @aesharpe in #653
• Net metering table eia861 by @aesharpe in #671
• Service territories by @zaneselvans in #670
• Non net metering function eia861 by @aesharpe in #680
• Categorize eia codes with either Util or BA priority by @zaneselvans in #687
• Add a new FERC 714 Output Module by @zaneselvans in #699
• 635: Datastore passes travis tests by @ptvirgo in #701
• Operational data table eia861 by @aesharpe in #691
• Add limit_by_state option to utility territory generation by @zaneselvans in #707
• Simplify datapkg_to_sqlite script by @zaneselvans in #712
• Clobber datapackage bundles not single datapackages by @zaneselvans in #714
• Reliability and utility data eia861 by @aesharpe in #710
• Datastore improvements by @ptvirgo in #715
• Distributed generation eia861 by @aesharpe in #724
• Set up GitHub Actions to run Tox/PyTest by @zaneselvans in #727
• Restore utility_assn() and other code wiped out by PR 724 by @zaneselvans in #730
• Energy efficiency eia861 by @aesharpe in #731
• Demand mapping by @yashkumar1803 in #717
• Ferc714 by @ptvirgo in #733
• Demand side management eia861 by @aesharpe in #732
• Some tweaks to table columns and data types by @aesharpe in #743
• get_census2010_gdf uses datastore by @zaneselvans in #764
• Datastore data package validation and updated DOIs by @zaneselvans in #761
• More robust flake8 linting by @zaneselvans in #768
• Validate new dois by @cmgosnell in #773
• Draft of ferc1 + eia860 + eia923 data integration for 2019 by @zaneselvans in #788
• Merge Sprint25 into dev branch by @zaneselvans in #800
• Add DOIs for production archives on Zenodo by @zaneselvans in #804
• Zipcode fix by @aesharpe in #820
• Better help messages and default to verbose logging by @zaneselvans in #825
• Add docker build scripts by @rousik in #826
• Fix few issues surfaced in the previous PR by @rousik in #827
• Automate docker image builds by @rousik in #829
• Bump build-push-action to @v2 and fix arguments. by @rousik in #831
• Draft documentation framework for data sources by @aesharpe in #821
• Eia epa crosswalk by @aesharpe in #822
• Integrate EIA-860 2008 data by @aesharpe in #838
• Integrate EIA 860 M into ETL by @cmgosnell in #824
• Add basic Datasette metadata and deployment script by @zaneselvans in #841
• Notebook land: intro notebooks for CEMS and output tables by @aesharpe in #823
• add output and access notebooks by @cmgosnell in #844
• Allocate generation_fuel_eia923 table data to generators by @cmgosnell in #785
• Notebook land by @zaneselvans in #853
• Ensure deterministic checksums on csv.gz outputs by @rousik in #856
• Add output methods for all remaining EIA 861 tables. by @zaneselvans in #862
• EIA860 old years (through 2004) by @aesharpe in #849
• Add high-performance timeseries anomaly detection and imputation module by @ezwelty in #871
• Speed up FERC 714 hourly demand transform by @ezwelty in #873
• Always run interim ETL tests b/c they're fast now. by @zaneselvans in #874
• Alaska is a thing by @rousik in #876
• Specify min/max versions for all dependencies in setup.py by @zaneselvans in #875
• Fix broken links in README by @kyleries in #864
• Datastore refactoring by @rousik in #880
• Add unit test environment that runs quick tests under src/pudl by @rousik in #867
• Regex future warning by @rousik in #883
• Adjust FERC 714 service territories by using modified versions of EIA 861 tables by @ezwelty in #881
• Timeseries unittest by @zaneselvans in #885
• Bugfixes for states=[ALL] and SQLite DB clobber check by @zaneselvans in #890
• Consolidate interim ETL / output tests by @zaneselvans in #892
• Jupyterhub beta by @zaneselvans in #894
• Clean up PyTest config, coverage generation, unit tests by @zaneselvans in #896
• Implementation of DataFrameCollection by @rousik in #887
• Sprint29 by @zaneselvans in #897
• Dev by @zaneselvans in #898
• Pyarrow v3 by @zaneselvans in #912
• Eia860 validation by @aesharpe in #911
• Improvements to the DataFrameCollection by @rousik in #916
• pudl_datastore --list-partitions by @rousik in #925
• Pudl rmi by @zaneselvans in #926
• Pytest scripts by @zaneselvans in #913
• Sprint30 by @zaneselvans in #933
• Integrate EIA-860m through Nov. 2020 + fixed PUDL Plant IDs by @zaneselvans in #934
• Update PUDL Development Docs by @zaneselvans in #940
• Metadata docs by @aesharpe in #907
• Update transform documentation by @aesharpe in #939
• Convert Census DP1 to SQLite by @zaneselvans in #948
• Sprint31 by @zaneselvans in #951
• Databeta by @zaneselvans in #956
• Dev by @zaneselvans in #957
• Dev docs setup updates by @cmgosnell in #955
• Data Source Docs / Overview / Usage Modes by @aesharpe in #953
• Remove leading zeroes from all numeric EIA generator_ids. by @zaneselvans in #968
• Merge work from April 12-26 sprint. by @zaneselvans in #970
• Fix pudl ids by @aesharpe in #1003
• Form instructions by @zaneselvans in #987
• Simplified demand allocation algorithm by @yashkumar1803 in #741
• final review of release documentation for grammar, punctuation, spelling, and clarity issues by @aesharpe in #1019
• Improve documentation and error messages related to use of EIA API keys. by @zaneselvans in #1030
• Convert spatial doctests to unit tests by @zaneselvans in #1021
• Merge accumulated dev changes from sprint ending 2021-07-05 by @zaneselvans in #1068
• Integrate eia 923 2001-2008 by @TrentonBush in #1035
• Remove pytest logging capture to avoid duplicate logging output by @zaneselvans in #1077
• Integrate EPA CEMS 2020 data by @zaneselvans in #1078
• WIP: update allocation of net gen by @cmgosnell in #989
• PUDL Unit ID Assignments by @zaneselvans in #1037
• update ferc1 documentation: by @aesharpe in #1053
• Release v0.4.0 by @zaneselvans in #1087
• add xfail test for ownership tbl and make count_records FASTER by @cmgosnell in #1118
• Replace janky merge_on_date_year with built-in merge_asof() by @zaneselvans in #1103
• Release v0.4.0 by @zaneselvans in #1124
• Potential PUDL v0.4.0 release by @zaneselvans in #1125

New Contributors

• @rousik made their first contribution in #566
• @yashkumar1803 made their first contribution in #594
• @ptvirgo made their first contribution in #701
• @kyleries made their first contribution in #864

Full Changelog: v0.3.2...v0.4.0