diff --git a/checksums.txt b/checksums.txt index 5571788..aab0591 100644 --- a/checksums.txt +++ b/checksums.txt @@ -1,13 +1,13 @@ -renewable_capacity_timeseries.csv,179586d0eb2b1dbebfd0a503395340ad57d33018c911069749783147e80feebf -renewable_power_plants.sqlite,5a9db104b1d80541f6fc3d11ca16c98fb913376501ddad0bdcb632dcca0b2a43 -renewable_power_plants.xlsx,7b3868663c2df0b7c3a4f5dfb0b695f420db821dc18a2bc082f03c7ea1b5bcba +renewable_capacity_timeseries.csv,a5a375e7b731c266f7dae36980a21c4f35c6cca8b428804b9c38f10fa4125b1c +renewable_power_plants.sqlite,724d830ff8e72cedd4d1dce8cbfb10004c8ec0002b42ffcb15de1ca33ec5fc09 +renewable_power_plants.xlsx,fafd9e9cd0b92139746d2a2647bbf5c0bae44aa42d51b2b6a919d78ab22120dd renewable_power_plants_CH.csv,2a16b65c02fdcf6a1776426eb24e92a64220259cebf03db31a64ed9f9124b900 -renewable_power_plants_DE.csv,1dfc79254809507206c53f2faf96977f7c55e095423f819e738443b275636117 -renewable_power_plants_DK.csv,13a3cf4e4e21d011ade71e30d1ea868eacd664a21344e73e7279a0311afbe2ad -renewable_power_plants_EU.csv,6d9be5502d224f0845110f67d2e70e04cd40f4e902e847b99eb39df353af4415 +renewable_power_plants_DE.csv,a132eb18219d8c7a044521f6f9678beb68b289c848996a3095289d781a040abd +renewable_power_plants_DK.csv,90fb83d378e9d61538fd4c45eece46d76880e2e1b9ca383bec01ccfe46ecc246 +renewable_power_plants_EU.csv,38f9523f65e850d7b8e2ddd7f4a539731543a21c2f197ae0f3abef6d648b9efe renewable_power_plants_FR.csv,f47a50e1826ac02eb59aed510000f769b1eba02200a27815aedc621e4a79aaf5 renewable_power_plants_PL.csv,dfe70c5ed3a8a2e244073072edd891a05e24e15c68bc3f7192f7341465eea68f renewable_power_plants_UK.csv,feb25a154c383444eed7cbb01cd09934c502a691878616a0df9c16cf44f14ba7 -res_plants_separated_DE_outvalidated_plants.csv,a43b577e61781ef592b20e78b5c2b93c7fa26dd9884da927f1b1f4a4e37cb30b +res_plants_separated_DE_outvalidated_plants.csv,10b501b89f18a1bcb17cd74bd25c3aa0dd965175064a58682e05a93cb8e417e0 res_plants_separated_FR_overseas_territories.csv,ca25ab4e1de51e334e8ac87521d1a3a635d15514c52309553d6b5881e0b3917a validation_marker.csv,c8eefd78ae9b2de1c6f308964c09c9cf5c79e3e29ca3aeeb4da15257913e83db diff --git a/download_and_process.ipynb b/download_and_process.ipynb index d860d3d..01a0a5d 100644 --- a/download_and_process.ipynb +++ b/download_and_process.ipynb @@ -31,7 +31,7 @@ }, "source": [ "

Table of Contents

\n", - "
" + "
" ] }, { @@ -40,7 +40,7 @@ "metadata": {}, "outputs": [], "source": [ - "version = '2019-04-01'" + "version = '2019-04-05'" ] }, { @@ -317,11 +317,14 @@ "source": [ "We will use function `get_download_urls` to get the urls as a dictionary `{'data':{source:url}, 'geo':url}` for a given country from the source list. If the highest-level key is `'data'`, the corresponding dictionary refers to the files containing actual data. If it is `'geo'`, the url points to the file with postcode geodata for the chosen country, provided that there is such an entry in the file. Otherwise, the `'geo' `url will be empty. The choice whether the links are going to lead to the original sources or the OPSD server is made by passing `'original_sources'` or `'opsd_server'` as the function's argument `download_from`.\n", "\n", - "Note that, as of February 2, 2019, the following sources are available only from the OPSD server and the data will be downloaded from it even if `download_from` is set to `'original_sources'`:\n", + "Note that, as of April 5, 2019, the following sources are available only from the OPSD server and the data will be downloaded from it even if `download_from` is set to `'original_sources'`:\n", "- Urzad Regulacji Energetyki (PL)\n", "- Energinet (DK)\n", "\n", - "The original links which are not available anymore are marked as inactive in the column `active` in the dataframe." + "Also, the BNetzA PV dataset will be loaded from OPSD server. The reason is that the \"April 2018\" sheet’s date columns needed \"date\" formatting to be manually re-applied from row 5221 onwards, otherwise the some date values would have been read as integer values. Therefore: This dataset will be loaded from OPSD server as well instead of its original source:\n", + "- Bundesnetzagentur BNetzA PV (DE): \"Meldungen_Juli17-Dez18.xlsx\"\n", + "\n", + "The original links which should be downloaded from OPSD are marked as inactive in the column `active` in the above dataframe." ] }, { @@ -462,7 +465,7 @@ "### Download and read\n", "The data which will be processed below is provided by the following data sources:\n", "\n", - "**[Netztransparenz.de](https://www.netztransparenz.de/de/Anlagenstammdaten.htm)** - Official grid transparency platform from the German TSOs (50Hertz, Amprion, TenneT and TransnetBW).\n", + "**[Netztransparenz.de](https://www.netztransparenz.de/de/Anlagenstammdaten.htm)** - Official grid transparency platform from the German Transmission System Operators (TSOs): 50Hertz, Amprion, TenneT and TransnetBW.\n", "\n", "**[Bundesnetzagentur (BNetzA)](https://www.bundesnetzagentur.de/DE/Sachgebiete/ElektrizitaetundGas/Unternehmen_Institutionen/ErneuerbareEnergien/ZahlenDatenInformationen/EEG_Registerdaten/EEG_Registerdaten_node.html)** - German Federal Network Agency for Electricity, Gas, Telecommunications, Posts and Railway (In separate files for **data for roof-mounted PV power plants** and for **all other renewable energy power plants**.)\n", "\n", @@ -545,26 +548,6 @@ " filename.close()" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### !!! Note !!! BNetzA_PV input Excel file needs manual editing before reading it !!!\n", - "\n", - "The \"April 2018\" sheet’s date columns need \"date\" formatting to be re-applied from row 5221 onwards. If this is not done, some date values will be read as integer values and the following will not work. If you don't yet know what to do, have a look at \"Meldungen_Juli17-Dez18.xlsx\" in the \"\\input\\original_data\\raw\" and the \"\\input\\original_data\\DE\" folder and go to the mentioned sheet and row. You will see that dates are formatted as intergers there. These must be formatted as dates, and then everything will work.\n", - "\n", - "*Manual work required!*" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "Script intentionally creashes here, to make users aware of manual work (see above)." - ] - }, { "cell_type": "code", "execution_count": null, diff --git a/input/sources.csv b/input/sources.csv index 89e96a1..1e9544e 100644 --- a/input/sources.csv +++ b/input/sources.csv @@ -1,7 +1,7 @@ source,country,url,filename,file_type,active bnetza,DE,https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Energie/Unternehmen_Institutionen/ErneuerbareEnergien/ZahlenDatenInformationen/VOeFF_Registerdaten/2018_12_Veroeff_RegDaten.xlsx?__blob=publicationFile&v=2,2017_12_Veroeff_RegDaten.xlsx,data,yes bnetza_pv_historic,DE,https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Energie/Unternehmen_Institutionen/ErneuerbareEnergien/ZahlenDatenInformationen/PV_Datenmeldungen/Archiv_PV/Meldungen_Aug-Juni2017.xlsx?__blob=publicationFile&v=2,Meldungen_Aug-Juni2017.xlsx,data,yes -bnetza_pv,DE,https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Energie/Unternehmen_Institutionen/ErneuerbareEnergien/ZahlenDatenInformationen/PV_Datenmeldungen/Meldungen_Juli17-Dez18.xlsx?__blob=publicationFile&v=2,Meldungen_Juli-Dez2017.xlsx,data,yes +bnetza_pv,DE,https://www.bundesnetzagentur.de/SharedDocs/Downloads/DE/Sachgebiete/Energie/Unternehmen_Institutionen/ErneuerbareEnergien/ZahlenDatenInformationen/PV_Datenmeldungen/Meldungen_Juli17-Dez18.xlsx?__blob=publicationFile&v=2,Meldungen_Juli17-Dez18.xlsx,data,no Amprion,DE,https://www.netztransparenz.de/portals/1/Content/Erneuerbare-Energien-Gesetz/EEG-Anlagestammdaten/Netztransparenz%20Anlagenstammdaten%202017%20Amprion%20GmbH_V03.zip,Netztransparenz%20Anlagenstammdaten%202017%20Amprion%20GmbH_V03,data,yes TransnetBW,DE,https://www.netztransparenz.de/portals/1/Content/Erneuerbare-Energien-Gesetz/EEG-Anlagestammdaten/Netztransparenz%20Anlagenstammdaten%202017%20TransnetBW%20GmbH.zip,Netztransparenz%20Anlagenstammdaten%202017%20TransnetBW%20GmbH,data,yes 50Hertz,DE,https://www.netztransparenz.de/portals/1/Content/Erneuerbare-Energien-Gesetz/EEG-Anlagestammdaten/Netztransparenz%20Anlagenstammdaten%202017%2050Hertz%20Transmission%20GmbH.zip,Netztransparenz%20Anlagenstammdaten%202017%2050Hertz%20Transmission%20GmbH,data,yes diff --git a/validation_and_output.ipynb b/validation_and_output.ipynb index 88dc4c4..2d1800d 100644 --- a/validation_and_output.ipynb +++ b/validation_and_output.ipynb @@ -49,7 +49,7 @@ "source": [ "settings = {\n", " 'version': '2019-04-05',\n", - " 'changes': 'Updated all countries with new data available (DE, FR, CH, DK) and added data for UK.'\n", + " 'changes': 'Updated all countries with new data available (DE, FR, CH, DK), added data for UK and expanded renewable capacity timeseries to more countries (DK, UK, CH in addition to DE).'\n", "}" ] }, @@ -1281,9 +1281,6 @@ " - name: address\n", " description: Street name or name of land parcel\n", " type: string\n", - " - name: address_number\n", - " description: House number or number of land parcel\n", - " type: string\n", " - name: lat\n", " description: Latitude coordinates\n", " type: geopoint\n", @@ -1591,12 +1588,6 @@ " - name: address\n", " description: Street name or name of land parcel\n", " type: string\n", - " - name: address_number\n", - " description: House number or number of land parcel\n", - " type: string\n", - " - name: utm_zone\n", - " description: Universal Transverse Mercator zone value\n", - " type: \n", " - name: lat\n", " description: Latitude coordinates\n", " type: geopoint\n", @@ -1725,216 +1716,315 @@ " - name: CH_bioenergy_capacity\n", " description: Cumulative bioenergy electrical capacity for Switzerland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Switzerland\n", + " Variable: Bioenergy\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Swiss Federal Office of Energy\n", " - name: CH_hydro_capacity\n", " description: Cumulative hydro electrical capacity for Switzerland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Switzerland\n", + " Variable: Hydro\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Swiss Federal Office of Energy\n", " - name: CH_solar_capacity\n", " description: Cumulative solar electrical capacity for Switzerland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Switzerland\n", + " Variable: Solar\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Swiss Federal Office of Energy\n", " - name: CH_wind_capacity \n", " ription: Cumulative total wind electrical capacity for Switzerland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Switzerland\n", + " Variable: Wind\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Swiss Federal Office of Energy\n", " - name: CH_wind_onshore_capacity\n", " description: Cumulative onshore wind electrical capacity for Switzerland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Switzerland\n", + " Variable: Wind onshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Swiss Federal Office of Energy\n", " - name: DE_bioenergy_capacity\n", " description: Cumulative bioenergy electrical capacity for Germany in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Bioenergy\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DE_geothermal_capacity\n", " description: Cumulative geothermal electrical capacity for Germany in MW\n", " unit: MW\n", - " type: number\n", - " source:\n", - " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", - " - name: DE_hydro_capacity\n", - " description: Cumulative hydro electrical capacity for Germany in MW\n", - " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Geothermal\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DE_solar_capacity\n", " description: Cumulative solar electrical capacity for Germany in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Solar\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DE_storage_capacity\n", " description: Cumulative storage electrical capacity for Germany in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Storage\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DE_wind_offshore_capacity\n", " description: Cumulative offshore wind electrical capacity for Germany in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Wind offshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DE_wind_capacity \n", " ription: Cumulative total wind electrical capacity for Germany in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Wind\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DE_wind_onshore_capacity\n", " description: Cumulative onshore wind electrical capacity for Germany in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Germany\n", + " Variable: Wind onshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BNetzA and Netztransparenz.de\n", " - name: DK_solar_capacity\n", " description: Cumulative solar electrical capacity for Denmark in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Denmark\n", + " Variable: Solar\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Energinet.dk\n", " - name: DK_wind_offshore_capacity\n", " description: Cumulative offshore wind electrical capacity for Denmark in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Denmark\n", + " Variable: Wind offshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Danish Energy Agency\n", " - name: DK_wind_capacity \n", " ription: Cumulative total wind electrical capacity for Denmark in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Denmark\n", + " Variable: Wind\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Danish Energy Agency\n", " - name: DK_wind_onshore_capacity\n", " description: Cumulative onshore wind electrical capacity for Denmark in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Denmark\n", + " Variable: Wind onshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from Danish Energy Agency\n", " - name: GB-GBN_bioenergy_capacity\n", " description: Cumulative bioenergy electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Bioenergy\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-GBN_hydro_capacity\n", " description: Cumulative hydro electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Hydro\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-GBN_marine_capacity\n", " description: Cumulative marine electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Marine\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-GBN_solar_capacity\n", " description: Cumulative solar electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Solar\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-GBN_wind_offshore_capacity\n", " description: Cumulative offshore wind electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Wind offshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-GBN_wind_capacity\n", " description: Cumulative total wind electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Wind\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-GBN_wind_onshore_capacity\n", " description: Cumulative onshore wind electrical capacity for Great Britain (England, Scotland, Wales) in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Great Britain (England, Scotland, Wales)\n", + " Variable: Wind onshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-NIR_bioenergy_capacity\n", " description: Cumulative bioenergy electrical capacity for Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Northern Ireland\n", + " Variable: Bioenergy\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-NIR_marine_capacity\n", " description: Cumulative marine electrical capacity for Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Northern Ireland\n", + " Variable: Marine\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-NIR_solar_capacity\n", " description: Cumulative solar electrical capacity for Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Northern Ireland\n", + " Variable: Solar\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-NIR_wind_capacity\n", " description: Cumulative total wind electrical capacity for Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Northern Ireland\n", + " Variable: Wind\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-NIR_wind_onshore_capacity\n", " description: Cumulative onshore wind electrical capacity for Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: Northern Ireland\n", + " Variable: Wind onshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_bioenergy_capacity\n", " description: Cumulative bioenergy electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Bioenergy\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_hydro_capacity\n", " description: Cumulative hydro electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Hydro\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_marine_capacity\n", " description: Cumulative marine electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Marine\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_solar_capacity\n", " description: Cumulative solar electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Solar\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_wind_offshore_capacity\n", " description: Cumulative offshore wind electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Wind offshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_wind_capacity\n", " description: Cumulative total wind electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Wind\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n", " - name: GB-UKM_wind_onshore_capacity\n", " description: Cumulative onshore wind electrical capacity for the United Kingdom of Great Britain and Northern Ireland in MW\n", " unit: MW\n", + " opsdProperties:\n", + " Region: United Kingdom of Great Britain and Northern Ireland\n", + " Variable: Wind onshore\n", " type: number\n", " source:\n", " name: Own calculation based on plant-level data from BEIS\n",