Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data generator folder #200

Merged
merged 21 commits into from
May 12, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
0fc9dc2
create extra folder
Ultramarin95 Mar 23, 2022
8ca4e0d
getting ready to rock
Ultramarin95 Mar 23, 2022
e0912e2
reworked Data generator folder
Ultramarin95 Mar 31, 2022
3623b9e
earlier ready-to-rock and started with renaming variables and functions
Ultramarin95 Apr 2, 2022
70afbb0
more renaming
Ultramarin95 Apr 2, 2022
0c3675f
added Data class to be able to set arbitrary cut-off dates
Ultramarin95 Apr 3, 2022
1c49e6a
deleted comment
Ultramarin95 Apr 3, 2022
932dddb
included some code suggestions
Ultramarin95 Apr 4, 2022
ae4a6d0
black + whitespaces
Ultramarin95 Apr 4, 2022
63a6a9c
Merge branch 'main' of https://github.com/GermanZero-de/localzero-gen…
Ultramarin95 Apr 6, 2022
2841dcd
Merge commit '309a831c3b5bb1b42a3e646046ea0d3ff729b8b7' of https://gi…
Ultramarin95 Apr 6, 2022
98ed720
Merge remote-tracking branch 'origin' into addDataGeneratorFolder
Ultramarin95 May 2, 2022
12f28a6
add Leonarts Version
Ultramarin95 May 3, 2022
88c8848
further work on update_ags needed
Ultramarin95 May 6, 2022
3b08a22
Revert "further work on update_ags needed"
Ultramarin95 May 7, 2022
3f5b81b
little extra comment
Ultramarin95 May 7, 2022
cdadf84
changed cut off date to the end of 2021 and some minor issues
Ultramarin95 May 8, 2022
1587ca5
adapt readme and store script in new folder
Ultramarin95 May 10, 2022
a4d9b57
change data repo production hash to new renewable data merge
Ultramarin95 May 11, 2022
e103eef
update expectations
Ultramarin95 May 11, 2022
d9cf339
whitespaces and black
Ultramarin95 May 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions data/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ on every data file you have changed, so minor encoding differences (such as
wether or not to put `"` around fields in a CSV) do not result in spurious
merge conflicts later on.

We also added the *dataGeneratorScripts* folder, that includes some python scripts
which generate data files for the data repositories from various sources.

Where to get the data: Public Domain
------------------------------------

Expand Down
File renamed without changes.
22 changes: 22 additions & 0 deletions data/dataGeneratorScripts/generateRenewables/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
renewable_energy import script
==============================

The importMarktStammDaten.py script imports data from the Marktstammdaten Register and saves them into the
renewable_energy data file (2018.csv). If one wants to work with the original data from the Marktstammdaten Register,
one should download the original Xml files and save them into a folder and adapt the path in the script. Also make sure to have the *reloadfromXML* boolean set as *True*.
Once you parsed the xml files once, you can also use the json files that are created during the xml parsing.
In order to use the json files set *reloadfromXML* boolean set as *False*.

Note that if you want the parse the xml data you need to install the third party lxml python package (Download and install https://lxml.de/).

The json file that is used to update all ags keys to 2018 is downloaded from https://www.xrepository.de/api/xrepository/urn:xoev-de:bund:destatis:bevoelkerungsstatistik:codeliste:ags.historie_2021-12-31/download/Destatis.AGS.Historie_2021-12-31.json
and holds data on all ags keys and changes that happend after 2006.

More info on the Marktstammdaten register:
The Marktstammdaten Register lists all local energy sources (alle Strom- und Gaserzeugungsanlagen)
in germany including wind turbines, solar cells etc. but also fossil fuel dependent energy producers
like small "Blockheizkraftwerke" (more than 2.8 million entries). (almost) All units come with ags keys,
which is why we use this source to generate our renewable_energy data. The Marktstammdaten Register
provides a full data download under https://www.marktstammdatenregister.de/MaStR/Datendownload.
However the zip-file is ~900 MByte and the unziped XML-files are more than 19 GB. This is why we dont
provide them here directly but feel free to download them on your own and play around with the python script.
Loading