Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a EEA Reader supporting the new parquet based format #43

Open
heikoklein opened this issue Jun 18, 2024 · 2 comments
Open

Add a EEA Reader supporting the new parquet based format #43

heikoklein opened this issue Jun 18, 2024 · 2 comments
Labels
enhancement New feature or request
Milestone

Comments

@heikoklein
Copy link
Member

heikoklein commented Jun 18, 2024

pyaerocom read EEA data currently still through a reader of it's own, which uses huge amount of memory and long processing time. In addition, the current EEA dataformat will be discontinued by the end of the year and replaced by a parquet based format.
A pyaro-reader for the new EEA data-format needs to be implemented. During the implementation it should be decided if the new format is efficient enough or should be converted to another format (e.g. netcdf-rw of pyaro-reader).

@heikoklein heikoklein added the enhancement New feature or request label Jun 18, 2024
@heikoklein heikoklein added this to the m2024-07 milestone Jun 18, 2024
@lewisblake lewisblake modified the milestones: m2024-07, m2024-09 Jun 25, 2024
@jgriesfeller
Copy link
Member

the current EEA data format will be discontinued at the end of the year (read https://discomap.eea.europa.eu/map/fme/AirQualityExport.htm).
New API is here:
https://eeadmz1-downloads-webapp.azurewebsites.net/
and uses Parqet files as format.
So I guess we don't do any further development here?

@heikoklein
Copy link
Member Author

Thanks for checking the documentation.
Doesn't this make this issue even more relevant in that we need a new EEA reader because the old one is soon to be discontinued? I will reformulate the issue slightly, but keep it.

@heikoklein heikoklein changed the title Add a EEA Reader Add a EEA Reader supporting the new parquet base format Jul 1, 2024
@heikoklein heikoklein changed the title Add a EEA Reader supporting the new parquet base format Add a EEA Reader supporting the new parquet based format Jul 1, 2024
@heikoklein heikoklein modified the milestones: m2024-09, m2024-10 Sep 10, 2024
@heikoklein heikoklein modified the milestones: m2024-10, m2024-11 Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants