Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix OpenAQ reader #116

Merged
merged 8 commits into from
Jul 11, 2023
Merged

Fix OpenAQ reader #116

merged 8 commits into from
Jul 11, 2023

Conversation

zmoon
Copy link
Member

@zmoon zmoon commented May 26, 2023 •
edited
Loading

  • local time calc

    • don't quite understand why it wasn't working properly (giving only a few unique UTC offsets), but looks more reasonable now (several unique UTC offsets and correct e.g. most common is +8 UTC which mostly corresponds to sites in China)

  • convert to ppmv

    • there was code to do so but it wasn't being used
    • updated and documented the conversion factors and added factors for CH4 and NO which can be in OpenAQ data

  • at least one test (maybe find a way to set things to load just one file for testing)

zmoon added 5 commits May 22, 2023 09:11
@zmoon
Different local time calc
c7a70f2 
this makes it more reasonable, though don't quite understand
why what was there wasn't working properly
@zmoon
Convert units
3b50eaf 
code was there but wasn't being used

OpenAQ allows ppm and ug m-3; rows of the same variable may
have different units
@zmoon
Update conversion factors
d92ac1d 
Currently in MONETIO:
CO   1145 -> mw_a / rho_a = 24.463 ->(mw_a=29) rho_a = 1.185
O3   2000 -> mw_a / rho_a = 23.998 ->(mw_a=29) rho_a = 1.208
SO2  2620 -> mw_a / rho_a = 24.453 ->(mw_a=29) rho_a = 1.186
NO2  1880 -> mw_a / rho_a = 24.471 ->(mw_a=29) rho_a = 1.185
Better accuracy (for rho_a=1.2, mw_a=29):
CO   1145 -> 1159
O3   2000 -> 1986
SO2  2620 -> 2651
NO2  1880 -> 1904
CH4       -> 663.8
NO        -> 1242
Rounded to 3 sf:
{'co': 1160, 'o3': 1990, 'so2': 2650, 'no2': 1900, 'ch4': 664, 'no': 1240}
@zmoon
Only find where once
45e4899
@zmoon
In case we have some CH4 or NO
1486417
@bbakernoaa bbakernoaa self-requested a review June 8, 2023 19:22
@zmoon
Copy link
Member Author

zmoon commented Jun 9, 2023 •
edited
Loading

Some initial timings suggest we can gain speed by using s3 URL scheme instead of current HTTPS:

In [8]: %timeit -n 10 pd.read_json("https://openaq-fetches.s3.amazonaws.com/realtime/2013-11-26/2013-11-26.ndjson",
   ...:  lines=True)
311 ms ± 37.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [9]: %timeit -n 10 pd.read_json("s3://openaq-fetches/realtime/2013-11-26/2013-11-26.ndjson", lines=True)
159 ms ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

similar to #113

@zmoon zmoon marked this pull request as ready for review June 9, 2023 20:26
@zmoon zmoon mentioned this pull request Jun 13, 2023
Closed
@zmoon zmoon merged commit 77497d4 into noaa-oar-arl:develop Jul 11, 2023
@zmoon zmoon deleted the fix/openaq branch July 11, 2023 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bbakernoaa bbakernoaa Awaiting requested review from bbakernoaa

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@zmoon