-
Notifications
You must be signed in to change notification settings - Fork 73
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to read ECMWF ensemble into xarray with FastHerbie #196
Comments
Hi @karlwx, Thanks for reporting this. This gives me a number of things to thing about; I can't promise I'll make any changes any time soon, but I will be looking into this. I admit, I haven't spent much time testing the ECMWF data with the FastHerbie implementation.
As you pointed out, the reason FastHerbie is not working here is because when cfgrib reads ds = Herbie(
"2023-05-25 00:00",
model="ecmwf",
product="enfo",
).xarray(":2t:", verbose=True, remove_grib=False)
len(ds) # list of 2 datasets This downloads about 30 MB of data (in ~2 mins!!) and the grib messages are spread throughout the file which requires multiple cURL downloads (which could probably be optimized with multithreading; I'll look into that). I'd suggest modifying your searchString to get only one of these two hypercubes. If you look at the inventory, you see ECMWF does provide the ensemble number. I am assuming H = Herbie("2023-05-25 00:00", model="ecmwf", product="enfo")
H.inventory(":2t:") # This will get you what must be the control run...
H.inventory(":2t:sfc:g:")
# This will get you member number 1...
H.inventory(":2t:sfc:1:g:")
# This will get you members 1-5 that are in the same hypercube...
H.inventory(":2t:sfc:[1-5]:g:") Now, using this searchString in FastHerbie, this does work (albeit slow), and gets the dataframe with number=0: FH = FastHerbie(
["2023-05-25 00:00"],
model="ecmwf",
product="enfo",
fxx=[0, 6, 12]
)
ds = FH.xarray(":2t:sfc:g:")
ds
You are right! I was having issues with using multithreading (run out of memory, jupyter crashing). Long story short, I don't yet know enough about Multithreading/Multiprocessing to keep things under control. By default you will notice that This example took 18 seconds, whereas FH = FastHerbie(
["2023-05-25 00:00"],
model="ecmwf",
product="enfo",
fxx=[0, 6, 12]
)
ds = FH.xarray(":2t:sfc:g:", max_threads=3)
ds Hope this gives you some answers! P.S. I'll have to look back at v0.0.10 and see why that was downloading stuff faster. I can't actually remember what changed since then. |
Thanks Brian! I am at least avoiding errors and seeing some improved speed by using your suggestions. I was wondering if something changed in the download process, or the process just became more verbose in recent versions. I was not seeing a bunch or curl requests in 0.10.0. The bottleneck seems to be the need for multiple requests to download parts of a single file. I remember playing around with python's requests package and I was able to download multiple byte ranges with a single request. If that's possible with curl, that would be a huge improvement in speed! |
This is a good idea. I seem to remember that the ability to do a multiple byte range request depends on the server the data is hosted, and AWS S3 didn't allow multiple byte range requests last I looked (several years ago). Just googled this...looks like they still don't (which is unfortunate)... But maybe Azure (where ECMWF data is) does???S |
That's unfortunate! I had the idea to try multiple byte ranges in a single request as a way to get around the hit limits on the NOMADS server. So I know it works there, maybe it would work directly on the ECMWF website (not Azure)? |
Hey Brian, I'm trying to download a whole run of ECMWF ensemble data with FastHerbie - it seems like the data format is causing things to break. Here is the error:
Also, a couple of suggestions regarding ECMWF data:
The text was updated successfully, but these errors were encountered: