Herbie 2023.3.0

Thanks again for using Herbie and your kind words. More than a few of you found me at the AMS annual meeting in Denver to thank me for publishing and maintaining Herbie. It's always encouraging to hear how I've helped someone. I'm still in shock how putting some code I wrote in grad school on the internet was discovered by so many. Hebie is still a project I work on in my free time, and I don't claim it is the most efficient or cleanly coded package, but the lessons I learn with Herbie have helped me in many ways in other work I do. So, I thank you for your involvement and use of Herbie and for helping me develop my skills.

Now I want to give a big "thank you" to these two people

1. Ryan May (@dopplershift), thanks for instructing me on how to publish Herbie on conda-forge. This really simplifies installing Herbie given its dependency on cgfrib and cartopy. Herbie can now be installed like this:

conda install herbie-data

Note: The name of this package on pypi and conda is herbie-data because herbie is taken on PyPI.

2. Andreas Motl (@amotl), thanks for your help with migrating the documentation to readthedocs and adding GitHub Actions to run the suite of tests with each pull request. Automating the doc builds and tests really helps.

These changes and other updates help make Herbie feel a little more "grown up" in this release.

Breaking Changes

• I renamed the file herbie/archive.py to herbie/core.py because the name "archive" was starting to lose its meaning. Please pay attention to how you import Herbie...

# DON'T do this anymore
from herbie.archive import Herbie

# DO this
from herbie import Herbie

• I renamed the Herbie method H.read_idx() to H.inventory() because the single word is easier for me to remember. If you use read_idx() you will get a warning for now; it will be removed later. Note that the inventory method is also available for FastHerbie. In fact, I'm migrating away from the term "index" to "inventory" because it I think it's a more accurate description of what a ".idx" file is--in inventory of the file contents. And besides, "index" means other things in python if you are a heavy Pandas user.

## DON'T do this anymore
H = Herbie(date, model)
H.read_idx()

# DO this
H = Herbie(date, model)
H.inventory()

New/Updated Model Templates

• Added rtma model template for RTMA (CONUS and Alaska). Template could be extended to include more products (Hawaii, guam, URMA products, etc.).
• Added gdas to model template by @blaylockbk in #107 Thanks, @alcoat!
• Updated gefs model template (not reforecast GEFS) by @blaylockbk in #133
• Updated ecmwf to include AWS as source for ECMWF forecasts by @blaylockbk in #159

Note: There will be additional changes to the GEFS and GFS model templates to make the more consistent. I also plan to add a template for the URMA model since it exists.

What's Changed

• Install: Herbie can be installed with conda.
• Bug Fix: Essentially turned off 'content-length' check in affdbda because files from NOMADS report a content-length of 20 bytes (something changed on NOMADS in the last year)
• Enhancement: Fix handling idx source location and generate missing idx with wgrib2 for local files. by @blaylockbk in #127
• Documentation: Made the Herbie logo a little more "shiny".
• Documentation: Overhaul of the Herbie documentation (thanks to the flu and being confined to bed for a few days).
• Documentation: Add configuration for "Read the Docs" by @amotl in #142
  • Fix RTD configuration by @amotl in #145
• CI: Run tests on GHA by @amotl in #139
• General updates, improvement, and clean up by @blaylockbk in #158
• Improvements to nearest_points required input. by @blaylockbk in #165
• Add a wrapper for wgrib2 in Herbie. by @blaylockbk in #161

New Contributors

• @amotl made their first contribution in #142

Full Changelog: 2022.09.0...2022.x.x

Herbie 2022.9.0

Huge shoutout to Ryan May for instructing me on how to publish Herbie on conda-forge. You can now install Herbie 2022.9.0.post1 with Conda

conda install -c conda-forge herbie-data

What's Changed

• Changed to CalVer versioning. Release versions will follow the YYYY.mm.micro format as I felt this more clearly highlights when the package was last updated.
• Changed default branch from master to main
• Add missing pygrib dependency by @haim0n in #74
• Remove unused import by @haim0n in #78
• Changes subset hash label to include date and fxx by @blaylockbk in #96
• Blaylockbk/issue98 change setup.py to setup.cfg and pyproject.toml by @blaylockbk in #99
• Added pygrib and cartopy as a dependency.
• Added NAM model #93
• Documentation: added dark mode documentation via PyData's Sphinx theme
• Let Herbie find the most recent GRIB2 file. by @blaylockbk in #103

New Contributors

• @haim0n made their first contribution in #74

Full Changelog: 0.0.10...2022.09.0

Herbie 0.0.10

⭐ Over 110 GitHub stars!

Wow, I had no idea so many people would find Herbie useful. Thanks everyone.

What's Changed

• Now you can import the Herbie class easier with from herbie import Herbie instead of from herbie.archive import Herbie
• Add template for GEFS by @blaylockbk in #56
• Changed how extending Herbie with custom templates works by @blaylockbk in #55
• Herbie helper tool to make an index file for all grib2 files in a directory. by @blaylockbk in #59
• Smarter cURL ranges for faster subset downloads by @blaylockbk in #62
• Moved the grib and index search in the Herbie class to their own functions (find_grib and find_idx)
• Cache the full index file Dataframe first time it is read.
• Added some ASCII art to __init__.py and some Easter eggs (because every project needs cool ASCII art and Easter eggs 😎)
• Add some multithreading tools to Herbie by @blaylockbk in #69 -- have not done extensive tests; watch your memory
• Fixed subset filename bug (de70cb7) that affected GFS files.
• Herbie can make an index file if one does not exist on the remote archive if the full grib2 file is downloaded locally and if wgrib2 is installed. 8653573
• Concat HRRR subh fields, if possible 8a9a862

Full Changelog: 0.0.9...0.0.10

Herbie 0.0.9

Changelog

• Fixed #42: bug where GFS index files could not be found.
• Speed up Herbie.xarray by Reusing local_file by @WToma in #46
• Removed old hrrrb API and old documentation.

Full Changelog: 0.0.8...0.0.9

Herbie 0.0.8

Add access to ECMWF open data forecast products

The main feature of this release is the ability to retrieve ECMWF open data forecast products (see tweet)

The big change was implementing a method to read the grib_ls-style index files to get the byte ranges for specific variables/parameters. This means that the searchString argument will need to be specified differently than that used for other models. Read more about the searchString argument for grib_ls-style index files: https://blaylockbk.github.io/Herbie/_build/html/user_guide/searchString.html#grib-ls-style-index-files

For example:

from herbie.archive import Herbie
# Create Herbie object to discover ECMWF operational forecast product
H = Herbie("2022-01-26", model="ecmwf", product="oper", fxx=12)

# Download the full grib2 file
H.download()

# Download just the 10-m u and v winds
H.download(searchString=":10(u|v):")

# Retrieve the 500 hPa temperature as an xarray.Dataset
ds = H.xarray(searchString=":t:500:")

---

🏹 More examples for retrieving ECMWF open data

---

Changelog

• Added access to ECMWF's open data products by @blaylockbk in #37

Full Changelog: 0.0.7...0.0.8

Pretty Pictures

Herbie 0.0.7

Changelog

• Default config value for priority is now None, which will make the download source priority to be the order of the SOURCE in the models template files.
• isort all imports
• Fixed Issue #24: Use hash labels to name subset files to make unique file names so filenames don't get too long (Pull Request #26).
• When loading data into xarray, Herbie will parse CF grid_mapping from grib file with pygrib/pyproj/metpy. (see CF Appendix F: Grid Mapping for more info). 2789859
• Lay the groundwork for local GRIB2 files that are not downloaded from remote sources. Suppose you ran a model locally (like WRF) and have that data stored locally. Herbie can be configured to find and read those local files! This assumes an index file also exists in the same directory with .idx appended to the file name.
  • Documentation needed.
• Added model template for rap_ncei. This is poorly implemented because the NCEI data doesn't match the other RAP model URLs.
• #24, implemented hashed filename for subset by @blaylockbk in #26
• Add a Gitter chat badge to README.md by @gitter-badger in #21
• Remove Gitter chat badge. I'm not committed to chatting on Gitter; just focus on discussions.
• Add support for NCEI historical RAP analyses by @djgagne in #30
• Handle different idx styles by @blaylockbk in #35 (more changes in addition to those by djgagne)
• Fixed #33 so that the herbie.tools.bulk_download will return a dict of Herbie objects that were successful and those that failed.

Full Changelog: 0.0.6...0.0.7

Herbie 0.0.6

Changelog

• #18 Use TOML as config file format instead of INI
• Expanded setting that can be set in the configuration file
• Adopt Black formatting
• Moved PyPI project from hrrrb to herbie-data

Herbie 0.0.5

New Name! HRRR-B 🠖 Herbie

I updated the GitHub repository name to Herbie and I'm slowly removing the old hrrrb API (but it's still there).

The most significant change is that the vision of Herbie has expanded. Herbie is being built do download many different model types, not just the HRRR model.

• Rename package to herbie. "Herbie is your model output download assistant with a mind of its own." Yes, this is named after a favorite childhood movie series.
• Implement new Herbie class
• Drop support for hrrrx (experimental HRRR no longer archived on Pando and ESRL is now developing RRFS)
• Added ability to download and read RAP model GRIB2 files.
• Less reliance on Pando, more on aws and google.
• New method for searchString index file search. Uses same regex search patterns as old API.
• Filename for GRIB2 subset includes all GRIB message numbers.
• Moved default download source to config file setting.
• Check local file copy on init. (Don't need to look for file on remote if we have local copy)
• Option to remove grib2 file when reading xarray if didn't already exist locally (don't clutter local disk).
• Attach index file DataFrame to object if it exists.
• If full file exists locally, use remote idx file to cURL local file instead of remote. (Can't create idx file locally because wgrib2 not available on windows)
• Added GFS data, though it isn't implemented as cleanly as HRRR or RAP
• Renamed 'field' argument to 'product'
• ✨ Moved the source URL templates to their own classes in the models folder
• Renamed GitHub repository to Herbie (changed from HRRR_archive_download)
• Added RRFS, NBM, GFS, RAP as models Herbie can download
• Reworked read_idx() to support index files with additional info (specifically for the NBM).

Herbie 0.0.4

New Herbie API

There are a few things about the hrrrb API that make it difficult to update, so I started changing things under the new name "herbie." But don't worry, both the hrrrb and herbie APIs are included. The setup.py file also is fixed.

To use the new Herbie API, refer to the documentation for some usage examples.

HRRR-B 0.0.3

Welcome to the world, HRRR-B 🎂

This is my first initial GitHub release ever! I have published on PyPi before, but this is my first here on GitHub. Certainly, a happy birthday for the HRRR-B package.

Be aware, this is v0.0.3, meaning it is subject to change at my leisure. The purpose of this repository is to serve as an example of how you can download HRRR data from archives, but I try to keep this package in a workable state that might be useful for you.

📔 Documentation