Data Seeding Scripts For Analysis Ready Dataset #53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alxmrs
reviewed
Sep 14, 2023
| def get_pressure_levels_arg(pressure_levels_group: str): | ||
| return PRESSURE_LEVELS_GROUPS[pressure_levels_group] | ||
|
|
||
| class LoadTemporalDataForDateDoFn(beam.DoFn): |
There was a problem hiding this comment.
Can we make this a PTransform instead?
| # Make sure we print the date as part of the error for easier debugging | ||
| # if something goes wrong. Note "from e" will also raise the details of the | ||
| # original exception. | ||
| raise Exception(f"Error loading {year}-{month}-{day}") from e |
src/arco_era5/update.py
Outdated
| init_date: str | ||
|
|
||
| def expand(self, pcoll: beam.PCollection) -> beam.PCollection: | ||
| return pcoll | beam.Map(update, target=self.target, init_date=self.init_date) |
There was a problem hiding this comment.
Do you want to make this MapTuple?
src/arco_era5/update.py
Outdated
|
|
||
| def update(offset_ds: Tuple[int, xr.Dataset, str], target: str, init_date: str): | ||
| """Generate region slice and update zarr array directly""" | ||
| key, ds = offset_ds |
There was a problem hiding this comment.
If this is called with MapTuple, then you wouldn't have to unpack this.
src/arco_era5/update.py
Outdated
Comment on lines
17
to
18
| date = datetime.datetime.strptime(init_date, '%Y-%m-%d') + datetime.timedelta(days=offset / HOURS_PER_DAY) | ||
| date_str = date.strftime('%Y-%m-%d') |
There was a problem hiding this comment.
Why do we parse a date string into a datetime, then write it back to a string?
There was a problem hiding this comment.
This is only for logs. So we can see the logs like below.
Started 10m_u_component_of_wind for 2014-06-29
Done 10m_u_component_of_wind for 2014-06-29
DarshanSP19
force-pushed
the
ar-backfill-scripts
branch
from
3a1329b
to
3d6adc5
Compare
|
Closing this PR as I've raised a separate PR including these changes and for CO data back fill scripts. Here #58 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Here are some changes in the current script to initialize the
zarrstore from some specific date and a script to seed the data later on in the dataset.Change Includes
netcdf_to_zarrscript.--init_date,--from_init_dateand--only_initialize_store.--temp_locationfrom arguments as the pipeline is ignoring it while running onDataflowRunner.zarrarray itself without involving thexarraylayer and chunking scheme.source_data.pyfromnetcdf_to_zarr.pyas it's reused in the data seeding script.netcdf_to_zarrscript can now be used in three different ways.start_datetoend_dateand write chunks. (Current flow)init_datetoend_dateand write chunks fromstart_datetoend_date. The other values which are falling beyond the range ofstart_dateandend_datewill remainnans. (Required--from_init_dateand optional--init_date.)update_data.py. (Required--only_initialize_storeand optional--init_date)Some defaults
init_datewill be1900-01-01. Can be changed via--init_datearg.--only_initialize_storewhich will only create stores and not write data.