{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":489109517,"defaultBranch":"main","name":"arco-era5","ownerLogin":"google-research","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2022-05-05T19:58:23.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/43830688?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1720614107.0","currentOid":""},"activityList":{"items":[{"before":"4e2f205dbd77f21bbbf40f2ba75707756d648876","after":"241e91805c56ebde2425cf8bc38bb0e2c1bd5fc6","ref":"refs/heads/main","pushedAt":"2024-07-10T15:53:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shoyer","name":"Stephan Hoyer","path":"/shoyer","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1217238?s=80&v=4"},"commit":{"message":"Update description of data update frequency (#79)\n\nUpdates the description of data updates based on\r\nhttps://github.com/google-research/arco-era5/issues/77#issuecomment-2217505230\r\n\r\n@dabhicusp Please verify that this looks correct to you.","shortMessageHtmlLink":"Update description of data update frequency (#79)"}},{"before":"eb464582a3aaa6f93892f419dcab8cff9250734f","after":null,"ref":"refs/heads/remove_inclusive","pushedAt":"2024-07-10T12:21:47.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"}},{"before":"15673454cae04dc2650187aea429a85e5d195960","after":"4e2f205dbd77f21bbbf40f2ba75707756d648876","ref":"refs/heads/main","pushedAt":"2024-07-10T12:21:43.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Remove inclusive from the source_data. (#80)\n\nRemove `inclusive=left` from the `daily_date_iterator` function of the\r\n`source_data.py` file. This will fixed the issue #78.\r\n\r\nReason: while running the script on the monthly basis this didn't\r\ninclude the last date of the month due to the `inclusive=left` so after\r\nremoving this it will added the last day's data, too.","shortMessageHtmlLink":"Remove inclusive from the source_data. (#80)"}},{"before":null,"after":"eb464582a3aaa6f93892f419dcab8cff9250734f","ref":"refs/heads/remove_inclusive","pushedAt":"2024-07-10T12:13:35.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Remove inclusive from the source_data.","shortMessageHtmlLink":"Remove inclusive from the source_data."}},{"before":"946edfab8b39cd7ef56adfc32e3f4490b11749df","after":"15673454cae04dc2650187aea429a85e5d195960","ref":"refs/heads/main","pushedAt":"2024-06-25T22:24:15.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"wundersooner","name":"Rob Carver","path":"/wundersooner","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2908070?s=80&v=4"},"commit":{"message":"Fix table of contents for README (#75)","shortMessageHtmlLink":"Fix table of contents for README (#75)"}},{"before":"276c52606aaf72a4edb3fe84c505ad1c7ec41923","after":"946edfab8b39cd7ef56adfc32e3f4490b11749df","ref":"refs/heads/main","pushedAt":"2024-06-25T22:16:26.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"wundersooner","name":"Rob Carver","path":"/wundersooner","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2908070?s=80&v=4"},"commit":{"message":"Update README dataset description (#74)\n\nI've reorganized the README to separately introduce \"Analysis Ready\" and\r\n\"Cloud Optimized\" datasets, which an expectation that users will be most\r\ninterested in the former.\r\n\r\nI've also updated all datasets with size and chunking information,\r\ngenerating with the following snippet:\r\n```python\r\nimport xarray_beam\r\nimport math\r\n\r\ndef get_size(x):\r\n for threshold, units in [\r\n (1e6, 'MB'),\r\n (1e9, 'GB'),\r\n (1e12, 'TB'),\r\n (1e15, 'PB'),\r\n ]:\r\n if x < threshold * 1000:\r\n return x/threshold, units\r\n raise RuntimeError('unhandled size')\r\n\r\nfor path in [\r\n 'gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3',\r\n 'gs://gcp-public-data-arco-era5/ar/model-level-1h-0p25deg.zarr-v1',\r\n 'gs://gcp-public-data-arco-era5/co/model-level-wind.zarr-v2',\r\n 'gs://gcp-public-data-arco-era5/co/model-level-moisture.zarr-v2',\r\n 'gs://gcp-public-data-arco-era5/co/single-level-surface.zarr-v2',\r\n 'gs://gcp-public-data-arco-era5/co/single-level-reanalysis.zarr-v2',\r\n 'gs://gcp-public-data-arco-era5/co/single-level-forecast.zarr-v2', \r\n]:\r\n ds, chunks = xarray_beam.open_zarr(\r\n path, storage_options=dict(token='anon')\r\n )\r\n print()\r\n print(path)\r\n size, units = get_size(ds.sel(time=slice(\"1940\", None)).nbytes)\r\n print(f'Total size (1940-present): {size:.3g} {units}')\r\n print('Chunks:', chunks)\r\n size, units = get_size(4*math.prod(chunks.values()))\r\n print(f'Chunk size: {size:.3g} {units}')\r\n print(f'Last time: {ds.indexes[\"time\"][-1]}')\r\n```\r\n\r\nThis currently outputs:\r\n```\r\ngs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3\r\nTotal size (1940-present): 2.05 PB\r\nChunks: {'time': 1, 'latitude': 721, 'longitude': 1440, 'level': 37}\r\nChunk size: 154 MB\r\nLast time: 2024-03-31 23:00:00\r\n\r\ngs://gcp-public-data-arco-era5/ar/model-level-1h-0p25deg.zarr-v1\r\nTotal size (1940-present): 5.88 PB\r\nChunks: {'time': 1, 'hybrid': 18, 'latitude': 721, 'longitude': 1440}\r\nChunk size: 74.8 MB\r\nLast time: 2024-03-31 23:00:00\r\n\r\ngs://gcp-public-data-arco-era5/co/model-level-wind.zarr-v2\r\nTotal size (1940-present): 664 TB\r\nChunks: {'time': 1, 'hybrid': 1, 'values': 410240}\r\nChunk size: 1.64 MB\r\nLast time: 2024-03-31 23:00:00\r\n\r\ngs://gcp-public-data-arco-era5/co/model-level-moisture.zarr-v2\r\nTotal size (1940-present): 1.54 PB\r\nChunks: {'time': 1, 'hybrid': 1, 'values': 542080}\r\nChunk size: 2.17 MB\r\nLast time: 2024-03-31 23:00:00\r\n\r\ngs://gcp-public-data-arco-era5/co/single-level-surface.zarr-v2\r\nTotal size (1940-present): 2.42 TB\r\nChunks: {'time': 1, 'values': 410240}\r\nChunk size: 1.64 MB\r\nLast time: 2024-03-31 23:00:00\r\n\r\ngs://gcp-public-data-arco-era5/co/single-level-reanalysis.zarr-v2\r\nTotal size (1940-present): 60.9 TB\r\nChunks: {'time': 1, 'values': 542080}\r\nChunk size: 2.17 MB\r\nLast time: 2024-03-31 23:00:00\r\n\r\ngs://gcp-public-data-arco-era5/co/single-level-forecast.zarr-v2\r\nTotal size (1940-present): 53.2 TB\r\nChunks: {'time': 1, 'step': 1, 'values': 542080}\r\nChunk size: 2.17 MB\r\nLast time: 2024-03-31 18:00:00\r\n```","shortMessageHtmlLink":"Update README dataset description (#74)"}},{"before":"97cdeee7a96e96544320c41b37eae6f39bd84bf7","after":"276c52606aaf72a4edb3fe84c505ad1c7ec41923","ref":"refs/heads/main","pushedAt":"2024-06-25T12:26:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"ERA5 Native vertical grid zarr(`model-level-1h-0p25deg.zarr-v1`) added. (#73)\n\nAs mentioned in the 5th phase of the roadmap, I derived a\r\nhigh-resolution version of the ERA5\r\ndataset(`gs://gcp-public-data-arco-era5/ar/model-level-1h-0p25deg.zarr-v1`).\r\nFor deriving this dataset, I regridded the datasets to lat/long grids,\r\nconverted the model level variables to pressure level variables, and\r\ncomputed some variables from the model level variables.","shortMessageHtmlLink":"ERA5 Native vertical grid zarr(model-level-1h-0p25deg.zarr-v1) adde…"}},{"before":"45565b1f93752f404cacd56942f196b8330a87e2","after":"977101a3c6e75fa10723514d7297df4f5f0df087","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-25T12:22:02.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"new code updated.","shortMessageHtmlLink":"new code updated."}},{"before":"950ad7d77126835bcb5437709d0e1908376f29bb","after":"45565b1f93752f404cacd56942f196b8330a87e2","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-25T11:41:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Doc string added into docker file.","shortMessageHtmlLink":"Doc string added into docker file."}},{"before":"b822f46c9b77535ed9bcca01adc0f381566124dd","after":"950ad7d77126835bcb5437709d0e1908376f29bb","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-25T10:36:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"timestamp updated in the code.","shortMessageHtmlLink":"timestamp updated in the code."}},{"before":"dc5661583128bac2cf2297ed4c9e054c30d0240e","after":"b822f46c9b77535ed9bcca01adc0f381566124dd","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-25T08:50:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"new constant file added.","shortMessageHtmlLink":"new constant file added."}},{"before":"7ca9c215def3df871b5689b842491f4177dfe2ab","after":null,"ref":"refs/heads/wundersooner-patch-1","pushedAt":"2024-06-25T07:03:36.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"DarshanSP19","name":"Darshan Prajapati","path":"/DarshanSP19","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/93967637?s=80&v=4"}},{"before":"3a9991abfdf989b1c6fa7becc2c09e9e22ae35a1","after":"97cdeee7a96e96544320c41b37eae6f39bd84bf7","ref":"refs/heads/main","pushedAt":"2024-06-25T07:03:32.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"DarshanSP19","name":"Darshan Prajapati","path":"/DarshanSP19","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/93967637?s=80&v=4"},"commit":{"message":"Update README.md to fix typo in roadmap (#63)","shortMessageHtmlLink":"Update README.md to fix typo in roadmap (#63)"}},{"before":"920c1a27f7e3e39373de912c5545e579e318c982","after":"dc5661583128bac2cf2297ed4c9e054c30d0240e","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-16T15:30:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Code refactored.","shortMessageHtmlLink":"Code refactored."}},{"before":"ab159dfb412165e9042b05458abfaf1f072503db","after":"920c1a27f7e3e39373de912c5545e579e318c982","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-16T15:16:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Code updated.","shortMessageHtmlLink":"Code updated."}},{"before":"72dd63fa1beee70b54bc5ec6eea3bb3682ac293c","after":"ab159dfb412165e9042b05458abfaf1f072503db","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-14T14:17:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Docker file and env file updated.","shortMessageHtmlLink":"Docker file and env file updated."}},{"before":"76d2dcd3e32564d37a5b4d21379195716479bbae","after":"72dd63fa1beee70b54bc5ec6eea3bb3682ac293c","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-06-04T11:34:32.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"final code added.","shortMessageHtmlLink":"final code added."}},{"before":"7f859065d0e4dc3cadb377200567152bf27d9db9","after":"76d2dcd3e32564d37a5b4d21379195716479bbae","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-31T11:56:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"zarr chunking scheme updated.","shortMessageHtmlLink":"zarr chunking scheme updated."}},{"before":"5c5cd4da42e74e567fa0eb560946c1e7433ee98b","after":"7f859065d0e4dc3cadb377200567152bf27d9db9","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-31T07:27:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"zarr store final design added.","shortMessageHtmlLink":"zarr store final design added."}},{"before":"fdff70d362f03acaff76d6386a35de5dff59eca9","after":"5c5cd4da42e74e567fa0eb560946c1e7433ee98b","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-31T06:30:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"final zarr store code updated.","shortMessageHtmlLink":"final zarr store code updated."}},{"before":"6784bdd7367e2dd5c11e2b95a9c41662d632e294","after":"fdff70d362f03acaff76d6386a35de5dff59eca9","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-30T08:52:26.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"sample pipeline code updated.","shortMessageHtmlLink":"sample pipeline code updated."}},{"before":"7c70603f437e6a1cb31f53c9ad7beb78b720ed87","after":"6784bdd7367e2dd5c11e2b95a9c41662d632e294","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-29T11:52:28.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"docker image added.","shortMessageHtmlLink":"docker image added."}},{"before":"1c2001280abbf3c3266643b10f2f302981102099","after":"7c70603f437e6a1cb31f53c9ad7beb78b720ed87","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-28T15:09:29.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"second pass of the code is added.","shortMessageHtmlLink":"second pass of the code is added."}},{"before":null,"after":"1c2001280abbf3c3266643b10f2f302981102099","ref":"refs/heads/native_vertical_grid","pushedAt":"2024-05-28T06:26:23.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"Code of the first pass is added.","shortMessageHtmlLink":"Code of the first pass is added."}},{"before":"72b90abc16e39d641a51c4c32dfb19b181c55fa7","after":"fc046d8792a78f2736df26c930c7b47e336465ab","ref":"refs/heads/avro-bq","pushedAt":"2024-04-30T08:21:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"necessary changes are done.","shortMessageHtmlLink":"necessary changes are done."}},{"before":"813358d8439e7d9f792e29695eb42bf57dd494ac","after":"72b90abc16e39d641a51c4c32dfb19b181c55fa7","ref":"refs/heads/avro-bq","pushedAt":"2024-04-29T08:02:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"save_main_session removed from the code.","shortMessageHtmlLink":"save_main_session removed from the code."}},{"before":"470352d4767bec5b3a6f307afaae4d551c5179f9","after":"813358d8439e7d9f792e29695eb42bf57dd494ac","ref":"refs/heads/avro-bq","pushedAt":"2024-04-29T06:41:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"zarr_to_avro_utils added.","shortMessageHtmlLink":"zarr_to_avro_utils added."}},{"before":"f5cb458bb5e98098b4ba15dacfcfa8869371ae11","after":"470352d4767bec5b3a6f307afaae4d551c5179f9","ref":"refs/heads/avro-bq","pushedAt":"2024-04-29T06:28:46.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"data type of variable is updated.","shortMessageHtmlLink":"data type of variable is updated."}},{"before":"a8376231e791b38963b49183f999d387d1445f5a","after":"f5cb458bb5e98098b4ba15dacfcfa8869371ae11","ref":"refs/heads/avro-bq","pushedAt":"2024-04-29T05:58:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"minor changes are done into the pipeline.","shortMessageHtmlLink":"minor changes are done into the pipeline."}},{"before":"2f26f80ba8784777510cf349ee028100f3f42fec","after":"a8376231e791b38963b49183f999d387d1445f5a","ref":"refs/heads/avro-bq","pushedAt":"2024-04-28T05:26:52.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"dabhicusp","name":"dabhi_cusp","path":"/dabhicusp","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/123355381?s=80&v=4"},"commit":{"message":"second pass of bigquery ingestion is done.","shortMessageHtmlLink":"second pass of bigquery ingestion is done."}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEe_FW3wA","startCursor":null,"endCursor":null}},"title":"Activity · google-research/arco-era5"}