diff --git a/docs/examples/parallel-computing-with-dask.ipynb b/docs/examples/parallel-computing-with-dask.ipynb
index 57baf8ce..08dd79ed 100644
--- a/docs/examples/parallel-computing-with-dask.ipynb
+++ b/docs/examples/parallel-computing-with-dask.ipynb
@@ -7,7 +7,9 @@
"source": [
"# General Guide for Parallelizing xCDAT Operations with Dask\n",
"\n",
- "Author: [Tom Vo](https://github.com/tomvothecoder)\n"
+ "Author: [Tom Vo](https://github.com/tomvothecoder)\n",
+ "\n",
+ "Last Updated: 04/16/24 (v0.7.0)\n"
]
},
{
@@ -22,11 +24,7 @@
"\n",
"- Basics of Dask Arrays\n",
"- General Dask Best Practice\n",
- "- How to use Dask with Xarray\n",
- "- How to use Dask with xCDAT, including real-world examples and performance metrics\n",
- "- Dask Schedulers and using a local distributed scheduler for more resource-intensive needs\n",
- "\n",
- "_The data used in the code examples can be found through the [Earth System Grid Federation (ESGF) search portal](https://aims2.llnl.gov/search)._\n",
+ "- How to use Dask with xCDAT, including real-world examples\n",
"\n",
"### More Resources\n",
"\n",
@@ -35,7 +33,9 @@
"- [Official Xarray Parallel Computing with Dask Guide](https://docs.xarray.dev/en/stable/user-guide/dask.html)\n",
"- [Official Xarray Parallel Computing with Dask Jupyter Notebook Tutorial](https://tutorial.xarray.dev/intermediate/xarray_and_dask.html)\n",
"- [Official Dask guide for Xarray with Dask Arrays](https://examples.dask.org/xarray.html)\n",
- "- [Project Pythia: Dask Arrays with Xarray](https://foundations.projectpythia.org/core/xarray/dask-arrays-xarray.html)\n"
+ "- [Project Pythia: Dask Arrays with Xarray](https://foundations.projectpythia.org/core/xarray/dask-arrays-xarray.html)\n",
+ "\n",
+ "_The data used in the code examples can be found through the [Earth System Grid Federation (ESGF) search portal](https://aims2.llnl.gov/search)._\n"
]
},
{
@@ -270,7 +270,32 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Code Example - Setup\n"
+ "## Using a Dask Cluster for Scalable Computations\n",
+ "\n",
+ "> All of the large-scale Dask collections like Dask Array, Dask DataFrame, and Dask Bag and the fine-grained APIs like delayed and futures generate task graphs where each node in the graph is a normal Python function and edges between nodes are normal Python objects that are created by one task as outputs and used as inputs in another task. After Dask generates these task graphs, it needs to execute them on parallel hardware. This is the job of a task scheduler. Different task schedulers exist, and each will consume a task graph and compute the same result, but with different performance characteristics.\n",
+ "> Dask has two families of task schedulers:\n",
+ ">\n",
+ "> 1. **Single-machine scheduler**: This scheduler provides basic features on a local process or thread pool. This scheduler was made first and is the default. It is simple and cheap to use, although it can only be used on a single machine and does not scale\n",
+ "> 2. **Distributed scheduler**: This scheduler is more sophisticated, offers more features, but also requires a bit more effort to set up. It can run locally or distributed across a cluster\n",
+ ">\n",
+ "> — https://docs.dask.org/en/stable/scheduling.html\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "\n",
+ "Xarray is setup to use 1. **Single-machine scheduler**. However, Dask advises users to use the Dask distributed scheduler for more advanced functionality and more resource-intensive needs.\n",
+ "\n",
+ "— https://docs.dask.org/en/stable/scheduling.html#dask-distributed-local\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Code Example - Setup\n",
+ "\n",
+ "# TODO: Use a larger dataset\n"
]
},
{
@@ -299,31 +324,425 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Code Example - Parallelism with xCDAT + Dask\n",
+ "### 1. Setup the Dask Cluster on your local machine\n",
"\n",
- "The code example below demonstrates chunking a dataset in Xarray and grouping the data\n",
- "in parallel across chunks.\n",
+ "We will quickly setup a local cluster using the Dask `Client` and `LocalCluster`\n",
+ "Python modules.\n",
"\n",
- "**By default, dask uses its multi-threaded scheduler**, which distributes work across multiple cores and allows for processing some datasets that do not fit into memory.\n",
+ "You can configure the Dask Client (e.g., memory limit) to your needs. In this case,\n",
+ "we are deploying a cluster with 4 workers, automatic memory limitation, and processors\n",
+ "(instead of threads). For info on cluster configurations, visit these links:\n",
"\n",
- "If you are interested in using a distributed scheduler (local or cluster) for more resource-intensive computational operations, there is more information below in this notebook.\n"
+ "- https://distributed.dask.org/en/latest/api.html#client\n",
+ "- https://distributed.dask.org/en/latest/api.html#distributed.LocalCluster\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "2024-04-16 15:14:38,008 [INFO]: scheduler.py(__init__:1709) >> State start\n",
+ "2024-04-16 15:14:38,008 [INFO]: scheduler.py(__init__:1709) >> State start\n",
+ "2024-04-16 15:14:38,014 [INFO]: diskutils.py(_check_lock_or_purge:252) >> Found stale lock file and directory '/tmp/dask-scratch-space/scheduler-0fbhq831', purging\n",
+ "2024-04-16 15:14:38,014 [INFO]: diskutils.py(_check_lock_or_purge:252) >> Found stale lock file and directory '/tmp/dask-scratch-space/scheduler-0fbhq831', purging\n",
+ "2024-04-16 15:14:38,017 [INFO]: diskutils.py(_check_lock_or_purge:252) >> Found stale lock file and directory '/tmp/dask-scratch-space/scheduler-cwzvww8f', purging\n",
+ "2024-04-16 15:14:38,017 [INFO]: diskutils.py(_check_lock_or_purge:252) >> Found stale lock file and directory '/tmp/dask-scratch-space/scheduler-cwzvww8f', purging\n",
+ "2024-04-16 15:14:38,025 [INFO]: scheduler.py(start_unsafe:4060) >> Scheduler at: tcp://127.0.0.1:37919\n",
+ "2024-04-16 15:14:38,025 [INFO]: scheduler.py(start_unsafe:4060) >> Scheduler at: tcp://127.0.0.1:37919\n",
+ "2024-04-16 15:14:38,027 [INFO]: scheduler.py(start_unsafe:4075) >> dashboard at: http://127.0.0.1:8787/status\n",
+ "2024-04-16 15:14:38,027 [INFO]: scheduler.py(start_unsafe:4075) >> dashboard at: http://127.0.0.1:8787/status\n",
+ "2024-04-16 15:14:38,029 [INFO]: scheduler.py(register_worker_plugin:7676) >> Registering Worker plugin shuffle\n",
+ "2024-04-16 15:14:38,029 [INFO]: scheduler.py(register_worker_plugin:7676) >> Registering Worker plugin shuffle\n",
+ "2024-04-16 15:14:38,051 [INFO]: nanny.py(start_unsafe:368) >> Start Nanny at: 'tcp://127.0.0.1:46038'\n",
+ "2024-04-16 15:14:38,051 [INFO]: nanny.py(start_unsafe:368) >> Start Nanny at: 'tcp://127.0.0.1:46038'\n",
+ "2024-04-16 15:14:38,063 [INFO]: nanny.py(start_unsafe:368) >> Start Nanny at: 'tcp://127.0.0.1:37970'\n",
+ "2024-04-16 15:14:38,063 [INFO]: nanny.py(start_unsafe:368) >> Start Nanny at: 'tcp://127.0.0.1:37970'\n",
+ "2024-04-16 15:14:40,516 [INFO]: scheduler.py(add_worker:4412) >> Register worker \n",
+ "2024-04-16 15:14:40,516 [INFO]: scheduler.py(add_worker:4412) >> Register worker \n",
+ "2024-04-16 15:14:40,523 [INFO]: scheduler.py(handle_worker:5892) >> Starting worker compute stream, tcp://127.0.0.1:45995\n",
+ "2024-04-16 15:14:40,523 [INFO]: scheduler.py(handle_worker:5892) >> Starting worker compute stream, tcp://127.0.0.1:45995\n",
+ "2024-04-16 15:14:40,525 [INFO]: core.py(handle_stream:1019) >> Starting established connection to tcp://127.0.0.1:37940\n",
+ "2024-04-16 15:14:40,525 [INFO]: core.py(handle_stream:1019) >> Starting established connection to tcp://127.0.0.1:37940\n",
+ "2024-04-16 15:14:40,530 [INFO]: scheduler.py(add_worker:4412) >> Register worker \n",
+ "2024-04-16 15:14:40,530 [INFO]: scheduler.py(add_worker:4412) >> Register worker \n",
+ "2024-04-16 15:14:40,532 [INFO]: scheduler.py(handle_worker:5892) >> Starting worker compute stream, tcp://127.0.0.1:34301\n",
+ "2024-04-16 15:14:40,532 [INFO]: scheduler.py(handle_worker:5892) >> Starting worker compute stream, tcp://127.0.0.1:34301\n",
+ "2024-04-16 15:14:40,534 [INFO]: core.py(handle_stream:1019) >> Starting established connection to tcp://127.0.0.1:37938\n",
+ "2024-04-16 15:14:40,534 [INFO]: core.py(handle_stream:1019) >> Starting established connection to tcp://127.0.0.1:37938\n",
+ "2024-04-16 15:14:40,557 [INFO]: scheduler.py(add_client:5650) >> Receive client connection: Client-b84f7600-fc3e-11ee-84a4-f4e9d4af2192\n",
+ "2024-04-16 15:14:40,557 [INFO]: scheduler.py(add_client:5650) >> Receive client connection: Client-b84f7600-fc3e-11ee-84a4-f4e9d4af2192\n",
+ "2024-04-16 15:14:40,560 [INFO]: core.py(handle_stream:1019) >> Starting established connection to tcp://127.0.0.1:37968\n",
+ "2024-04-16 15:14:40,560 [INFO]: core.py(handle_stream:1019) >> Starting established connection to tcp://127.0.0.1:37968\n"
+ ]
+ }
+ ],
+ "source": [
+ "from dask.distributed import Client, LocalCluster\n",
+ "\n",
+ "# This line of code will automatically start a local cluster\n",
+ "# https://docs.dask.org/en/stable/deploying.html#local-machine\n",
+ "\n",
+ "cluster = LocalCluster(\n",
+ " n_workers=2, threads_per_worker=1, memory_limit=\"auto\", processes=True\n",
+ ")\n",
+ "\n",
+ "client = Client(cluster)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "e77c4df4b8bc4a468cf508b648e73a59",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/html": [
+ "
"
+ ],
+ "text/plain": [
+ "LocalCluster(f9a5f35e, 'tcp://127.0.0.1:42380', workers=4, threads=4, memory=20.99 GiB)"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "client.cluster"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2. Open the Dask Dashboard UI\n",
+ "\n",
+ "The Dask distributed scheduler provides an interactive dashboard containing many plots\n",
+ "and tables with live information.\n",
+ "\n",
+ "Check this [Dask documentation page](https://docs.dask.org/en/stable/dashboard.html) to learn how to interpret the information.\n",
+ "\n",
+ "Here's an example:\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n"
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
- "We're letting Dask auto-scale all dimensions to get a good chunk size using `chunks=\"auto\"`, which references the `.chunks` attribute.\n"
+ "#### Open the link to the dashboard\n"
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'http://127.0.0.1:8787/status'"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "client.dashboard_link"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
"metadata": {},
- "outputs": [],
"source": [
- "ds = xr.open_dataset(filepath, chunks=\"auto\")"
+ "### 3. Open a dataset with xCDAT and chunk it\n",
+ "\n",
+ "We're letting Dask auto-scale all dimensions to get a good chunk size using `chunks=\"auto\"`, which references the `.chunks` attribute.\n",
+ "\n",
+ "It tries to reach chunk equal to the config value of `array.chunk-size`, which is set to\n",
+ "128MiB by default, which you can change in your [configuration](https://docs.dask.org/en/stable/configuration.html).\n",
+ "\n",
+ "— https://docs.dask.org/en/stable/array-chunks.html#automatic-chunking\n"
]
},
{
@@ -723,10 +1142,10 @@
" license: CMIP6 model data produced by CSIRO is li...\n",
" cmor_version: 3.4.0\n",
" tracking_id: hdl:21.14100/af78ae5e-f3a6-4e99-8cfe-5f2...\n",
- " DODS_EXTRA.Unlimited_Dimension: time
xarray.Dataset
time: 1980
bnds: 2
lat: 145
lon: 192
time
(time)
datetime64[ns]
1850-01-16T12:00:00 ... 2014-12-...
bounds :
time_bnds
axis :
T
long_name :
time
standard_name :
time
_ChunkSizes :
1
array(['1850-01-16T12:00:00.000000000', '1850-02-15T00:00:00.000000000',\n",
+ " DODS_EXTRA.Unlimited_Dimension: time
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
ACCESS-ESM1.5 (2019): \n",
"aerosol: CLASSIC (v1.0)\n",
"atmos: HadGAM2 (r1.1, N96; 192 x 145 longitude/latitude; 38 levels; top level 39255 m)\n",
"atmosChem: none\n",
@@ -1081,18 +1500,36 @@
}
],
"source": [
+ "ds = xr.open_dataset(filepath, chunks=\"auto\")\n",
"ds"
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
- "Now we perform a daily average using the `groupby` API.\n",
+ "### 4. Run your computations while viewing the dashboards\n",
+ "\n",
+ "Bad signs to watch out for in the dashboard include:\n",
+ "\n",
+ "- Lots of white space in the task stream plot is a bad sign. White space means nothing is happening. Chunks may be too small.\n",
+ "- Lots and lots of red in the task stream plot is a bad sign. Red means worker communication. Dask workers need some communication, but if they are doing almost nothing except communication then there is not much productive work going on.\n",
+ "- On the worker memory plot, watch out for orange bars which are a sign you are getting close to the memory limit. Chunks may be too big.\n",
+ "- On the worker memory plot, watch out for grey bars which mean data is being spilled to disk. Chunks may be too big.\n",
+ "\n",
+ "— https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we perform a monthly grouped average using the `group_average` API.\n",
"\n",
- "`flox` must be installed to make this API parallelizable and Xarray will use `flox` by\n",
- "default if it is installed.\n"
+ "`flox` must be installed to make this API parallelizable and Xarray will use `flox` by default if it is installed.\n",
+ "\n",
+ "> Note: No computations are performed yet. The task graph is built and the computation\n",
+ "> is queued up (lazy)\n"
]
},
{
@@ -1466,104 +1903,32 @@
" stroke: currentColor;\n",
" fill: currentColor;\n",
"}\n",
- "
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
Creation Date:(30 April 2019) MD5:5bd755e94c2173cb3050a0f3480f60c4
title :
ACCESS-ESM1-5 output prepared for CMIP6
variable_id :
tas
variant_label :
r10i1p1f1
version :
v20200605
license :
CMIP6 model data produced by CSIRO is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (https://creativecommons.org/licenses/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file). The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
Creation Date:(30 April 2019) MD5:5bd755e94c2173cb3050a0f3480f60c4
title :
ACCESS-ESM1-5 output prepared for CMIP6
variable_id :
tas
variant_label :
r10i1p1f1
version :
v20200605
license :
CMIP6 model data produced by CSIRO is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (https://creativecommons.org/licenses/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file). The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.
cmor_version :
3.4.0
tracking_id :
hdl:21.14100/af78ae5e-f3a6-4e99-8cfe-5f29eb467cd1
DODS_EXTRA.Unlimited_Dimension :
time
"
- ],
- "text/plain": [
- " Size: 441MB\n",
- "Dimensions: (lat: 145, bnds: 2, lon: 192, time: 1980)\n",
- "Coordinates:\n",
- " * lat (lat) float64 1kB -90.0 -88.75 -87.5 -86.25 ... 87.5 88.75 90.0\n",
- " * lon (lon) float64 2kB 0.0 1.875 3.75 5.625 ... 352.5 354.4 356.2 358.1\n",
- " height float64 8B 2.0\n",
- " * time (time) object 16kB 1850-01-01 00:00:00 ... 2014-12-01 00:00:00\n",
- "Dimensions without coordinates: bnds\n",
- "Data variables:\n",
- " lat_bnds (lat, bnds) float64 2kB -90.0 -89.38 -89.38 ... 89.38 89.38 90.0\n",
- " lon_bnds (lon, bnds) float64 3kB -0.9375 0.9375 0.9375 ... 357.2 359.1\n",
- " tas (time, lat, lon) float64 441MB 246.0 246.0 246.0 ... 247.9 247.9\n",
- "Attributes: (12/48)\n",
- " Conventions: CF-1.7 CMIP-6.2\n",
- " activity_id: CMIP\n",
- " branch_method: standard\n",
- " branch_time_in_child: 0.0\n",
- " branch_time_in_parent: 87658.0\n",
- " creation_date: 2020-06-05T04:06:11Z\n",
- " ... ...\n",
- " variant_label: r10i1p1f1\n",
- " version: v20200605\n",
- " license: CMIP6 model data produced by CSIRO is li...\n",
- " cmor_version: 3.4.0\n",
- " tracking_id: hdl:21.14100/af78ae5e-f3a6-4e99-8cfe-5f2...\n",
- " DODS_EXTRA.Unlimited_Dimension: time"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "tas_avg_xc.compute()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Dask Task Scheduling\n",
- "\n",
- "> All of the large-scale Dask collections like Dask Array, Dask DataFrame, and Dask Bag and the fine-grained APIs like delayed and futures generate task graphs where each node in the graph is a normal Python function and edges between nodes are normal Python objects that are created by one task as outputs and used as inputs in another task. After Dask generates these task graphs, it needs to execute them on parallel hardware. This is the job of a task scheduler. Different task schedulers exist, and each will consume a task graph and compute the same result, but with different performance characteristics.\n",
- "> Dask has two families of task schedulers:\n",
- ">\n",
- "> 1. **Single-machine scheduler**: This scheduler provides basic features on a local process or thread pool. This scheduler was made first and is the default. It is simple and cheap to use, although it can only be used on a single machine and does not scale\n",
- "> 2. **Distributed scheduler**: This scheduler is more sophisticated, offers more features, but also requires a bit more effort to set up. It can run locally or distributed across a cluster\n",
- ">\n",
- "> — https://docs.dask.org/en/stable/scheduling.html\n",
- "\n",
- "
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
Creation Date:(30 April 2019) MD5:5bd755e94c2173cb3050a0f3480f60c4
title :
ACCESS-ESM1-5 output prepared for CMIP6
variable_id :
tas
variant_label :
r10i1p1f1
version :
v20200605
license :
CMIP6 model data produced by CSIRO is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (https://creativecommons.org/licenses/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file). The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
2020-06-05T04:06:10Z altered by CMOR: Treated scalar dimension: 'height'. 2020-06-05T04:06:10Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20).
Creation Date:(30 April 2019) MD5:5bd755e94c2173cb3050a0f3480f60c4
title :
ACCESS-ESM1-5 output prepared for CMIP6
variable_id :
tas
variant_label :
r10i1p1f1
version :
v20200605
license :
CMIP6 model data produced by CSIRO is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (https://creativecommons.org/licenses/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file). The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.
cmor_version :
3.4.0
tracking_id :
hdl:21.14100/af78ae5e-f3a6-4e99-8cfe-5f29eb467cd1
DODS_EXTRA.Unlimited_Dimension :
time
"
],
"text/plain": [
- " Size: 231kB\n",
- "Dimensions: (lat: 145, bnds: 2, lon: 192)\n",
+ " Size: 441MB\n",
+ "Dimensions: (lat: 145, bnds: 2, lon: 192, time: 1980)\n",
"Coordinates:\n",
" * lat (lat) float64 1kB -90.0 -88.75 -87.5 -86.25 ... 87.5 88.75 90.0\n",
" * lon (lon) float64 2kB 0.0 1.875 3.75 5.625 ... 352.5 354.4 356.2 358.1\n",
" height float64 8B 2.0\n",
+ " * time (time) object 16kB 1850-01-01 00:00:00 ... 2014-12-01 00:00:00\n",
"Dimensions without coordinates: bnds\n",
"Data variables:\n",
" lat_bnds (lat, bnds) float64 2kB -90.0 -89.38 -89.38 ... 89.38 89.38 90.0\n",
" lon_bnds (lon, bnds) float64 3kB -0.9375 0.9375 0.9375 ... 357.2 359.1\n",
- " tas (lat, lon) float64 223kB 225.1 225.1 225.1 ... 254.1 254.1 254.1\n",
+ " tas (time, lat, lon) float64 441MB 246.0 246.0 246.0 ... 247.9 247.9\n",
"Attributes: (12/48)\n",
" Conventions: CF-1.7 CMIP-6.2\n",
" activity_id: CMIP\n",
@@ -5089,13 +2881,121 @@
" DODS_EXTRA.Unlimited_Dimension: time"
]
},
- "execution_count": 11,
+ "execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
- "tas_avg_xc2.compute()"
+ "tas_avg_xc.compute()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Code Example - Setting up Dask on HPC\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 1. Start an interactive job\n",
+ "\n",
+ "The first step if you're an HPC environment is to start a Dask job to get nodes.\n",
+ "\n",
+ "Depending on your HPC environment, you might need to run `salloc`, `srun`, etc. to\n",
+ "start an interactive job. Please refer to the documentation for your HPC environment.\n",
+ "\n",
+ "In the example below on the NERSC Perlmutter machine, we're requesting 1 CPU node from\n",
+ "the reservation and divide the resources on that node among 10 Dask workers. If you're\n",
+ "working on a reservation, request the respective number of hours needed for\n",
+ "your node (e.g., 4 hrs/240 mins). When you are working outside a reservation, its best\n",
+ "to limit your requested time to your expected working window.\n",
+ "\n",
+ "`salloc --reservation=dask_day1 -C cpu -N 1 -n 10 -A ntrain5 -t 240`\n",
+ "\n",
+ "It may take a moment to start, but once it does, you'll get your prompt back.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Processes or Threads?\n",
+ "\n",
+ "Python has the [global interpreter lock (GIL)](https://wiki.python.org/moin/GlobalInterpreterLock),\n",
+ "which basically means that Python does not leverage multiple threads well.\n",
+ "\n",
+ "The general exceptions to this rule include code that is mainly I/O (e.g., downloading\n",
+ "data) or code that leverages mostly C++ and other non-Python libraries (e.g., NumPy).\n",
+ "\n",
+ "For most use cases, using processes over threads might make more sense. Here's an example\n",
+ "with `dask-worker`:\n",
+ "\n",
+ "- I have 40 GB of RAM\n",
+ "- I have 4 cores. So I want 4 workers.\n",
+ " - This means each worker can consume 10 GB of RAM.\n",
+ "\n",
+ "```bash\n",
+ "$ dask-worker tcp://127.0.0.1:8786 --nprocs 4 --memory-limit 10GB\n",
+ "```\n",
+ "\n",
+ "— https://saturncloud.io/blog/local-cluster/\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2. Start the Dask Cluster\n",
+ "\n",
+ "There are several ways to deploy a Dask Cluster. You can check them out [here](https://docs.dask.org/en/stable/deploying.html). For this section, we'll be manually deploying a cluster in your HPC environment.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Open a terminal and start the Dask scheduler:\n",
+ "\n",
+ "```bash\n",
+ "$ dask-scheduler\n",
+ "```\n",
+ "\n",
+ "Open a second terminal and start a Dask worker:\n",
+ "\n",
+ "```bash\n",
+ "$ dask-worker tcp://127.0.0.1:8786 --nworkers 4 --nthreads=1 --memory-limit 10GB\n",
+ "```\n",
+ "\n",
+ "The default address for the `dask-scheduler` is `tcp://127.0.0.1:8786`.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 3. Connect the Dask Client to the Cluster\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client_hpc = Client(\"tcp://127.0.0.1:8786\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client_hpc"
]
},
{
![\"Dask](\"../_static/dask-overview-schedulers.svg\")
\n",
+ "![\"Dask](\"../_static/dask-dashboard-example.png\")
\n",
+ "