This repository contains a case study of parallel I/O kernel from the E3SM climate simulation model. The E3SM I/O module, Scorpio, can be built on top of PnetCDF, NetCDF-4, HDF5, and ADIOS. The benchmark program in this repository is designed to evaluate the E3SM I/O kernel when using the above mentioned libraries to perform the I/O task. The challenge of E3SM I/O patterns is that the problem domain is represented by cubed sphere grids which produce long lists of small and noncontiguous requests in each of all MPI processes.
The I/O patterns (describing the data decomposition of multi-dimensional arrays representing the problem domain among MPI processes) used in this case study were captured by the Scorpio library during the E3SM production runs. A data decomposition file records the data access patterns at the array element level by each MPI process. The access offsets are stored in a text file, referred to as the "decomposition map file". This benchmark currently studies three cases from E3SM, namely F, G and I cases. Information about the climate variables written in these three case studies and their decomposition maps can be found in variables.md.
BLOB_IO.md describes the designs of blob I/O methods implemented in this benchmark. Blob I/O methods store data in a timely log-based layout in files.
- See INSTALL.md
- Current build status:
Below shows the execution times of four blob I/O methods collected on July 2022 on Cori at NERSC.
Below shows the execution times of four blob I/O methods collected in September 2022 on Summit at OLCF.
- Wei-keng Liao <[email protected]>
- Kai-yuan Hou <[email protected]>
Copyright (C) 2021, Northwestern University. See COPYRIGHT notice in top-level directory.
This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy's Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation's exascale computing imperative.