Skip to content

Latest commit

 

History

History
346 lines (328 loc) · 15.4 KB

README.md

File metadata and controls

346 lines (328 loc) · 15.4 KB
Raw

Utility Programs for E3SM-IO benchmark

  • dat2nc -- converts multiple decomposition map .dat files in text format into a file in NetCDF CDF5 format.
  • dat2decomp -- converts decomposition map .dat files in text format into a file in either NetCDF CDF5, HDF5, NetCDF4, or BP format.
  • decomp_copy -- copies a decomposition map file in CDF5/HDF5/NetCDF4/BP format to a new file in a different file format.
  • datstat -- displays statistics of a decomposition map .dat file.
  • bpstat -- displays the size of attributes and variables in an ADIOS2 BP3 file.
  • pnetcdf_blob_replay -- converts the subfiles produced by the e3sm_io running with PnetCDF blob I/O strategy into a regular NetCDF CDF5 file.

dat2nc

dat2nc is a utility program that converts a set of decomposition files (in a text format) generated by PIO or Scorpio to a NetCDF CDF5 file, which can be used as input files to the e3sm_io benchmark to evaluate its I/O performance. The instructions below explain how to convert the data decomposition file into the NetCDF file.

  • For the F case, there are three unique data decomposition maps, thus three .dat files, generated during E3SM production runs. The .dat files are in the text format. Converting the decomposition maps and merged into a single NetCDF file enables e3sm_io to read the decomposition maps in parallel.

  • For the G case, there are 6 unique decomposition maps, thus 6 .dat files.

  • For the I case, there are 5 unique decomposition map files.

  • Utility program, dat2nc.c, can be built by running command make, which will build the executable file named dat2nc.

  • In addition to converting the text-based decomposition maps into NetCDF array variables, dat2nc also performs the following tasks.

    • Removes 0 offsets - In a decomposition map .dat file, the index starts from 1, i.e. a Fortran convension. Indices with 0 values indicate the process does not write to any element. They can be ignored and hence are removed in the output NetCDF file.
    • Coalesces consecutive offsets - The decomposition map records the offsets of individual array elements accessed by a process. dat2nc detects consecutive offsets and merges them into a contiguous access region, represented by a pair of offset and length.
    • Option -s - Sorts the offset and length pairs into an increasing order of offsets. The offsets in a decomposition map generated from PIO or Scorpio are NOT necessarily sorted in an increasing order.
    • Option -r - The original decomposition maps, i.e. offsets only, can also be included in the NetCDF file if using the command-line option -r. When this option is enabled, dat2nc will save a copy of the original decomposition maps into variables with prefix name "raw_".
    • Option -f - Assigns the uncovered elements in decomposition maps. Not all decomposition maps cover the enire array elements they decompose.

  • Three decomposition map files from a small E3SM F case running 16 MPI processes are provided in folder datasets.

    • piodecomp16tasks16io01dims_ioid_514.dat (decomposition along the fastest dimensions)
    • piodecomp16tasks16io01dims_ioid_516.dat (decomposition along the fastest dimensions)
    • piodecomp16tasks16io02dims_ioid_548.dat (decomposition along the fastest two dimensions)

  • The input file to dat2nc is a text file contains a list of decomposition map file names. An example is provided in ./input_files_f_case.txt.

    % cat ./input_files_f_case.txt
../datasets/piodecomp16tasks16io01dims_ioid_514.dat
../datasets/piodecomp16tasks16io01dims_ioid_516.dat
../datasets/piodecomp16tasks16io02dims_ioid_548.dat

  • The command to convert and combine the three .dat files from the small F case example to a NetCDF file is:

    % ./dat2nc -i input_files_f_case.txt -o map_f_case_16p.nc

  • Full list of command-line options of ./dat2nc:

    % ./dat2nc -h
Usage: dat2nc [-h|-v|-r|-f|-s|-l num] -i input_file -o out_file
-h             Print help
-v             Verbose mode
-r             Include original decomposition maps
-f             Fill in unassigned elements in decomposition maps
-s             Sort offsets of decomposition maps increasingly
-l num         max number of characters per line in input file
-i input_file  a text file containing a list of decomposition
               map .dat file names
-o out_file    name of output netCDF file

  • The output NetCDF file, map_f_case_16p.nc, converted from these 3 decomposition .dat files is also provided in folder ../datasets. Its metadata can be obtained from running NetCDF utility program ncdump or ncmpidump and is shown below.

      % cd ./datasets
  % ncmpidump -h map_f_case_16p.nc
  netcdf map_f_case_16p {
  // file format: CDF-5 (big variables)
  dimensions:
      num_decomp = 3 ;
      decomp_nprocs = 16 ;
      D1.total_nreqs = 47 ;
      D1.total_raw_nreqs = 866 ;
      D2.total_nreqs = 866 ;
      D2.total_raw_nreqs = 866 ;
      D3.total_nreqs = 62352 ;
      D3.total_raw_nreqs = 62352 ;
  variables:
      int D1.nreqs(decomp_nprocs) ;
              D1.nreqs:description = "Number of noncontiguous requests per process" ;
      int D1.fill_starts(decomp_nprocs) ;
              D1.fill_starts:description = "Start index in offsets[] and lengths[] for requests to be written with fill values" ;
      int D1.offsets(D1.total_nreqs) ;
              D1.offsets:description = "Flattened starting indices of noncontiguous requests" ;
      int D1.lengths(D1.total_nreqs) ;
              D1.lengths:description = "Lengths of noncontiguous requests" ;
              D1.lengths:max = 36 ;
              D1.lengths:min = 9 ;
      int D1.raw_nreqs(decomp_nprocs) ;
              D1.raw_nreqs:description = "Number of file offsets accessed per process before merge" ;
      int D1.raw_offsets(D1.total_raw_nreqs) ;
              D1.raw_offsets:description = "File offsets accessed before merge" ;
      int D2.nreqs(decomp_nprocs) ;
              D2.nreqs:description = "Number of noncontiguous requests per process" ;
      int D2.fill_starts(decomp_nprocs) ;
              D2.fill_starts:description = "Start index in offsets[] and lengths[] for requests to be written with fill values" ;
      int D2.offsets(D2.total_nreqs) ;
              D2.offsets:description = "Flattened starting indices of noncontiguous requests" ;
      int D2.lengths(D2.total_nreqs) ;
              D2.lengths:description = "Lengths of noncontiguous requests" ;
              D2.lengths:max = 1 ;
              D2.lengths:min = 1 ;
      int D2.raw_nreqs(decomp_nprocs) ;
              D2.raw_nreqs:description = "Number of file offsets accessed per process before merge" ;
      int D2.raw_offsets(D2.total_raw_nreqs) ;
              D2.raw_offsets:description = "File offsets accessed before merge" ;
      int D3.nreqs(decomp_nprocs) ;
              D3.nreqs:description = "Number of noncontiguous requests per process" ;
      int D3.fill_starts(decomp_nprocs) ;
              D3.fill_starts:description = "Start index in offsets[] and lengths[] for requests to be written with fill values" ;
      int D3.offsets(D3.total_nreqs) ;
              D3.offsets:description = "Flattened starting indices of noncontiguous requests" ;
      int D3.lengths(D3.total_nreqs) ;
              D3.lengths:description = "Lengths of noncontiguous requests" ;
              D3.lengths:max = 1 ;
              D3.lengths:min = 1 ;
      int D3.raw_nreqs(decomp_nprocs) ;
              D3.raw_nreqs:description = "Number of file offsets accessed per process before merge" ;
      int D3.raw_offsets(D3.total_raw_nreqs) ;
              D3.raw_offsets:description = "File offsets accessed before merge" ;

  // global attributes:
              :command_line = "./dat2decomp -v -r -a cdf5 -i input_files_f.txt -o map_f_case_16p.nc " ;
              :D1.ndims = 1 ;
              :D1.dims = 866 ;
              :D1.max_nreqs = 4 ;
              :D1.min_nreqs = 2 ;
              :D1.fully_covered = "true" ;
              :D2.ndims = 1 ;
              :D2.dims = 866 ;
              :D2.max_nreqs = 56 ;
              :D2.min_nreqs = 52 ;
              :D2.fully_covered = "true" ;
              :D3.ndims = 2 ;
              :D3.dims = 72, 866 ;
              :D3.max_nreqs = 4032 ;
              :D3.min_nreqs = 3744 ;
              :D3.fully_covered = "true" ;
  }

  • The NetCDF file containing the 6 decompositions from a small E3SM G case running 16 MPI processes is also available in folder ../datasets with file named map_g_case_16p.nc.

  • The NetCDF file containing the 5 decompositions from a small E3SM I case running 16 MPI processes is also available in folder ../datasets with file named map_i_case_16p.nc.

dat2decomp

dat2decomp is generalized from dat2nc to convert the decomposition map .dat files into HDF5, NetCDF4, or BP file format, in addition to the NetCDF classic 64-bit data format. The instructions below explain how to convert the data decomposition map files into CDF5, HDF5, NetCDF4, or ADIOS file format.

  • Running command make under to root folder should have created the executable dat2decomp in folder utils.
    • Use -a option to select the output file format.
      • cdf5 - NetCDF CDF5 file format, same as dat2nc
      • hdf5 - HDF5 format
      • netcdf4 - NetCDF 4 format
      • bp - ADIOS2 BP format
  • The command to combine the three .dat files for the small F case to a HDF5 file is: 
    % ./dat2decomp -a hdf5 -i input_files_f_case.txt -o map_f_case_16p.nc
  • Command-line options of ./dat2decomp: 
    % ./dat2decomp -h
Usage: ./dat2decomp [-h|-v|-r|-f|-s|-l num] -a fmt -i input_file -o out_file
-h             Print help
-v             Verbose mode
-r             Include original decomposition maps
-f             Fill in unassigned elements in decomposition maps
-s             Sort offsets of decomposition maps increasingly
-l num         max number of characters per line in input file
-a fmt         output file format, fmt is one of the followings
   cdf5:       NetCDF classic 64-bit data format
   netcdf4:    NetCDF-4 (HDF5-based) format
   hdf5:       HDF5 format
   bp:         ADIOS2 BP format
-i input_file  a text file containing a list of decomposition
               map .dat file names
-o out_file    output file name

decomp_copy

decomp_copy copies a decomposition file already in the NetCDF CDF5, HDF5, NetCDF4, or ADIOS BP format to a new file in a different format.

  • Note that e3sm_io must be configured with the API options enabled for the corresponding input and output file formats. For instance, to convert a NetCDF CDF5 file to an HDF5 file, e3sm_io must be configured with --with-pnetcdf and --with-hdf5. Below is an example command. 
    % ./decomp_copy -a hdf5 -i map_f_case_16p.nc -o map_f_case_16p.h5
  • Command-line options of ./decomp_copy: 
    % ./decomp_copy -h
Usage: ./decomp_copy [-h|-v] -a fmt -i input_file -o out_file
   [-h]            Print help
   [-v]            verbose mode
   -a fmt          output file format, fmt is one of the followings
      cdf5:        NetCDF classic 64-bit data format
      netcdf4:     NetCDF-4 (HDF5-based) format
      hdf5:        HDF5 format
      bp:          ADIOS2 BP format
   -i input_file   input file name
   -o out_file     output file name

datstat

datstat reads a decomposition map .dat file in its original text format and report some statistics of the decomposition.

  • Command-line options of datstat:

    % ./datstat -h
Usage: ./datstat [OPTION]...
       -h             Print help
       -d input_file  decomposition map .dat file to be analyzed

  • Example run:

      % ./datstat -d ../datasets/piodecomp16tasks16io01dims_ioid_514.dat
  =========================================================================
  Deomposition file ../datasets/piodecomp16tasks16io01dims_ioid_514.dat
  =========================================================================
  version: 2001
  npes: 16
  ndims: 1
  Dims: 866,
  total_cells: 1536
  total_zero_cells: 670
  max_cells: 96 @ rank 0
  max_zero_cells: 56 @ rank 12
  zero ratio: 0.436198

bpstat

bpstat counts and displays the size of attributes and variables in an ADIOS2 BP3 file. If the input path name is a folder containing sub-files, the path name should be without the .dir extension.

  • Command-line options of bpstat:

    % ./bpstat -h
Usage: ./bpstat [OPTION]...
       -v    Verbose mode
       -h    Print help
       path  Name of BP file or folder to be analyzed

  • Example run:

    % ./bpstat ./f_case_h0.bp
Num subfiles: 1
Num variables: 1203
Total variable size: 17115880
Num attributes: 3501
Total attribute size: 302738
Total object size (MiB): 16.6117
Total object size (GiB): 0.0162224

pnetcdf_blob_replay

pnetcdf_blob_replay reads the subfiles produced by the e3sm_io benchmark when running with pnetcdf blob method into a single, regular NetCDF file. Currently, it must be run with the same number of MPI processes that produced the subfiles. The future work will remove such limitation to allow less number of MPI processes to convert.

  • Command-line options of pnetcdf_blob_replay: 
    % ./pnetcdf_blob_replay -h
Usage: pnetcdf_blob_replay [OPTION]... FILE
    [-h] Print help
    [-v] Verbose mode
    -i file  Base name of input subfiles
    -o file  Output file name
  • Example run: 
      % mpiexec -n 16 ./pnetcdf_blob_replay -i blob_F_out_h0.nc -o F_out_h0.nc
    Input subfile base name            = blob_F_out_h0.nc
    Output file name                   = F_out_h0.nc
    No. decompositions                 =   3
    No. variables                      = 429
    No. records (time steps)           =   1
    Max Time of file open/create       = 0.0494 sec
    Max Time of define variables       = 0.0446 sec
    Max Time of read                   = 0.0090 sec
    Max Time of write                  = 0.3435 sec
    Max Time of close                  = 0.0249 sec
    Max end-to-end time                = 0.4714 sec
    -----------------------------------------------------------
    Total read  amount                 = 16.80 MiB = 0.02 GiB
    Read  bandwidth                    = 1869.3614 MiB/sec
    Total write amount                 = 16.16 MiB = 0.02 GiB
    Write bandwidth                    = 47.0496 MiB/sec
    -----------------------------------------------------------

Developers

Copyright (C) 2021, Northwestern University. See COPYRIGHT notice in top-level directory.

Project funding supports:

This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy's Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation's exascale computing imperative.