Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Subfile Read Issue #62

Open
wants to merge 22 commits into
base: master
Choose a base branch
from
Open

Fix Subfile Read Issue #62

wants to merge 22 commits into from

Commits on May 30, 2023

  1. Configuration menu
    Copy the full SHA
    ba718a1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f7049e4 View commit details
    Browse the repository at this point in the history
  3. check return error

    wkliao authored and yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    d905666 View commit details
    Browse the repository at this point in the history
  4. check env H5VL_LOG_NSUBFILES

    wkliao authored and yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    493ea35 View commit details
    Browse the repository at this point in the history
  5. add test program subfile_dwrite.cpp

    wkliao authored and yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    63963d0 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    5d1fed5 View commit details
    Browse the repository at this point in the history
  7. clean test output subfile folders

    wkliao authored and yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    ebd19c2 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    0e11f4f View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    d21c041 View commit details
    Browse the repository at this point in the history
  10. fix: correct nldset and nmdset for subfiles

    when opening a subfile, fp->nldset and fp->nmdset are not read from
    subfiles. Instead, they are read from the master file. This commit
    fix this issue and read from the subfile correctly.
    yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    eeca154 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    9436082 View commit details
    Browse the repository at this point in the history
  12. add subfile read testcases

    yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    3601bbc View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    b21a80c View commit details
    Browse the repository at this point in the history
  14. fix: subfile read: mismatch of nsubfiles and nproc

    It is possible that, for example, a file is created with 8 subfiles.
    But when openning and reading the file, we only use 4 processes.
    
    In the original implementation before this fix, the info of 8 subfiles
    is not saved. Only the first fp->ngroup subfiles will be opened for
    read, where fp->ngroup is a number bounded by the number of processes
    (i.e. <= 4 in this case).
    
    In this fix, we use fp->nsubfiles to store the number of subfiles for an
    opened file. All fp->nsubfiles subfiles will be opened for read.
    yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    c32690c View commit details
    Browse the repository at this point in the history
  15. fix: possible hangs during dataset read.

    MPI_FILE_set_view is a collective call. Befroe this fix, not all
    processes call this function during dataset read, introducing possible
    hangs. A subfile read test case may trigger this issue more easiliy.
    This commit fix this issue.
    yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    6c0a053 View commit details
    Browse the repository at this point in the history
  16. testcase: subfile_dread tests multiple scenarios

    Test the following:
    1. nsbufile > nproc
    2. nsubfile == nproc
    3. nsubfile < nproc
    
    For each of the above, test:
    1. read pattern same as write pattern (row wise)
    2. read pattern is row wise, but each process read a different row than
    it writes. (read from one subfile that process is not responsible for)
    3. read pattern is column wise (read from several subfiles)
    4. read all dataset.
    
    This means a total of 12 scenarios are tested.
    
    Also, we test each scenario using 1 to 12 number of processes. This
    makes sures Log VOL also works for odd number of processes.
    yzanhua committed May 30, 2023
    Configuration menu
    Copy the full SHA
    817fd11 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    c15c360 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. DEBUG PRINT

    yzanhua committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    3b3a0a2 View commit details
    Browse the repository at this point in the history
  2. add print internally

    yzanhua committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    4e61d18 View commit details
    Browse the repository at this point in the history
  3. add more print internally

    yzanhua committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    2f85ab9 View commit details
    Browse the repository at this point in the history
  4. test smaller mem

    yzanhua committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    7f368c6 View commit details
    Browse the repository at this point in the history
  5. possible fix 1

    yzanhua committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    815918f View commit details
    Browse the repository at this point in the history