Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy mppnccombine-fast to vk83 #27

Open
aidanheerdegen opened this issue Mar 12, 2024 · 1 comment
Open

Deploy mppnccombine-fast to vk83 #27

aidanheerdegen opened this issue Mar 12, 2024 · 1 comment

Comments

@aidanheerdegen
Copy link
Member

mppnccombine-fast is a fast version of mppnccombine (the clue is in the name).

https://github.com/coecms/mppnccombine-fast

It is necessary for the high resolution versions of ACCESS-OM2 to use this fast version of mppnccombine to collate the tiled outputs (and restarts) otherwise the resource requirements blow out, particularly memory and runtime. To quote from the README

Uses HDF5's raw IO functions to speed up collating large datasets - a 0.1 degree model goes from taking 4 hours to collate a compressed variable with mppnccombine, to 6 minutes with mppnccombine-fast running with 16 processes

This will require a spack package and deployment infrastructure to make it available in vk83.

@aidanheerdegen
Copy link
Member Author

As a temporary measure I have installed the same version of mppnccombine-fast currently used by COSIMA here:

$ md5sum  /g/data/ik11/inputs/access-om2/bin/mppnccombine-fast                                                
875366d620a04ea4ec28e8ec5871efb7  /g/data/ik11/inputs/access-om2/bin/mppnccombine-fast

into vk83

$ md5sum /g/data/vk83/apps/mppnccombine-fast/0.2/bin/mppnccombine-fast
875366d620a04ea4ec28e8ec5871efb7  /g/data/vk83/apps/mppnccombine-fast/0.2/bin/mppnccombine-fast

I built this executable in 2020 with this source version:

commit 025c7193bd43418319db378f6553d1d115eec1d5 (HEAD -> master, tag: 0.2, origin/master, origin/HEAD)
Author: Scott Wales <[email protected]>
Date:   Fri Apr 17 10:22:40 2020 +1000

    Compressnc3 (#42)
    
    * Compress nc3 input files
    
    Fixes #39

I have pushed a 0.2 tag (as you can see above) so we have a tagged version for the current package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant