[INSTALL] Library modules to support GFSv16.2.0 on Hera/Orion #379
Comments
|
Some comments about a few of these packages:
Otherwise the other packages can be installed. |
|
|
In addition, for this purpose, on Hera, the compiler will still be
intel/18.0.5.274.
We are also adding an installation of stack with intel/2021.
…On Tue, Feb 8, 2022 at 1:40 PM Kyle Gerheiser ***@***.***> wrote:
Some comments about a few of these packages:
- gempack - that's not something we have a package for
- cmake - we don't usually handle this and prefer to use the system
CMake. Orion has cmake/3.22.1 and Hera cmake/3.20.1. I would expect
them to be sufficient.
- libpng - We call the module png and have 1.6.35instead of1.6.37`. We
can install 1.6.37 if necessary.
Otherwise the other packages can be installed.
—
Reply to this email directly, view it on GitHub
<#379 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKWSMFBDAZOHNOBYKYP64U3U2FPSRANCNFSM5N3IETPA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
|
I have been working on UPP changes. I will provide the UPP tag for installing upp/8.1.0 when it is ready. |
|
Thanks guys! We want to match the libraries in operations that will be on WCOSS2 with the ones on HERA and Orion (plus the additional libraries we need on the development platforms for the next gen systems) under hpc-stack so we can remove the old installations. I dont think we have to worry about gempak as that is not run in dev mode We will live with intel/18.0.5.274 for now. And intel/2021 when available Thanks ! |
|
The variable "Z_LIB" is not defined in module zlib/1.2.11 on the hpc-stack on Hera. |
Noted, thanks! Won't need based on what @arunchawla-NOAA said above.
Also noted, thanks!
Yes, please install
Excellent, thanks! |
|
The official name for the PNG library is libpng (http://www.libpng.org/pub/png/libpng.html), spack also knows it by this name. Maybe |
|
@climbfuji that's a pretty good idea. Just re-name it libpng, install it as a new package and use that. |
|
I would prefer it be renamed to libpng as well, since that is what we have on WCOSS2 and I'd prefer to use the same name everywhere. Thanks! |
|
Based on the discussions above, may I confirm all modules across different computers must have the same name and version numbers? In other words, will we use the same build.ver and run.ver for all computers? |
@YaliMao-NOAA Yes, we will use the same build.ver and run.ver files for all machines but we will need to accommodate some differences. Therefore, for the workflow side, I am now creating $target.ver files (e.g. orion.ver) which will set the hpc-stack module variables/versions to support those machines. I am going to source them after sourcing build.ver or run.ver, so the stack module names/versions get set for the specific machine. Here is what I just sent in the email thread about updating our Hera/Orion modulefiles to LUA format and hpc-stack: The top of one of the workflow modulefiles will now look like this for Orion: ...and then the rest of the module versions will be set, just as on WCOSS2. |
|
@kgerheiser We also load the following modules on WCOSS2, are they from hpc-stack and if so, can we get these versions on Hera/Orion too? Thanks!
|
|
@KateFriedman-NOAA It's a smart solution to introduce $target.ver to differentiate the platforms. |
@YaliMao-NOAA Yes, |
|
The module zlib/1.2.11 settings on WCOSS2 and Orion/Hera are not consistent. Hera: The UPP for GFSV16 needs "ZLIB_LIB" for GNU makefile building. |
|
@WenMeng-NOAA you are looking at the wrong module on Orion ( |
|
@kgerheiser On Orion, Here are what I check on Orion: Please let me know the correct path. Thanks! |
|
@kgerheiser The UPP is looking for variable "ZLIB_LIB" for GNU makefile building. It is defined on WCOSS2 but not on Orion/Hera. |
|
@kgerheiser Similar to Wen's comment, I need |
|
It looks like the external packages have not been installed using hpc-stack but using spack |
|
The UPP also got building failure in netcdf as: My working version on hera is at /scratch1/NCEPDEV/stmp2/Wen.Meng/post_gfsv16_hpc/UPP/sorc. The GNU make file is |
|
@WenMeng-NOAA , @KateFriedman-NOAA solved this problem, reach out to her. |
|
@WenMeng-NOAA Not sure if it's related but I had to make the following change to the gaussian_sfcanl makefile to get it building on Orion with the static libraries there: |
|
@KateFriedman-NOAA our solution works for UPP. Thanks! |
|
@kgerheiser Is it possible to get these two modules available on Hera and Orion too? |
|
It looks like we will need to have an intel 18.5 stack to support gfs v16.2.0 on orion and hera, but we should create an issue to solve this problem with intel 2022.1, perhaps by the time gfs v16.2.1 comes along we can move to intel 2022.1 compiler. This is not an issue with intel 19 compilers on wcoss2 Mike ? |
|
@arunchawla-NOAA What would the "intel 18.5" stack have compared to the current intel 2018 stack that Hang/Kyle have installed for us? The fixed crtm version? @MichaelLueken-NOAA |
|
@arunchawla-NOAA There is no need for an intel 18.5 stack on Orion/Hera. Kyle and Hang have Intel 2018 hpc-stack built libraries on both machines. There is no need to add a fixed crtm version, since the fix is already in crtm/2.4.0 (though, the GSI hasn't been updated to use crtm/2.4.0 as of this time). As for the Intel 2019 on WCOSS2, the GSI has been running fine on that machine with the 2019 compiler. This issue is only for Intel 2021 and 2022 compilers. During our meeting last Thursday with the UFS RRFS app folks, Chris told us that the GSI had issues with these compilers, so it shouldn't be a surprise that the GSI is segfaulting with 2022. |
|
@MichaelLueken-NOAA Gotcha...if we stick with the intel 2018 stack on Hera/Orion should I point global-workflow to use @junwang-noaa @WenMeng-NOAA @GeorgeGayno-NOAA @HelinWei-NOAA @YaliMao-NOAA Given the issues with the GSI moving to the newer intel we are going to have to stick with the intel 2018 stack on Hera/Orion (even knowing it's now unsupported should new issues crop up). We will work to resolve the issues with intel 2022 and possibly move the whole GFS system to the newer version for v16.2.1+ (post-WCOSS2-go-live). @WenMeng-NOAA reconfirmed in email that intel 2018 is fine for UPP...please also reconfirm if intel 2018 has any outstanding issues for your respective components. Thanks again for testing both versions, we've uncovered some issues that we can now work to fix because of that effort. |
|
The intel/2018 compiler on Hera is called as intel/18.0.5.274 versus the
other old compiler intel/18.0.1.163, It is briefed as intel/18.5.
On orion, the intel/2018 compiler is called as intel/2018.4
I think Arun use the hera name.
The crtm v2.3.0 is different from v2.4.0. However, all fix files have been
added to the old v2.3.0 (fixed/update) and v2.4.0 installations.
…On Thu, Mar 10, 2022 at 9:54 AM MichaelLueken-NOAA ***@***.***> wrote:
@arunchawla-NOAA <https://github.com/arunchawla-NOAA> There is no need
for an intel 18.5 stack on Orion/Hera. Kyle and Hang have Intel 2018
hpc-stack built libraries on both machines. There is no need to add a fixed
crtm version, since the fix is already in crtm/2.4.0 (though, the GSI
hasn't been updated to use crtm/2.4.0 as of this time). As for the Intel
2019 on WCOSS2, the GSI has been running fine on that machine with the 2019
compiler. This issue is only for Intel 2021 and 2022 compilers. During our
meeting last Thursday with the UFS RRFS app folks, Chris told us that the
GSI had issues with these compilers, so it shouldn't be a surprise that the
GSI is segfaulting with 2022.
—
Reply to this email directly, view it on GitHub
<#379 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKWSMFGWIAUHZN6KFRHVK5TU7IEH5ANCNFSM5N3IETPA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Ok, sounds like we should be ok to stick with |
|
Ben, Haixia and I, we worked together till last Friday, finally get the
last issue for crtm v2.4.0 fixed.
So, now the GSI tests based on crtm v2.3.0 and v2.4.0 get the same result.
…On Thu, Mar 10, 2022 at 10:05 AM Kate Friedman ***@***.***> wrote:
The crtm v2.3.0 is different from v2.4.0. However, all fix files have been
added to the old v2.3.0 (fixed/update) and v2.4.0 installations.
Ok, sounds like we should be ok to stick with crtm/2.3.0 then. Thanks
@Hang-Lei-NOAA <https://github.com/Hang-Lei-NOAA> ! Please confirm
@MichaelLueken-NOAA <https://github.com/MichaelLueken-NOAA> , thanks!
—
Reply to this email directly, view it on GitHub
<#379 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKWSMFC5XG62WH3L4VRCEADU7IFSNANCNFSM5N3IETPA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Oh ok, excellent! Will see which version @MichaelLueken-NOAA tells me to use for the GSI within the GFS and will go forward with that version for GFSv16.2 on Hera/Orion. Thanks @Hang-Lei-NOAA ! |
|
@KateFriedman-NOAA I tested Intel 2022 version on hera/intel with atm only configuration, the model works fine. I will test atm only with Intel 2018 version and get wave component updated when the changes are ready. |
|
@KateFriedman-NOAA UPP supporting GFS V16.2.0 uses crtm/2.3.0. |
Thanks @junwang-noaa ! I can start testing cycling with intel 2018 and atmos-only (waves turned off) next week if you have something that works while waiting for wave updates. Let me know, thanks!
Gotcha, thanks for confirming @WenMeng-NOAA ! |
@KateFriedman-NOAA We should continue pointing at crtm/2.3.0 until the GSI repository has been updated to handle crtm/2.4.0. |
Roger that, will stick wtih `crtm/2.3.0'. Thanks @MichaelLueken-NOAA ! |
|
@KateFriedman-NOAA I tested the Intel 2018 hpc-stack in my branch: https://github.com/junwang-noaa/ufs-weather-model/tree/modulefileRnD. It works for atmos-only runs on hera and orion. Please let me know if you have any questions. |
Thanks @junwang-noaa ! I may try this before the WW3 stuff is sorted out, will let you know if I encounter any issues when I do try it. |
|
@junwang-noaa I built your Hera: /scratch1/NCEPDEV/global/Kate.Friedman/git/dev_v16/sorc/logs/build_fv3.log_modulefileRnD_2 Thanks! |
|
@KateFriedman-NOAA Thanks for testing. I checked the compile log files, it looks to me the model was built successfully. |
Thanks @junwang-noaa ! I will begin testing with this copy on Hera/Orion. Will report any runtime issues. |
|
@kgerheiser @Hang-Lei-NOAA Can we get a prod_util v2+ installed in the hpc-stack-gfsv16 installs on Hera and Orion? I ask because the new obsproc/prepobs packages for WCOSS2 use prod_util/2+ and those newer versions now use |
|
@KateFriedman-NOAA do you know where prod_util comes from? We have a repo, but apparently it's not being used by whoever is maintaining this code. |
Good question...I think it's NCO that maintains |
|
We have it in the global workflow repository don't we?. I remember
building it from there in the past and snagged it for my obsolete tarball
NCEPLIBS distribution in mid 2019 after getting sick of trying to keep
track of where it was on the various platforms. I also did this with
grib_utils and that's the same vintage.
I no longer work with the nceplibs tarball; it's been (sensibly)
superseded by the HPC_STACK API .
…On Tue, Apr 5, 2022 at 11:23 AM Kate Friedman ***@***.***> wrote:
@KateFriedman-NOAA <https://github.com/KateFriedman-NOAA> do you know
where prod_util comes from? We have a repo
<https://github.com/NOAA-EMC/NCEPLIBS-prod_util>, but apparently it's not
being used by whoever is maintaining this code.
Good question...I think it's NCO that maintains prod_util. The v2.0.9
module on WCOSS2 lives in /apps/ops/prod/nco space:
***@***.***:/lfs/h2/emc/global/noscrub/Kate.Friedman/git/dev_v16> module show prod_util/2.0.9
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
/apps/ops/prod/nco/modulefiles/core/prod_util/2.0.9:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("This module loads and sets environment variables for the NCEP Central Operations production utilities. ")
conflict("prod_util")
setenv("UTILROOT","/apps/ops/prod/nco/core/prod_util.v2.0.9")
setenv("MDATE","/apps/ops/prod/nco/core/prod_util.v2.0.9/exec/mdate")
setenv("NDATE","/apps/ops/prod/nco/core/prod_util.v2.0.9/exec/ndate")
setenv("NHOUR","/apps/ops/prod/nco/core/prod_util.v2.0.9/exec/nhour")
setenv("FSYNC","/apps/ops/prod/nco/core/prod_util.v2.0.9/exec/fsync_file")
setenv("COMDATEROOT","/lfs/h1/ops/prod/com")
setenv("COMLISTROOT","/lfs/h1/ops/prod/config")
setenv("COMLOGSROOT","/lfs/h1/ops/prod/com")
prepend_path("PATH","/apps/ops/prod/nco/core/prod_util.v2.0.9/ush")
prepend_path("PYTHONPATH","/apps/ops/prod/nco/core/prod_util.v2.0.9/ush")
help([[Set environment variables for production utilities
]])
—
Reply to this email directly, view it on GitHub
<#379 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANDS4FUKAN3S37RFNIOVQFTVDRLERANCNFSM5N3IETPA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
George W Vandenberghe
*IMSG* at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
***@***.***
301-683-3769(work) 3017751547(cell)
|
It doesn't live in global-workflow. It may have been in a release package years ago, can't remember. Doesn't live with global-workflow anymore though. |
|
@kgerheiser Checking to see if it will be possible to get a v2+ version of prod_util in hpc-stack-gfsv16 on Hera and Orion. We'll need a v2+ version in the non-gfsv16 hpc-stack soon after as well for supporting GFSv17+ development. Let me know if I can help. Thanks! |
|
Looking into getting the v2 source. |
Thanks @kgerheiser ! Appreciate it! |
|
Does it run with intel 2021?
…On Tue, Mar 8, 2022 at 9:29 AM MichaelLueken-NOAA ***@***.***> wrote:
@KateFriedman-NOAA <https://github.com/KateFriedman-NOAA> While the GSI
will compile with Intel 2022, it will not run. So, we will need to move
forward with Intel 2018.
—
Reply to this email directly, view it on GitHub
<#379 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANDS4FSD7HZZKLVICQZOD5LU65P23ANCNFSM5N3IETPA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
George W Vandenberghe
*IMSG* at NOAA/NWS/NCEP/EMC
5830 University Research Ct., Rm. 2141
College Park, MD 20740
***@***.***
301-683-3769(work) 3017751547(cell)
|
|
I think we have both hpc-stack installations with intel/18.0.5.274 and
intel/2022.1 on Hera and orion for gfsv16.2. I installed them on Hera and
Kyle made them on orion.
…On Wed, Mar 9, 2022 at 2:35 PM arun chawla ***@***.***> wrote:
It looks like we will need to have an intel 18.5 stack to support gfs
v16.2.0 on orion and hera, but we should create an issue to solve this
problem with intel 2022.1, perhaps by the time gfs v16.2.1 comes along we
can move to intel 2022.1 compiler. This is not an issue with intel 19
compilers on wcoss2 Mike ?
—
Reply to this email directly, view it on GitHub
<#379 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKWSMFB4YTNW7OMDN3SHOTDU7D4PPANCNFSM5N3IETPA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@GeorgeVandenberghe-NOAA The GFSv16 only runs with intel 2018 on the R&Ds. There was an issue moving the GSI to intel 2021/2022 so we stuck with 2018. We're waiting for the GSI/JEDI to resolve this so we can move the entire GFSv17 system to intel 2021/2022. |
In order to support the new GFSv16.2.0 (WCOSS2 port version) on Hera and Orion we need the same library module versions available. Below I list the versions that are currently being used in the new operational GFSv16.2.0 and which ones are missing on Hera/Orion.
Which software (and version) in the stack would you like installed?
Hera & Orion:
gempak/7.14.1cmake/3.20.2Which machines would you like to have the software installed?
Hera, Orion
Additional context
Here are the build.ver module versions for GFSv16.2.0:
https://github.com/NOAA-EMC/global-workflow/blob/feature/ops-wcoss2/versions/build.ver
Refs: NOAA-EMC/global-workflow#639
The text was updated successfully, but these errors were encountered: