Changes needed to build on Hera

[CoryMartin-NOAA]

Without these additions, this repo will not build on NOAA RDHPCS Hera.

Now builds with the following command:
INSTALL_TARGET=hera ./build.sh

[CoryMartin-NOAA]

@aerorahul

[aerorahul]

A similar change is perhaps needed for Orion.
Looks good otherwise.

[aerorahul]

looks good.

[ShelleyMelchior-NOAA]

@KateFriedman-NOAA -- do you have this change accounted for in the changes you are working on for which you're soon to issue a PR? Kate is working on a bunch of changes to run on R&D. I would prefer these changes for R&D all be coordinated.

[KateFriedman-NOAA]

I built the release/obsproc.v1.0.0 branch on Hera (although I have not yet tested it in GFS cycling tests for v16.2, I've been working on Orion first). That obsproc branch has the needed hpc-impi/2018.0.4 module load:

https://github.com/NOAA-EMC/obsproc/blob/release/obsproc.v1.0.0/modulefiles/obsproc_hera.lua#L10

It doesn't have setenv("FC", "mpiifort") but I didn't encounter any build issues without it. See the obsproc install I did on Hera here:

/scratch1/NCEPDEV/global/glopara/git/obsproc/v1.0.0

I agree that the develop branch copy should have the hpc-impi load though.

I recloned a copy of the release/obsproc.v1.0.0 branch on Hera and saved the build output for viewing:

[glopara@hfe04 v1.0.0_test]$ pwd
/scratch1/NCEPDEV/global/glopara/git/obsproc/v1.0.0_test
[glopara@hfe04 v1.0.0_test]$ ll
total 48
drwxr-sr-x  3 glopara global 4096 Apr  8 12:57 build-obsproc
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 docs
drwxr-sr-x 12 glopara global 4096 Apr  8 12:57 ecf
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 fix
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 jobs
drwxr-sr-x  3 glopara global 4096 Apr  8 12:57 modulefiles
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 parm
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 scripts
drwxr-sr-x  8 glopara global 4096 Apr  8 12:57 sorc
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 triggers
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 ush
drwxr-sr-x  2 glopara global 4096 Apr  8 12:57 versions
[glopara@hfe04 v1.0.0_test]$ cd build-obsproc/
[glopara@hfe04 build-obsproc]$ INSTALL_TARGET=hera ./build.sh > build.log 2>&1

Please review the build.log for errors:
/scratch1/NCEPDEV/global/glopara/git/obsproc/v1.0.0_test/build-obsproc/build.log

[CoryMartin-NOAA]

Very interesting,

I agree it shouldn't be needed. But if I don't include it, I get errors such as:

/scratch2/NCEPDEV/stmp1/Cory.R.Martin/obsproc/github/obsproc/sorc/prepobs_oiqcbufr.fd/oiqcbufr.f(2031): error #5102: Cannot open include file 'mpif.h'
      include "mpif.h"
--------------^

[CoryMartin-NOAA]

@KateFriedman-NOAA just a point of clarification, I am able to build the branch you specified above, but develop will fail unless both the hpc-impi and FC=mpiifort additions are included.

[KateFriedman-NOAA]

I have mpiifort in the glopara account environment from loaded modules. Perhaps that's why it works for me? Although the obsproc build should load that at build-time if it needs it.

[glopara@hfe04 build-obsproc]$ module list

Currently Loaded Modules:
  1) hpss/hpss      3) grads/2.0.2        5) impi/2018.0.4   7) nccmp/1.8.5   9) ncview/2.1.7   11) imagemagick/7.0.8-53  13) matlab/R2017b    15) grib_util/1.1.1
  2) rocoto/1.3.1   4) intel/18.0.5.274   6) netcdf/4.7.0    8) nco/4.9.3    10) xxdiff/3.2.Z1  12) contrib               14) prod_util/1.1.0

 
[glopara@hfe04 build-obsproc]$ which mpiifort
/apps/intel/compilers_and_libraries_2018/linux/mpi/intel64/bin/mpiifort

[KateFriedman-NOAA]

@KateFriedman-NOAA just a point of clarification, I am able to build the branch you specified above, but develop will fail unless both the hpc-impi and FC=mpiifort additions are included.

Gotcha...definitely in agreement on needing to add hpc-impi load into the develop modulefile for Hera (and others that may be missing it). I noted that same issue in an issue (#24) I opened in February and the change got into the release branch.

[KateFriedman-NOAA]

So to make sure my environment isn't forgiving (with respect to mpiifort) I tried the following:

1. module purge interactively and then run build (built ok)
   /scratch1/NCEPDEV/global/glopara/git/obsproc/v1.0.0_test/build-obsproc/build.log_2

2. add module purge to build.sh (built ok)

if [[ "$target" =~ ^(wcoss2|hera|orion)$ ]]; then
  source $pkg_root/versions/build.ver
  set +x
  module purge
  module use $pkg_root/modulefiles
  module load obsproc_$target
  module list
  set -x
fi

/scratch1/NCEPDEV/global/glopara/git/obsproc/v1.0.0_test/build-obsproc/build.log_3

3. redid build 2 without set -x/+x to see the purge and such (built ok)
   /scratch1/NCEPDEV/global/glopara/git/obsproc/v1.0.0_test/build-obsproc/build.log_4

Not arguing against adding FC=mpiifort, just showing it builds for me without it while under the glopara account on Hera. Similar build success on Orion with the release branch while under my personal account.

[KateFriedman-NOAA]

@CoryMartin-NOAA @aerorahul FYI, @ShelleyMelchior-NOAA has made a develop-rd branch in both the obsproc and prepobs repos to collect updates to support the two packages on the R&D machines and for us running them outside of operations. I am about to open issues to document updates that I have been working on and will be sending back to the respective develop-rd branches soon. I can include the FC update in the modulefiles if we're in agreement on that.

@ShelleyMelchior-NOAA We should get some of these updates I will be making (particularly the build ones that @CoryMartin-NOAA noted) into the main develop branches at some point. We won't want the R&D branch to diverge from develop and, assuming the R&D changes don't impact use in operations, we should move the develop-rd changes into the develop branch at some point. It's good to cordon off those diffs for now though so we can discuss them before moving them into develop.

[KateFriedman-NOAA]

FYI, I have opened the noted issues in both repos:

obsproc: #34
prepobs: NOAA-EMC/prepobs#6

I will now work on committing my tested changes from Orion into forks of the develop-rd branches and send those changes back to the auth repos via PRs.

[ShelleyMelchior-NOAA]

For the record, @ilianagenkova made the develop-rd branches, not me. ;-)
@CoryMartin-NOAA , @aerorahul thanks for this PR and discussion. Since this change is included in @KateFriedman-NOAA 's future PR into develop-rd, I will plan to close this PR without merge. Fair?

[ilianagenkova]

@KateFriedman-NOAA @CoryMartin-NOAA @aerorahul, I'd like to be upfront about this - we do not intend to merge develop-rd into develop (for either obsproc and prepobs) unless it resolves a bug (demonstrated on WCOSS) . Anything unrelated to obsproc's performance in NCO will stay in develop-rd. It will fall upon the users/developers of develop-rd to keep it up to date with develop.

[aerorahul]

@ilianagenkova
Is there a reason develop can't work for both WCOSS and elsewhere while maintaining WCOSS performance metrics?

[ShelleyMelchior-NOAA]

Changes to platform specific modulefiles are not a problem to take into develop, like this change in @CoryMartin-NOAA 's PR. But @KateFriedman-NOAA is having to make changes to actual processing code, especially for NCO specific things like module prod_util, which does not exist on R&D machines. Those changes need to be thoroughly vetted before ever coming into develop. Obsproc's priority is operational support and has historically never supported obsproc code on R&D machines. Obsproc does not have the resources to ensure working software across non-operational platforms. So while we are not saying "no way!" the priority is very low, and only after thorough vetting. develop-rd provides the landscape for R&D specific needs.

[KateFriedman-NOAA]

So the develop-rd branch obsproc_hera.lua modulefile already has load("hpc-impi/2018.0.4"). I also see it in develop. Confused how that diff shows in this PR. Am I missing something?

[arunchawla-NOAA]

having two development branches -- one for rd and one main one is going to be more work in the long run

[ilianagenkova]

@arunchawla-NOAA , when ObsProc reacts to operational failures, we investigate, develop, test and implement solutions in a matter of hours. NCO has made it clear that any EMC use/testing configurations and settings for obsproc should be limited to our trigger scripts. The idea is to keep develop NCO-ready when time is of the essence. I hope that makes sense. Also see Shelley's comment earlier.

[ilianagenkova]

Changes are confined to Hera-centric file, therefore ok to pull.

[ilianagenkova]

@CoryMartin-NOAA , @KateFriedman-NOAA, @ShelleyMelchior-NOAA , I merged this as it's a hera-centric file change. Going forward, please point PRs to merge to develop-rd. Thanks!
Kate, Shelley, Daryl and I will meet later this week to flesh out the details about this approach.