Document Polaris PrgEnv-nvhpc edge case for compiling OpenACC codes with NVTX

[felker]

This came up during the ALCF INCITE Hackathon this year (team that @cms21 was mentoring). This is a corner case in which a user 1) is using PrgEnv-nvhpc compilers 2) is using OpenACC 3) is not using CUDA but wants NVTX.

If you compile CUDA code with the default modules (PrgEnv-nvhpc and nvhpc/21.9 right now), you should be to compile and link agains the NVTX (host-based, header only) library without modifying any environment variables or specifying any paths to the library or header. E.g. for test.c

#include "nvToolsExt.h"
#include <stdio.h>

int main() {
    nvtxRangePushA("testing");
    printf("Hello World\n");
    nvtxRangePop();
}

E.g. these compile fine:

❯ nvcc -lnvToolsExt test.c
❯ nvc -cuda -lnvToolsExt test.c

However, without -cuda for nvc, it will fail to find nvToolsExt.h. This group was trying to build and run an OpenACC code and profile it with NVTX markers.

Just adding -cuda is a hacky workaround that works with nvc and nvc++ to get NVTX working with an OpenACC code, but it also links in a bunch of CUDA libraries you might not need (-lcudadevrt -lcudart -lcudanvhpc). Passing -I and -L paths for NVTX is the correct approach for NVHPC compilers:

nvc -c test.c -I/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/include
nvc test.o -o test -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/lib64 -lnvToolsExt

But we should document this in the user guide.

No need to modify the existing modulefiles as far as I can see since it works out of the box for CUDA applications, except:

---

Somewhat related: the nvhpc/21.9 Lmod modulefile supplied by HPE has two more erroneous lines, each missing a forward slash:

prepend_path("LD_LIBRARY_PATH","/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cudaextras/CUPTI/lib64")
prepend_path("LD_LIBRARY_PATH","/opt/nvidia/hpc_sdk/Linux_x86_64/21.9/compilersextras/qd/lib")

So the default LD_LIBRARY_PATH is messed up.