Module Style Install for Perlmutter Users

[etpalmer63]

Hi VisIt Community!

My name is Erik and I'm a Software Integration Engineer at NERSC. I would like to support a site-wide install of VisIt for our users on Perlmutter and I would like to ask if you have any advice about what install options would work in this case?

I had success following the directions here to install with Spack develop. However, it's not clear to me that doing this style of install will translate into something that could be offered to multiple users. (i.e. if I make frontendlauncher.py available to all users, is that all that's needed? Since it seems to be setting and modifying some environment variables, which I thought would normally be reserved for the module itself.) Therefore I would like to ask if someone has some advice or guidance on the best way to offer VisIt as a module to a wide group of users?

Thank you!

Erik

[markcmiller86]

@etpalmer63 thanks for reaching out.

As you likely know, VisIt has a lot of configuration options and our Spack recipe for building VisIt is still a work in progress. AFAIK, it does not produce a fully featured install. Its better than it was originally but I think a number of database readers are not yet included in it. And, it may not properly configure the desired MPI libs for the HPC platform(s) you are targetting.

It is likely that working directly with build_visit might be better.

As far as how an installed VisIt is actually launched once you've got it installed, users do not run frontendlauncher.py directly. There is a small shell script which finds the desired python to run, sets various enviornment variables and then turns around and starts frontendlauncher.py

As far has having it installed for multiple users, I think all that is required is that it is installed to a publicly readable portion of the file system(s) supporting the targetted HPC platforms. There is a visit-install script that facilitates this once you get a completed build you want to install.

Do any of these resources help...

• https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/getting_started/Installing_VisIt.html
• https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/building_visit/index.html

[etpalmer63]

Thanks @markcmiller86 ! This is very helpful, I was about to go in the completely wrong direction.

[cyrush]

Hi @etpalmer63

I believe Eric Brugger (@brugger1) has been working on a system install for Perlmutter using spack.
You should touch base with him for the latest and the best path forward.

On systems that don't provide X11 libs, we have been using spack b/c build_visit doesn't take care of building that entire stack.

[etpalmer63]

Hi @cyrush,

Yes. I've been in contact with Eric previously. He recommended the Spack build.

When I build from source I'm running into some missing needed packages which I would need someone else to help me install. I can request it, but it slows things down. I suppose I'll pursue the two paths simultaneously and see where I end up.

Thanks for your input!

Erik

[brugger1]

Hi Erik,

I used spack to build VisIt on permutter and documented the necessary steps here. I haven't tried build_visit, but it relies on the X development packages being installed and those don't seem to be installed anymore on most HPC systems. Spack builds all the X libraries and avoids those issues.

Spack's default is to install the packages in $spack/opt/spack. To install it in a public location that is accessible to everyone, you can have spack install it elsewhere be modifying the file etc/spack/defaults/config.yaml. You would modify the following line:

    root: $spack/opt/spack

changing $spack/opt/spack to the path of the directory you want to install VisIt in (documented here).

Once spack has built VisIt, you should be able to run it in serial on one of the login nodes by running $path_to_visit_install_dir/$spack_hardware_description/$spack_compiler_description/visit-3.3.3-$spack_visit_hash/bin/visit

where

$spack_hardware_description is something like linux-sles15-zen3
$spack_compiler_description is something like gcc-11.2.0
$spack_visit_has is a sha that spack creates for VisIt

You can find all those strings by inspecting where spack installed VisIt.

Once you have verified running VisIt in serial, you can create a host profile for running in parallel.

[etpalmer63]

I have a first draft in place. If you're interested in giving it a try you can access it before I put it up for the general user population by doing the following:

• Login to Perlmutter via NoMachine
• perlmutter:login07:~> $ module use /global/common/software/nersc/pm-2023q3/sw/visit/modulefile
• perlmutter:login07:~> $ module load visit
• perlmutter:login07:~> $ visit

This was an experiment in manually translating the Spack install over to the user.

@brugger1 , I haven't followed all the steps in your directions yet. So there is still additional work to do, but I think this should be good for serial so far.

[brugger1]

@etpalmer63 I gave it a try and got the following linking error.

brugger@perlmutter:login14:~> module use /global/common/software/nersc/pm-2023q3/sw/visit/modulefile
brugger@perlmutter:login14:~> module load visit
Note: The VisIt module is designed to be used within an X Window System such as
NoMachine. 
brugger@perlmutter:login14:~> visit
/global/common/software/nersc/pm-2023q3/sw/visit/gcc-11.2.0/visit-3.3.3-xgtah5gxarvann6yefgbwjvs3gaszw2x/bin/../current/linux-x86_64/bin/python3: error while loading shared libraries: libcrypt.so.2: cannot open shared object file: No such file or directory

[etpalmer63]

Thanks @brugger1 for trying that!

I made some progress on the missing libraries, but currently getting the error below:

e4s:login05> visit
Running: gui3.3.3
Qt: Warning: could not connect to display 
Qt: Info: Could not load the Qt platform plugin "xcb" in "" even though it was found.
Qt: Fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: linuxfb, minimal, offscreen, vnc, webgl, xcb.

Is this an error people have hit before? I'm looking for a way to resolve it without reinstalling any packages. I attempted all the other platform plugins by setting the environment variable export QT_QPA_PLATFORM="<platform>" with no luck.

Thanks!

Erik

[brugger1]

@etpalmer63 Yes, I've seen that before. Is this using build_visit? I don't remember the exact cause, but I'm pretty sure it is related to an X Windows related package not being installed. I don't think there is any way around it other than to install the appropriate package. @biagas do you concur?

[biagas]

Yes., I concur.

You might want to do an ldd on libqxcb.so (from Visit's install lib/qtplugins/platforms/ to find out exactly which packages are missing.

[etpalmer63]

Thanks for that feedback! I think I overlooked the way the shared libraries would be linked when I did the install and therefore when I moved things around they became inaccessible. I'm retrying now, will report back when I have something new to try out.

[etpalmer63]

I have another attempt ready for testing by any brave soul 😄

ssh -X <yourself>@perlmutter.nersc.gov 
cd /global/common/software/nersc/pm-2023q3/sw/visit
module use modulefile
module load visit/3.x 
visit

If that works, it is still quite a few iterations from a final product. I'd appreciate any suggestions you can offer for shaking out the bugs.

Thanks again!

Erik

[brugger1]

Hi Erik,

We're getting closer!! The GUI came up fine and I was able to submit a parallel job. Below is the output from my latest attempt.

Eric

visit
Running: gui3.3.3
Running: viewer3.3.3 -geometry 1499x986+421+28 -borders 45,10,10,10 -shift 0,20 -preshift 0,17 -defer -host 127.0.0.1 -port 5600
Running: mdserver3.3.3 -host 127.0.0.1 -port 5601
Running: salloc -A m3090 -t 01:00:00 -N 1 -n 128 --ntasks-per-node=128 --constraint cpu --qos interactive srun /global/u2/b/brugger/spack/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/visit-3.3.3-th5ncewyylv4p4gszkpciogov3ky5cb6/3.3.3/linux-x86_64/bin/engine_par -dir /global/homes/b/brugger/spack/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/visit-3.3.3-th5ncewyylv4p4gszkpciogov3ky5cb6 -forcestatic -idle-timeout 480 -noloopback -host login39 -port 5600
salloc: Pending job allocation 13010727
salloc: job 13010727 queued and waiting for resources
salloc: job 13010727 has been allocated resources
salloc: Granted job allocation 13010727
salloc: Nodes nid007059 are ready for job
srun: error: plugin_load_from_file: dlopen(/usr/lib64/slurm/switch_hpe_slingshot.so): /usr/lib64/libssh.so.4: undefined symbol: EVP_KDF_CTX_new_id, version OPENSSL_1_1_1d
srun: error: Couldn't load specified plugin name for switch/hpe_slingshot: Dlopen of plugin file failed
srun: fatal: Can't find plugin for switch/hpe_slingshot
salloc: Relinquishing job allocation 13010727

[etpalmer63]

Today I added the libtiff library in the proper location and added a bash script to the module load to copy the hosts file into place.

I'll look into that error next.

[brugger1]

The salloc commands are the same except for path differences.

[brugger1]

I found the following paragraph at the page https://docs.nersc.gov/current/ that looks like it may be relevant:

Due to changes in the SLES SP4 system libraries, changes may be required for conda environments built or invoked without using the NERSC provided python module. Users may see errors like ImportError: /usr/lib64/libssh.so.4: undefined symbol: EVP_KDF_CTX_new_id, version OPENSSL_1_1_1d. Please see our Perlmutter python documentation for more information.

It has to do with conda, but it's the same error message. I'm not sure why you're hitting it and I'm not.

[etpalmer63]

👍 -- This is helpful, I'll investigate and report back

[etpalmer63]

Hi Eric,

I did two things. I edited the ~/.visit/hosts/host_nersc_perlmutter.xml file to point to the public install location of visit. This didn't fix the error but seemed like it should be this way anyways.

Then I tried the workaround listed in the docs --doing module load evp-patch before running VisIt. Now I can access the interactive queue without an error message, but VisIt doesn't seem to do anything pass that.

Any other ideas for further trouble shooting? Maybe a debug mode?

[brugger1]

If you run VisIt with -debug 5 on the command line, VisIt will generate log files. The log files will be in the directory you started VisIt and if you are running client server, the debug logs will end up in your home directory on the server.

[etpalmer63]

Gunther did some nice work to get an install of VisIt running on Perlmutter for us. I updated our docs to include how to access it in client-server mode: https://docs.nersc.gov/applications/visit/

Since this issue is no longer relevant, I'm going to go ahead and close it. Thanks!