Shell Startup File (.zshrc) Not Run When Connecting to Remote Cluster

[jacobjivanov]

I am a researcher trying to connect Visit to a cluster, and have configured the Host Profiles on my local machine. However, when I connect to the cluster, it fails to initialize the shell with my .zshrc file. As a result, my module files/environment variables are not loaded and I have the following error when I try to open the .h5 files on the cluster (which I can navigate through the tree within the VisIt GUI):

[cluster log in message]
[MODULES SHOULD SHOW THEY WERE LOADED HERE, BUT ARE NOT DONE]
Custom Launcher:Customize:Setting paths on [cluster name] 3.3.3 new arch
Running: vcl3.3.3 -dir [path]/visit/visit3.3.3 -idle-timeout 480 -noloopback -sshtunneling -host localhost -port 25625
Custom Launcher:Customize:Setting paths on [cluster name] 3.3.3 new arch
Running: mdserver3.3.3 -dir [path]/visit/visit3.3.3 -idle-timeout 480 -noloopback -sshtunneling -host localhost -port 16091
Custom Launcher:Customize:Setting paths on [cluster name] 3.3.3 new arch
I am here
Running: qsub -v HOME=[path],LIBPATH=[path]/visit/visit3.3.3/3.3.3/linux-x86_64/lib,VISITHOME=[path]/visit/visit3.3.3/3.3.3,VISITARCHHOME=[path]/visit/visit3.3.3/3.3.3/linux-x86_64,VISITPLUGINDIR=[path]/.visit/3.3.3/linux-x86_64/plugins:[path]/visit/visit3.3.3/3.3.3/linux-x86_64/plugins -j oe -k oe -l application=visit -N VisIt -l place=scatter:excl -l select=1:ncpus=128:mpiprocs=128 -l walltime=0:10:00 -q debug -A [bank name] /tmp/[path].Thu-Oct-24-00:17:19-2024
ERROR: Can't launch using qsub/mpiexec. No such file or directory: 'qsub'

I wanted to verify that the problem was not with my local machine VisIt configuration, and I had another researcher in my group log into the same cluster from my machine. They had no issue. I compared our .zshrc, .visit files on the cluster, and was not able to find any differences, nor were there any other relevant files in our cluster home directories.

Are there any other files that I should check? I would appreciate any help this community might be able to provide.

[markcmiller86]

@jacobjivanov can you confirm you are running 3.3.3 client on your local machine also and not, maybe, a 3.4.0 or 3.4.1 client?

I am a little confused by the new arch in Custom Launcher messages.

Can you also confirm qsub and mpiexec are correct too?

[jacobjivanov]

Confirmed that I am running Visit 3.3.3 on my local machine:

I don't know what the new arch refers to. How should I confirm that qsub and mpiexec are correct?

[markcmiller86]

Hmmm...I don't think you can create a valid host profile for a remote host without knowing how parallel jobs are launched there. Well, I mean if you plan to use parallel engines there. If you plan to only use a serial engine, that might be ok. But, the trace output above you shared showed its trying to use qsub/mpiexec there. If you were actually logged into that remote system and wanted to run a parallel job, would you wind up using qsub and mpiexec to do it? There are many options for this kind of thing depending on how your job scheduler on the remote machine works (srun, psub, msub, poe, sbatch, aprun, etc., etc.). This is a detail about the remote system you have to know to create a valid host profile for it.

[jacobjivanov]

qsub was the correct Job Scheduler. Changing my preferred shell to CSH from ZSH fixed this problem, and I am now able to connect VisIt to this cluster. Why? I don't know. I didn't change anything between my .zshrc and .cshrc files aside from pure syntax and didn't touch my .visit file.

[markcmiller86]

Hmm...well, that is a tad odd. I would expect the specific shell to NOT have any impact on this. To investigate further, it may be useful to have VisIt debug logs (which you can generate by adding -debug 5 to the command line where you started the VisIt client). In client/server scenarios, some logs are generated on the client and some in ~ on the server (remote compute resource).

I don't like that you had to change your preferred shell to fix this. That said, we will need someone else on the VisIt team with a bit more experience with launcher issues to weigh in.