Document pitfalls with PBS output spooling, etc.

[felker]

From a recent experience helping a user on Polaris, there are a few points that could use some clarification in our docs:

• If you submit a job script that specifies #PBS -o /home/user/missing_dir/, and the target directory does not exist, the job will ping-pong between Running ---> Error --> Hold status, with confusing error messages, first claiming that Job run at Wed Oct 16 at 05:37, then job held, too many failed attempts to run

• I don't think we ever explicitly say that a script submitted through qsub starts in $HOME?

• Similarly PBS_O_WORKDIR is referenced a lot in the docs in example scripts, but deserves to be discussed in text alongside the above point.

• Some users have tried to format the output file naming schema with variables like #PBS -o pbs_logs/R-$PBS_JOBID-$PBS_JOBNAME.out but I dont they are defined/ this is possible at job submission time? Might be a Slurm thing

Need to revisit the PBS Big Book on some of this