Trouble with trying to zstash data to NERSC tape

[ndkeen]

I've actually had several issues using zstash, but here is the latest. I first tried a zstash create that seemed to have completed, but I wasn't sure. When I tried checking it, many errors. So I decided to simply try a fresh zstash with a new dir name:

cori03% pwd
/global/cfs/cdirs/e3smdata/simulations/production-ssp-unc12

source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-knl.csh

nohup zstash create --hpss=2021/20211021-maint-1.0-tro.A_WCYCLSSP585_CMIP6_HR.ne120_oRRS18v3_ICG.unc12-second-zstash  --maxsize 128 . >& zout-second-zstash.txt &

After several days, I noticed that this process seems to have stopped (couple of days since output last updated) without any error message.

cori03% tail zout-second-zstash.txt
INFO: Archiving run/timing.145858.220210-153808.tar.gz
INFO: Archiving run/timing.146282.220211-192011.tar.gz
INFO: Archiving run/timing.146569.220212-210137.tar.gz
INFO: Archiving run/timing.147073.220215-024253.tar.gz
INFO: Archiving run/timing.147332.220216-024257.tar.gz
INFO: Archiving run/wav_modelio.nml
INFO: Archiving zstash-first-attempt/index.db
INFO: tar name=000332.tar, tar size=64899747840, tar md5=7d183eb23fa736fa3cc8ae5fb1a0b827
INFO: Transferring file to HPSS: zstash/000332.tar
INFO: Transferring file to HPSS: zstash/index.db

At this point, I don't know what to do. So then I tried to update:

nohup zstash update --hpss=2021/20211021-maint-1.0-tro.A_WCYCLSSP585_CMIP6_HR.ne120_oRRS18v3_ICG.unc12-second-zstash  > & zout-second-zstash.update.txt &

Is that command correct?

cori03% cat  zout-second-zstash.update.txt
INFO: Gathering list of files to archive
INFO: Creating new tar archive 000333.tar
INFO: Archiving lo
INFO: Archiving zcheck.out.txt
INFO: Archiving zout-second-zstash.txt
INFO: Archiving zout-second-zstash.update.txt
INFO: tar name=000333.tar, tar size=4352000, tar md5=f9e691b8c65fc1c8ecc4f29b57d6e660
INFO: Transferring file to HPSS: zstash/000333.tar
INFO: Transferring file to HPSS: zstash/index.db

Which sat there for many hours, and the process was no longer there. Again something stopped without an error message.

Looking at the data on tape, first notice that the original zstash create has:

cori03% hsi -q ls -l 2021/20211021-maint-1.0-tro.A_WCYCLSSP585_CMIP6_HR.ne120_oRRS18v3_ICG.unc12 >& od
cori03% tail od
-rw-r-----    1 ndk       ndk      186206883840 Mar  4 09:46 000322.tar
-rw-r-----    1 ndk       ndk      205541191680 Mar  4 10:12 000323.tar
-rw-r-----    1 ndk       ndk      186206883840 Mar  4 10:32 000324.tar
-rw-r-----    1 ndk       ndk      186206883840 Mar  4 10:51 000325.tar
-rw-r-----    1 ndk       ndk      186206883840 Mar  4 11:09 000326.tar
-rw-r-----    1 ndk       ndk      186206883840 Mar  4 11:27 000327.tar
-rw-r-----    1 ndk       ndk      186206883840 Mar  4 11:43 000328.tar
-rw-r-----    1 ndk       ndk      205541191680 Mar  4 11:59 000329.tar
-rw-r-----    1 ndk       ndk      137131499520 Mar  4 12:12 00032a.tar
-rw-r-----    1 ndk       ndk      137363712000 Mar  4 12:24 00032b.tar

and then this second-zstash shows:

cori03% hsi -q ls -l 2021/20211021-maint-1.0-tro.A_WCYCLSSP585_CMIP6_HR.ne120_oRRS18v3_ICG.unc12-second-zstash >& od
cori03% tail od
-rw-r-----    1 ndk       ndk      137363712000 Mar 29 11:06 00032b.tar
-rw-r-----    1 ndk       ndk      137062522880 Mar 29 11:20 00032c.tar
-rw-r-----    1 ndk       ndk      119964723200 Mar 29 11:34 00032d.tar
-rw-r-----    1 ndk       ndk      135340216320 Mar 29 11:48 00032e.tar
-rw-r-----    1 ndk       ndk      135340216320 Mar 29 12:01 00032f.tar
-rw-r-----    1 ndk       ndk      135340216320 Mar 29 12:14 000330.tar
-rw-r-----    1 ndk       ndk      146048153600 Mar 29 12:28 000331.tar
-rw-r-----    1 ndk       ndk      64899747840 Mar 29 12:34 000332.tar
-rw-r-----    1 ndk       ndk          4352000 Mar 31 13:28 000333.tar
-rw-r-----    1 ndk       ndk          3903488 Mar 31 13:28 index.db

Where clearly the 000332.tar and 000333.tar look too small. Am I supposed to "clean up" by identifying those files and remove them first? Will zstash update know what to do?

or just, how to proceed?

[forsyth2]

Is that command correct?

I don't know about the output redirection part but the zstash update --hpss=2021/20211021-maint-1.0-tro.A_WCYCLSSP585_CMIP6_HR.ne120_oRRS18v3_ICG.unc12-second-zstash part looks right to me. The --hpss= matches the one from create (zstash create --hpss=2021/20211021-maint-1.0-tro.A_WCYCLSSP585_CMIP6_HR.ne120_oRRS18v3_ICG.unc12-second-zstash).

Am I supposed to "clean up" by identifying those files and remove them first? Will zstash update know what to do?

zstash update should add any new files and updates to any existing files, irrespective of the tars. I'm not sure why it hung for you. @golaz do you have any input on this?