Stage marine DA files and data to common directory

[JohnSteffen-NOAA]

Stage marine DA related data files for verification etc. from Shastri's local directories to a common place:

/scratch2/NCEPDEV/marine/Shastri.Paturi/
/scratch2/NCEPDEV/stmp1/Shastri.Paturi/forLydiaSulagna/validation/
/scratch1/NCEPDEV/stmp4/Shastri.Paturi/

/scratch1/NCEPDEV/da/Shastri.Paturi

Clean up unnecessary data files as well.

[guillaumevernieres]

this needs to be rm'ed

/scratch2/NCEPDEV/marineda/validation

Duplicated ...

[apchoiCMD]

Is it also, correct? since da/common has it as well,

/scratch2/NCEPDEV/stmp1/Shastri.Paturi/forLydiaSulagna/validation/

[guillaumevernieres]

That's 3 places with the exact same stuff ... We might want to start digging into the fs and look for "Shastri" and "validation".

[apchoiCMD]

Group Name : stmp1

• /scratch2/NCEPDEV/stmp1/Shastri.Paturi/forZulema => many folders for obs but most of folders are empty : RM'ED
• /scratch2/NCEPDEV/stmp1/Shastri.Paturi/forKatherine => duplicated validation files for ostia2mom6 : RM'ED
• /scratch2/NCEPDEV/stmp1/Shastri.Paturi/forLydiaSulagna => duplicated validation files : RM'ED
• /scratch2/NCEPDEV/stmp1/Shastri.Paturi/aws => empty : RM'ED
• /scratch2/NCEPDEV/stmp1/Shastri.Paturi/data => empty : RM'ED

Group Name : marine

• /scratch2/NCEPDEV/marine/Shastri.Paturi/2021 => 2021's altkob, bathy, subpfl, tesac, trkob & xbtctd : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/CFSR2DATM_P1 => CFSR Data from 1979 to 1987 : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/COAPS => NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/DATA_realtime => Some real time obs to bump stuff : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/DATM_INPUT => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/ForJim => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/GEFS => Some data only : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/GIOMAS => NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/GOMb0.08 => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/IODA_00z => Perhaps Old IODA converter sh file with data : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/JEDI => Just JEDI build : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/Latency_table => Latency related but Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/NCEPLIBS-bufr => Build NCEPLIBS-bufr : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/NG-GODAS => NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/NSIDC => Some seaice files : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/OISST => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/OceanObsQC => NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/PIOMAS => NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/QC_task => NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/WCOSS1 => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/anaconda => Related to anaconda stuff : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/aws => Unnecessary files : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/for_maria => Some logs from Maria : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/frommars => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/historical_profiles => http://www.metoffice.gov.uk/hadobs/en4/data/en4-2-1/EN.4.2.1.profiles.g10 from 1978 to 2020 - LIST ONLY, No files : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/ice.L2.emc => Some ice data files : NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/ice.cryosat_GDR.esa => CS_OFFL_SIR_GDR_2 from 20100715 to 20200829 : NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/icec.nsidc => SeaIce related data files : NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/insitu.fnmoc => FNMOC data files : NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/ioda_observations => Seems some old files there : NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/ncoda_code => Permission Denied
• /scratch2/NCEPDEV/marine/Shastri.Paturi/plots => plots only : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/rtofs => Nothing important : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/rtofs_025=> Some files : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/rtofs_prod => RTOFSDA.zip : NEEDED TO LOOK AT
• /scratch2/NCEPDEV/marine/Shastri.Paturi/rtofsv2_NAVYrst => Nothing important : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/sandbox => His sandbox for WCDA, g-w, etc : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/scripts => Some scripts : RM'ED
• /scratch2/NCEPDEV/marine/Shastri.Paturi/sst_2019_2020=> Duplicated SST data sets (I guess) : RM'ED

---

All folders and files are inspected and we will need to decide them to be deleted or kept to common : @guillaumevernieres @AndrewEichmann-NOAA @JohnSteffen-NOAA , please have them to make sure-

[apchoiCMD]

@guillaumevernieres @AndrewEichmann-NOAA @JohnSteffen-NOAA I just get an approvals getting permission from Daryl, Vijay and Avichal on Shastri's folder so, I kindly ask you to look in the his directory and let me know what should be kept/moved to common folder based on what I wrote. Thanks!

[AndrewEichmann-NOAA]

The only thing I remember touching was /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew, and that for samples of the DMPDIR for testing preoceanobs

[apchoiCMD]

The only thing I remember touching was /scratch1/NCEPDEV/stmp4/Shastri.Paturi/forAndrew, and that for samples of the DMPDIR for testing preoceanobs

@AndrewEichmann-NOAA I also remember stmp4 also is need to take care of- , all in his folder are duplicated to common

[apchoiCMD]

Final Arrangement

Group Name : stmp1
/scratch2/NCEPDEV/stmp1/Shastri.Paturi/forKatherine => duplicated validation files for ostia2mom6 : RM'ED

Group Name : stmp4
/scratch1/NCEPDEV/stmp4/Shastri.Paturi/ => RM'ED

Group Name : marine
/scratch2/NCEPDEV/marine/Shastri.Paturi/COAPS => NEEDED TO LOOK AT -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/DATM_INPUT => Permission Denied -> gefs data RM'ED
/scratch2/NCEPDEV/marine/Shastri.Paturi/ForJim => Permission Denied -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/GIOMAS => NEEDED TO LOOK AT -> moved to common
/scratch2/NCEPDEV/marine/Shastri.Paturi/GOMb0.08 => Permission Denied -> RM'ED
/scratch2/NCEPDEV/marine/Shastri.Paturi/Latency_table => Latency related but Permission Denied -> RM'ED
/scratch2/NCEPDEV/marine/Shastri.Paturi/NG-GODAS => NEEDED TO LOOK AT -> RM'ED
/scratch2/NCEPDEV/marine/Shastri.Paturi/OISST => Permission Denied -> moved to common
/scratch2/NCEPDEV/marine/Shastri.Paturi/OceanObsQC => NEEDED TO LOOK AT -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/PIOMAS => NEEDED TO LOOK AT -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/QC_task => NEEDED TO LOOK AT -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/WCOSS1 => Permission Denied -> RM'ED
/scratch2/NCEPDEV/marine/Shastri.Paturi/frommars => Permission Denied -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/ice.L2.emc => Some ice data files : NEEDED TO LOOK AT -> moved to common
/scratch2/NCEPDEV/marine/Shastri.Paturi/ice.cryosat_GDR.esa => CS_OFFL_SIR_GDR_2 from 20100715 to 20200829 : NEEDED TO LOOK AT -> RM'ED
/scratch2/NCEPDEV/marine/Shastri.Paturi/icec.nsidc => SeaIce related data files : NEEDED TO LOOK AT -> moved to common
/scratch2/NCEPDEV/marine/Shastri.Paturi/insitu.fnmoc => FNMOC data files : NEEDED TO LOOK AT -> moved to common
/scratch2/NCEPDEV/marine/Shastri.Paturi/ioda_observations => Seems some old files there : NEEDED TO LOOK AT -> moved to common
/scratch2/NCEPDEV/marine/Shastri.Paturi/ncoda_code => Permission Denied -> copied
/scratch2/NCEPDEV/marine/Shastri.Paturi/rtofs_prod => RTOFSDA.zip : NEEDED TO LOOK AT -> moved to common

[apchoiCMD]

Another big folder is /scratch1/NCEPDEV/da/Shastri.Paturi/ -> 3.6 TB

[guillaumevernieres]

I assume this was closed unintentionally? Do we need to keep this open @apchoiCMD ?

[apchoiCMD]

I would like to discuss it during tomorrow's meeting since I have not touched one of Shastri's folder where is at /scratch1/NCEPDEV/da/Shastri.Paturi/ , I am sure that we need to remove it sooner or later-

[apchoiCMD]

Those below are removed

/scratch1/NCEPDEV/da/Shastri.Paturi/
/scratch2/NCEPDEV/marine/Shastri.Paturi/