New file transfer guide #824
Conversation
Added TOC, tarballs, minor formatting
Small file transfer and SQUID are archived
|
Confirmed that |
Co-authored-by: Christina K. <[email protected]>
Co-authored-by: Christina K. <[email protected]>
Co-authored-by: Christina K. <[email protected]>
Co-authored-by: Christina K. <[email protected]>
| The default quota allotment in your `/home` directory is 20 GB with a hard | ||
| limit of 30 GB (at which point you cannot write more files). |
There was a problem hiding this comment.
I think this got upped when we were upgrading the OS last spring.
| The default quota allotment in your `/home` directory is 20 GB with a hard | |
| limit of 30 GB (at which point you cannot write more files). | |
| The default quota allotment in your `/home` directory is 40 GB with a hard | |
| limit of 50 GB (at which point you cannot write more files). |
| be allowed to save data. Each value is given in 1-kilobyte | ||
| blocks, so you can divide each number by 1024 to get megabytes (MB), and | ||
| again for gigabytes (GB). (It also lists information for ` files`, but | ||
| we don\'t typically allocate disk space by file count.) | ||
| again for gigabytes (GB). (It also lists information for number of `files`, but |
There was a problem hiding this comment.
The -s option for quotas should return mebibytes, not kibibytes.
Also, bytes are multiples of 1000 while bibytes are multiples of 1024 (bi is used to denote that the unit is a power of 2; 1024 = 2^10).
We use bibytes for setting quotas.
In general, I think that we use bytes when communicating with users, but if math is involved for getting their quota values, will need to use bibytes instead..
|
|
||
| To transfer a file or folder back to `/staging`, you will need an additional line in your HTCondor submit file: | ||
| ``` | ||
| transfer_output_remaps = "file1.txt = file:///staging/NetID/output1.txt; file2.txt = /home/NetId/outputs/output2.txt" |
There was a problem hiding this comment.
Suggest to omit file going to home, for clarity.
| ``` | ||
| {:.sub} | ||
|
|
||
| In this example above, `file1.txt` is remapped to the staging directory using the `file:///` transfer protocol and simultaneously renamed `output1.txt`. In addition, `file2.txt` is renamed to `output2.txt`and will be transferred to a different directory on `/home`. Ensure you have the right file transfer syntax (`osdf:///` or `file:///` depending on the anticipated file size). |
There was a problem hiding this comment.
In addition,
file2.txtis renamed tooutput2.txtand will be transferred to a different directory on/home. Ensure you have the right file transfer syntax (osdf:///orfile:///depending on the anticipated file size).
A version of this should be moved to after the next example.
| | Input Sizes | File Location | Submit File Syntax to Transfer to Jobs | | ||
| | ----------- | ----------- | ----------- | ----------- | | ||
| | 0 - 100 MB | `/home` | `transfer_input_files = input.txt` | | ||
| | 100 MB - 30 GB | `/staging` | `transfer_input_files = osdf:///chtc/staging/NetID/input.txt` | | ||
| | > 30 GB | `/staging` | `transfer_input_files = file:///staging/NetID/input.txt` | | ||
| | > 100 GB | | For larger datasets (100GB+ per job), contact the facilitation team about the best strategy to stage your data | |
There was a problem hiding this comment.
May want to include /projects in here as well
| {:.sub} | ||
|
|
||
| Ensure you are using the correct file transfer protocol for efficiency. Failure to use the right protocol can result in slow file transfers or overloading the system. | ||
|
|
There was a problem hiding this comment.
Suggest adding a couple subsections here to expound upon the syntax, i.e., ### file:/// Syntax and ### osdf:/// Syntax.
Major changes
osdf://andfile:///syntax, including mentions of caching (https://chtc.github.io/web-preview/preview-xalim-osdf-guide/uw-research-computing/htc-job-file-transfer)osdf://orfile:///. Removed references to copying to and from/stagingin the executable script. I can put it back if we want to keep it.file_availlayout (it puts a table on top of these data guides) to reflect our current file systems and the new file transfer syntax.I'm planning to test
osdf://file transfer, so don't merge this yet until it's confirmed to work.