Skip to content

Commit

Permalink
2 pages updated (#50)
Browse files Browse the repository at this point in the history
* File recovery updates

* Data Transfer using Globus updates

---------

Co-authored-by: Kahu Anderson <[email protected]>
Co-authored-by: Cal <[email protected]>
  • Loading branch information
3 people authored Dec 5, 2023
1 parent b0c377e commit b400124
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 70 deletions.
20 changes: 5 additions & 15 deletions docs/Storage/Data_Recovery/File_Recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,6 @@ zendesk_article_id: 360000207315
zendesk_section_id: 360000042215
---



[//]: <> (REMOVE ME IF PAGE VALIDATED)
[//]: <> (vvvvvvvvvvvvvvvvvvvv)
!!! warning
This page has been automatically migrated and may contain formatting errors.
[//]: <> (^^^^^^^^^^^^^^^^^^^^)
[//]: <> (REMOVE ME IF PAGE VALIDATED)

## Snapshots

Snapshots are read only copies of the file system taken every day at
Expand All @@ -37,17 +28,16 @@ project code, e.g;
``` sl
/home/username/.snapshots/Sunday/
```
!!! prerequisite Warning
Files in `/nesi/nobackup/` are not snapshotted.


!!! Warning
Files in `/nesi/nobackup/` are not snapshotted.

Recovering a file or a directory from the snapshot is as simple as
copying it over, e.g.

``` sl
cp /nesi/project/.snapshots/Sunday/nesi99999/file.txt /nesi/project/nesi99999/file.txt
```
!!! prerequisite Tip
For copying directories use the flag -ir or just -r if you don't want
to be prompted before overwriting.

!!! Tip
For copying directories use the flag -ir or just -r if you don't want to be prompted before overwriting
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,6 @@ zendesk_article_id: 4405623380751
zendesk_section_id: 360000040596
---



[//]: <> (REMOVE ME IF PAGE VALIDATED)
[//]: <> (vvvvvvvvvvvvvvvvvvvv)
!!! warning
This page has been automatically migrated and may contain formatting errors.
[//]: <> (^^^^^^^^^^^^^^^^^^^^)
[//]: <> (REMOVE ME IF PAGE VALIDATED)

## Globus

Globus is a third-party service for transferring large amounts of data
Expand All @@ -28,26 +19,24 @@ endpoint, or an endpoint from your institution. With Globus, very high
data transfer rates are achievable. This service allows data to be
accessible to any person who has a Globus account. The newest
implementation (v5) provides extra features and some key differences
from the previous setup that you can find
[here](https://docs.globus.org/globus-connect-server/)
from the previous setup that you can find [here](https://docs.globus.org/globus-connect-server/).

To use Globus on NeSI platforms, you need:

1. A Globus account (see [Initial Globus Sign-Up and Globus
ID](../../Storage/Data_Transfer_Services/Initial_Globus_Sign_Up-and_your_Globus_Identities.md))
2. An active NeSI account (see [Creating a NeSI
Account](../../Getting_Started/Accounts-Projects_and_Allocations/Creating_a_NeSI_Account_Profile.md))
3. Access privileges on the non-NeSI Globus endpoint/collection you
plan on transferring data from or to. This other endpoint/collection
could be a personal one on your workstation, or it could be managed
by your institution or a third party.
1. A Globus account (see [Initial Globus Sign-Up and Globus
ID](../../Storage/Data_Transfer_Services/Initial_Globus_Sign_Up-and_your_Globus_Identities.md))
2. An active NeSI account (see [Creating a NeSI
Account](../../Getting_Started/Accounts-Projects_and_Allocations/Creating_a_NeSI_Account_Profile.md))
3. Access privileges on the non-NeSI Globus endpoint/collection you
plan on transferring data from or to. This other endpoint/collection
could be a personal one on your workstation, or it could be managed
by your institution or a third party.

- *Note that a NeSI user account does not create a Globus account, and
similarly a Globus account does not create a NeSI user account. Nor
can you, as the end user, link the two through any website.*
- *Note that a NeSI user account does not create a Globus account, and
similarly a Globus account does not create a NeSI user account. Nor
can you, as the end user, link the two through any website.*

Both your accounts (NeSI and Globus) must exist before you try to use
our DTN.
Both your accounts (NeSI and Globus) must exist before you try to use our DTN.

The NeSI Wellington DTN endpoint is protected by a second factor
authentication (2FA).  Also note, your NeSI username and password are
Expand Down Expand Up @@ -102,23 +91,23 @@ As an example, to move files between the NeSI HPC Storage (accessible
from Māui and Mahuika) and the Otago University high-capacity central
file storage (another managed server endpoint):

!!! info
Log in to the [NeSI File Manager](https://transfer.nesi.org.nz/file-manager) where you are able to search for DTNs in the Collection field.
[Here](https://support.nesi.org.nz/hc/en-gb/articles/360000931775) is a listing of available endpoints on the New Zealand Data Transfer Platform.

Find the NeSI endpoint by typing in "NeSI Wellington DTN V5". Select the
endpoint "NeSI Wellington DTN V5" from the list, and you will be asked
to authenticate your access to the endpoint. Click Continue to the next
step.

![mceclip0.png](../../assets/images/Data_Transfer_using_Globus_V5.png)



You can choose either of **&lt;username&gt;@wlg-dtn-oidc.nesi.org.nz**
or NeSI Wellington OIDC Server (wlg-dtn-oidc.nesi.org.nz), they are all
linked to the same website. If this is your first time login, you may
ask to *bind* your primary identity to the OIDC login, you need to allow
that.



![mceclip1.png](../../assets/images/Data_Transfer_using_Globus_V6.png)

The NeSI Wellington DTN V5 endpoint is protected by a second factor
Expand All @@ -130,7 +119,6 @@ authentication (2FA-same as accessing NeSI clusters).  In the
not*** use any additional characters or spaces between your password and
the token number.)


![mceclip0.png](../../assets/images/Data_Transfer_using_Globus_V7.png)

After the login, you will navigate to the default root(display as "/")
Expand All @@ -139,7 +127,7 @@ path, then you could change the path to
\(1\) your ***/home/&lt;username&gt;*** directory,

\(2\) project directory (read-only)
***/nesi/project/&lt;project\_code&gt;*** 
***/nesi/project/&lt;project\_code&gt;***

\(3\) project sub-directories of
***/nesi/nobackup/&lt;project\_code&gt;***  - see [Globus Paths,
Expand All @@ -164,37 +152,35 @@ button:

![mceclip5.png](../../assets/images/Data_Transfer_using_Globus_V10.png)



To find other NeSI endpoints, type in "nesi#":

![filemanage\_nesi.png](../../assets/images/Data_Transfer_using_Globus_V11.png)

## In brief:

- Sign in to the NeSI Globus Web App <https://transfer.nesi.org.nz/>.
You will be taken to the *File Manager* page
<https://transfer.nesi.org.nz/file-manager>
- If this is your first time, you will need to create a Globus
account.
- Open the two-endpoint panel
![two\_endpoint.png](../../assets/images/Data_Transfer_using_Globus_V12.png)located
on the top-right of the *File Manager* page.
- Select the Endpoints you wish to move files between (start typing
"nesi#" to see the list of NeSI DTNs to select from).
[Authenticate](../../Storage/Data_Transfer_Services/Globus_V5_endpoint_activation.md)
at both endpoints.
- At Globus.org the** **endpoint **defaults to
"/home/&lt;username&gt;" path** (represented by "/~/") on Mahuika or
Māui. We do not recommend uploading data to your home directory, as
home directories are very small. Instead, navigate to an appropriate
project directory under /nobackup (see [Globus Paths, Permissions,
Storage
Allocation](../../Storage/Data_Transfer_Services/Globus_V5_Paths-Permissions-Storage_Allocation.md)).
- Transfer the files by clicking the appropriate
![start.png](../../assets/images/Data_Transfer_using_Globus_V13.png)button
depending on the direction of the transfer.
- Check your email for confirmation about the job completion report.
- Sign in to the NeSI Globus Web App <https://transfer.nesi.org.nz/>.
You will be taken to the *File Manager* page
<https://transfer.nesi.org.nz/file-manager>
- If this is your first time, you will need to create a Globus
account.
- Open the two-endpoint panel
![two\_endpoint.png](../../assets/images/Data_Transfer_using_Globus_V12.png){: style="height:2em;"} located
on the top-right of the *File Manager* page.
- Select the Endpoints you wish to move files between (start typing
"nesi#" to see the list of NeSI DTNs to select from).
[Authenticate](../../Storage/Data_Transfer_Services/Globus_V5_endpoint_activation.md)
at both endpoints.
- At Globus.org the endpoint **defaults to
"/home/&lt;username&gt;" path** (represented by "/~/") on Mahuika or
Māui. We do not recommend uploading data to your home directory, as
home directories are very small. Instead, navigate to an appropriate
project directory under /nobackup (see [Globus Paths, Permissions,
Storage
Allocation](../../Storage/Data_Transfer_Services/Globus_V5_Paths-Permissions-Storage_Allocation.md)).
- Transfer the files by clicking the appropriate
![start.png](../../assets/images/Data_Transfer_using_Globus_V13.png){: style="height:1em;"} button
depending on the direction of the transfer.
- Check your email for confirmation about the job completion report.

## Transferring data using a personal endpoint

Expand Down

0 comments on commit b400124

Please sign in to comment.