Skip to content

Commit

Permalink
Removed uneeded man dump for rclone and rsync. Added prequisites. (#364)
Browse files Browse the repository at this point in the history
  • Loading branch information
@CallumWalley
CallumWalley authored Oct 11, 2024
2 parents 48ab927 + 6dff651 commit 62f3f5c
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 56 deletions.
44 changes: 19 additions & 25 deletions docs/Getting_Started/Next_Steps/Moving_files_to_and_from_the_cluster.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The Download button is at the bottom.
## Standard Terminal

!!! prerequisite
Have SSH [configured as recommended](../../Scientific_Computing/Terminal_Setup/Standard_Terminal_Setup.md)
Have SSH setup as described in [Standard Terminal Setup](../../Scientific_Computing/Terminal_Setup/Standard_Terminal_Setup.md)

In a local terminal the following commands can be used to:

Expand Down Expand Up @@ -77,27 +77,32 @@ your password.

## File Managers

!!! prerequisite
Have SSH setup as described in [Standard Terminal Setup](../../Scientific_Computing/Terminal_Setup/Standard_Terminal_Setup.md)

Most file managers can be used to connect to a remote directory simply
by typing in the address bar (provided your have an active connection to
the cluster and your ssh config file is set up as described in
[Standard_Terminal_Setup](../../Scientific_Computing/Terminal_Setup/Standard_Terminal_Setup.md)).
by typing in the address bar provided your have an active connection to
the cluster.

For Nautilus (Ubuntu default) just prepend the path you want to connect
to with `sftp://mahuika`. (ctrl + L opens address bar)

This does not work for File Explorer (Windows default)
This **does not** work for File Explorer (Windows default)

This does not work for Finder (Mac default)
This **does not** work for Finder (Mac default)

![files](../../assets/images/Moving_files_to_and_from_the_cluster_1.png)

If your default file manager does not support mounting over sftp, see
If your default file manager does not support mounting over SFTP, see
our documentation
on [SSHFS](../../General/FAQs/Can_I_use_SSHFS_to_mount_the_cluster_filesystem_on_my_local_machine.md).

## MobaXterm

Clicking the "*Scp*" tab (located on the left-hand side of the MobaXTerm
!!! prerequisite
[MobaXterm Setup Windows](../../Scientific_Computing/Terminal_Setup/MobaXterm_Setup_Windows.md)

Clicking the "*Scp*" tab (located on the left-hand side of the
window) opens up a graphical user interface that can be used for basic
file operations. You can drag and drop files in the file explorer or use
the up and down arrows on the toolbar to upload and download files.
Expand All @@ -111,6 +116,9 @@ is enabled).

## WinSCP

!!! prerequisite
[WinSCP-PuTTY Setup Windows](../../Scientific_Computing/Terminal_Setup/WinSCP-PuTTY_Setup_Windows.md)

As WinSCP uses multiple tunnels for file transfer you will be required
to authenticate again on your first file operation of the session. The
second prompt for your 2FA can be skipped, just the same as with login
Expand All @@ -120,8 +128,8 @@ authentication.

Globus is available for those with large amounts of data, security
concerns, or connection consistency issues.
You can find more details on its use on our [Globus support
page](../../Storage/Data_Transfer_Services/Data_Transfer_using_Globus_V5.md).
You can find more details on its use on our
[Globus support page](../../Storage/Data_Transfer_Services/Data_Transfer_using_Globus_V5.md).

## Rclone

Expand All @@ -138,19 +146,8 @@ The most frequently used Rclone subcommands:

- `rclone copy` - Copy files from the source to the destination, skipping what has already been copied.
- `rclone sync` - Make the source and destination identical, modifying only the destination.
- `rclone mov`e - Move files from the source to the destination.
- `rclone move` - Move files from the source to the destination.
- `rclone delete` - Remove the contents of a path.
- `rclone mkdir` - Create the path if it does not already exist.
- `rclone rmdir` - Remove the path.
- `rclone check` - Check if the files in the source and destination match.
- `rclone ls` - List all objects in the path, including size and path.
- `rclone lsd` - List all directories/containers/buckets in the path.
- `rclone lsl` - List all objects in the path, including size, modification time and path.
- `rclone lsf` - List the objects using the virtual directory structure based on the object names.
- `rclone cat` - Concatenate files and send them to stdout.
- `rclone copyto` - Copy files from the source to the destination, skipping what has already been copied.
- `rclone moveto` - Move the file or directory from the source to the destination.
- `rclone copyurl` - Copy the URL's content to the destination without saving it in the tmp storage.

A more extensive list can be found in the [Rclone documentation](https://rclone.org/docs).

Expand All @@ -175,11 +172,8 @@ The most frequently used Rsync options:

- `-r` - Recurse into directories.
- `-a` - Use archive mode: copy files and directories recursively and preserve access permissions and time stamps.
- `-v` - Verbose mode.
- `-z` - Compress.
- `-e ssh` - Specify the remote shell to use.
- `-n` - Show what files would be transferred.
- `--partial` - Keep partially transferred files.
- `--progress` - Show progress during transfer.

A more extensive list can be found in the [Rsync documentation](https://download.samba.org/pub/rsync/rsync.1).
39 changes: 8 additions & 31 deletions docs/Scientific_Computing/Terminal_Setup/Standard_Terminal_Setup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,28 @@ tags:
- howto
vote_count: 8
vote_sum: 6
zendesk_article_id: 360000625535
zendesk_section_id: 360000189696
description: How to setup your ssh config file in order to connect to the NeSI cluster.
---

!!! prerequisite
- Have an [active account and project.](../../Getting_Started/Accounts-Projects_and_Allocations/Creating_a_NeSI_Account_Profile.md)
- Set up your [Linux Password.](../../Getting_Started/Accessing_the_HPCs/Setting_Up_and_Resetting_Your_Password.md)
- Set up Second [Factor Authentication.](../../Getting_Started/Accessing_the_HPCs/Setting_Up_Two_Factor_Authentication.md)
- Using standard Linux/Mac terminal *or* [Windows Subsystem for Linux](../../Scientific_Computing/Terminal_Setup/Windows_Subsystem_for_Linux_WSL.md)
with [Ubuntu terminal](../../Scientific_Computing/Terminal_Setup/Ubuntu_LTS_terminal_Windows.md).
- Set up [Second Factor Authentication.](../../Getting_Started/Accessing_the_HPCs/Setting_Up_Two_Factor_Authentication.md)
- Have one of:
- Built in Linux/Mac terminal
- [Windows Subsystem for Linux](../../Scientific_Computing/Terminal_Setup/Windows_Subsystem_for_Linux_WSL.md) with [Ubuntu terminal](../../Scientific_Computing/Terminal_Setup/Ubuntu_LTS_terminal_Windows.md).
- [VSCode](../../Scientific_Computing/Terminal_Setup/VSCode.md)

## First time setup

The login process can be simplified significantly with a few easy
configurations.

1. In a new local terminal run; `mkdir -p ~/.ssh/sockets` this will
create a hidden file in your home directory to store socket
create a subdirectory in your home directory to store socket
configurations.

2. Open your ssh config file with `nano ~/.ssh/config` and add the
2. Open your ssh config file (e.g. `nano ~/.ssh/config` to open with the text editor `nano`) and add the
following (replacing **`username`** with your username):

```sh
Expand All @@ -38,15 +39,6 @@ configurations.
ServerAliveInterval 300
ServerAliveCountMax 2

Host maui
User username
Hostname login.maui.nesi.org.nz
ProxyCommand ssh -W %h:%p lander
ForwardX11 yes
ForwardX11Trusted yes
ServerAliveInterval 300
ServerAliveCountMax 2

Host lander
User username
HostName lander.nesi.org.nz
Expand Down Expand Up @@ -75,21 +67,6 @@ to the clusters directly using;
ssh mahuika
```

or

```sh
ssh maui
```

Subsequent local terminals opened will be able to `scp` files without
having to re-enter authentication e.g.

```sh
scp <path/filename> mahuika:~/
```

For more info visit [data transfer](../../Getting_Started/Next_Steps/Moving_files_to_and_from_the_cluster.md).

!!! prerequisite "What Next?"
- [Moving files to/from a cluster.](../../Getting_Started/Next_Steps/Moving_files_to_and_from_the_cluster.md)
- Setting up an [X-Server](../../Scientific_Computing/Terminal_Setup/X11_on_NeSI.md) (optional).

0 comments on commit 62f3f5c

Please sign in to comment.