Setup on the alternative bastion (bastion-alt.gigadb.host) a daily rclone sync job for keeping user dropboxes in sync between live and alternative

[rija]

## User Story

As a curator
I want to ensure the user dropboxes area is backed up
So that I have piece of mind when I work on files on the dropboxes

Acceptance criteria

Given there is a dataset being curated
When there are new files added
then the backup should have the new files

Given there is a dataset being curated
When there are new files modified
then the backup should have the updated files

Given there is a dataset being curated
When files are deleted from the user dropbox by the curators
then the backup should also delete the files

Additional infos

• sync files from the live infrastructure into the alternative infrastructure.
• the process runs on the alternative infrastructure
• runs daily
• rclone sync does the sync
• bats test to test the correctness of the configuration

Tasks

• Create a shell script wrapping rclone sync #2004
• Generate rclone configuration for source of rclone sync #2005
• Update bastion playbook to add a new cronjob to execute the shell script and to create the rclone configuration #2006
• Write bats tests for correctness of rclone configuration and shell scripts #2007

[rija]

Hi @pli888, @kencho51,

the command I used to do the manual sync:

$ rclone --config rclone_upstream.conf -v  sync live-bastion:/share/dropbox/ /share/dropbox/

and the config used for it, rclone_upstream.conf looks like this:

[live-bastion]
type = sftp
host = bastion.gigadb.host
user = centos
key_file = /home/centos/.ssh/upstream_from_alt-live
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

The public part of that ssh key was manually added to the current production bastion's authorized_key file.

I'll update the various tasks with more details based on the above.