Check quickly that we have Fedora copr-backend backup

[praiskup]

The backups should be on storinator box.

[praiskup]

We need a howto document (output from this ticket).

[praiskup]

# for i in $(ls -1 /var/log/cron-*.xz | tac); do xzcat $i | grep rsnapshot; done
Sep 17 21:06:26 copr-be CROND[1470129]: (copr) CMDOUT (rsnapshot encountered an error! The program was invoked with these options:)
Sep 17 21:06:26 copr-be CROND[1470129]: (copr) CMDOUT (/bin/rsnapshot -c /srv/nfs/copr-be/copr-be-copr-user/rsnapshot.conf push )
Sep 17 21:06:26 copr-be CROND[1470129]: (copr) CMDOUT (ERROR: Could not write lockfile /srv/nfs/copr-be/copr-be-copr-user/rsnapshot.pid: No space left on device)
Sep 17 21:06:27 copr-be CROND[1470129]: (copr) CMDOUT (  File "/srv/nfs/copr-be/copr-be-copr-user/rsnapshot", line 58, in <module>)
Sep 17 21:06:27 copr-be CROND[1470129]: (copr) CMDOUT (  File "/srv/nfs/copr-be/copr-be-copr-user/rsnapshot", line 51, in _main)
Sep 17 21:06:27 copr-be CROND[1470129]: (copr) CMDOUT (  File "/srv/nfs/copr-be/copr-be-copr-user/rsnapshot", line 42, in rotate)
Sep 17 21:06:27 copr-be CROND[1470129]: (copr) CMDOUT (subprocess.CalledProcessError: Command '['/bin/rsnapshot', '-c', '/srv/nfs/copr-be/copr-be-copr-user/rsnapshot.conf', 'push']' returned non-zero exit status 1.)
Sep 17 21:06:28 copr-be CROND[1470129]: (copr) CMDEND (ionice --class=idle /usr/local/bin/rsnapshot_copr_backend >/dev/null)
Sep 14 01:01:02 copr-be CROND[1470229]: (copr) CMD (ionice --class=idle /usr/local/bin/rsnapshot_copr_backend >/dev/null)

[praiskup]

$ lvresize /dev/VG_nfs/copr-be -L +8TB
$ xfs_growfs /srv/nfs/copr-be/
$ df -h /srv/nfs/copr-be/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VG_nfs-copr--be 48T 40T 8.1T 84% /srv/nfs/copr-be

[praiskup]

Running ionice --class=idle /usr/local/bin/rsnapshot_copr_backend manually.

[praiskup]

Still doing the rsync :-( and we seem to run out of space again:
/dev/mapper/VG_nfs-copr--be 48T 47T 1.7T 97% /srv/nfs/copr-be

[praiskup]

I would remove the old increments, but that would probably break the current rsnapshot process. I'll keep the sync going for now, and wait for the potential failure (if it really fails, I'll remove old increments, and then restart rsnapshot).