5 add pgdump as provisioning for backup data (#8)

* a lot of work1

* 5: add backup scripts

* 5: add comments

* 5: fix for long messages

* 5: restart option was buggy, fixxed

* 5: add provisioning mode for manager

* 5: cleaning

* 5: adjust template for env and build in Variable checker

* 5: add additional way for debugging

* 5: create folders

* 5: first step of new images

* 5: first step of new images II

* 5: update images

* 5: add version bump file

* 5: init repo when not existing

* 5: not needed

* 5: cleaning

* 5: fix bug #10

* 5: adjustments

* 5: done

* 5: update

* 5: update

.env

@@ -1,20 +1,44 @@
-### data in which nextcloud is synched and used from restic as backup source
-DATA_TO_BACKUP=/path/to/data/to/backup
+### for run_backup.sh itself not for the image
+# data in which provisioing places the data to backup, needs to be fullpath
+# uncomment the line when you don't want to bind any data to the container (optional)
+# when not explicitly set, defaults to creating a folder near the script and mount it into it
+# options: 'fullpath' or 'none'
+SCRIPT_DATA_TO_BACKUP=/path/to/data/to/backup
 
-### storagebox settings
+# data in which restic places the restore of the backup, needs to be fullpath
+# when not set the run script will create a folder near the script and mount it into it
+SCRIPT_RESTORE_DATA_TO=/path/to/restore/data
+
+# to access log files outside of the container, default not needed (optional)
+SCRIPT_LOG_PATH=/path/to/logfiles
+
+### storagebox settings (needed)
 ENV_TARGET_DOMAIN=name_of_your_storagebox_domain
 ENV_TARGET_DOMAIN_USER=login_name_of_your_storagebox
-ENV_PATH_OF_PRIVATE_KEY=/path/to/your/private/key
+ENV_SSH_PRIVATE_KEY='your private key between single quotes'
 
-### nextcloud
-ENV_NC_URL=url_of_your_nextcloud
-ENV_NC_USER=your_nextcloud_user
-ENV_NC_PASS='password to your nextcloud between single quotes'
-
-### restic
+### restic (needed)
 ENV_RESTIC_REPOSITORY_NAME=name_of_your_repository
 ENV_RESTIC_PASSWORD='password of your restic repository between single quotes'
 
-### telegram
+### telegram (needed)
 ENV_TELEGRAM_TOKEN=your_telegram_token
-ENV_TELEGRAM_CHAT_ID=your_telegram_chat_id
+ENV_TELEGRAM_CHAT_ID=your_telegram_chat_id
+
+### CRON, defaults to 1am (needed)
+ENV_CRON='0 1 * * *'
+
+### provision mode (needed)
+### possible values: "postgres", "nextcloud", "none"
+ENV_PROVISION_MODE="nextcloud"
+
+### nextcloud when used
+ENV_NC_URL=url_of_your_nextcloud
+ENV_NC_USER=your_nextcloud_user
+ENV_NC_PASS='password to your nextcloud between single quotes'
+
+### postgres when used
+ENV_POSTGRES_USER="user of your postgres database"
+ENV_POSTGRES_PASSWORD="password of your postgres database"
+ENV_POSTGRES_DATABASE="name of your postgres database"
+ENV_POSTGRES_HOST="host of your postgres database"

.gitignore

@@ -234,3 +234,4 @@ fabric.properties
 !.idea/runConfigurations
 
 # End of https://www.toptal.com/developers/gitignore/api/intellij+all
+/local.env

README.md

@@ -1,76 +1,108 @@
-[![Pipeline](https://github.com/realAP/backup/actions/workflows/pipeline.yml/badge.svg?branch=main)](https://github.com/realAP/backup/actions/workflows/pipeline.yml)
-# backup
-Use this image to sync your nextcloud to a local directory. This will be mounted in your container and used as input for restic. Restic creates a backup and upload it to your sftp host.
-All of it in a container based manner.
-
-# Overview
-![backup_overview.drawio.svg](backup_overview.drawio.svg)
-
-
-# General
-There are two modes.
-1. Default mode is running the script without any arguments.
-    * e.g. `./run_backup.sh`
-1. Argument mode. Use the script as you would use restic. The script will run the container in which restic is started and places every argument behind it.
-You have access to all the environment variables set in the `.env` file.
-      * `./run_backup.sh snapshots`
-      * `./run_backup.sh init`
-      * and more...
-
-# How to use it
-### Prerequisite
-These things are needed:
-* Nextcloud Server
-* SFTP Server
-* Docker
-* a device where the backup is running
-
-Place your public key at the sftp server and use the private key to log into it.
-> Currently, the private key should not have a password, it is not supported yet.
-> This is not ideal, in further versions private key with passwords are supported and recommended to use.
-
-### Build image
-1. clone the repository `https://github.com/realAP/backup.git`
-1. `cd backup/docker` 
-1. build the docker image `docker build . -t restic`
-
-### Set .env file
-Fill all variables in the `.env` file, it is provided with example values.
-
-### Create Repository
-In first place a repository has to be created on the remote (sftp).
-
-1. `./run_backup.sh init` this will initialize a repository
-1. exit the container 
-
-### Run Backup
-Just run the script `./run_backup.sh` it will immediately sync the nextcloud and creates a backup, the status report will be sent via telegram.
-This repeats every day at 1am (default), feel free to adjust the `cron` variable in the run_backup.sh
-
-# Restore data
-Just work as you would do it with restic.
-When cloned the repo there is an empty folder called `restore` which is mounted into the container under /restore.
-This folder can be used as target for your restored data.
-Example: to get the latest snapshot from your data
-`./run_backup.sh restore latest --target /restore`
-
----
-# Debugging
-The `run_backup.sh` has a flag 
-* `"DEBUG"=0` which disables
-* `"DEBUG"=1` which enables debugging
-  * the container will start and opens a bash terminal
-
-# Logging
-The `log` folder is mounted to `/var/log` which enables to have access to different kinds of log files.
-
-# ToDo
-TBD
-
-# Why | Motivation
-I have used https://duplicati.com/ which i can recommend. 
-My problem is duplicati is not supported for my hardware anymore.
-This is the reason for this project.
-
-I have nextcloud hosted by hetzner (https://www.hetzner.com/storage/storage-share/) and storagebox (https://www.hetzner.com/storage/storage-box/) a sftp hosted platform.
-The backup should work without hetzner it just needs a nextcloud server and sftp access.
+[![Pipeline](https://github.com/realAP/backup/actions/workflows/pipeline.yml/badge.svg?branch=main)](https://github.com/realAP/backup/actions/workflows/pipeline.yml)
+[![Docker Image Version](https://img.shields.io/docker/v/devp1337/backup?sort=semver)](https://hub.docker.com/r/devp1337/backup)
+# backup
+Use this image to load your data from a **provider** into your **binded** folder. This folder will be used as input for restic. Restic creates a backup and upload it to your sftp host.
+You can configure a cron job to run the backup every day at a specific time and telegram will be used to send status reports.
+All of it in a container based manner.
+
+## Overview
+![backup_overview.drawio.svg](resources/backup_overview.svg)
+
+## Prerequisite
+These things are needed:
+* Provider (Nextcloud, PostgreSQL, None)
+* SFTP Server to store (e.g. Hetzner Storagebox)
+* Docker
+* a device where backup is running
+
+## Configuration
+**1. Bind**  
+Restic looks for a folder `/source` to back up data. The behavior of this folder depends on the mode you select at the beginning of the script:
+
+- **Bind Mounting**:
+    - You can mount a folder from the host system into `/source`.
+    - This approach is recommended because the data is persistent even if the container is deleted.
+
+- **Internal Folder**:
+    - If no folder is mounted, the `/source` directory exists within the container.
+    - **Warning**: By default the container deletes itself after it stops. Any data stored in the container's `/source` directory will be lost.
+---
+**2. Provider**  
+The provider mode dynamically writes data into the `/source` directory before restic creates a backup. This mode is useful when the data is generated by another service or application.
+
+Providers currently supported:
+- **Nextcloud**: Synchronizes files into `/source`.
+- **PostgreSQL**: Dumps database content into `/source`.
+- **None**: No additional data is written to `/source`.
+
+---
+**3. Environment Variables**
+
+There is a `.env` file where you can set all needed variables.
+It describes every variable and provides an example value.
+
+
+## How to backup
+### 1. Place your public key at the sftp server and use the private key to log into it.
+When used with Hetzner Storagebox, follow this [guide](https://docs.hetzner.com/storage/storage-box/backup-space-ssh-keys)
+> Currently, the private key should not have a password, it is not supported yet.
+
+### 2. Download the script and .env file
+* `curl -O https://raw.githubusercontent.com/realAP/backup/main/run_backup.sh`
+* `curl -0 https://raw.githubusercontent.com/realAP/backup/main/.env`
+* make the script executable `chmod +x run_backup.sh`
+
+### 3. Set .env file
+Fill all needed variables in the `.env` file, it is provided with example values.
+
+### 4. Run Backup
+Just run the script `./run_backup.sh` the backup will immediately start and repeat every day at 1am (default).
+For the first run, the script will initialize the restic repository.
+
+## Restore data
+Just work as you would do it with restic.
+
+Example: to get the latest snapshot from your data
+* `./run_backup.sh restore latest --target /restore`
+  * this will restore the latest snapshot to the `/restore` folder in the container which is binded to the host
+> as target use `/restore` to restore the data to the binded folder. For more information read the .env file example
+
+
+## Operations
+You can use the script in two ways:
+1. Default is running the script without any arguments. As shown in the example above.
+    * e.g. `./run_backup.sh`
+1. With arguments. Use the script as you would use restic. The script will run the container in which restic is started and places every argument behind it.
+   You have access to all the environment variables set in the `.env` file. Remember `/restore` is **always** binded to the host. And `/source` folder is binded to the host when not set to `none`.
+    * `./run_backup.sh snapshots`
+    * `./run_backup.sh init`
+    * and more...
+    >  Attention: the script will run the container which will be deleted after the command is executed. The data in the container is lost.
+
+---
+# Debugging
+When you want to debug the container you can run the container in interactive mode.
+Just run the script as follows `./run_backup.sh DEBUG` and the container will start in interactive mode.
+This is just used in the development phase.
+
+# Logging
+You can bind the `/var/log` folder to your host to get the logs. For this set the `SCRIPT_LOG_PATH` variable in the `.env` file.
+When the variable is not set then log files will be stored in the container.
+This is just for debugging purposes.
+
+# Why | Motivation
+I have used https://duplicati.com/ which i can recommend. 
+My problem is duplicati is not supported for my hardware anymore.
+This is the reason for this project.
+
+I have nextcloud hosted by hetzner (https://www.hetzner.com/storage/storage-share/) and storagebox (https://www.hetzner.com/storage/storage-box/) a sftp hosted platform.
+The backup should work without hetzner it just needs a nextcloud server and sftp access.
+
+## Example for Provider Nextcloud
+![backup_overview_nextcloud.svg](resources/backup_overview_nextcloud.svg)
+* TBD
+
+## Example for Provider PostgreSQL
+![backup_overview_postgres.svg](resources/backup_overview_postgres.svg)
+* TBD
+

VERSION

@@ -1 +1 @@
-1.0.0
+1.1.0

docker/Dockerfile

@@ -1,19 +1,26 @@
 FROM ubuntu:24.04
 
-RUN apt update && apt install cron restic nextcloud-desktop-cmd ssh curl -y
+RUN apt update && apt install cron restic nextcloud-desktop-cmd ssh curl postgresql-client -y
 
 # user permissions?
 ADD backup.sh /bin
 ADD entry.sh /bin
 ADD manager.sh /bin
 ADD nextcloud.sh /bin
+ADD postgres_backup.sh /bin
+ADD postgres_restore.sh /bin
+ADD prepare.sh /bin
+ADD prepare_ssh.sh /bin
 ADD telegram.sh /bin
-ADD create_ssh_config.sh /bin
+
 RUN chmod 0744 /bin/backup.sh
 RUN chmod 0744 /bin/entry.sh
 RUN chmod 0744 /bin/manager.sh
 RUN chmod 0744 /bin/nextcloud.sh
+RUN chmod 0744 /bin/postgres_backup.sh
+RUN chmod 0744 /bin/postgres_restore.sh
+RUN chmod 0744 /bin/prepare.sh
+RUN chmod 0744 /bin/prepare_ssh.sh
 RUN chmod 0744 /bin/telegram.sh
-RUN chmod 0744 /bin/create_ssh_config.sh
 
 ENTRYPOINT ["entry.sh"]

docker/backup.sh

@@ -1,27 +1,27 @@
-#!/bin/bash -l
-set -e
-echo "=========LET'S DO THIS, BACKUP THE SHIT!!!========="
-date
-hasRepo="restic cat config"
-$hasRepo || exit 1
-
-echo "+------------------------------------------+"
-echo "|::::::::::::::::call backup:::::::::::::::|"
-echo "+------------------------------------------+"
-restic backup /source
-
-echo "+------------------------------------------+"
-echo "|::::::::::::::::call check::::::::::::::::|"
-echo "+------------------------------------------+"
-restic check
-
-echo "+------------------------------------------+"
-echo "|::::::::::::::::call forget:::::::::::::::|"
-echo "+------------------------------------------+"
-restic forget --keep-weekly 52 --keep-monthly 12 --keep-yearly 100
-
-echo "+------------------------------------------+"
-echo "|::::::::::::::::call check::::::::::::::::|"
-echo "+------------------------------------------+"
-restic check
-echo "==================DONE====================="
+#!/usr/bin/env bash
+set -e
+echo "=========LET'S DO THIS, BACKUP THE SHIT!!!========="
+date
+hasRepo="restic cat config"
+$hasRepo || restic init || exit 1
+
+echo "+------------------------------------------+"
+echo "|::::::::::::::::call backup:::::::::::::::|"
+echo "+------------------------------------------+"
+restic backup /source
+
+echo "+------------------------------------------+"
+echo "|::::::::::::::::call check::::::::::::::::|"
+echo "+------------------------------------------+"
+restic check
+
+echo "+------------------------------------------+"
+echo "|::::::::::::::::call forget:::::::::::::::|"
+echo "+------------------------------------------+"
+restic forget --keep-weekly 52 --keep-monthly 12 --keep-yearly 100
+
+echo "+------------------------------------------+"
+echo "|::::::::::::::::call check::::::::::::::::|"
+echo "+------------------------------------------+"
+restic check
+echo "==================DONE====================="