Downloading a backblaze snapshot yields ciphertext verification fails

[benjamin051000]

Here's my setup:

version: "3.3"

services:
  backup:
    image: mazzolino/restic
    hostname: docker
    restart: unless-stopped
    environment:
      RUN_ON_STARTUP: "true"
      BACKUP_CRON: "0 0 3 * * *"
      RESTIC_REPOSITORY: b2:rpi-homelab-backup:/restic-repo
      RESTIC_PASSWORD: [redacted]
      RESTIC_BACKUP_SOURCES: /mnt/volumes
      RESTIC_BACKUP_ARGS: --tag docker-volumes
      RESTIC_FORGET_ARGS: --keep-daily 7 --keep-weekly 4
      B2_ACCOUNT_ID: [redacted]
      B2_ACCOUNT_KEY: [redacted]
    volumes:
      - /var/lib/docker/volumes:/mnt/volumes:ro

  check:
    image: mazzolino/restic
    hostname: docker
    restart: unless-stopped
    environment:
      RUN_ON_STARTUP: "false"
      CHECK_CRON: "0 15 5 * * *"
      RESTIC_CHECK_ARGS: >-
        --read-data-subset=10%
      RESTIC_REPOSITORY: b2:rpi-homelab-backup:/restic-repo
      RESTIC_PASSWORD: [redacted]
      B2_ACCOUNT_ID: [redacted]
      B2_ACCOUNT_KEY: [redacted]

Backup task logs:

Checking configured repository 'b2:rpi-homelab-backup:/restic-repo' ...
Fatal: unable to open config file: Stat: b2_download_file_by_name: 404: 
Is there a repository at the following location?
b2:rpi-homelab-backup:/restic-repo
Could not access the configured repository. Trying to initialize (in case it has not been initialized yet) ...
created restic repository d724a9802c at b2:rpi-homelab-backup:/restic-repo
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
Repository successfully initialized.
Executing backup on startup ...
Starting Backup at 2023-07-22 02:40:41
no parent snapshot found, will read all files
Files:       43857 new,     0 changed,     0 unmodified
Dirs:        13599 new,     0 changed,     0 unmodified
Added to the repository: 17.466 GiB (16.668 GiB stored)
processed 43857 files, 29.772 GiB in 30:39
snapshot f7487f77 saved
Backup successful
Forget about old snapshots based on RESTIC_FORGET_ARGS = --keep-daily 7 --keep-weekly 4
Applying Policy: keep 7 daily, 4 weekly snapshots
keep 1 snapshots:
ID        Time                 Host        Tags            Reasons          Paths
----------------------------------------------------------------------------------------
f7487f77  2023-07-22 02:40:41  docker      docker-volumes  daily snapshot   /mnt/volumes
                                                           weekly snapshot
----------------------------------------------------------------------------------------
1 snapshots
Finished backup at 2023-07-22 03:11:27 after 1846 seconds
Scheduling backup job according to cron expression.
new cron: 0 0 3 * * *

Check logs:

Checking configured repository 'b2:rpi-homelab-backup:/restic-repo' ...
Fatal: unable to open config file: Stat: b2_download_file_by_name: 404: 
Is there a repository at the following location?
b2:rpi-homelab-backup:/restic-repo
Could not access the configured repository. Trying to initialize (in case it has not been initialized yet) ...
created restic repository c3bef4a1ac at b2:rpi-homelab-backup:/restic-repo
Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
Repository successfully initialized.
Scheduling check job according to cron expression.
new cron: 0 15 5 * * *
executing: /usr/local/bin/check 
Starting check at 2023-07-22 05:15:00
using temporary cache in /tmp/restic-check-cache-960397027
create exclusive lock for repository
load indexes
check all packs
[0:02] 100.00%  1 / 1 snapshots
check snapshots, trees and blobs
read 10.0% of data packs
[1:19] 100.00%  101 / 101 packs
no errors were found
Check successful
Finished check at 2023-07-22 05:16:32 after 92 seconds

I see the restic repo folder in my Backblaze bucket.

So, I created a snapshot (which is how you download files from backblaze), and downloaded/extracted it. Here's what I see:

❯ restic -r restic-repo snapshots
enter password for repository: 
Fatal: config or key f704f3952dd1d9fd3c8e35e4c4feed5797e3237608e60a1c564a4ab242c0f7aa is damaged: ciphertext verification failed. Try again

I am not sure what to do. I have already tried deleting the repo from the cloud and restarting the resticker containers to rebuild the repo from scratch. Both times I get this exact error and am unable to open the vault. I know the password I'm entering is correct because the error changes when I enter something wrong.

I also am not sure this is a drive issue since the check job states no errors. Does the check job run locally, though? Or does it run on the files in the cloud? That may affect my opinion about it being a drive issue.

Any help would be greatly appreciated, thanks!

[benjamin051000]

Please let me know if I need to transfer this issue to restic, too. I really want to get to the bottom of this.

[benjamin051000]

I don't know what changed, but I deleted everything, started over, and now I'm able to open the repo downloaded from backblaze. I wonder what the issue may have been...

[benjamin051000]

Just updating: I still get this issue spontaneously :(

[5a5cha]

Hi, in a previous versions there was a bug when starting more then one service in a compose file.
Both services tried to init the repo.
Maybe this could be the reason for the first issue. Since there is no start time in the log it's a little bit complicated to debug.

Regarding your second issue with the B2 bucket....
I'm not an B2 expert. But why do you download a snapshot?
Are you sure you downloading the complete repo? Since restic uses incremental backup you need to download all files in the bucket.
Could you create a extra "restore" container and mount this to your B2 bucket on the source side and to a restore volume on the other side. And try to restore some files?