RED ALERT: DISCOURSE BACKUPS DON'T WORK (S3 Related)

[FarisZR]

Discourse backups don't work.

When importing backups, files that are stored on R2 are inline linked as if S3 isn't enabled for some reason, rebaking the posts doesn't change anything.
Affected files are most optimized profile pics, site logos and new files uploaded since enabling S3.

Notable: migrate_to_s3 task also doesn't run correctly.

Even if I wanted to include S3 uploads into the backups, there is something missing in the R2-S3 compatible API stopping discourse from downloading the images in the backup process.

I can currently only move the server using by just copying the bind volumes, which isn't great and isn't how discourse backups work.

The plan

write a post about the S3 content link issue on meta

Maybe there's something that can fix this, because the files actually exist on R2, but the new discourse instance isn't using the R2 CDN-link for some reason.

Move off S3

the only officially supported S3 storage solution is AWS, to avoid further issues, we either move to AWS S3, which is very unpredictable in terms of monthly costs, or we disable S3 in discourse completely.

Because Discourse can't currently download uploads from R2 to include in the backups, we first need to move to a fully S3 compliant solution.
Cloning the bucket to a local minio instance, and then letting discourse use that as the new S3 host seems to be the best workaround.

[FarisZR]

Cloudflare R2 should actually be working, even with backups.
here are few details about this:
https://meta.discourse.org/t/cloudflare-r2-navigating-setup-and-handling-configuration-errors/305617/40

[FarisZR]

Update:
downloading S3 media seems to work for backups, but still images are missing from the archive.
need to test stuff more, as this was tested with the broken s3 settings from the app.yml file.

I think it should work when migrate to s3 actually uploads all files to R2

[FarisZR]

migrate to s3 still crashes when migrating.
it doesnt actually upload all files for some reason.
need to post this on meta.

[FarisZR]

Current situation:

• Migrate to s3 doesn't work
  Meaning we can't continue to use R2 currently
• S3 assets seem to be included currently in the backup process
  Previously Discourse couldn't download things from R2, but this seems to be fixed.
• Even after including S3 Assets in backups, Images are still missing after restoring from a discourse backup.
  This is the big issue, as it means that the issue isn't R2 or S3, even when downloading assets from R2, they are still missing in the final backups.
  I don't know what's going on here, and whether this has something to do with the failed migrate to s3 runs.

Actions Items:

• Create a post on meta with logs from migrate_to_s3
• Upload a photo of a restored instance from the backups showing the broken images (even when including S3 Assets)
• (Maybe?) upload logs of the backup process.