Signature failure during initial upload of large database

[RaghavSood]

I have an existing ~400GB sqlite3 db I'm trying to add litestream to.

I'm using fly.io and cloudflare r2.

About 20 minutes into the first-time run, I can see a multipart object being uploaded in the cloudflare dashboard. Then, I suddenly get the error:

time=2024-06-15T09:36:23.746Z level=ERROR msg="monitor error" db=/data/prod.sqlite replica=s3 error="MultipartUpload: upload multipart failed\n\tupload id: ADxwtS8v0QJ1gjtXvMoyXRTF_fazQ3SrKHp20IPaA8RTOXSdxsUh-TIvJRBiz8ld1uORx34X3hnoNiyDeHIzRoYsaeYyTetFfjyiuv7xXJbCSdI6LacIJKql496xYl7my5pjmVpAsI-tVKcpLHBSetw-8heCvK6mioMByQMULt_2hQWS_bU0vpB2NtZKzl9aGOd3cHy-LDAdz96t8-GKdxwQrKlO12-bqMt6LqdfKNvADMJml0a8XE0QOXtKQwc-kL_E4agOp0HRAFtQFO2PlWiTtxKJEAVDUddn0f7o5jkB50M1BquzN1246TeEEz2tkrPINcTlMjnwW9VvVf0cACE\ncaused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. \n\tstatus code: 403, request id: , host id: "
time=2024-06-15T09:36:23.812Z level=INFO msg="write snapshot" db=/data/prod.sqlite replica=s3 position=f011d7572eac33e3/00000000:7217045232

litestream does not exit, and seems to keep continuing to upload other parts - cloudflare shows tens of thousands of operations on the bucket (this is the only user of that bucket)

I'm unclear on whether or not the failed part was retried, or what the final state will look like.

Is this expected when syncing large databases?

Am I pushing my luck by trying to sync a large db from scratch, while the app is running?

[RaghavSood]

I am now trying with a smaller DB (~20GB), with no other process accessing it while litestream is doing the initial snapshot upload. However, it still fails with the same signature error:

Jun 23 04:31:41 btcsupply systemd[1]: Started Litestream.
Jun 23 04:31:41 btcsupply litestream[153404]: time=2024-06-23T04:31:41.814Z level=INFO msg=litestream version=0.3.13
Jun 23 04:31:41 btcsupply litestream[153404]: time=2024-06-23T04:31:41.815Z level=INFO msg="initialized db" path=/var/lib/btcsupply/prod.sqlite
Jun 23 04:31:41 btcsupply litestream[153404]: time=2024-06-23T04:31:41.815Z level=INFO msg="replicating to" name=s3 type=s3 sync-interval=1s bucket=burned-money path=btcsupply region="" endpoint=https://bee8c5132606211d468114d3.r2.cloudflarestorage.com
Jun 23 04:31:42 btcsupply litestream[153404]: time=2024-06-23T04:31:42.831Z level=INFO msg="sync: new generation" db=/var/lib/btcsupply/prod.sqlite generation=e45893188df90ce9 reason="no generation exists"
Jun 23 04:31:42 btcsupply litestream[153404]: time=2024-06-23T04:31:42.982Z level=INFO msg="write snapshot" db=/var/lib/btcsupply/prod.sqlite replica=s3 position=e45893188df90ce9/00000000:4152
Jun 23 04:32:12 btcsupply litestream[153404]: time=2024-06-23T04:32:12.655Z level=ERROR msg="monitor error" db=/var/lib/btcsupply/prod.sqlite replica=s3 error="MultipartUpload: upload multipart failed\n\tupload id: ADHZtsU-_mEVgn6Q4hgOZa-ivEbfheaTj982Czsb3CA9_bEUycPuAPlvyxwrE_J3K8tnCWbwF7fcT-JXgK9GHgPNpYf4tWa1o_2vnzVnV3p_dg-Mq01zYGbpmxX3AmrrmFAvRW41iAjTGo7l4gCzG6QPFLpLINhXEcbxEeAKZbrJ5gka9dQggY_mVm3eq1MOU6VvkC02gPAbaW3cSXC_YK_ZEXoFIXuTW1gXFyupk8c3a3sg34HpBfDKExQ80Z11EwiSHFVLOtEQe0iaF9O8FVLZpFrkyi-MgXbEvwiDzKylAbYlVKhTL13Nck_nB3YdQr7_HdTB4LsZVGLy6_EjyFM\ncaused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. \n\tstatus code: 403, request id: , host id: "
Jun 23 04:32:12 btcsupply litestream[153404]: time=2024-06-23T04:32:12.713Z level=INFO msg="write snapshot" db=/var/lib/btcsupply/prod.sqlite replica=s3 position=e45893188df90ce9/00000000:4152