Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on writing (deleting?) snapshot #558

Open
liffiton opened this issue Jan 6, 2024 · 1 comment
Open

Crash on writing (deleting?) snapshot #558

liffiton opened this issue Jan 6, 2024 · 1 comment

Comments

@liffiton
Copy link

liffiton commented Jan 6, 2024

I've been running Litestream for months with no change to the configuration and no issues, when a few days ago it crashed on a normal snapshot operation. It had just written a new snapshot, and it looks like something went wrong when deleting a previous one. I'm using v0.3.13 and writing to Backblaze B2 with the S3 API.

litestream[411]: time=2024-01-04T19:25:23.637-06:00 level=INFO msg="write snapshot" db=<my DB file> replica=s3 position=5ef78de18fe20421/00004a76:4152
litestream[411]: time=2024-01-04T19:25:25.204-06:00 level=INFO msg="snapshot written" db=<my DB file> replica=s3 position=5ef78de18fe20421/00004a76:4152 elapsed=1.560554481s sz=23182400
litestream[411]: panic: runtime error: invalid memory address or nil pointer dereference
litestream[411]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xea55a3]
litestream[411]: goroutine 36 [running]:
litestream[411]: github.com/benbjohnson/litestream/s3.deleteOutputError(0xc00042c940)
litestream[411]:         /home/runner/work/litestream/litestream/s3/replica_client.go:766 +0x63
litestream[411]: github.com/benbjohnson/litestream/s3.(*ReplicaClient).DeleteSnapshot(0xc00062b100, {0x17de4f8, 0xc00013e190}, {0xc00054b5af, 0x10}, 0xc000763fc0?)
litestream[411]:         /home/runner/work/litestream/litestream/s3/replica_client.go:318 +0x29a
litestream[411]: github.com/benbjohnson/litestream.(*Replica).deleteSnapshotsBeforeIndex(0xc0005de0f0, {0x17de4f8, 0xc00013e190}, {0xc00054b5af, 0x10}, 0x4a76)
litestream[411]:         /home/runner/work/litestream/litestream/replica.go:627 +0x225
litestream[411]: github.com/benbjohnson/litestream.(*Replica).EnforceRetention(0xc0005de0f0, {0x17de4f8, 0xc00013e190})
litestream[411]:         /home/runner/work/litestream/litestream/replica.go:604 +0x38b
litestream[411]: github.com/benbjohnson/litestream.(*Replica).retainer(0xc0005de0f0, {0x17de4f8, 0xc00013e190})
litestream[411]:         /home/runner/work/litestream/litestream/replica.go:726 +0x138
litestream[411]: github.com/benbjohnson/litestream.(*Replica).Start.func2()
litestream[411]:         /home/runner/work/litestream/litestream/replica.go:128 +0x57
litestream[411]: created by github.com/benbjohnson/litestream.(*Replica).Start in goroutine 30
litestream[411]:         /home/runner/work/litestream/litestream/replica.go:128 +0x165
@hifi
Copy link
Collaborator

hifi commented Jan 6, 2024

Hi, #557 fixes this crash. I'll merge it soon but can't promise a release date.

It seems B2 broke their API and they return an error complaining about malformed XML randomly to delete operations made by Litestream.

That PR doesn't fix the issue B2 is having but it does prevent the crash and Litestream should eventually succeed and only show an error in logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants