Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GITHUB-ISSUE-1847: Fix Azure binlog upload for large files #1848

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+4 −1
Open

GITHUB-ISSUE-1847: Fix Azure binlog upload for large files #1848

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ee0c387
Fix Azure binlog upload for large files
dcaputo-harmoni Oct 16, 2024
d86af6e
Merge branch 'main' into GITHUB-ISSUE-1847-fix-azure-binlog-upload-fo…
hors Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion pkg/pxc/backup/storage/storage.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,10 @@ func (a *Azure) GetObject(ctx context.Context, name string) (io.ReadCloser, erro

func (a *Azure) PutObject(ctx context.Context, name string, data io.Reader, _ int64) error {
objPath := path.Join(a.prefix, name)
_, err := a.client.UploadStream(ctx, a.container, objPath, data, nil)
uploadOption := azblob.UploadStreamOptions{
Concurrency: 4,
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dcaputo-harmoni Hi! Thank you for your contribution.
Is it possible to make this value configurable? I believe it will super useful for all our users.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible, just don't have the bandwidth to do it at the moment. Given that it's currently broken for large files what I would suggest is that we release as is to fix the issue and then just open a feature request / issue to track it and I'll get that in down the line.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, time is always the problem.
We can take over this PR and make value configurable.

_, err := a.client.UploadStream(ctx, a.container, objPath, data, &uploadOption)
if err != nil {
return errors.Wrapf(err, "upload stream: %s", objPath)
}
Expand Down
Loading