Skip to content

Commit

Permalink
Sidestep multipart upload failures on GCS
Browse files Browse the repository at this point in the history
- Attempting to upload a file to Google Cloud Storage larger than 5 MB
  (the default multipart upload size) will return:
```
error running command: InvalidArgument: Invalid argument.
  status code: 400, request id:
```
- This PR avoids the issue by disabling multipart uploads when the GCS
  endpoint is used, didn't have time to investigate an actual fix
- Didn't add an integration test as it seemed like a lot of overhead to
  require a GCS account in CI just for this change, but let me know if
  you'd rather have integration coverage around this.
  • Loading branch information
ljfranklin committed Apr 10, 2017
1 parent af1b1aa commit 12c0a52
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions s3client.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"fmt"
"io"
"os"
"strings"
"time"

"github.com/aws/aws-sdk-go/aws"
Expand Down Expand Up @@ -152,6 +153,11 @@ func (client *s3client) BucketFileVersions(bucketName string, remotePath string)
func (client *s3client) UploadFile(bucketName string, remotePath string, localPath string, options UploadFileOptions) (string, error) {
uploader := s3manager.NewUploader(client.session)

if client.isGCSHost() {
// GCS returns `InvalidArgument` on multipart uploads
uploader.MaxUploadParts = 1
}

stat, err := os.Stat(localPath)
if err != nil {
return "", err
Expand Down Expand Up @@ -411,3 +417,7 @@ func (client *s3client) newProgressBar(total int64) *pb.ProgressBar {

return progress.SetWidth(80)
}

func (client *s3client) isGCSHost() bool {
return (client.session.Config.Endpoint != nil && strings.Contains(*client.session.Config.Endpoint, "storage.googleapis.com"))
}

0 comments on commit 12c0a52

Please sign in to comment.