Skip to content

Commit

Permalink
aws: disable client-side rate limiting by default
Browse files Browse the repository at this point in the history
Issuing many S3 uploads in quick succession causes the following
error:

operation error S3: PutObject, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

Disable this to restore the behavior of SDK v1.

Relates to #3496
  • Loading branch information
stanhu committed Oct 9, 2024
1 parent bbdd0b3 commit a54e8c8
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions aws/aws.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ import (
"strconv"

awsv2 "github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/aws/ratelimit"
"github.com/aws/aws-sdk-go-v2/aws/retry"
"github.com/aws/aws-sdk-go-v2/config"
awsv2cfg "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/client"
Expand Down Expand Up @@ -251,5 +254,14 @@ func V2ConfigFromURLParams(ctx context.Context, q url.Values) (awsv2.Config, err
opts = append(opts, awsv2cfg.WithEndpointResolverWithOptions(customResolver))
}

// SDK v2 adds client-side rate limiting: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/#client-side-rate-limiting
// This can cause "failed to get rate limit token" errors. Disable this to restore
// the behavior of SDK v1.
opts = append(opts, config.WithRetryer(func() awsv2.Retryer {
return retry.NewStandard(func(o *retry.StandardOptions) {
o.RateLimiter = ratelimit.None
})
}))

return awsv2cfg.LoadDefaultConfig(ctx, opts...)
}

0 comments on commit a54e8c8

Please sign in to comment.