How to handle rate limiting for glacier?

[andig]

Confirm by changing [ ] to [x] below:

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers

Describe the question

As asked on SO: https://stackoverflow.com/questions/69089033/how-to-handle-rate-limiting-using-the-aws-glacier-sdk

I have a glacier client like this:

cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("eu-west-1"),
    config.WithRetryer(func() aws.Retryer {
        return retry.AddWithMaxBackoffDelay(retry.NewStandard(), time.Second*5)
    }),
)

client := glacier.NewFromConfig(cfg)

trying to delete ~1Mio archives from glacier vault in a loop, using 16 workers, I'm running into:

operation error Glacier: DeleteArchive, exceeded maximum number of attempts, 3, https response error StatusCode: 400, 
RequestID: pmkKm45AI3tNzMSUp5PKP84QoanLYHHEpaZ-WXsxsv1v8mc, api error ThrottlingException: Rate exceeded

How can I configure the SDK to handle the rate limiting?

Note: what seems strange is that the API responds with HTTP 400 for rate limiting, I would have expected HTTP 429?

[KaibaLopez]

Hi @andig ,
So the exponential backoff is meant to handle this but you are now hitting the max retry attempts so, the next thing I would suggest is specifying a bigger max retries so that the exponential wait gets to grow enough to avoid the throttling.
Look here for documenation, and let me know if this helped or if you got more questions on this.

[andig]

Thanks for the idea! Unfortunately, it seems as if this is still hard to use (adding MaxAttempts = 5):

2021/09/10 22:52:42 operation error Glacier: DeleteArchive, retry quota exceeded, 0 available, 5 requested

[KaibaLopez]

I mean there is not a huge difference between 3 attempts and 5, I was thinking more on like 30 so that the exponential backoff comes in.
But yea it's up to you at that point, how you play with the waits and the throttling limit.

[andig]

Sorry, I may not be doing what you've suggested, here's the code now with 60 attempts:

cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(*region),
	config.WithRetryer(func() aws.Retryer {
		return retry.AddWithMaxAttempts(
			retry.AddWithMaxBackoffDelay(retry.NewStandard(), 30*time.Second),
			60)
	}),
)

The error is still the same though:

2021/09/14 19:40:18 operation error Glacier: DeleteArchive, retry quota exceeded, 1 available, 5 requested

The retry quota error here is different from the initial error which was

operation error Glacier: DeleteArchive, exceeded maximum number of attempts, 3, https response error StatusCode: 400, 

So it doesn't seem to be that the attempts are exhausted.

I'm failing to unterstand which know to turn to make the backoff behave like you've suggested?

[andig]

Might this somehow be related to #1413?

[Skarlso]

That is super weird, because it's not immediately apparent that that Max value is actually applied there correctly. It might not be related to that issue but the "problem" if it exists, is certainly somewhere around that code.

[andig]

I can share the entire code- its just boilerplate leading up to whats shown above.

[Skarlso]

cheers, that might help me writing some test code

[Skarlso]

@andig Ah mate, I think your stuff is different. That ticket is about the middleware retrier which is configured through AddRetryMiddlewares from the API. What you are using is the global config retrier.

Here is some more documentation about that with examples: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry

That said, it's not that yours shouldn't be working..

[andig]

The retry docs didn't gibt enlightenment. I'm still stuck with this- any more insights?

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.