Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rate Limitation not reported #390

Closed
kcphysics opened this issue Jan 24, 2024 · 3 comments · May be fixed by #391
Closed

Rate Limitation not reported #390

kcphysics opened this issue Jan 24, 2024 · 3 comments · May be fixed by #391
Assignees
@justinpolygon
Labels
bug Something isn't working

Comments

@kcphysics
Copy link

Describe the bug
Rate Limitation is not correctly being detected and reported

To Reproduce
Run the snippet of code below multiple times until you hit your rate limit:

        API_KEY := os.Getenv("POLYGON_API_KEY")
	client := polygon.New(API_KEY)
	adjusted := true
	limit := 50000
	from := time.Now().AddDate(-2, 0, 0)
	to := time.Now()
	params := models.ListAggsParams{
		Ticker:     "AAPL",
		From:       models.Millis(from),
		To:         models.Millis(to),
		Limit:      &limit,
		Adjusted:   &adjusted,
		Timespan:   models.Day,
		Multiplier: 1,
	}
	iter := client.ListAggs(context.TODO(), &params, models.WithTrace(true))
	count := 0
	for iter.Next() {
		if iter.Err() != nil {
			log.Fatal(iter.Err())
		}
		count += 1
	}
	log.Printf("Found %d results", count)

Expected behavior
When rate limitation is reached, iter.Err contains an error stating that the rate limitation has been encountered

Screenshots
Output from the snipped above:
image

Additional context
I've tried this several different ways, but there's not "good" way to detect the rate limitation, and we cannot inspect the response inside of the Polygon.io client.
@kcphysics kcphysics added the bug Something isn't working label Jan 24, 2024
@justinpolygon justinpolygon self-assigned this Jan 24, 2024
@justinpolygon
Copy link
Contributor

Thanks for reporting this @kcphysics. I'll check it out.

@justinpolygon justinpolygon mentioned this issue Jan 24, 2024
Open
@justinpolygon
Copy link
Contributor

justinpolygon commented Jan 24, 2024 •
edited
Loading

Hey @kcphysics, there is a couple things happening here.

There was a bug in the tracing code where we were not reporting the trace when there was an error. Obviously, this information is most needed when there is an API error so you can debug things. Thank you for reporting this as it'll really help people going forward.

Here's what the trace will look like now (with status code):

Request URL: /v2/aggs/ticker/AAPL/range/1/day/1643037491376/1706109491376?adjusted=true&limit=50000
Request Headers: map[Accept-Encoding:[gzip] Authorization:[REDACTED] User-Agent:[Polygon.io GoClient/v1.16.0]]
Response Status Code: 429
Response Headers: map[Content-Length:[206] Content-Type:[application/json] Date:[Wed, 24 Jan 2024 15:18:11 GMT] Server:[nginx/1.19.2] Strict-Transport-Security:[max-age=15724800; includeSubDomains] X-Request-Id:[60a62c691e2f820a45ba7213e3f6b40d]]

There was a second issue though with your example in that the iter.Err() check was nested under for iter.Next() {. So, if there wasn't any results you'd never print the error message.

	for iter.Next() {
		if iter.Err() != nil {
			log.Fatal(iter.Err())
		}
		count += 1
	}

You can change the code to look like this you'll capture the error.

        for iter.Next() {
                count += 1
        }
        if iter.Err() != nil {
                log.Fatal(iter.Err())
        }

With the added trace fix you'll see something like this following (although you don't need the trace fix to still see the error):

Request URL: /v2/aggs/ticker/AAPL/range/1/day/1643037491376/1706109491376?adjusted=true&limit=50000
Request Headers: map[Accept-Encoding:[gzip] Authorization:[REDACTED] User-Agent:[Polygon.io GoClient/v1.16.0]]
Response Status Code: 429
Response Headers: map[Content-Length:[206] Content-Type:[application/json] Date:[Wed, 24 Jan 2024 15:18:11 GMT] Server:[nginx/1.19.2] Strict-Transport-Security:[max-age=15724800; includeSubDomains] X-Request-Id:[60a62c691e2f820a45ba7213e3f6b40d]]
2024/01/24 07:18:11 bad status with code '429': message 'You've exceeded the maximum requests per minute, please wait or upgrade your subscription to continue. https://polygon.io/pricing': request ID '60a62c691e2f820a45ba7213e3f6b40d': internal status: 'ERROR'
exit status 1

Again, thanks again for reporting this.

@kcphysics
Copy link
Author

Thanks! I appreciate the clarity and action!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@justinpolygon justinpolygon

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Show trace on error polygon-io/client-go
2 participants
@kcphysics @justinpolygon