-
Notifications
You must be signed in to change notification settings - Fork 204
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
use max limit
to conserve api quota in all cases
#1775
base: master
Are you sure you want to change the base?
Conversation
we were previously only using the maximum limit value if the `q` query param was used, now it should be used in all cases
okta/data_source_okta_groups.go didn't do pagination in it's original implementation and was never updated to do so correctly. Okta users has a better technique @exitcode0 https://github.com/okta/terraform-provider-okta/blob/master/okta/data_source_okta_users.go#L100-L102 https://github.com/okta/terraform-provider-okta/blob/master/okta/data_source_okta_users.go#L138-L152 |
My intent here is not so much to resolve a pagination bug, but more that we're using more API calls than needed to retrieve a given cohort of groups I'll tweak my PR to add pagination |
Being able to set the default limit parameter https://github.com/okta/terraform-provider-okta/blob/master/okta/utils.go#L28C1-L28C1 should probably a setting on the provider itself and then have it be readable by any resource/datasource via the global config. However, that might be too generic. Or maybe something generic to the otka provider itself where any resource can have a setting similar to |
I thought about adding the parameter at the provider level, I even attempted a PR last night but was a bit out of my depth. Some thoughts/questions regarding `limit
|
Right, I think limit should probably be a provider config value that makes it into the global config. The limit value is already set to 200, which is a mirror of the same value the Okta API has for itself internally. So, let the operator choose their default. There must be a max, I don't know what it is, perhaps @exitcode0 inferred using Postman and found it to be 10K? The best value depends on your use case. To that end, the best way to do it would be to have api limit be a setting on every data source and resource (like given new code // in utils.go?
var apiLimitSchema = map[string]*schema.Schema{
"limit": {
Type: schema.TypeInt,
Default: 200,
Description: "API request objects limit",
},
} for example, the brand data source schema https://github.com/okta/terraform-provider-okta/blob/master/okta/data_source_okta_brand.go#L14-L23 would be this: Schema: buildSchema(
map[string]*schema.Schema{
"brand_id": {
Type: schema.TypeString,
Required: true,
Description: "Brand ID",
},
},
brandDataSourceSchema,
apiLimitSchema
), but then each resource and data source would have to pass that value into the SDK client to use. Having it as a global configuration would be much easier implement and be closer to what is desired. |
@arvindkrishnakumar-okta is going to follow up on this. The API docs are inconsistent. On the one hand it states the max is 10000 but the docs also recommend not using a value greater than 200 because of eventual consistency concerns. |
I'd imagine the limit for the /groups endpoint is 10k as trying to retrieve all groups using pages of 200 items could take quite some time in large Okta instances (e.g >100k groups) |
@exitcode0 I'm reading this issue and have a basic question - are you running into API rate limit quota soon (or are the results taking too long to page through multiple pages?) because of a lower limit value? In other words, is the motivation to increase it to a large number |
on
data_source_okta_groups.go
we were previously only using the maximumlimit
value if theq
query parram was used, now it should be used in all cases