Exporter scrapes not all metrics (only 100 by default)

[SklyarHappens]

Rancher api by default returns only 100 objects for endpoint (/hosts, /services, etc.)

root@3326ef74ddf4:/# curl -s -H "Cookie: token=<API_TOKEN>" -H "Accept: application/json" http://<RANCHER_URL>/v1/projects/PROJECT_ID/services | tr '}' '\n' |grep -c "\"type\":\"service\""
97
root@3326ef74ddf4:/# curl -s -H "Cookie: token=<API_TOKEN>" -H "Accept: application/json" http://<RANCHER_URL>/v1/projects/<PROJECT_ID>/services?limit=1000 | tr '}' '\n' |grep -c "\"type\":\"service\""
399
root@3326ef74ddf4:/#

[onedr0p]

Duplicate of #28

[mikequentel]

I know this is an old issue, but in case anyone else is facing a similar problem, I would like to suggest looking at the example of config.go of github-exporter where one can include multiple target URLs with different URL parameters.

I had to scrape Bitbucket API 1.0 where one can request only 100 objects at a time. Using the config.go TargetURLs it is possible to specify multiple URLs with different parameters where one URL might have the first 100 items, then the second would have the next 100, etc; thus one can paginate and collect the metrics at the same time.

https://github.com/infinityworks/github-exporter/blob/d94272fb546222c2127767a45757b65c5ee8be7b/config/config.go#L62

By the way, github-exporter is one of the best examples of a reference implementation for scraping REST APIs. I have used this as a pattern many times when creating Exporters.