Add a scrape timeout parameter in docker compose

[demeringo]

Problem

Producing metrics for a large number of instances takes too long (something like 25 seconds for 100+ instances).
As a result promethues times out before cloud scanner returns metrics and we see no data in the dashboard (nor in prometheus).

Solution

As a short terme workaround we can increase the scrape_timeout in prometheus config. The default is 10 second, we could include an example of setting the timeout to 60 seconds.

Also needs to be mentionned in the docs.

Long term solution is to optimize the way we gather data and return metrics but this is another story #392

Alternatives

Additional context or elements

This condition can be detected in the prometheus UI, by checking the status / targets page which returns details about scrape time for different targets.

[demeringo]

In term of documentation I believe we should do 2 things:

• add a chapter about this in the 'how-to' section
• mention this potential issue in the 'common-issues' and link to above page

[demeringo]

Note: this setting is also briefly mentionned in https://boavizta.github.io/cloud-scanner/how-to/set-up-dashboard.html#adapting-configuration-for-production-use, we may have to link to the newly created page and remove the details from this paragraph.

[demeringo]

The example scrape interval is already mentioned in the sample prometheus config file.

cloud-scanner/dashboard-config/prometheus/prometheus.yml

Line 2 in ecca1ac

scrape_interval: 30s # By default, scrape targets every 30 seconds.

So this issue is more a documentaion issue (but we could be more explicit in the comment of the prometheus config file)

[demeringo]

To help debug the possible timeout when scrapping metrics (in docker compose example), you can check the status for individual scraping targets here:

http://localhost:9090/targets?search=

The global prometheus UI is at: http://localhost:9090/

[jnioche]

See 8d4ab37 which explicits the config in Prometheus