Custom collectors and multiprocess mode

[tillprochaska]

Hi there,

I’m trying to collect metrics for a Python web application. I’ve instrumented my app using the metric classes and utilities exported by the Python client (e.g. Gauge and timers) to collect a few basic HTTP request metrics. As the application is served using Gunicorn with multiple worker processes, I’ve followed the multiprocess instructions from the docs. This all works perfectly fine!

I’d also like to expose some metrics based on database records (e.g. number of users) or queue lengths (e.g. number of queued tasks). In order to do that, I have implemented a few custom collectors that fetch data from data stores and return Prometheus metrics.

As the docs state that the multiprocess mode doesn’t support custom collectors, I’m currently running a sidecar service to export metrics using my custom collectors:

• app service: uses MultiProcessCollector to expose request metrics from multiple worker processes
• prom-collector service: runs a single process and exposes only metrics from the databases, queues using custom collectors

I was wondering whether this is the best/recommend way to handle this use case? I had a look at the issues/discussion in this repository and searched other resources on the web, but haven’t found any recommendations/guidance for this use case

Is the fact that custom collectors are not supported in a multi-process environment a fundamental limitation or is there a possibility to make this work at some point (at least if the metrics collected by custom collectors are not dependent on the current process)?

Thanks a lot!

[tillprochaska]

I’ve actually ended up not using a sidecar service to export metrics from custom collectors. Instead, I’ve simply created two separate registries. The first one includes only a MultiProcessCollector, and the second one includes all my custom collectors.

My metrics endpoint basically simply concatenates the metrics from both registries. This way I can collect metrics across processes, but can still use custom collectors (as long as the custom collectors return the same data no matter what process executes them).

This seems to work fine, but I’m not sure if this approach has any significant disadvantages I might be overlooking?

[csmarchbanks]

I think your approach will work for most cases, but it might be safer to use two endpoints if you are simply concatenating the strings returned from each registry. That should work for the Prometheus exposition format (though double check to make sure that there is a newline after the end of the first result), but for OpenMetrics exposition you could run into some trouble as the responses would be separated by a # EOF line when it is not actually the end of the file.

[tillprochaska]

Hi @csmarchbanks, thanks, that is very helpful. I’ll make sure to check that the two responses are concatenated properly.