Problems with one push_to_gateway flushing metrics created by previous push_to_gateway for different counters

[Jacek-ghub]

I am having problems with push_to_gateway from one script flushing stats for the same job that came through another script. Here are my scripts:

from prometheus_client import Gauge, push_to_gateway, CollectorRegistry

GATEWAY  = '192.168.1.51:9091'
JOB_NAME = '_gauge_test'

registry = CollectorRegistry ()

g1 = Gauge ('_g1', 'g1 gauge', registry = registry)
g1.set (1)
push_to_gateway (GATEWAY, job = JOB_NAME, registry = registry)

and

from prometheus_client import Gauge, push_to_gateway, CollectorRegistry

GATEWAY  = '192.168.1.51:9091'
JOB_NAME = '_gauge_test'

registry = CollectorRegistry ()

g2 = Gauge ('_g2', 'g2 gauge', registry = registry)
g2.set (2)
push_to_gateway (GATEWAY, job = JOB_NAME, registry = registry)

Those scripts are hit by apache server based on its activity.

Not sure whether I understand it right, but it looks like I would need to create CollectorRegistry in a separate process and let those two scripts somehow piggy back to that registry. Although, looking through the push_to_gateway code I didn't see anything that would help me to do it.

What I am trying to achieve is to have just one job in PushGateway that would have all my counters. However, maybe this is not a good way to do it, and just let those batch files to create unique job names (JOB_NAME in those two scripts above).

I am not sure how to approach this problem. Any help would be greatly appreciated.

[csmarchbanks]

Hello,

This is how Push Gateway works, if a push is sent with the same job it will overwrite anything for that job. Your options are to either have separate job names, or you can use the grouping_key argument to push_to_gateway to have a secondary set of labels used for grouping. For example you could specify the script or an instance running the script (only if that is static) as the label and then the job label could be the same still.

Hope that helps, you can find more information on grouping in the Push Gateway Docs.

[Jacek-ghub]

Hi @csmarchbanks Thank you for your response.

As mentioned, I am using your prometheus_client library. Looks like when CollectroRegistry is instantiated, it may be using that process PID as identification, thus destroying the previous metrics (to me, overriding would mean only replacing the same metrics as being used in the current instantiation, so I would really want to see overriding, not destroying).

Do you think that CollectorRegistry constructor could be extended to eventually provide a name / ID used for the push binding (class CollectorRegistry(Collector, binding_name: str = None))?

Having that addition would simplify a lot the client-side development, as right now all of us in the similar situation need to add some way of making the push_gateway part static. I do hear you about those other options, but this one takes just one extra string to get it working and not worrying about the extra service being there all the time.

If possible, I would also like to see a possibility to specify expiration time for metrics. Although, most likely my understanding of how it works is just not on the level to see how it could be done otherwise (I am just starting with it).

[csmarchbanks]

Another option that you could try is the pushadd_to_gateway method. This will only replace metrics with the same name, but it means that stale metrics would not be cleaned up and you would have to handle that manually.

I don't think we should have a binding_name on a collector, mostly as I would not want to tie a registry to something about pushing, a registry should just have metrics defined. Would it be helpful if a registry had a common label that could be shared among all metrics defined on it?

Expiration time would probably need to be a request for pushgateway, or at least I don't know of that feature today.

[Jacek-ghub]

I don't think we should have a binding_name on a collector, mostly as I would not want to tie a registry to something about pushing, a registry should just have metrics defined.

Not sure if I understand this concern correctly. My understanding is that with the current implementation a collector is implicitly binding to the process (ID). What is a difference between using a process ID for binding vs. an arbitrary string or a random number? Using process ID for case like mine is clearly providing unexpected results (subsequent instantiation of push containing process is destroying previous information). On the other hand, using user supplied one (if elected to do that) would overcome this issue. For sure, as I am new to Prometheus, I am missing something.

Also, the second problem (for me) is that as collector binds to process id, there is basically no way to start two collectors in one script. This further convolutes a solution where for instance a couple or so collectors would be beneficial, but in order to do so some extra voodoo needs to be done to run multiple push processes and addressing them.

In my case, I have a handful of clients that talk to a single PHP client via REST. That module starts a Python script with each received data, and that script talks to another Python script that runs as a daemon thus having one collector open. Once the data is passed, that script exits. It would really simplify that design, if I could have multiple collectors open in that daemon script, instead of trying to have a dispatcher for a dispatcher to hit multiple daemon scripts. (Actually, it would be even simpler in my case, if just one collector could be used, where the push_to_gateway call would clearly separate pushes using different job names. Although, again most likely this is just my misunderstanding of push_to_gateway working.)