Dynamically creating and deleting prdcrs (as part of streams)

[oceandlr]

Has anyone dynamically removed a prdcr? I believe the calls in order should be prdcr_unsubscribe, prdcr_stop, and prdcr_del. I am running into issues with the prdcr_del. I will wait and see who has done this before writing further. Thanks!

[nichamon]

Yes, you can dynamically remove a producer. Let me show you a working example:

Here's my aggregator's configuration file (agg.conf):

After starting the aggregator, here's the initial prdcr_status output showing all producers:

To remove producer node-1, I executed these commands:

Notes:

1. In the updtr_prdcr_del command, I used ^node-1$ as the regular expression to specifically match and remove just that producer
2. The follow-up prdcr_status command confirms the producer was successfully removed
3. While you mentioned using prdcr_unsubscribe, it wasn't strictly necessary in this case since the sampler daemon automatically removes the aggregator from the stream my_stream's subscriber list when the connection is disconnected. However, including prdcr_unsubscribe in the removal process is also acceptable

Please correct me if any of these assumptions are wrong:

1. The ldmsd version you are using is v4.4.4
2. After a producer is added to the aggregator (whether at startup or dynamically while running), updtr_prdcr_add is called with a regex that matches the producer name to add it to the updater's producer list (i.e., the producer's sets will be updated by the updater)

If the error you got from prdcr_del is "producer is in use", it was because you needed to remove the producer from the updater's producer list by calling updtr_prdcr_del, as shown in my example

I hope this helps.

[oceandlr]

Sorry about the delay in getting back to you. I'll have to show you this, live --- I have something inconsistent going on. Here is what is happening:

0. I am dynamically (in a sampler) calling ldmsd_controller (via 'system') and adding a stream and a producer for that stream only. Therefore there is no updatr.

1. I issue:

• prdcr_add
• prdcr_subscribe
• prdcr_start
  and
• ldmsd_stream_subscribe

2. after this, if I manually check status in a controller, I see what I would expect.
   2a) sock:localhost:52001> prdcr_status
   Name Host Port Transport State

---

zedprdcr localhost 52002 sock CONNECTED

with no updtr

Also, everything works as planned.

3. Eventually, i will tear (via the sampler) the new stream down and will want to get rid of the prdcr's and subscriptions that I added
   I assumed I would want to issue:

• prdcr_unsubscribe
• prdcr_stop
• prdcr_del

4. If I manually issue the commands in the controller, then everything is fine in ldmsd_controller:
   4a) after the prdcr_stop:
   sock:localhost:52001> prdcr_status
   Name Host Port Transport State

---

zedprdcr localhost 52002 sock STOPPED
4b) and after the prdcr_del there is no prdcr

And everything appears to work as planned.

5. However, if I do this via calls to ldmsd_controller in the sampler, then the prdcr_stop is ok, but the prdcr_del prints out "Error deleting prdcr zedprdcr: " in the log, with no explanation of what the error is.
   5a) if i also try to manually connect via ldmsd_controller to check the status, I get an error:
   Welcome to the LDMSD control processor
   sock:localhost:52001> prdcr_status
   Error getting prdcr status:

I wonder if there is something going on because I am doing it from within the code and changing some state??? @nichamon @tom95858

[nichamon]

@oceandlr Thanks for the information.

I wonder if there is something going on because I am doing it from within the code and changing some state???

Since you system call ldmsd_controller from your sampler plugin code, it shouldn't cause any differences, except that the prdcr_del is sent much closer to the prdcr_stop in your sampler plugin, comparing with doing it manually.

As you mentioned, missing explanation makes it hard to know what is going on. The explanation is missing and is supposed to follow the colon and a space. For example,

If the cause of prdcr_del failure is the timing, we should be able to see "The poducer is in use" after the zedprdcr: .

Did your sampler plugin print "Error deleting prdcr zedprdcr: " from the ldmsd_contoller system call output to the log file?

5a) if i also try to manually connect via ldmsd_controller to check the status, I get an error:
Welcome to the LDMSD control processor
sock:localhost:52001> prdcr_status
Error getting prdcr status:

This makes me think that there could be something going on that cause 'prdcr_status' to fail, besides the timing thing.

I'm up for a Zoom call this week.

[oceandlr]

@nichamon and @oceandlr met today. It looks like this is a result of an intentional locking issue to keep from deleting the producer while the stream message is being processed. It looks like the error message without a reason is because the ldmsd_controller times out.

We have some options for workarounds and will note the behavior to inform the eventual production implementation.

[oceandlr]

Closing discussion as a result of the meeting above.