[fix] [broker] Do not record a bundle-unloading into the topic load failed metrics

[poorbarcode]

Motivation & Modifications

Pulsar has a metric that indicates load topic failed: topic_load_failed_total, it reflects something is not expected, such as

• Not enough bookies
• Failed to connect to the Metadata store

But the topic loaded failed due to a bundle in unloading is expected, we should not record it into this metrics.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

since multiple threads are involved, shouldn't these be AtomicIntegers?

[lhotari]

I don't currently understand the reason to use MutableInts.
If the reason is to that a final variable is needed for an inner lambda, it might be better to use a final variable.
One trick that could be useful is to isolate variable scopes with a {....} block. That would allow to introduce the same variable name multiple times in a single unit test without variable name collisions since the variables would be in different scopes.

[poorbarcode]

Fixed

[heesung-sn]

But the topic loaded failed due to a bundle in unloading is expected, we should not record it into this metrics.

Don't we need to count this case as failure as well? To me if the topic load failed on the broker and client lookup failed, it should be counted as a failure.

I think this is one area that Extensible Load Manager can shine as it handles lookups more gracefully during unloading -- the topic lookups are deferred until the unloading completes in the service unit state channel. Hence, the client won't see failure, and the lookup response should be slightly delayed instead of returning error code(no retry as well).

[poorbarcode]

@heesung-sn

Don't we need to count this case as failure as well? To me if the topic load failed on the broker and client lookup failed, it should be counted as a failure.

It is useful to check if there is a bug in LoadBalancer, such as: if the variable keep increasing fastly for a long time, maybe the a broker cached a wrong bundle owner or DNS issue

Sorry, I misunderstood the comment. I reply again:

• Increasespulsar_concurrency_load_topic_and_unload_bundle_count when a topic loaded failed due to the bundle unloading
• Increasestopic_load_failed_total when an unexpected error occurs

If users assumed it should be combined into failed to load topic, they can get the counter this way pulsar_concurrency_load_topic_and_unload_bundle_count + topic_load_failed_total

[heesung-sn]

I still think this is a still failure case. Can we add a "reason" label instead of adding a new metric?

[poorbarcode]

@heesung-sn

I still think this is a still failure case. Can we add a "reason" label instead of adding a new metric?

I will start a PIP for this metric, it is useful to detect the following issue:

• [fix] [broker] Internal reader of __change_events can not started after metadata store session rebuilt #23018
• DNS error: The client wants to connect to broker-0.io, but the client gets an IP from broker-1