output_status_retry_wait means the elapsed time from the first retry

[daipom]

• From retry_type periodic not working for http fluentd#4233.

The document says output_status_retry_wait means current retry_wait computed from last retry time and next retry time.

fluent-plugin-prometheus/README.md

Lines 122 to 123 in 41fa2df

	- `fluentd_output_status_retry_wait`
	- current retry_wait computed from last retry time and next retry time

However, it actually means the elapsed time from the first retry.
We need to fix the value or the document.

fluent-plugin-prometheus/lib/fluent/plugin/in_prometheus_output_monitor.rb

Lines 194 to 207 in 41fa2df

	if info['retry']
	next_time = info['retry']['next_time']
	start_time = info['retry']['start']
	if start_time.nil? && info['instance_variables']
	# v0.12 does not include start, use last_retry_time instead
	start_time = info['instance_variables'][:last_retry_time]
	end
	wait = 0
	if next_time && start_time
	wait = next_time - start_time
	end
	@metrics[:retry_wait].set(wait.to_f, labels: label)
	end

It is calculated as next_time - start.
start is the time of the first retry, so this value means the elapsed time from the first retry.

• https://github.com/fluent/fluentd/blob/aaa40f71758eb5fa7b3a89084e69a00c0910fb3b/lib/fluent/plugin/in_monitor_agent.rb#L368-L384
• https://github.com/fluent/fluentd/blob/aaa40f71758eb5fa7b3a89084e69a00c0910fb3b/lib/fluent/plugin_helper/retry_state.rb

[raulgupto]

thanks for your effort @daipom. This fix/clarification is important imo as many dashboard metrics depend on prometheus and can cause noise if someone interprets this in a wrong way.

[raulgupto]

thanks for your effort @daipom. This fix/clarification is important imo as many dashboard metrics depend on prometheus and can cause noise if someone interprets this in a wrong way.