[Housekeeping] Ensure accurate ActiveNodeExecutions and ActiveTaskExecutions metrics

[hamersaw]

Describe the issue

Currently the only ActiveNodeExecutions and ActiveTaskExecutions metrics exported are from flyteadmin. This can result in inaccuracies if, for example, flyteadmin restarts or there is an uncommon inconsistency in data and control plane state (ex. FlyteWorkflow CR manually deleted).

What if we do not do this?

If we do nothing, then these metrics are often inaccurate and therefore useless.

Related component(s)

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

🤖

Hello @hamersaw!

I'm Dosu, a bot designed to assist you with bugs, answer your queries, and help you journey through the world of open-source contribution. While waiting for a human maintainer, I'm here to make sure your issue doesn't go unnoticed.

I'm taking a quick look at your issue regarding the ActiveNodeExecutions and ActiveTaskExecutions metrics. We'll delve into it in a moment. Thanks for your patience!

[hamersaw]

One proposal, is to move these metrics to FlytePropeller so they can more accurately reflect actual execution status. A relatively high-level design is as follows:

• Add NodeExecutionCount and TaskExecutionCount values to the ExecutionContext through the ControlFlow struct and increment these accordingly as FlytePropeller progresses through DAG execution (entrypoint here).
• Maintain an in-memory mapping of execution_id to node and task execution counts that is updated as FlytePropeller evaluations executions. This update will in-turn update the prometheus gauge metric.
• Use a separate go routine to periodically iterate over the in-memory mapping and remove executions that have been manually deleted (decrementing the gauge metric).

With this solution the metric under FlytePropeller restarts will be eventually consistent. Although, as the workflow reeval duration is, by default, 30s it should converge within that timeframe.

[Future-Outlier]

@hamersaw Are you working on this issue?
If not, can I try it?

[hamersaw]

@Future-Outlier this is already being addressed, will assign accordingly. Thanks for the initiative!