Configurable TimeMonitor interval

[boaz-wiesner]

At the moment, the TimeMonitor interval is hardcoded to 1 minute. In druid clusters with many datasources, running a full cycle of time checks can take longer than a minute, and lead to continuous polling of the cluster, which isn't optimal.

Moving this to a cluster level configurable parameter would definitely solve the problem.

The relevant line seems to be:

turnilo/src/common/utils/time-monitor/time-monitor.ts

Line 36 in f8c7af2

this.regularCheckInterval = 60000;

Thanks!

[boaz-wiesner]

Another default that is currently hardcoded is the default "time" dimension (timeAttribute). Introspected sources end up using this default, and currently there's no way to configure it.

Allowing this default to be changed cluster-wide would enable automatic introspection of datasources not using __time as their time dimension. At the moment, if you have an introspected source that uses a different dimension, it won't work.

turnilo/src/common/models/data-cube/data-cube.ts

Line 291 in a78295a

const timeAttribute = $("__time");

[adrianmroz-allegro]

I think cluster level property in config for regularCheckInterval sounds good. As always we need to figure out good name. sourceMaxTimeRefreshInterval? MaxTime is turnilo centric, that how we talk about it in codebase. Maybe there is better term in Druid lingo? @mkuthan wanna weigh in? sourceTimeBoundaryRefreshInterval?

[adrianmroz-allegro]

Another default that is currently hardcoded is the default "time" dimension (timeAttribute).

That's deliberate. To my understanding __time attribute name is required by Druid. It is a special dimension used to partition data. In theory you could have additional "time" dimensions but turnilo can't work with them, we handle only one time dimension that's partitioning column.

[boaz-wiesner]

As always we need to figure out good name. sourceMaxTimeRefreshInterval? MaxTime is turnilo centric, that how we talk about it in codebase. Maybe there is better term in Druid lingo? sourceTimeBoundaryRefreshInterval?

I'm for something like timeBoundaryQueryInterval or sourceTimeBoundaryRefreshInterval, but as long as the name is documented then I don't have a strong opinion one way or the other.

That's deliberate. To my understanding __time attribute name is required by Druid

Yep, my bad - we use alternative fields for additional partitioning and applicative uses, but for Turnilo's use case of running a TimeBoundary query it should be enough.