Bootstrap an Apache Druid operator

[lfrancke]

Implement initial Druid Operator for all Server-/Process Types (https://druid.apache.org/docs/latest/design/processes.html) (ACs: )

Acceptance Criteria

• Operator can start/stop/restart a Druid Cluster

• Druid configs can be applied and updated

• Monitoring is integrated

• all Process types are supported (Coordinator, Overlord, Broker, Historical, MiddleManager and Peons, Indexer (Optional), Router (Optional)

• all Server types are supported (Master, Query, Data)

• support Maturity Level 1 (Is there more todo than in AC 1?)

tbd

• ships with license compatible JDBC driver for S3 (Does this really apply? Maybe not necessary (https://druid.apache.org/docs/latest/ingestion/native-batch.html#s3-input-source)

[stefanigel]

Jim tries to figure out if JDBC driver is needed - most probably not needed!

[Jimvin]

There's no suggestion of needing a JDBC driver for S3 support from what I can see in the documentation link.

[fhennig]

Note: "Druid officially supports Java 8 only. Support for later major versions of Java is currently in experimental status."

zookeeper is a dependency

in a production setting a SQL DB is needed: https://druid.apache.org/docs/latest/tutorials/cluster.html#metadata-storage

[fhennig]

Each process (Coordinator, Overlord, Broker, Historical, MiddleManager and Peons, Indexer (Optional), Router (Optional)) will be scheduled as a separate pod. "Server grouping" (Master, Query, Data) can be achieved by pod affinity so the coordinator and overlord get scheduled together on the same node.

[fhennig]

Things that are missing:

• [ ] indexer
• peons (althrough it looks like that looks like it is internal only)
• [ ] standalone overlord (See Standalone overlord #12)
• SQL config
• zookeeper discovery (zookeeper config properties, more info)
• [ ] pod affinity to model the master/query/data setup
• druid.host needs to be set dynamically. This is required so the services can find each other

What works so far:
The "micro quickstart" setup can be deployed with a coordinator/overlord, a broker, middlemanager, historical and router.

[fhennig]

jvm config:
Most options are common to all processes, but the memory allocation differs and should be configurable. There are two settings, the heap size and the direct memory size. heap size is relevant to all processes, and direct memory only to the Broker, Historical and Router.

[fhennig]

A lot of runtime properties are buffer sizes, thread counts, memory allocation etc. As a user, I wouldn't really care about all that, it should just be matching to my hardware.

Druid supplies sample configs that are called 'micro, small, medium, large, xlarge' etc, with different sizes.

The numbers also need to be balanced. This is quite a complex thing to configure without prior knowledge and it would be nice to be able to pre-populate these settings. Also, if they are all mapped out into the CRD, it's a lot of settings. Not quite sure what to do yet.

[fhennig]

The indexer was spun out into a seperate issue #8

[fhennig]

I've put the pod affinity in a seperate issue as well #9 , I believe the pod affinity is not critical to a minimal working version

[fhennig]

Druid configs can be applied and updated

a cluster config can be changed and with a restart the configs will be applied too