KSQL quickstart for Kubernetes

[solsson]

The sequel to #68, using confluent's images because I failed to build 0.5 using earler Dockerfiles. No luck with starting ksql-cli in k8s yet though.

I'd like to set up the equivalent of https://github.com/confluentinc/ksql/tree/master/docs/quickstart.

@pavel-agarkov @clintfred I've tried to refresh #68 with a new build of ksql but the old build steps failed on various test crashes. I'd happily switch to confluent's images, but my quick attempt here fails to start ksql and fails to get any logs out of the failed container. Unfortunately I don't have more time to spend on this in the near future. I thought converting the docker-compose.yml in quick start would be a no-brainer.

I set up a fresh kafka + avro-tools from master with no tweaks at all.

[solsson]

If I exec ksql-cli local --botstrap-server $STREAMS_BOOTSTRAP_SERVERS --schema-registry-url http://$STREAMS_SCHEMA_REGISTRY_HOST:$STREAMS_SCHEMA_REGISTRY_PORT I get the error message Found unexpected parameters: [--botstrap-server, bootstrap.kafka:9092] which is odd because ksql-cli help local and the quickstart lists those arguments.

[solsson]

Did some catching up, watched https://www.confluent.io/kafka-summit-london18/ksql-201-a-deep-dive-into-query-processing. KSQL is highly interesting because unlike the other streaming platform tools it is

• Language neutral - REST API mode
• Supports JSON messages (actually supports it, as in SELECT my_json_prop)

I really need to get this up and running. The experimental web api sounds interesting too. CLI isn't of much use in kubernetes.

Edit: my player wouldn't give me a link, but at around 25:00 in the video there's a very enlightening description of KSQL as abstraction on top of Kafka Streams.

[solsson]

There's an amazing infinite loop in the docs on how to download KSQL. I fail to find a way to download a compiled package. The confluent platform download requires an e-mail registration, so I wouldn't want to redistribute it in docker.

Edit: In fact the loop wasn't infinite. After a few rounds I understood that one instruction is the naming of individual deb packages. Result: StreamingMicroservicesPlatform/docker-kafka#4

[pavel-agarkov]

If I understood it right then KSQL Rest API is not for production use.
It is just a way to experiment with it or write at hoc queries on dev environment.
For production use case it is recommended to disable Rest Api and to create separate KSQL docker image with injection of you custom queries/streams file. And there is no way to update this file without stoping KSQL...
This opinion is based on the conversation with Viktor Gamov (Confluent solution architect) and can be my misunderstanding.
I'm not sure yet but I might prefer to use a simple consumer and do projections and joins by my own...
I would be glad to be wrong!

[solsson]

@pavel-agarkov I too would be glad if you're wrong :) https://docs.confluent.io/current/ksql/docs/installation/server-config/index.html#non-interactive-headless-ksql-usage may indicate that you're right though. Also "proper response codes" in https://docs.confluent.io/5.0.0-beta1/release-notes.html#ksql indicate that 4.x is immature wrt REST.

[solsson]

Actually experiments using CLI + production as a ConfigMap with mounted .sql files isn't too bad, is it? I'd use #175 to mirror to a test cluster and develop queries there. But... the language neutral part... how do we consume the results?

Edit: Result topics and regular consumers would be practical enough for most of our use cases.

[pavel-agarkov]

Yes, to consume results the regular consumers can be used to read from the topics created by KSQL.
The main problem with mounted files from my point of view is that I have to put a lot of different scripts in one file or I would need a new KSQL instance per script. With the first way I cannot update some script without interrupting the others in the same file (potentially).
Probably there is a way to use multiple files per instance. I haven't investigated much yet. Still working with a front-end :)

[solsson]

Maybe we could come up with something similar to how https://github.com/coreos/prometheus-operator combines rules files into a single config? Could start as a client side script instead of an operator.

I'm not sure KSQL will be valuable to us, an organization where only programmers use Kafka. As an abstraction on top of Kafka Streams, KSQL can only have a subset or equal feature set, so we could just as well use java. Two potential advantages:

• Faster iterations (and possibly innovation) by not having to build and deploy streams apps.
• Streaming the result of SELECT over HTTP

Because https://www.confluent.io/blog/introducing-confluent-platform-preview-releases/ says "We encourage developers to build applications against the API shipped in this preview" I've updated the docker image to 5.0.0-beta1.

[clintfred]

I did a little experimenting with KSQL using

https://hub.docker.com/r/confluentinc/cp-ksql-cli/
confluentinc/cp-ksql-cli:5.0.0-beta30

https://hub.docker.com/r/confluentinc/cp-ksql-server/
confluentinc/cp-ksql-server:5.0.0-beta30

The experiment was done using a single node Kafka cluster, hosted in minikube on my local machine. Two KSQL processes were started outside of minikube, in docker. Both the brokers and schema registry were configured to make their IPs accessible outside of the cluster using #160.

Maybe I missed something, but the ksql-rc branch only seems to have ksql-cli and not ksql-server, so what's why I just ran the images in docker. The dockerhub images worked great and seem like they would be easy to wrap into deployments. I think having the CLI deployable for debugging would be good.

[solsson]

I think Confluent recently started shipping -server docker images. Did you try updating the image and entrypoint in ksql-rc?

[jeffbeagley]

I was able to get the server spun up utilizing the following (not sure if this is good practice or not, I'm very new to K8 but wanted to get this fired up to test with)

https://gist.github.com/jeffbeagley/7934fb90d3778519914ccd4521da2ca4