kube-events goes into Completed state without RBAC

[solsson]

Remaining issue from #92.

There's no meaningful error message. Pod start looks like:

%7|1513582363.494|BROKER|rdkafka#producer-1| [thrd:app]: bootstrap.kafka:9092/bootstrap: Added new broker with NodeId -1
%7|1513582363.494|TOPIC|rdkafka#producer-1| [thrd:app]: New local topic: ops.kube-events-all.stream.json.001
%7|1513582363.494|TOPPARNEW|rdkafka#producer-1| [thrd:app]: NEW ops.kube-events-all.stream.json.001 [-1] 0x56108fa42840 (at rd_kafka_topic_new0:282)
%7|1513582363.498|BRKMAIN|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: Enter main broker thread
%7|1513582363.498|CONNECT|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: broker in state INIT connecting
%7|1513582363.499|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1513582363.499|STATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Broker changed state INIT -> UP
%7|1513582363.514|DESTROY|rdkafka#producer-1| [thrd:app]: Terminating instance
%7|1513582363.514|DESTROY|rdkafka#producer-1| [thrd:main]: Destroy internal
%7|1513582363.514|DESTROY|rdkafka#producer-1| [thrd:main]: Removing all topics
%7|1513582363.514|TOPPARREMOVE|rdkafka#producer-1| [thrd:main]: Removing toppar ops.kube-events-all.stream.json.001 [-1] 0x56108fa42840
%7|1513582363.514|DESTROY|rdkafka#producer-1| [thrd:main]: ops.kube-events-all.stream.json.001 [-1]: 0x56108fa42840 DESTROY_FINAL
%7|1513582363.514|TERM|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Received TERMINATE op in state UP: 1 refcnts, 0 toppar(s), 0 fetch toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1513582363.561|CONNECT|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: Connecting to ipv4#10.3.252.81:9092 (plaintext) with socket 7
%7|1513582363.561|STATE|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: Broker changed state INIT -> CONNECT
%7|1513582363.561|TERMINATE|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1513582363.561|BROKERFAIL|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: Operation now in progress)
%7|1513582363.561|STATE|rdkafka#producer-1| [thrd:bootstrap.kafka:9092/bootstrap]: bootstrap.kafka:9092/bootstrap: Broker changed state CONNECT -> DOWN
%7|1513582363.614|TERMINATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Handle is terminating: failed 0 request(s) in retry+outbuf
%7|1513582363.614|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: Success)
%7|1513582363.614|STATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Broker changed state UP -> DOWN

To add RBAC, apply recursively: kubectl apply -R -f events-kube/

I found no obvious solution to how to print curl's errors. Do they end up in the topic?

[solsson]

Actually the test events-topic is quite unhelpful too. To pass, it requires a fresh event every minute. This means that when everything runs perfectly and you're not working on anything, you get an alert for events-kube not working. An effort to raise test quality based on #116 should probably include this one.

[solsson]

The test needs to trigger some event in order to be useful. But with abe3491 it might be useful in real clusters. Very annoying during setup though.

[solsson]

Did some searching for alternative approaches today, as it will be difficult to get at-least-once using this curl-based solution.

• https://github.com/heptio/eventrouter

• https://github.com/kubeless/kubeless/tree/master/docker/event-sources/kubernetes

• https://github.com/operator-framework/operator-sdk
  https://banzaicloud.com/blog/operator-sdk/