Sink is not able to use kubernetes service

[nitishkumar71]

I am trying to deploy sink inside a kubernetes cluster. Kubernetes cluster host both kafka and clickhouse but in different namespace. I am actuallly using strimizi to deploy kafka cluster and kafka connect. Both the kafka and connect cluster are being deployed properly.

Sink is being loaded properly and class is being detected com.clickhouse.kafka.connect.ClickHouseSinkConnector

Here is the yaml config for connector

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: clickhouse-sink-connector
  labels:
    strimzi.io/cluster: clickhouse-sink-connect-worker
spec:
  class: com.clickhouse.kafka.connect.ClickHouseSinkConnector
  tasksMax: 1
  config:
    ssl: false
    client_version: v2
    timeoutSeconds: 60
    retryCount: 10
    topics: activity
    hostname: cluster-demo-cluster.clickhouse.svc.cluster.local
    database: demodb
    password: password
    username: admin
    port: 8123
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter: org.apache.kafka.connect.storage.StringConverter
    errors.retry.timeout: 60

But the connector is throwing error Connection to ClickHouse is not active. Here is the full error

2025-01-12 12:08:12,669 INFO [clickhouse-sink-connector|task-0] Using new http client implementation (com.clickhouse.client.api.Client) [task-thread-clickhouse-sink-connector-0]                              
2025-01-12 12:08:12,670 INFO [clickhouse-sink-connector|task-0] ClickHouse URL: http://cluster-demo-cluster.clickhouse.svc.cluster.local:8123/demodb? (com.clickhouse.kafka.connect.sink.db.helper.ClickHouseHelperClient) [task-thread-clickhouse-sink-connector-0]                                                                                                                                             
2025-01-12 12:08:12,765 INFO 10.244.1.3 - - [12/Jan/2025:12:08:12 +0000] "GET /connectors/clickhouse-sink-connector/config HTTP/1.1" 200 469 "-" "-" 5 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp2
85723608-31]                                                                                                                                                                                                   
2025-01-12 12:08:12,790 WARN [clickhouse-sink-connector|task-0] Ping retry 1 out of 3 (com.clickhouse.kafka.connect.sink.db.helper.ClickHouseHelperClient) [task-thread-clickhouse-sink-connector-0]           
2025-01-12 12:08:12,794 WARN [clickhouse-sink-connector|task-0] Ping retry 2 out of 3 (com.clickhouse.kafka.connect.sink.db.helper.ClickHouseHelperClient) [task-thread-clickhouse-sink-connector-0]           
2025-01-12 12:08:12,798 WARN [clickhouse-sink-connector|task-0] Ping retry 3 out of 3 (com.clickhouse.kafka.connect.sink.db.helper.ClickHouseHelperClient) [task-thread-clickhouse-sink-connector-0]           
2025-01-12 12:08:12,798 ERROR [clickhouse-sink-connector|task-0] Unable to ping ClickHouse instance. (com.clickhouse.kafka.connect.sink.db.helper.ClickHouseHelperClient) [task-thread-clickhouse-sink-connecto
r-0]                                                                                                                                                                                                           
2025-01-12 12:08:12,799 ERROR [clickhouse-sink-connector|task-0] Unable to ping Clickhouse server. (com.clickhouse.kafka.connect.sink.db.ClickHouseWriter) [task-thread-clickhouse-sink-connector-0]           
2025-01-12 12:08:12,800 ERROR [clickhouse-sink-connector|task-0] WorkerSinkTask{id=clickhouse-sink-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover u
ntil manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-clickhouse-sink-connector-0]                                                                                                
java.lang.RuntimeException: Connection to ClickHouse is not active.                                                                                                                                            
    at com.clickhouse.kafka.connect.sink.ProxySinkTask.<init>(ProxySinkTask.java:63)                                                                                                                           
    at com.clickhouse.kafka.connect.sink.ClickHouseSinkTask.start(ClickHouseSinkTask.java:40)                                                                                                                  
    at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:324)                                                                                                             
    at org.apache.kafka.connect.runtime.WorkerTask.doStart(WorkerTask.java:176)                                                                                                                                
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:225)                                                                                                                                  
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)                                                                                                                                    
    at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)                                                                                                           
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)                                                                                                                       
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)                                                                                                                                      
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)                                                                                                               
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)                                                                                                               
    at java.base/java.lang.Thread.run(Thread.java:840) 

Please note, i tried to curl endpoint http://cluster-demo-cluster.clickhouse.svc.cluster.local:8123/ping from same pod i get Ok. So it means clickhouse service is reachable.

Also, I am able to connect to clickhouse cluster using credentials provided with command kubectl exec -n clickhouse -it <POD_NAME> -- clickhouse-client --user admin --password password

Edit

Used the HTTP interface using below command and it works

echo 'SELECT 1' | curl -H 'X-ClickHouse-User: admin' -H 'X-ClickHouse-Key: password' 'http://cluster-demo-cluster.clickhouse.svc.cluster.local:8123/?database=demodb' -d @-

But when tried with clickhouse URL which is generated in sink, it did not work and throws a 404 status with error There is no handle /demodb

echo 'SELECT 1' | curl -H 'X-ClickHouse-User: admin' -H 'X-ClickHouse-Key: password' 'http://cluster-demo-cluster.clickhouse.svc.cluster.local:8123/demodb' -d @-

[nitishkumar71]

Nothing wrong with config here. Recreating minikube cluster solved the issue.