Skip to content
This repository has been archived by the owner on May 2, 2023. It is now read-only.

Getting Unkown error when installing MQTT Connector with confluent-hub install behind a firewall #895

Open
langalibalele opened this issue Feb 4, 2021 · 0 comments

Comments

@langalibalele
Copy link

I expanded the connect container in docker-composite.yml in order to install, create and deploy the MQTT Connector (kafka-connect-mqtt):
[see Apache Kafka / Kafka Connect / MQTT / Mosquitto Live Demo]

` command:
- bash
- -c
- |
# Install connector plugins
# This will by default install into /usr/share/confluent-hub-components/ so make
# sure that this path is added to the plugin.path in the environment variables
confluent-hub install --no-prompt confluentinc/kafka-connect-mqtt:latest
# Launch the Kafka Connect worker
/etc/confluent/docker/run &

    echo "Waiting for Kafka Connect to start listening on localhost:8083 ⏳"
    while : ; do
        curl_status=$$(curl -s -o /dev/null -w %{http_code} http://localhost:8083/connectors)
        echo -e $$(date) " Kafka Connect listener HTTP state: " $$curl_status " (waiting for 200)"
        if [ $$curl_status -eq 200 ] ; then
        break
        fi
        sleep 5
    done
    echo -e "\n--\n+> Creating Kafka Connect source connectors"
    # Create a Kafka Topic for consuming the MQTT messages via the MQTT Connector (needs to be the same as in the Connector Config):
    #kafka-topics --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic mqtt.temperature
    #curl -s -X POST -H "Content-Type:application/json" --data @/c/temp/mqttconnector/mqtt-connector-config.json http://connect:8083/connectors
    echo -e "\n--\n+> config mqtt connector"
    curl -s -X POST -H 'Content-Type: application/json' http://localhost:8083/connectors -d '{    "name" : "mqtt-source",	"config" : {		"connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector",		"tasks.max" : "1",		"mqtt.server.uri" : "tcp://10.230.145.42:1883",		"mqtt.topics" : "temperature",		"kafka.topic" : "mqtt.temperature",		"io.confluent.topic.bootstrap.servers": "broker:29092",		"io.confluent.topic.replication.factor": "1",		"io.confluent.license":""    }}'
    echo -e "\n--\n+> test mqtt connector"
    #kafka-topics --create --bootstrap-server broker:29092 --replication-factor 1 --partitions 1 --topic mqtt.temperature
    #kafka-console-consumer --bootstrap-server broker:29092 --topic mqtt.temperature --property print.key=true --from-beginning
    #curl -s -X POST -H 'Content-Type: application/json' http://connect:8083/connectors -d
    
    # Don't exit
    sleep infinity`

When directly conected to the internet, docker-compose up -d yields correct installation under the conect container:

Downloading component Kafka Connect MQTT 1.4.0, provided by Confluent, Inc. from Confluent Hub and installing into /usr/share/confluent-hub-components Adding installation directory to plugin path in the following files: /etc/kafka/connect-distributed.properties /etc/kafka/connect-standalone.properties /etc/schema-registry/connect-avro-distributed.properties /etc/schema-registry/connect-avro-standalone.properties Completed Waiting for Kafka Connect to start listening on localhost:8083 ⏳ .... Thu Feb 4 19:44:40 UTC 2021 Kafka Connect listener HTTP state: 000 (waiting for 200)

When connecting through our company firewall, calling docker-compose up -d yields the follwing in the connect container:
Downloading component Kafka Connect MQTT 1.4.0, provided by Confluent, Inc. from Confluent Hub and installing into /usr/share/confluent-hub-components javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Error: Unknown error Waiting for Kafka Connect to start listening on localhost:8083 ⏳

I have not found a solution to this yet. Apparantly adding the *.confluent.io cert to java cacerts does not resolve the issue. I suspect the firewall is blocking download of the jar files in \lib of the connector

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant