Problem starting a MQTT adapter

[leniervm]

Hello,

I have high expectations for the integration of StreamPipes into my project, but I am encountering a particularly frustrating issue. I am utilizing Apache StreamPipes version 0.95.1 in a Docker-based setup, employing the minimal compose file which runs the backend, UI, CouchDB, InfluxDB, and extensions-iiot-minimal containers. However, whenever I attempt to start the MQTT connector with a correct and validated schema, the adapter displays a "Read timeout" error. I have verified the connection using a test broker, but the issue appears to be related to reaching the subnet at container ip:8090.

Adapter creation:

Adapter list:

Error:

Full detail:

Transcription: org.apache.streampipes.commons.exceptions.connect.AdapterException: Adapter was not started successfully with url http://172.31.0.4:8090/api/v1/worker/stream/invoke

Env file config with subnet default value:

To check the Docker subnet, I used the command docker inspect streampipes_spnet and obtained the following information:

The subnet IP 172.31.0.4 corresponds to the extensions-iiot-minimal container or any container replacing it, such as extensions-all-jvm. I have tested both configurations with the same result: the adapter attempts to reach the container IP at port 8090 via its API but times out. As I have not found any topics related to this issue, I am requesting your support to resolve this problem and ensure the application runs correctly.

When I run docker compose down, the extensions-iiot-minimal container continues to run for an unexplained reason, and I must stop it manually:

Thank you very much.

[dominikriemer]

Hi @leniervm that's a strange error - can you please post the full stack trace of the error you get when starting the adapter?

[leniervm]

Thank you for you support @dominikriemer. I have tested in a cloud VPS and my local machine with Ubuntu 22.04 with same results. The last point that refers to docker compose down was corrected just changing the compose file to docker compose -f docker-compose.minimal.yml down what is actually using.

[dominikriemer]

Thanks @leniervm!
I had a look at the setup - I think the problem is that the docker-compose.minimal.yml only includes Nats as a message broker, while you chose mqtt as SP_PRIORITIZED_PROTOCOL (which defines to use MQTT as an internal broker). So I guess the adapter tries to send data to MQTT due to the config but can't reach the broker and no data is sent.

You can either try setting SP_PRIORITIZED_PROTOCOL to nats which will then use Nats as an internal messaging system. If you want to use MQTT, you can add an MQTT broker to the internal docker network. The default connection settings are mosquitto as hostname and 1883 as port (can be overriden in the UI under Configuration -> Messaging -> Messaging Layers). The internally used MQTT broker currently doesn't support authenticated connections, so the port shouldn't be mapped.

Both settings are considered setup-time configurations and don't affect existing resources, so you might need to create new adapters or start from a plain system.

Thanks for testing this and please ask if you don't think this is the actual problem you're encountering.

[leniervm]

Thank you, @dominikriemer, for the clear information. I will attempt the implementation today. I have a functioning MQTT broker, either EMQX or Mosquitto, that I can add to the Docker subnet. However, my question is: What are the minimum requirements to obtain a functional MQTT adapter? It is my understanding that this adapter should be able to communicate with any internal or external broker.

[dominikriemer]

@leniervm your understanding is correct, we are just talking about two different concepts:

• You can create an MQTT adapter or sink from any internal or external broker. When you create a new adapter, the source (e.g., the MQTT broker you're creating an adapter for) data is fetched from the broker, sent through the internal pre-processing pipeline and than sent to the internal StreamPipes messaging system for downstream computations.
• The internal system is used for routing data between data streams, processors and sinks. This internal system can also be configured and can be selected from one of the supported brokers (Kafka, Pulsar, MQTT, Nats).

So the easiest way to solve your problem is to rely on the default configuration in the docker-compose.minimal.yml file and setting SP_PRIORITIZED_PROTOCOL to nats. You will still be able to create MQTT adapters and sinks from any broker which is reachable from the extension service, the only difference is that internal messages will be sent over the internal Nats broker.

Hope this helps!

[leniervm]

Successful result, @dominikriemer. The issue was that I had commented out the NATS service in the docker-compose.minimal.yml file with a # before understanding the internal flow of the application. Additionally, I believe there is an error in how the backend service is called:

# StreamPipes services backend: image: "${SP_DOCKER_REGISTRY}/backend-nats:${SP_VERSION}" depends_on: - couchdb environment: - SP_PRIORITIZED_PROTOCOL=nats volumes: - backend:/root/.streampipes logging: *default-logging restart: unless-stopped networks: spnet:

Specifically, the $SP_VERSION in the .env files is set to 0.95.1, but on Docker Hub, it appears as apachestreampipes/backend-nats:0.95.0-SNAPSHOT. We need to manually modify this to apachestreampipes/backend:0.95.1. Also, in the extensions-iiotminimal file, there is a depends-on attribute without a list, which we need to correct. Because We can run the compose file.

extensions-iiot-minimal: image: "${SP_DOCKER_REGISTRY}/extensions-iiot-minimal:${SP_VERSION}" depends_on: logging: *default-logging restart: unless-stopped networks: spnet:

Lastly, as you mentioned, changing - SP_PRIORITIZED_PROTOCOL=nats is the best choice for me due to the lightweight nature of the internal communication broker.

Thanks a lot for your support, have a great day!