Kafka schema guessing timeout error

[mihau1211]

Hi guys, I have an issue reading data from the Kafka topic.

The thing is that I have Kafka connected to the Kafka Adapter, all topics are listed in the Adapter configuration, but when it comes to configuring fields, I receive a 'Read timed out' error. I'm using 'connect-configs' to check if it will work; there is definitely a single JSON object with 3 fields, so there is no chance of a lack of data. It was also checked with a topic with some data from SQL (JDBC connector data).

I've tried all JSON settings, and nothing works for me.

I'm using StreamPipes 0.93.0 and Kafka 3.6.0.

Timeout error message:

[tenthe]

Hi @mihau1211,
Thank you for your detailed description. Actually, everything you have described looks good to me. I don't see the problem yet, so here are a few more questions:

• If I understand you correctly, the topics are recognized correctly, right?
• While the schema is supposed to be recognized new events are published on the topic? (To recognize the schema, the system needs to receive a new event)
• Do other adapters (e.g. the machine data simulator) work with your instance?
• Can you possibly send an example json to see if there could be a problem.

Cheers,
Philipp

[mihau1211]

Hi @tenthe,
thanks for responding. Answering your questions:

1. Yes, topics are recognized correctly.
2. Firstly, I was just waiting to recognize the schema, now I also checked if publishing new records on the topic would help, but it didn't. Still the same issue.
3. Yes, checked the data simulator and some custom adapters and they were working fine.
4. Yes, here you go:
   { "schema": { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "int64", "optional": true, "name": "org.apache.kafka.connect.data.Timestamp", "version": 1, "field": "timestamp" }, { "type": "string", "optional": true, "field": "name" } ], "optional": false, "name": "test" }, "payload": { "id": 1, "timestamp": 1701875846033, "name": "name1" } }

Additionally, I'm using the new topic that is produced by Kafka Connect JDBC Source connector.

EDIT:

In the first post, I didn't notice that it was a different error than usual. Most time I receive that kind of error:

[mihau1211]

Update:

The issue seems to be caused by missing the guess schema endpoint. In the backend container logs, there was a line saying that there was a request for http://172.31.0.6:8090/api/v1/worker/guess/schema, but I couldn't find it on http://172.31.0.6:8090 docs. Accessing the endpoint through the browser returns Method Not Allowed.

I found also a similar issue on the repository but it was closed without a solution: 316

[tenthe]

Hi @mihau1211,

first of all, your json event looks good. I tested it with the file stream adapter and it worked.
Maybe a little hint. Your original event contains schema and payload information. In StreamPipes, we mainly focus on events that contain payload key/value pairs in a flat structure.
To transform the event schema, you can drag and drop the nested event properties to the top level and remove the other properties. See the screenshots below.

As for the Kafka error, I need to investigate it further to better understand the problem. It seems that the adapter is not able to read the events from the Kafka theme.

Original event schema

Move properties with drag and drop

Resulting event schema & event

[dominikriemer]

Hi @mihau1211 the Method Not Allowed message should be ok since the endpoint only support POST requests, and the browser usually performs a GET.

Thanks for the detailed description of the error you get - one thing which could be worth to check is the Kafka configuration. Sometimes, when the metadata discovery works (which seems to be the case since you get the list of topics), but data cannot be retrieved (which happens when the connect module starts to "guess" the schema, this is due to Kafka settings missing an advertised_listener which is accessible from inside the Docker network.

Could this be the case? We can try to figure out where you're running Kafka and check that the broker address returned by Kafka is reachable from the StreamPipes container.

Here is some information on that:
https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

Just a guess...if it's another problem, we can investigate further ;-)

[mihau1211]

Thanks a lot for your time guys, I only changed the advertised_listener property in Kafka server.properties and it started to work. It was really strange because I was able to connect to the Kafka server from Streampipes containers via curl.