Problems with extending StreamPipes with a new Data Processor

[KarenjanM]

Hello,
I am getting familiar with StreamPipes and I wanted also to try extending it with a new Data Processor as it stays in documentation. I did everything according to documentation:

• Created a project with the corresponding Maven archetype, added all needed environment variables as EnvFile in IntelliJ and launched it.
  It all seems to work, because I can go to localhost:8005 and find there links for the example processor and sink, also with appropriate IP-address. However when I try to start a pipeline with this ExampleProcessor I get 400 Bad Request error.

Logs from backend:
streampipes-backend-1 | 2023-07-12T07:34:48.357Z ERROR 1 --- [nio-8030-exec-6] o.a.s.m.e.http.InvokeHttpRequest : Could not perform invocation request at http://172.27.30.183:8005/sepa/deep5.pe.example.processor for pipeline element ExampleProcessor: status code: 400, reason phrase: Bad Request
streampipes-backend-1 | 2023-07-12T07:34:48.358Z INFO 1 --- [nio-8030-exec-6] o.a.s.m.e.http.InvokeHttpRequest : Invoking element: http://172.31.0.7:8090/sec/org.apache.streampipes.sinks.internal.jvm.datalake
streampipes-backend-1 | 2023-07-12T07:34:48.369Z INFO 1 --- [nio-8030-exec-6] a.s.m.e.h.InvokePipelineElementSubmitter : Could not start pipeline, initializing rollback...
streampipes-backend-1 | 2023-07-12T07:35:00.962Z INFO 1 --- [pool-3-thread-1] o.a.s.m.m.p.ExtensionsServiceLogExecutor : Could not fetch log info from endpoint http://172.27.30.183:8005

Also I get some not critical errors in ExampleProcessor itself:
Jul 12, 2023 9:56:56 AM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
09:57:00.485 SP [OkHttp http://localhost:8500/...] ERROR com.orbitz.consul.cache.ConsulCache - Error getting response from consul for health.service "core", will retry in 10000 MILLISECONDS
com.orbitz.consul.ConsulException: Consul cluster has no elected leader
at com.orbitz.consul.cache.ConsulCache$1.onComplete(ConsulCache.java:180)
at com.orbitz.consul.util.Http$1.onResponse(Http.java:80)
at retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:161)
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
09:57:01.573 SP [OkHttp http://localhost:8500/...] ERROR com.orbitz.consul.cache.ConsulCache - Error getting response from consul for health.service "ext", will retry in 10000 MILLISECONDS
com.orbitz.consul.ConsulException: Consul cluster has no elected leader
at com.orbitz.consul.cache.ConsulCache$1.onComplete(ConsulCache.java:180)
at com.orbitz.consul.util.Http$1.onResponse(Http.java:80)
at retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:161)

I would be very thankful, if somebody helps me with the problem

[dominikriemer]

Hi @KarenjanM, happy to help.
Which version of StreamPipes are you running?
Can you please check and reply which version is your StreamPipes Core version (click the user icon in StreamPipes and then info>system info) and which version is defined in your pom file?

Have you started the StreamPipes core components with the CLI or any other deployment option?

[KarenjanM]

Thank you for such a quick answer, @dominikriemer !
I start StreamPipes with CLI in lite environment
The version of StreamPipes Core is 0.92.0
The version in pom file is 0.69.0. I tried changing it to 0.92.0, but it didn't change the situation

[dominikriemer]

Thanks for the information.
The pom version of your extensions file should match the StreamPipes version of the core, 0.92.0. The Consul warning should disappear after updating the version.

Concerning the error upon pipeline start, I suggest to try the following:

• Try to start a pipeline from the pre-installed elements (e.g., connect a machine simulator to the data lake sink) - does this work or do you also get an error?
• If this is working, I think that the core (running in Docker) cannot properly access your locally running pipeline element (which seems to use the IP 172.27.30.183). The IP is usually auto-discovered when you start your custom service. Sometimes the auto-discovery chooses an IP which cannot be accessed from within the Docker network. You can try so by pinging to this IP from the core's Docker container. If you can't reach the IP from there, you can start your extensions service by manually providing an IP or hostname. This is done by providing an environment variable SP_HOST (to the extensions service you are developing).

Please answer in case these ideas don't work for you.

[KarenjanM]

I tried your solutions:

• After I changed the version of StreamPipes Core to 0.92.0 in my pom file Consul problem did not go away
• The pipeline from pre-installed elements works just fine
• Initially I set ip for my service manually with SP_HOST=172.27.30.183, but without this env.variable service auto-discovers the same ip
• I tried to ping 172.27.30.183 from backend container and it showed no problems

Docker container has following ip-address: 172.31.0.6
Could it be a problem, that it has different address? I tried changing with SP_HOST=172.31.0.6, but it didn't work at all

[dominikriemer]

Ok - thanks for trying.
If the service can be reached then I suggest we take a look for the Bad Request error code, which could come previously installing an older version. Can you go to Install Pipeline Elements in the UI and either delete/reinstall or update your custom pipeline element there?

Afterwards, delete and recreate the pipeline.

[KarenjanM]

I reinstalled it and created a new pipeline - still Bad Request.
How can I reinsure that my custom service updated dependencies, after I changed Core version in pom file? ( it has still the Consul problem - may be it didn't get updated after a change )

[dominikriemer]

You should be able to see the version in IntelliJ's Maven settings (see screenshot below, my example uses 0.93.0-SNAPSHOT).

As you say it, there is a good chance that the dependencies didn't update since I think the archetype code has changed quite a bit since v0.69.0.

So you could also try to create a new Maven archetype with the latest version 0.92.0 or you modify your pom file so that it matches the versions and dependencies of the 0.92.0 archetype version.

[KarenjanM]

Thank you very much.
I just created new project with 0.92.0 Core version and it has finally worked.
It's just that in documentation stays that the latest available version of Maven archetype is 0.69.0, that's why I never even tried to install newer version

[dominikriemer]

Thanks for the feedback! We'll update the documentation accordingly.