Error: ConnectError: [unknown] network error on the package execution logs screen

[leoporoli]

What's your CLI version?

0.86.18

Description & steps to reproduce

Execute the MERN package in the cloud and you will see this error in the logs:

Error: ConnectError: [unknown] network error

And you can see this error in the browser's console

Failed to load resource: net::ERR_HTTP2_PROTOCOL_ERROR

Desired behavior

We should see all the package's logs on the screen

What is the severity of this bug?

Painful; this is causing significant friction in my workflow.

What area of the product does this pertain to?

Frontend: the Graphical User Interface, including the Enclave Manager

[mieubrisse]

I just hit this today while trying to give a demo

[leoporoli]

we are discussing this issue here: https://kurtosistech.slack.com/archives/C05DZ4UABDZ/p1708114238397159

it could be related to this one: #1768

another possible relation: connectrpc/connect-es#907

[leoporoli]

It seems there is a timeout somewhere because it always fails when the call lasts more than 1 minute:

And it seems the timeout came from Connect http client when calling to the GRPC server, we can see the "Stream was closed by client" in the APIC's logs

[leoporoli]

I don't think that the problem is in ConnectGo or in the Kurtosis Engine because I've tested it locally with the UI and it runs the MERN package completely without any problem and even downloading the container images (I removed the images before the test).

If we see the following image the RunStarlarkPackage calls took more than 2 min and it was success while in the cloud it always is interrupted in 1min

I think that the problem, in cloud, could be in some layer above of this.

[leoporoli]

I see this error log in the cloud gateway when the call fails:

[error] 35#35: *16517 upstream timed out (110: Connection timed out) while reading upstream, client: 10.0.8.135, server: ~^(?<port_number>[0-9]{2,5})-(?<service_short_uuid>[0-9a-f]{12})-(?<enclave_short_uuid>[0-9a-f]{12})-(?<instance_short_uuid>[0-9a-f]{12}).*$, request: "POST /gateway/ips/54-88-173-250/ports/8081/kurtosis_enclave_manager.KurtosisEnclaveManagerServer/RunStarlarkPackage HTTP/1.1", upstream: "http://10.0.5.188:8081/kurtosis_enclave_manager.KurtosisEnclaveManagerServer/RunStarlarkPackage", host: "cloud.kurtosis.com", referrer: "https://cloud.kurtosis.com/gateway/ips/54-88-173-250/ports/9711/enclave/2bf955f3b89a/logs"

[leoporoli]

The error was fixed in the PR linked above

I was able to run the Jupyter package (including the images download part) without any issue, and the "RunStarlarkPackage" calls took ~2.5 minutes

Same result for the MERN package:

[leoporoli]

@barnabasbusa could you confirm if this fix also fix the use case that you reported on this ticket: #2231 ?