2025-01-02 06:15:40.118 [error] [channel.cc:Proc:104] SendImpl error [external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[38570eeb2e46782d];[content-length]:[14];[kuscia-error-message]:[<bob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout>];[x-accel-buffering]:[no];[x-b3-spanid]:[38570eeb2e46782d];[date]:[Thu, 02 Jan 2025 06:15:40 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'

[lvying0019]

Issue Type

Running

Search for existing issues similar to yours

Yes

OS Platform and Distribution

Ubuntu 22.04.1

Kuscia Version

secretpadImage版本：0.11.0b0 secretflowServingImage版本：0.7.0b0 kusciaImage版本：0.12.0b0 secretflowImage版本：1.10.0b1 dataProxyImage版本：0.1.0b0

Deployment

docker

deployment Version

docker

App Running type

secretflow

App Running version

secretpadImage版本：0.11.0b0 secretflowServingImage版本：0.7.0b0 kusciaImage版本：0.12.0b0 secretflowImage版本：1.10.0b1 dataProxyImage版本：0.1.0b0

Configuration file used to run kuscia.

使用all in one前端界面

What happend and What you expected to happen.

100W * 5亿数据求交，限速20Mbps，延时50ms，使用ECDH运行求交后，出现如下报错
2025-01-02 06:15:40.118 [error] [channel.cc:Proc:104] SendImpl error [external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[38570eeb2e46782d];[content-length]:[14];[kuscia-error-message]:[<bob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout>];[x-accel-buffering]:[no];[x-b3-spanid]:[38570eeb2e46782d];[date]:[Thu, 02 Jan 2025 06:15:40 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'

Kuscia log output.

2025-01-02 08:56:30.510 INFO nlog/nlog.go:77 W0102 08:56:30.510046       8 logging.go:59] [core] [Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: "dataproxy-grpc:8023", ServerName: "dataproxy-grpc:8023", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.88.0.19:8023: connect: connection refused"
2025-01-02 08:56:30.510 INFO nlog/nlog.go:77 W0102 08:56:30.510046       8 logging.go:59] [core] [Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: "dataproxy-grpc:8023", ServerName: "dataproxy-grpc:8023", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.88.0.19:8023: connect: connection refused"
2025-01-02 08:56:33.012 INFO nlog/nlog.go:77 W0102 08:56:33.012255       8 logging.go:59] [core] [Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: "dataproxy-grpc:8023", ServerName: "dataproxy-grpc:8023", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.88.0.19:8023: connect: connection refused"
2025-01-02 08:56:33.012 INFO nlog/nlog.go:77 W0102 08:56:33.012255       8 logging.go:59] [core] [Channel #12 SubChannel #13] grpc: addrConn.createTransport failed to connect to {Addr: "dataproxy-grpc:8023", ServerName: "dataproxy-grpc:8023", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.88.0.19:8023: connect: connection refused"
2025-01-02 14:15:43.075 INFO status/status_manager.go:625 Patch status for pod "vyfe-dgojzenk-node-36-0_bob(87588ab2-1dc0-4881-9440-7af665db5e2b)", patch={"metadata":{"uid":"87588ab2-1dc0-4881-9440-7af665db5e2b"},"status":{"$setElementOrder/conditions":[{"type":"Initialized"},{"type":"Ready"},{"type":"ContainersReady"},{"type":"PodScheduled"}],"conditions":[{"lastTransitionTime":"2025-01-02T06:15:43Z","reason":"PodFailed","status":"False","type":"Ready"},{"lastTransitionTime":"2025-01-02T06:15:43Z","reason":"PodFailed","status":"False","type":"ContainersReady"}],"containerStatuses":[{"containerID":"containerd://8920438ebb2809faad65bbe14c3ae73d15bb4bc8feb6720a41ced73ff658d2b1","image":"secretflow-registry.cn-hangzhou.cr.aliyuncs.com/secretflow/secretflow-lite-anolis8:1.10.0b1","imageID":"sha256:e8314d7ee41a02c9df46b3e28ff2ec786fad7a206d5dacad661743c6b631e24c","lastState":{},"name":"secretflow","ready":false,"restartCount":0,"started":false,"state":{"terminated":{"containerID":"containerd://8920438ebb2809faad65bbe14c3ae73d15bb4bc8feb6720a41ced73ff658d2b1","exitCode":255,"finishedAt":"2025-01-02T06:15:41Z","message":"];[x-accel-buffering]:[no];[x-b3-spanid]:[cfb147184e37ba65];[date]:[Thu, 02 Jan 2025 06:14:25 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:14:27.353 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=2, max_retry=3, interval_ms=3000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[d500c04213023cac];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[d500c04213023cac];[date]:[Thu, 02 Jan 2025 06:14:26 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:14:27.353 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=2, max_retry=3, interval_ms=3000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[edc309e74cc4e62a];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[edc309e74cc4e62a];[date]:[Thu, 02 Jan 2025 06:14:26 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:14:35.105 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=3, max_retry=3, interval_ms=5000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[e34c69079311d5bf];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[e34c69079311d5bf];[date]:[Thu, 02 Jan 2025 06:14:35 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:14:45.203 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=2, max_retry=3, interval_ms=3000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[4e49383370a53980];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[4e49383370a53980];[date]:[Thu, 02 Jan 2025 06:14:45 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:06.659 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[7117ff3adfe06f75];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[7117ff3adfe06f75];[date]:[Thu, 02 Jan 2025 06:15:06 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:09.704 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[15d1c03cc56a2aad];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[15d1c03cc56a2aad];[date]:[Thu, 02 Jan 2025 06:15:09 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:11.679 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[00ab35d3b0d74dbc];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[00ab35d3b0d74dbc];[date]:[Thu, 02 Jan 2025 06:15:11 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:14.201 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[d043f668a7ca58e1];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[d043f668a7ca58e1];[date]:[Thu, 02 Jan 2025 06:15:14 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:18.812 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[97db12f17d2a35e3];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[97db12f17d2a35e3];[date]:[Thu, 02 Jan 2025 06:15:18 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:19.745 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[752871d08665190f];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[752871d08665190f];[date]:[Thu, 02 Jan 2025 06:15:19 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n[2025-01-02 06:15:29.154] [info] [ecdh_psi.cc:122] MaskSelf:bob, batch_count=280, self_item_count=293601280\n2025-01-02 06:15:34.500 [info] [channel.cc:SendRequestWithRetry:359] send request failed and retry, retry_count=1, max_retry=3, interval_ms=1000, message=[external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[baaf8d2c173540b3];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[baaf8d2c173540b3];[date]:[Thu, 02 Jan 2025 06:15:34 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n2025-01-02 06:15:40.118 [error] [channel.cc:Proc:104] SendImpl error [external/yacl/yacl/link/transport/interconnection_link.cc:56] cntl ErrorCode '1010', http status code '504', response header '[x-b3-traceid]:[38570eeb2e46782d];[content-length]:[14];[kuscia-error-message]:[\u003cbob/root-kuscia-autonomy-bob-mpc-hp-prodesk-680-g4-mt/internal $stream_idle_timeout$ Gateway Timeout\u003e];[x-accel-buffering]:[no];[x-b3-spanid]:[38570eeb2e46782d];[date]:[Thu, 02 Jan 2025 06:15:40 GMT];[server]:[envoy];', response body '', error msg '[E1010]HTTP/1.1 504 Gateway Timeout: stream timeout'\n","reason":"Error","startedAt":"2025-01-02T04:00:03Z"}}}]}}

[popai9527]

检查下对端带宽，原来遇到过相关情况，可以建联，但是求交超时，对端带宽无法满足要求，导致大量网络请求积压超时

[lvying0019]

两侧限速20Mbps，请问最低带宽要求是多少？

[lanyy9527]

带宽受限场景下，可以尝试调整task_input_config.yacl_link的配置，throttle_window_size调整为2，如果仍然失败，可以试着调整http_max_payload_size的大小，从1M往下调整；

[lvying0019]

请问，有修改的文档吗？