-
Notifications
You must be signed in to change notification settings - Fork 126
RSDK-12499: better error log when host is offline #5471
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
Could we potentially miss any useful info with the new conditional? Also would there be any significant performance improvement if we use the |
for this error, the wrapping removes noisy information and returns the important message from the original log (host appears offline), so no useful info is lost. Also the err that gets returned includes a lot of other strings in it so cannot use errors.is() for this case (full err is |
benjirewis
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like a fine enough change to me. I agree that we'll lose some information, but I don't think that information was particularly useful. When "host appears to be offline" is in the message, that is definitely the signaling server bubbling back to the user (in some form) that no answerer for the machine is actively connected to an operator. I think it's fine to reduce the message to that.
(with a pretty long pause in CLI after the command is as it attempts 3x, is this bad?)
Was the "pause" not there before? I don't see how you changing the string that's returned here could cause a new delay in the command's execution.
instead of sending the two warning logs, it just runs silently in the background. So no new pause, but in case it may be confusing to users when nothing happens for a couple seconds? |
Which warning logs are you referring to? This PR is just returning a shortened error message in some cases, right? |
| // Connect will close any existing connection and try to reconnect to the remote. | ||
| func (rc *RobotClient) Connect(ctx context.Context) error { | ||
| if err := rc.connectWithLock(ctx); err != nil { | ||
| if strings.Contains(err.Error(), "host appears to be offline") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By lowering the "warn" logs to "debug", it doesn't show up in the the CLI return anymore. Should have linked this PR (which makes the changes), sorry! |
Want to make error log when host is offline less noisy and very easy to debug by hiding warnings and simplifying the final error. this only affects CLI and go SDK
gosdk before:
2025-11-10T22:52:37.769Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:52:37.769Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:52:37.770Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:52:38.000Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-10T22:52:38.205Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-10T22:52:43.340Z WARN client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=36f9c85ce6b4af03185062f41240fd4c: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-10T22:52:43.445Z WARN client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=415629694d25a8a89414992060fbf2a7: no active offer for """}
2025-11-10T22:52:43.446Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.446Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.446Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.447Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.597Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-10T22:53:03.783Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-10T22:53:08.900Z WARN client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=59feb2be38f021b3fb1f7457e17bace7: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-10T22:53:09.021Z WARN client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=007b778a356b3b60b190dd3d836b2f08: no active offer for """}
2025-11-10T22:53:09.022Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.023Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.024Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.023Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.096Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-10T22:53:29.328Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-10T22:53:34.484Z WARN client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=99cc1420b3f4e658458dc61ec70cb033: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-10T22:53:34.588Z WARN client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=a52f6da5cfa5b248843480d985652eb1: no active offer for """}
2025-11-10T22:53:34.590Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:54.591Z ERROR client gosdk/gosdk.go:28 rpc error: code = Unavailable desc = requestID=99cc1420b3f4e658458dc61ec70cb033: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again; context deadline exceeded; mDNS query failed to find a candidate
##sdk after:
2025-11-12T21:04:21.662Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:21.662Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:21.663Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:21.774Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-12T21:04:21.972Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-12T21:04:27.097Z DEBUG client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=f4b174889f0212c81455ee108a5afa93: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-12T21:04:27.181Z DEBUG client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=dfa4f4fe7dee92d04c0622ed2da9f523: no active offer for """}
2025-11-12T21:04:27.182Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.183Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.184Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.184Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.271Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-12T21:04:47.470Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-12T21:04:52.636Z DEBUG client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=98f3439123ae0067ecd679a2fd5dbe32: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-12T21:04:52.734Z DEBUG client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=3a8db48f990d2b07ef630afb89091879: no active offer for """}
2025-11-12T21:04:52.734Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.736Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.736Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.736Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.824Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-12T21:05:13.017Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-12T21:05:18.167Z DEBUG client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=26fd609d69a85310c9db8cc569cb2acc: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-12T21:05:18.239Z DEBUG client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=1f98f5caf3e73bfdec8ddbb12eb0ac50: no active offer for """}
2025-11-12T21:05:18.240Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:38.242Z ERROR client gosdk/gosdk.go:29 host appears to be offline; ensure machine is online and try again
##CLI before:
2025-11-11T20:27:20.786Z WARN cli.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=1c361402cad40f6e7f59db7f90bfc12c: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-11T20:27:20.950Z WARN cli.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=a531fb14357c6da53421e280bb2c1196: no active offer for """}
2025-11-11T20:27:46.440Z WARN cli.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=9d6dd586a8833a81107dfec659c5a22d: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-11T20:27:46.567Z WARN cli.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=f4fd29c3eeb0d6ad5e8fc6ee6283518b: no active offer for """}
2025-11-11T20:28:12.020Z WARN cli.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=a7bfcf784a4473f81f9148847047936a: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-11T20:28:12.139Z WARN cli.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=dacffd2e31b4880fae9a7d73724db1da: no active offer for """}
Error: Could not connect to machine part: rpc error: code = Unavailable desc = requestID=a7bfcf784a4473f81f9148847047936a: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again; context deadline exceeded; mDNS query failed to find a candidate
##Cli after
Error: Could not connect to machine part: host appears to be offline; ensure machine is online and try again
(with a pretty long pause in CLI after the command is as it attempts 3x, is this bad?)