Skip to content

Conversation

@allisonschiang
Copy link
Contributor

@allisonschiang allisonschiang commented Nov 11, 2025

Want to make error log when host is offline less noisy and very easy to debug by hiding warnings and simplifying the final error. this only affects CLI and go SDK

gosdk before:

2025-11-10T22:52:37.769Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:52:37.769Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:52:37.770Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:52:38.000Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-10T22:52:38.205Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-10T22:52:43.340Z WARN client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=36f9c85ce6b4af03185062f41240fd4c: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-10T22:52:43.445Z WARN client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=415629694d25a8a89414992060fbf2a7: no active offer for """}
2025-11-10T22:52:43.446Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.446Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.446Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.447Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:03.597Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-10T22:53:03.783Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-10T22:53:08.900Z WARN client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=59feb2be38f021b3fb1f7457e17bace7: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-10T22:53:09.021Z WARN client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=007b778a356b3b60b190dd3d836b2f08: no active offer for """}
2025-11-10T22:53:09.022Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.023Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.024Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.023Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:29.096Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-10T22:53:29.328Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-10T22:53:34.484Z WARN client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=99cc1420b3f4e658458dc61ec70cb033: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-10T22:53:34.588Z WARN client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=a52f6da5cfa5b248843480d985652eb1: no active offer for """}
2025-11-10T22:53:34.590Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"netcode-main.ugzewyfnm7.viam.cloud"}
2025-11-10T22:53:54.591Z ERROR client gosdk/gosdk.go:28 rpc error: code = Unavailable desc = requestID=99cc1420b3f4e658458dc61ec70cb033: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again; context deadline exceeded; mDNS query failed to find a candidate

##sdk after:
2025-11-12T21:04:21.662Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:21.662Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:21.663Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:21.774Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-12T21:04:21.972Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-12T21:04:27.097Z DEBUG client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=f4b174889f0212c81455ee108a5afa93: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-12T21:04:27.181Z DEBUG client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=dfa4f4fe7dee92d04c0622ed2da9f523: no active offer for """}
2025-11-12T21:04:27.182Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.183Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.184Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.184Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:04:47.271Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-12T21:04:47.470Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-12T21:04:52.636Z DEBUG client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=98f3439123ae0067ecd679a2fd5dbe32: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-12T21:04:52.734Z DEBUG client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=3a8db48f990d2b07ef630afb89091879: no active offer for """}
2025-11-12T21:04:52.734Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.736Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.736Z DEBUG client.networking.webrtc rpc/dial.go:188 trying WebRTC {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.736Z DEBUG client.networking.webrtc rpc/wrtc_client.go:108 connecting to signaling server {"signaling_server":"app.viam.com:443","host":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:12.824Z DEBUG client.networking.webrtc rpc/wrtc_client.go:124 connected to signaling server {"signaling_server":"app.viam.com:443"}
2025-11-12T21:05:13.017Z INFO client.networking.webrtc rpc/wrtc_signaling.go:125 extended WebRTC config {"iceURLs":["stun:global.stun.twilio.com:3478","turn:global.turn.twilio.com:3478?transport=udp","stun:turn.viam.com:443","turn:turn.viam.com:443?transport=udp"]}
2025-11-12T21:05:18.167Z DEBUG client.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=26fd609d69a85310c9db8cc569cb2acc: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-12T21:05:18.239Z DEBUG client.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=1f98f5caf3e73bfdec8ddbb12eb0ac50: no active offer for """}
2025-11-12T21:05:18.240Z DEBUG client.networking.mdns rpc/dial.go:144 trying mDNS {"address":"mac-server-main.ugzewyfnm7.viam.cloud"}
2025-11-12T21:05:38.242Z ERROR client gosdk/gosdk.go:29 host appears to be offline; ensure machine is online and try again

##CLI before:
2025-11-11T20:27:20.786Z WARN cli.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=1c361402cad40f6e7f59db7f90bfc12c: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-11T20:27:20.950Z WARN cli.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=a531fb14357c6da53421e280bb2c1196: no active offer for """}
2025-11-11T20:27:46.440Z WARN cli.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=9d6dd586a8833a81107dfec659c5a22d: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-11T20:27:46.567Z WARN cli.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=f4fd29c3eeb0d6ad5e8fc6ee6283518b: no active offer for """}
2025-11-11T20:28:12.020Z WARN cli.networking.webrtc rpc/wrtc_client.go:391 Failed to exchange candidates {"err":"rpc error: code = Unavailable desc = requestID=a7bfcf784a4473f81f9148847047936a: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again"}
2025-11-11T20:28:12.139Z WARN cli.networking.webrtc rpc/wrtc_client.go:414 Problem sending error to signaling server {"err":"rpc error: code = Unknown desc = requestID=dacffd2e31b4880fae9a7d73724db1da: no active offer for """}
Error: Could not connect to machine part: rpc error: code = Unavailable desc = requestID=a7bfcf784a4473f81f9148847047936a: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again; context deadline exceeded; mDNS query failed to find a candidate

##Cli after
Error: Could not connect to machine part: host appears to be offline; ensure machine is online and try again
(with a pretty long pause in CLI after the command is as it attempts 3x, is this bad?)

@viambot viambot added the safe to test This pull request is marked safe to test from a trusted zone label Nov 11, 2025
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 11, 2025
@aldenh-viam
Copy link
Contributor

Could we potentially miss any useful info with the new conditional? Also would there be any significant performance improvement if we use the errors.Is pattern instead of strings.Contains for these?

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 12, 2025
@benjirewis benjirewis requested a review from a team November 12, 2025 21:39
@allisonschiang
Copy link
Contributor Author

Could we potentially miss any useful info with the new conditional? Also would there be any significant performance improvement if we use the errors.Is pattern instead of strings.Contains for these?

for this error, the wrapping removes noisy information and returns the important message from the original log (host appears offline), so no useful info is lost. Also the err that gets returned includes a lot of other strings in it so cannot use errors.is() for this case (full err is code = Unavailable desc = requestID=a7bfcf784a4473f81f9148847047936a: rpc error: code = Unavailable desc = host appears to be offline; ensure machine is online and try again; context deadline exceeded; mDNS query failed to find a candidate)

Copy link
Member

@benjirewis benjirewis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a fine enough change to me. I agree that we'll lose some information, but I don't think that information was particularly useful. When "host appears to be offline" is in the message, that is definitely the signaling server bubbling back to the user (in some form) that no answerer for the machine is actively connected to an operator. I think it's fine to reduce the message to that.

(with a pretty long pause in CLI after the command is as it attempts 3x, is this bad?)

Was the "pause" not there before? I don't see how you changing the string that's returned here could cause a new delay in the command's execution.

@allisonschiang
Copy link
Contributor Author

Seems like a fine enough change to me. I agree that we'll lose some information, but I don't think that information was particularly useful. When "host appears to be offline" is in the message, that is definitely the signaling server bubbling back to the user (in some form) that no answerer for the machine is actively connected to an operator. I think it's fine to reduce the message to that.

(with a pretty long pause in CLI after the command is as it attempts 3x, is this bad?)

Was the "pause" not there before? I don't see how you changing the string that's returned here could cause a new delay in the command's execution.

instead of sending the two warning logs, it just runs silently in the background. So no new pause, but in case it may be confusing to users when nothing happens for a couple seconds?

@benjirewis
Copy link
Member

sending the two warning logs

Which warning logs are you referring to? This PR is just returning a shortened error message in some cases, right?

// Connect will close any existing connection and try to reconnect to the remote.
func (rc *RobotClient) Connect(ctx context.Context) error {
if err := rc.connectWithLock(ctx); err != nil {
if strings.Contains(err.Error(), "host appears to be offline") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cheukt 's comment on your other PR may be relevant here, too.

@allisonschiang
Copy link
Contributor Author

sending the two warning logs

Which warning logs are you referring to? This PR is just returning a shortened error message in some cases, right?

By lowering the "warn" logs to "debug", it doesn't show up in the the CLI return anymore. Should have linked this PR (which makes the changes), sorry!
viamrobotics/goutils#505

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test This pull request is marked safe to test from a trusted zone

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants