Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve errors and edge case handling in model proxy #1336

Closed
wants to merge 3 commits into from

Conversation

kian99
Copy link
Contributor

@kian99 kian99 commented Sep 2, 2024

Description

This PR modifies the model proxy to improve the error logging and simplify logic in some edge cases.

The reason for this change was that when running JIMM in a test setup there were constant debug logs about "error reading from client" or "error reading from controller" which were completely expected, every time the client closes the websocket our routines waiting to read from the websocket stop and return. In these cases, there were debug and error level logs which were not necessary because this was the normal mode of operation.

The first commit in this PR simply makes the client/controller proxy return a nil error when they fail to read from their respective websocket connections.

The second commit in this PR was done after taking a look at whether we could simplify the complicated logic when handling ctx.Done() involving a mutex. It was thankfully easy to clean this up because we already handle things gracefully when the client connection is closed so we can simply close the client connection to clean things up.

Finally, I noticed an edge case that wasn't handled and added a test for it. If the connection to the controller was closed unexpectedly, the model proxy would not handle this gracefully. Now if the controller proxy routine ends early, we handle this by stopping the model proxy.

Engineering checklist

Check only items that apply

  • Documentation updated
  • Covered by unit tests
  • Covered by integration tests

Test instructions

Ran the relevant tests with -race and/or -count=1000. E.g.

go test -race -timeout 30s github.com/canonical/jimm/v3/internal/rpc -count=10
go test -timeout 30s -run ^TestProxySockets$ github.com/canonical/jimm/v3/internal/rpc -count=1000

Notes for code reviewers

I suggest reviewing this PR by each commit individually to see the changes more clearly.

@kian99 kian99 requested a review from a team as a code owner September 2, 2024 08:01
- Simplified the model proxy to remove the need for a mutex.
- Handle an edge case in the model proxy where if the controller routine returned we don't stop the proxy.
@@ -339,8 +332,8 @@ func (p *clientProxy) start(ctx context.Context) error {
zapctx.Debug(ctx, "Reading on client connection")
msg := new(message)
if err := p.src.readJson(&msg); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reading at gorilla/websocket/json.go can return an error for two reasons:

  • error on marshaling the json
  • error on closing the connection.

I think your PR wants to hide error on closing connection, and that is fine from what I understand, but we are also masking errors on json marshaling, which could potentially lead to situation where the clients sends us non-json messages and we are not seeing the error.

Wouldn't be better to check here for marshaling error? And if not ignoring?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes good point

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #1345, I tried updating this PR but Github just hangs, I believe because this PR is still from my old private fork of JIMM.

@kian99 kian99 mentioned this pull request Sep 4, 2024
3 tasks
@kian99 kian99 closed this Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants