fix: add delays in ray application cloud build test #787
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: Ibd5f789d29dd91ecd140389b367f040bde8eb99b
genlu2011
changed the title
roberthbailey
approved these changes
Aug 28, 2024
|
/gcbrun |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The ray application is quite flaky with the following error, adding sleep to wait for pods to be more stable.
Forwarding from 127.0.0.1:8265 -> 8265
Handling connection for 8265
E0828 07:18:32.742675 386 portforward.go:409] an error occurred forwarding 8265 -> 8265: error forwarding port 8265 to pod 3f1f339204911c524c01d65e02ec3e19e20b2c3fdb5dcbb900d80be80ba97a11, uid : failed to execute portforward in network namespace "/var/run/netns/cni-a9192461-40d0-574a-1a1a-756308c62f70": failed to connect to localhost:8265 inside namespace "3f1f339204911c524c01d65e02ec3e19e20b2c3fdb5dcbb900d80be80ba97a11", IPv4: dial tcp4 127.0.0.1:8265: connect: connection refused IPv6 dial tcp6 [::1]:8265: connect: cannot assign requested address
error: lost connection to pod