Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Daisy cannot recover from failed GetSerialPortOutput failures. #1557

Open
jeremyje opened this issue Mar 5, 2021 · 3 comments
Open

Daisy cannot recover from failed GetSerialPortOutput failures. #1557

jeremyje opened this issue Mar 5, 2021 · 3 comments

Comments

@jeremyje
Copy link

jeremyje commented Mar 5, 2021

It looks like if there's multiple GetSerialPortOutput failures, Daisy will halt the process.

// Retry up to 3 times in a row on any error if we successfully got InstanceStatus.

@EricEdens
Copy link
Contributor

Hi Jeremy, this is correct. Are you affected by this? If so, thoughts on alternatives?

@jeremyje
Copy link
Author

jeremyje commented Mar 5, 2021

Interim fix:

  • Use exponential backoff with randomized jitter. It looks like something sorta like that is going on:
    func shouldRetryWithWait(tripper http.RoundTripper, err error, multiplier int) bool {
    but I'm not sure I'd roll my own retry loops there's plenty of golang libraries for this.
  • Make the retry logic not fail after 3 times. Make it time bound like say if I keep getting failures for 5 minutes then give up.

Permanent Fix:
Read from Stackdriver logs as the primary source. If there's no access fallback to GetSerialPortOutput with a less aggressive polling internal. Once per minute instead of once every 10 seconds.

@EricEdens
Copy link
Contributor

These all sound like reasonable approaches. If you're interested in writing a PR, we could chat about specifics, and I think @adjackura would have some thoughts too.

Before that, I'm curious: How often do your builds fail due to this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants