Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent Hangs in CI #288

Open
insertinterestingnamehere opened this issue Sep 27, 2024 · 5 comments
Open

Intermittent Hangs in CI #288

insertinterestingnamehere opened this issue Sep 27, 2024 · 5 comments

Comments

@insertinterestingnamehere
Copy link
Collaborator

After some kind of runtime environment update to the github actions CI runners (as of Wednesday 9/25) we're now seeing very occasional hangs during the CI tests. This affects all Linux build configurations and happens either during the basics/arbitrary_blocking_operation test or the features/qlfqueue test.

@insertinterestingnamehere
Copy link
Collaborator Author

This will probably be a pain to debug so I don't think it's feasible to get it fixed before the upcoming release, however I have confirmed that these new failures are not caused by the new code in the release. It's an existing issue that only surfaces in the newer build environment.

@insertinterestingnamehere insertinterestingnamehere added this to the 1.22 Release milestone Oct 3, 2024
@insertinterestingnamehere
Copy link
Collaborator Author

This seems to have gone away on its own. My best bet is that #297 fixed it, but I'm still not entirely sure.

@insertinterestingnamehere
Copy link
Collaborator Author

Just saw this again the day after closing it, but it's not showing up today. Perhaps the timeouts just need to be increased. Running on a busier CI instance could probably cause this. Leaving it closed for now, but I'm still keeping an eye out for it.

@insertinterestingnamehere
Copy link
Collaborator Author

Alright, we're still seeing these even after increasing the timeout thresholds. See https://github.com/sandialabs/qthreads/actions/runs/11448461793/job/31852093932?pr=305 for an example.

@insertinterestingnamehere
Copy link
Collaborator Author

I wonder if this is related to #267 (comment) as well. Plausible, but currently unclear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant