Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

grpc-backend running out of memory #2765

Open
another-rex opened this issue Oct 22, 2024 · 3 comments
Open

grpc-backend running out of memory #2765

another-rex opened this issue Oct 22, 2024 · 3 comments
Labels
bug Something isn't working infra infrastructure bugs/FRs

Comments

@another-rex
Copy link
Contributor

Tracking issue for the GRPC out of memory error. We likely need to bump the memory given the recent large OSV records we are now serving.
@another-rex another-rex added bug Something isn't working infra infrastructure bugs/FRs labels Oct 22, 2024
@another-rex
Copy link
Contributor Author

Manually bumped the memory to 8Gb from 4Gb, and will continue monitoring

@another-rex
Copy link
Contributor Author

This seems to have stopped OOM errors now, though there is a lot of 20 second queries now at 99%> . Theory is that this is caused by the paging limit correctly kicking in and preventing a timeout, and now that there is more memory these requests are not killed by OOM.

@andrewpollock
Copy link
Contributor

@another-rex is this okay to close now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working infra infrastructure bugs/FRs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andrewpollock @another-rex