-
Notifications
You must be signed in to change notification settings - Fork 257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory usage increasing with each permissions check call #1934
Comments
A few things could be going on here. Can you share more details of how you are generating load? That might give us more of a clue. One thing to look at is the caches themselves. There are multiple caches in SpiceDB; the sum of their defaults + overhead for normal running may greater than the 1GB you have available during your load test. I see you have set By default SpiceDB (grpc, really) spawns a goroutine per request. If you are sending a huge number of parallel requests, it may be spinning up too many goroutines and eating up your overhead. There are flags to control this as well ( It's also possible that something is interfering with connection management on the database side. Are you running e.g. pgbouncer? If none of these are the issue, it may be that you are hitting some edge case in a specific request that is causing a true memory leak, so knowing what requests you are making would be very helpful. At first glance though I don't think this is a leak - it seems like we just need to do some right-sizing for your test. |
Thank you @ecordell for your quick response.
I left the default value of
I am testing it right now. Once finished, I will be back with some details.
As far as I know, we do not. |
The problem still appears even with the limited amount of workers. This is my current configuration:
the memory usage is increasing during the load
also with the current config the load test execution time has increased a lot. I am generating the load by running e2e tests, which are executing various scenarios on my application. Those tests are running for about 20hours. We are checking permissions (opa-istio sidecar is intercepting the request and validate it against policies in spicedb) per each request made to and within the application.
I have attached also the logs from spicedb. |
What is the QPS of checks during your load test? You will need to size the node to handle the maximum number of parallel requests. It also looks like you're using Do you have the ability to scrape and graph prometheus metrics? We have a lot of metrics that can help diagnose including cache size metrics. It's perfectly normal for the memory to increase with Checks up to a certain point; that's normal and expected as the caches fill. The caches will hit a limit, and then memory use on top will be dictated by the number of in-flight queries, which you control in your load test. |
@janekmichalik did you get to check what Evan posted above? Is the the memory under control? Shall we close this? |
What platforms are affected?
linux
What architectures are affected?
amd64
What SpiceDB version are you using?
v1.30.0
v1.33.0
Steps to Reproduce
Use spicedb for permissions check.
Spicedb runtime configuration:
but I have also tried with the default one:
it is running within k3s pod.
Expected Result
Memory not increasing -> memory released.
Actual Result
The memory use is increasing with each permissions check during performance/load tests.
My pod has 1GB memory limit, and when it reaches the limit, it starts to throw 403 errors.
It looks like the cache is not released.
The text was updated successfully, but these errors were encountered: