Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tree service requests sometimes fail during high load #797

Open
cthulhu-rider opened this issue Jul 13, 2023 · 0 comments
Open

Tree service requests sometimes fail during high load #797

cthulhu-rider opened this issue Jul 13, 2023 · 0 comments
Labels
blocked Can't be done because of something bug Something isn't working I3 Minimal impact S4 Routine tree-service Implementing tree-service features U3 Regular

Comments

@cthulhu-rider
Copy link
Contributor

when xk6 loads NeoFS through S3 gateway with PutObject / GetObject stream, some Tree service requests fail

Jul 13 09:54:41 neofs-s3-gw[580041]: 2023-07-13T09:54:41.245Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "5b93dd2a-d244-476d-8d0a-ae1b225ad6c6", "method": "PutObject", "bucket": "48969b24-1e7c-4a29-8a40-8962542605fb", "object": "4dad9708-4309-4630-906e-ed49e851ce09", "description": "could not upload 
object", "error": "couldn't add new verion to tree service: failed to add node by path: rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:57298->[<ipv6>]:8080: use of closed network connection", "body close errors": []}

Jul 13 09:54:40 neofs-s3-gw[580041]: 2023-07-13T09:54:40.927Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "64848ab0-bd31-4e37-b7a5-b048395bc1f5", "method": "GetObject", "bucket": "8d80a780-b5f0-4ab2-9c68-e85f6501381d", "object": "8d04d4e2-e2a9-4b17-896e-1ef96e8807b4", "description": "could not find ob
ject", "error": "failed to get node by path: rpc error: code = Unavailable desc = error reading from server: read tcp [<ipv6>]:44100->[<ipv6>]:8080: use of closed network connection"}

this happens periodically during load test

we need to explore how this (attempt to use closed TCP conn) happens and which side - client or server - should be fixed. I searched a bit for similar cases, they are mostly caused by premature server shutdown, but it doesn't seem to be our case.

@cthulhu-rider cthulhu-rider added bug Something isn't working tree-service Implementing tree-service features labels Jul 13, 2023
@roman-khimov roman-khimov added U3 Regular S4 Routine I3 Minimal impact labels Dec 20, 2023
@roman-khimov roman-khimov added the blocked Can't be done because of something label Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Can't be done because of something bug Something isn't working I3 Minimal impact S4 Routine tree-service Implementing tree-service features U3 Regular
Projects
None yet
Development

No branches or pull requests

2 participants