blobstor: Try another sub-storage when the current one is full #2450
Conversation
Previously, `BlobStor.Put` always failed when the first sub-storage that matched the policy failed. In particular, when `Blobovnicza` tree (limited by size) was full, BlobStor didn' try to store data in the `FSTree`. This behavior led to storage service denial. Make `BlobStor.Put` to try next sub-storage when currently processed one is full (returns `common.ErrNoSpace` error). The method still immediately fails on any other error. Closes nspcc-dev#2382. Signed-off-by: Leonard Lyubich <[email protected]>
Codecov Report
@@ Coverage Diff @@
## master #2450 +/- ##
==========================================
+ Coverage 29.41% 29.43% +0.02%
==========================================
Files 399 399
Lines 30346 30353 +7
==========================================
+ Hits 8925 8934 +9
+ Misses 20680 20678 -2
Partials 741 741
... and 1 file with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
cthulhu-rider
force-pushed
the
bugfix/2382-blobovnicza-overflow
branch
from
ff1a428
to
7a3896b
Compare
Previously, `BlobStor` returned `ErrNoPlaceFound` when all sub-storages were full. This error is related to policy compliance instead of free space. To make component more responsive, the `ErrNoSpace` error should be returned in these cases. Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider
force-pushed
the
bugfix/2382-blobovnicza-overflow
branch
from
7a3896b
to
9cabb44
Compare
|
To me it fixes #2382, object will be stored successfully and it'll be found easily as well. bbczs refactoring is a different story. |
u can create an issue for this, imo such routine would complicate things rather than optimize smth (data removal isn't so frequent like write load spikes are)
simplification doesn't prevent no space cases, so these changes are independent |
There was a problem hiding this comment.
simplification doesn't prevent no space cases, so these changes are independent
But we are not sure that it is a "no space left" bug. Simplification would solve that misunderstanding.
imo such routine would complicate things rather than optimize smth
Without any error logs it may store dozen of gigabytes on the wrong (nonoptimal) storage.
this PR fixes particular problem: when blobovnicza says "i'm full" we must not deny the request and must fallback to any free sub-storage
in practice, when blobovnicza is full, FS tree is unlikely to save us if it is on the same drive. In theory, blobstor doesn't operate with "optimality", it has list of sub-storages which can or can not accept the object. Right now, when blobovnicza overflowed, there is no clear event when we should overtake data from FS tree to it. If u can suggest any particular behavioral model, pls fill an issue. Right now service downtime is worse than optimality anyway |
general improvement mentioned #2382 (comment)