Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store objects in "header+payload" format #2925

Open
roman-khimov opened this issue Aug 28, 2024 · 0 comments
Open

Store objects in "header+payload" format #2925

roman-khimov opened this issue Aug 28, 2024 · 0 comments
Labels
enhancement Improving existing functionality I4 No visible changes neofs-storage Storage node application issues performance More of something per second S1 Highly significant U3 Regular

Comments

@roman-khimov
Copy link
Member

Is your feature request related to a problem? Please describe.

I'm always frustrated when I look at how we allocate a lot of memory to fetch data from storage. #2316, #2178, #1398. #2814 also reminds about it.

Describe the solution you'd like

#1932 has some valid input for these problems, although it's localized to CLI, whereas I'm more concerned about SN. We can store/fetch objects in a bit different manner, serializing header and payload separately right at the FSTree level. Then object get operation would return a header and an io.ReaderCloser pointing to the file (if it's not too small). This can then be streamed to the outside as needed (but likely API changes are required as well).

Describe alternatives you've considered

Keep pulling MBs of data from the storage is no fun.

@roman-khimov roman-khimov added enhancement Improving existing functionality neofs-storage Storage node application issues performance More of something per second U3 Regular S1 Highly significant I4 No visible changes labels Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improving existing functionality I4 No visible changes neofs-storage Storage node application issues performance More of something per second S1 Highly significant U3 Regular
Projects
None yet
Development

No branches or pull requests

1 participant