Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage for receive buffers #350

Closed
Stebalien opened this issue Jun 11, 2020 · 4 comments · Fixed by #500
Closed

Reduce memory usage for receive buffers #350

Stebalien opened this issue Jun 11, 2020 · 4 comments · Fixed by #500
Labels
kind/enhancement A net-new feature or improvement to an existing feature status/ready Ready to be worked

Comments

@Stebalien
Copy link
Member

We don't need to keep around large receive buffers. Instead, we can use msgio's message reader to:

  1. Read the message length with a small buffer.
  2. Get a larger buffer from the buffer pool to read the rest of the message.
  3. Put the larger buffer back into the pool.

See https://github.com/libp2p/go-libp2p-kad-dht/blob/c245d836ab82922314dcbc364b3763d114938aad/dht_net.go#L76-L116 for how we do this in the DHT.

buffers

@Stebalien Stebalien added the kind/enhancement A net-new feature or improvement to an existing feature label Jun 11, 2020
@jacobheun jacobheun added the status/ready Ready to be worked label Jul 10, 2020
@Winterhuman
Copy link

@Stebalien Any update on this? Looks like it's been untouched for a while and is blocking ipfs/kubo#6621 (comment).

@amirylm
Copy link

amirylm commented Aug 21, 2022

Any update on this? we experienced the same issue ssvlabs/ssv#688

@vyzo
Copy link
Collaborator

vyzo commented Aug 21, 2022

@MarcoPolo would you like to take a swipe at this?

@Wondertan
Copy link
Contributor

Wondertan commented Aug 21, 2022

Happy to volunteer unless @MarcoPolo wants to tackle this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature status/ready Ready to be worked
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants