Move block / block assembly into Mnesia or ETS?

[mpope9]

I wouldn't say this is crucial, but it seems if block sizes reach 512 kB or 1 MB you might want to move them from Task statedata to being in Mnesia (or maybe just an ETS table because only a single process would interact with them). If you're really going for it and you have a bunch of torrents going at once you might hit your VM memory limit and get OOM'd by the kernel. ETS is offheap and fast!

Or if you want to support recovery from a partially downloaded block, you could use Mnesia with disk copies, or use this: https://github.com/klarna/mnesia_eleveldb Bindings for LevelDB K/V store. Doesn't have a 2GB on-disk limit and is highly optimized C++ code.

[bessey]

Previously we were writing blocks straight to disk, stupid because blocks are at most 2^14. I've since rearchitected massively so that each PeerDownloader builds up an entire Piece and dispatches multiple requests for blocks of that piece until it has a complete buffer, then writes the whole lot to disk. With that in mind, it could well be time to move the buffer implementation to ETS. Mnesia seems little better than writing files to disk for my purposes though Matt

…

On Sat, 25 Jan 2020 at 23:35, Matthew Pope ***@***.***> wrote: I wouldn't say this is crucial, but it seems if block sizes reach 512 kB or 1 MB you might want to move them from Task statedata to being in Mnesia (or maybe just an ETS table because only a single process would interact with them). If you're really going for it and you have a bunch of torrents going at once you might hit your VM memory limit and get OOM'd by the kernel. ETS is offheap and fast! Or if you want to support recovery from a partially downloaded block, you could use Mnesia with disk copies, or use this: https://github.com/klarna/mnesia_eleveldb Bindings for LevelDB K/V store. Doesn't have a 2GB on-disk limit and is highly optimized C++ code. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4?email_source=notifications&email_token=AAFM42EXRKI4FHX6BHL25DTQ7TEEVA5CNFSM4KLTVBO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IIW7GIA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFM42CDD53JE6SOSCLOWVDQ7TEEVANCNFSM4KLTVBOQ> .