Figure out larger file support (up to 5GB) #39
Conversation
The old queuing mechanism mostly worked by putting all the message in the `gen_statem`'s `next_event` queue. This is not a great mechanism because it it has some limitations: > The stored events are inserted in the queue as the next to process > before any already queued events. The order of these stored events is > preserved, so the first next_event in the containing list becomes the > first to process. The issue here is that every time we send ourselves a `ping` message, for example, it will go the back of the whole queue. In some cases I suspect this could lead to issues in flow control (eg. the TLS server waits for a `pong` to send messages, but the FSM is stuck trying to send more data). Instead, we start using a queue data structure to more literally hold the messages that are to be processed later. This may also become useful if we try to support sending large files by breaking them in sub-messages (eg. 15mb per batch) which would just pop them in front of the FSM's queue. Now with a data structure that's more explicit, we'll have the ability to control the scheduling without preventing other messages from being handled in between.
This avoids messing with regular instances running on docs' sample ports and accidentally failing tests if the software is running in the background already.
The data wrapper change introduces a new operation, which means we need to bump the internal version. At this point in time, the significance of this is that we eventually need to test the failures of reading the proper messages, but first let's change the version. This instantly highlighted shared concerns across _a lot_ of files for all wrappers, and some inconsistencies, so these get fixed up and brought under a better centralized umbrella. Existing tests pass, showing that we're at least consistent within a single version.
|
After some experimentation and checking the docs, there's a bit of bad news for S3 support:
And from the CopyObject page:
Essentially, it appears that while I can use multipart for better memory usage in file transfers, the way I use full-file hashes for version tracking implies that I'll need to limit myself to 5GB files on S3 by forcing another copy, and ignoring anything bigger than this. This is a bit annoying, but probably an acceptable trade-off for now. |
ferd
changed the title
Wire up multipart both for clients and servers. In writing this patch, I found out that there was a missing synchronization of conflict files coming from the client while being unknown to the server. It seems like tests had hidden that fact based on the ease of setup. The opposed syncs were tried but not that one. This commit adds a test that validates both single-transfer and multipart conflict portions, and makes the code support it. the code is messy, and is going to need a refactor, which writing this helped clarify in my mind.
Pick the comon aspects to the client and the server, where files are read and broken up to be sent, and unite them within a shared structure. This further splits up the handling of scheduling, serializing, and the callback management. It also temporarily undoes tracing support, which will need to be reintroduced later.
There's some fun stuff to deal with for the multipart checksums. If we want to use S3 at all, we need to limit all file uploads to 5GB otherwise we'll break the hashing mechanism we rely on.
ferd
force-pushed
the
larger-file-support
branch
from
93b14cd
to
a1d1290
Compare
ferd
force-pushed
the
larger-file-support
branch
from
3124936
to
933b521
Compare
Right now, big files need to be loaded all in memory to be sent and managed. This is going to be a problem for larger files one might wish to synchronize.
This long-standing PR/branch will aim to add in sub-file transfer messages to break down any sync into manageable bits.
The first step is to rework the file transfer queuing mechanism
The old queuing mechanism mostly worked by putting all the message in the
gen_statem'snext_eventqueue. This is not a great mechanism because it it has some limitations:The issue here is that every time we send ourselves a
pingmessage, for example, it will go the back of the whole queue. In some cases I suspect this could lead to issues in flow control (eg. the TLS server waits for apongto send messages, but the FSM is stuck trying to send more data).Instead, we start using a queue data structure to more literally hold the messages that are to be processed later.
This may also become useful if we try to support sending large files by breaking them in sub-messages (eg. 15mb per batch) which would just pop them in front of the FSM's queue. Now with a data structure that's more explicit, we'll have the ability to control the scheduling without preventing other messages from being handled in between.