Zero copy zip construction.

[stuaxo]

The sendfile API, uses the kernel to send all the bytes of a file over the network, with an optional offset to start at.

If the entries of the zip file that represent each file were actual files, sendfile could be used to send them using very little resources. We could probably still send other stuff like metadata manually as there's not much of it.

[michalc]

A few questions to kick off more discussion...

• Are you thinking these would be files in the filesystem?
• And I'm guessing they would not be compressed (or encrypted now we support that) in the final ZIP?
• What would the API for this look like? Just like now but the iterable of bytes could be a "real" file somehow?
• And I'm guessing this would be linux only?
• And do you have a concrete use case?

Have to admit: I think I'm initially skeptical that this would be useful enough to be worth it. It's partially from ignorance about sendfile, but from a short nosey around https://man7.org/linux/man-pages/man2/sendfile.2.html the source I don't think can be a socket. So you have to have it as a "real" file on the filesystem. So it sort of defeats a lot of the purpose of stream-zip - not having to have anything on disk (unless it's a ramdisk-y thing, but they would have to be copied to get there...)

[stuaxo]

• Are you thinking these would be files in the filesystem?

When I wrote the above I was thinking that, though now I noticed sendfile has got a count parameter, these could just as well be directory entries inside an existing zip file (sendfile would be called sending just the file extents for a the particular file you wanted to send at the time),

I'll stick with my original idea for now, though - where the mechanism relies on a folder containing dumps of the directory entries for each file you want to send.

And I'm guessing they would not be compressed (or encrypted now we support that) in the final ZIP?

Let's look at example.zip, it contains two ordinary files and one file "generated.csv" which was generated by a user of streamzip:

There would be a folder corresponding to the directory entries in the zip (for the ordinary files):

example
+ README.txt.direntry
+ hello.png.direntry

If you looked in README.txt.direntry you wouldn't see text since it's a dump of the zip directory entry, it's still compressed.

generated.csv would not be in the directory - unless the user had specified they wanted it to be cached.

When we want to send a zip file containing README.txt, hello.png and our generated.csv, we:
sendfile README.txt.direntry
sendfile hello.png.direntry
stream the content for generated.csv as usual
stream the zip central directory as usual.

What would the API for this look like? Just like now but the iterable of bytes could be a "real" file somehow?
There would probably be a new API for sending files, and the existing API might grow a parameter to let things be cached as files.

I'm pretty sure that sendfile can do socket to socket - (it's a while since I looked at this, it's possible the API is spelled differently, but I did do this for CSV download a long time ago) - so you could go further and enable this if there was a .direntry file on the other end.

And I'm guessing this would be linux only?

BSD (and thus OSX) is covered, I believe it's all the main Unixes. Windows may have issues (needs investigation).

And do you have a concrete use case?

My guess is that streamzip isn't only used to create zips of generated content, it's probably also handy mixed archives containing streamed data, along with data from files.

• Per-file caching granularity.
  This is handy where you want to do things like click a random selection of files in a file manager and get a zip of them, then another user may want a zip with some, but not-all of the same files contained, in that case if we already have .direntry files we dont need to recompress those.

• Much more efficient resource usage
  sendfile is one of the most efficient APIs available for sending large amounts of data over the network, since it operates in kernel space it avoids a lot of data copies and context switches, beating even a loop in C by quite a bit.
  This is good for keeping servers cooler, and bills lower.

• Consideration: Additional latency
  If it was made to always use this mechanism you would get a one-file latency penalty, but this could be avoided by streaming the first time (and saving the file somewhere), then using sendfile other times (if the .direntry file is available).

So it sort of defeats a lot of the purpose of stream-zip - not having to have anything on disk (unless it's a ramdisk-y thing, but they would have to be copied to get there...)

I didn't realise this the purpose :) I looked at is a way of efficiently streaming zips, in which case some of the data may be file based.

[michalc]

So it sort of defeats a lot of the purpose of stream-zip - not having to have anything on disk (unless it's a ramdisk-y thing, but they would have to be copied to get there...)
I didn't realise this the purpose :) I looked at is a way of efficiently streaming zips, in which case some of the data may be file based.

Ah yes - so not just streaming the output, but the fact that this happens while streaming input. So right now, stream-zip doesn't depend on even having all the source data of even the first file locally when you start to output. There is no random access to any source data. It's iterables of bytes in -> an iterable of bytes out.

And I'm guessing this would be linux only?

BSD (and thus OSX) is covered, I believe it's all the main Unixes. Windows may have issues (needs investigation)

Ah understood

I'm pretty sure that sendfile can do socket to socket

From https://man7.org/linux/man-pages/man2/sendfile.2.html it looks like the answer is no:

The in_fd argument must correspond to a file which supports mmap(2)-like operations (i.e., it cannot be a socket).

So I think main main thoughts are around:

Much more efficient resource usage
sendfile is one of the most efficient APIs available for sending large amounts of data over the network, since it operates in kernel space it avoids a lot of data copies and context switches, beating even a loop in C by quite a bit.
This is good for keeping servers cooler, and bills lower.

But... you would have to get the files onto the disk in the first place. If this is all in a web server of some application, by and large the source data would not already be on that web server(?) It'll be elsewhere: S3, some other remote HTTP service, maybe a database etc. You could save them to the local disk deliberately, but that'll negate the benefits? And then also each file would have to be on the disk in their entirety before starting to output its corresponding part of the ZIP, so adding a potential timeout issue for big files.

And: I have just realised as well: you need the CRC32 of each member which adds another layer of complication and stuff to happen before it hits that disk...

So I think cases where this would have a performance benefit all of the below would have to be true:

1. It's where the data will be on local disk anyway
2. Before/as it got to disk, you were able to work out its CRC32 (and length)
3. And where you don't want to add subsequent compression or any processing as it goes to the socket

This feels like such a specific case. Maybe (as in your example if I'm understanding) a few "static" files might satisfy these, but I guess I am thinking the benefits would be extremely minor in all but the most extreme cases.

Plus...

We would need some way to be able to sendfile multiple files, and with some other bytes in between them. I'm sort of assuming this is all for HTTP responses (or requests?). Do any existing HTTP servers (or clients?) support this? I am still a bit curious as to what the code would look like that uses this feature, were it to exist.

[michalc]

Consideration: Additional latency
If it was made to always use this mechanism you would get a one-file latency penalty, but this could be avoided by streaming the first time (and saving the file somewhere), then using sendfile other times (if the .direntry file is available).

So I might only now be understanding some of this... So yes: could stream and save to disk somewhere at the same time, ready to make it more efficient next time by using sendfile.

But: it's not exactly friendly to horizontally scaling? The user would need to want some of the same member files, and they would have to hit the same server as they did before for there to be a benefit. And, you've hit the problem of cache invalidation too. And, you need to make really sure you never end up sending someone else's files accidentally... And you're then also limited by the disk space on each server. And, it would all have to be shared between all users. Or put another way - it's not very 12 factor app?

I'm skeptical doing all this just for sendfile is worth it for the vast majority of cases... Or the right direction of travel to depend more on local disk rather than less. But also, even if there is a case out there where it's the best thing for some reason, unless the change in stream-zip to support this is almost trivial somehow (where I would probably say "why not?") I think I'm leaning to it being beyond the scope of the project. stream-zip is (streaming) memory to (streaming) memory. This is wonderfully both performant and flexible enough for quite a lot of cases, with a reasonably simple API. I think I fear anything that'll complicate this API or the internals, and ultimately result in things being worse for the memory -> memory case.