Upload 0 byte files on file closure

[muddyfish]

Tell us more about this new feature.

When a file is closed with 0 bytes written, Mountpoint currently waits until all references are removed before committing to S3. This is a sharp edge customers have faced in case they actually want to write an empty file to S3.

On a filesystem level, Mountpoint only knows about file handles, but on the user side, there are file descriptors. The user can duplicate file descriptors pointing to the same file handle. When the customer closes a file handle, MP receives a flush call. When the last file descriptor is closed, MP receives a release. Mountpoint needs to decide when to do a MPU complete, and currently we try to do this on flush. Mountpoint generally doesn't allow further writes after a flush except if the current size is 0 bytes to support cases such as touch, and shell redirects.

This is a deliberate special case to our semantics for these common cases (touch & shell redirects).

We're proposing modifying this behavior by writing to S3 when the file size is 0, by double-committing and actually writing 0 bytes to S3. If we later get more data, we can start a new upload as semantically replacing is the same as appending to a zero byte file.

We should consider the usage of If-Match headers to verify the file hasn't been changed between the two commits.