Remove content length from upload to enable chunked #208
Conversation
|
@djw8605 - is there a way to check for server compatibility? What happens when you try a chunk encoded PUT against an older server? |
|
I suppose you could do a separate request to the xrootd server to determine the version. If you do a chunked encoded PUT against an older (well, current as of now) server, it may or may not work. It all depends on timing, buffers, bandwidth... The larger the file, the more of a chance of failure. Though, even small files frequently failed in my tests. |
|
@djw8605 - I'm tempted to merge this one. The relevant server version has been released for about a month and I haven't seen any major complaints. If we merge now, it'd make it into the v7.3.0 (early December) release, giving a few weeks for any straggling origins to upgrade. Thoughts? |
|
Sounds good to me, setting for ready for review. |
|
I suggest doing a rebase on main and pushing in order to run all the tests again with updated code. It's been over two months and many things have changed since then and I'd like the extra sanity check before merging in (I doubt it would fail, but just in case). |
Removing the content length from a put changes the upload style from full file upload to a chunked encoded upload. A full file upload causes go to read in the entire file in memory while uploading, chunked does not. **NOTE** chunked uploads only work after xrootd has merged and put into production xrootd/xrootd#2102 Corresponds to htcondor/osdf-client#88
bbockelm
force-pushed
the
use-chunked-upload
branch
from
a660a4f to
2273cc9
Compare
Removing the content length from a put changes the upload style from full file upload to a chunked encoded upload. A full file upload causes go to read in the entire file in memory while uploading, chunked does not.
NOTE chunked uploads only work after xrootd has merged and put into production xrootd/xrootd#2102
Corresponds to htcondor/osdf-client#88