Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept output buffer in Response::Body#readpartial #472

Closed

Conversation

janko
Copy link
Member

@janko janko commented May 11, 2018

This makes it possible to use a HTTP::Response::Body object as a source argument in IO.copy_stream, allowing easy streaming. It also allows it to be used in the request body for another request (as HTTP::Request::Body::Writer uses IO.copy_stream), should anyone want that.

response = HTTP.get("http://example.com/download")
IO.copy_stream(response.body, destination)

Since IO.copy_stream uses an output buffer, it makes retrieving the response body much more memory efficient, because we're able to deallocate retrieved chunks. I tested with a 50MB response body, and streaming with HTTP::Response::Body#each allocated about 52MB of strings, whereas the IO.copy_stream version allocated only 3MB.

IO.copy_stream requires #readpartial to have IO#readpartial semantics, which among others includes raising and EOFError when end of file was reached. Since IO.copy_stream always uses an output buffer, the exception is raised only when output buffer is specified.

@janko janko force-pushed the support-output-buffer-in-body-readpartial branch 2 times, most recently from cc19592 to 1062b26 Compare May 11, 2018 13:27
This makes it possible to use a `HTTP::Response::Body` object as a
source argument in `IO.copy_stream`, allowing easy streaming.

  response = HTTP.get("http://example.com/download")
  IO.copy_stream(response.body, "destination.txt")

Since `IO.copy_stream` uses an output buffer, it makes retrieving the
response body much more memory efficient, because we can now deallocate
retrieved chunks.
@janko janko force-pushed the support-output-buffer-in-body-readpartial branch from 1062b26 to a06be0b Compare May 25, 2018 21:43
@janko
Copy link
Member Author

janko commented May 25, 2018

This one needs JRuby 9.2.0.0 to pass the tests (which has the fix for jruby/jruby#5167), but it's currently failing because of #475.

@janko
Copy link
Member Author

janko commented May 29, 2018

Now that I think about this more, this is probably an overkill. I don't like the idea of HTTP::Response::Body#readpartial partially mimicking IO#readpartial, it doesn't need to act as an IO object. People can just clear yielded chunks themselves if they want to save on memory.

@janko janko closed this May 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant