Skip to content
This repository has been archived by the owner on Nov 30, 2021. It is now read-only.

Allowing use of proxy with http streaming #78

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions lib/looker-sdk/client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -351,11 +351,15 @@ def build_response(connection, request)
OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE,
}

# TODO: figure out how/if to support proxies
# TODO: figure out how to test this comprehensively

progress = nil
Net::HTTP.start(uri.host, uri.port, connect_opts) do |http|
http_klass = if connection.proxy.present?
Net::HTTP::Proxy(connection.proxy.uri.host, connection.proxy.uri.port)
else
Net::HTTP
end
http_klass.start(uri.host, uri.port, connect_opts) do |http|
http.open_timeout = connection.options.open_timeout rescue 30
http.read_timeout = connection.options.timeout rescue 60

Expand Down
26 changes: 17 additions & 9 deletions streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,23 @@

#### Beta Feature - Experimental!

This SDK makes it easy to fetch a response from a Looker API and hydrate it into a Ruby object.This convenience is great for working with configuration and administrative data. However, when the response is gigabytes of row data, pulling it all into memory doesn't work so well - you can't begin processing the data until after it has all downloaded, for example, and chewing up tons of memory will put a serious strain on the entire system - even crash it.
This SDK makes it easy to fetch a response from a Looker API and hydrate it into a Ruby object.This convenience is great for working with configuration and administrative data. However, when the response is gigabytes of row data, pulling it all into memory doesn't work so well - you can't begin processing the data until after it has all downloaded, for example, and chewing up tons of memory will put a serious strain on the entire system - even crash it.

One solution to all this is to use streaming downloads to process the data in chunks as it is downloaded. Streaming requires a little more code to set up but the benefits can be significant.
One solution to all this is to use streaming downloads to process the data in chunks as it is downloaded. Streaming requires a little more code to set up but the benefits can be significant.

To use streaming downloads with the Looker SDK, simply add a block to an SDK call. The block will be called with chunks of data as they are downloaded instead of the method returning a complete result.

For example:

```ruby
```ruby
def run_look_to_file(look_id, filename, format, opts = {})
File.open(filename, 'w') do |file|
sdk.run_look(look_id, format, opts) do |data, progress|
file.write(data)
puts "Wrote #{data.length} bytes of #{progress.length} total"
end
end
end
end
end
end

run_look_to_file(38, 'out.csv', 'csv', limit: 10000)
```
Expand All @@ -40,7 +40,7 @@ You can also abort a streaming download by calling `progress.stop` within the bl
else
process_data(data)
end
end
end

```

Expand All @@ -53,7 +53,15 @@ These caveats can be mitigated by knowing the structure of the data being stream

#### A Tale of Two Stacks

The Looker Ruby SDK is built on top of Sawyer, and Sawyer sits on top of Faraday. Faraday does not support HTTP Streaming, so to do our streaming stuff we have to bypass Faraday and talk directly to the Net:HTTP stack. Our streaming implementation gathers connection settings from the Faraday stack so there's no additional config required for http streaming.
The Looker Ruby SDK is built on top of Sawyer, and Sawyer sits on top of Faraday. Faraday does not support HTTP Streaming, so to do our streaming stuff we have to bypass Faraday and talk directly to the Net:HTTP stack. Our streaming implementation gathers connection settings from the Faraday stack so there's no additional config required for http streaming.

Streaming downloads have not been tested with proxy connections. We assume attempting to stream across an http proxy will not work.
#### Proxy Connections
In order to stream with a proxy connection, you can define your proxy url in your client configs like so:
```
LookerSDK::Client.new(
client_id: ENV['CLIENT_ID'],
client_secret: ENV['CLIENT_SECRET'],
proxy: ENV['HTTP_PROXY']
)
```

9 changes: 9 additions & 0 deletions test/looker/test_client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,15 @@
client.client_secret.wont_be_nil
end
end

it "sets proxy with module methods" do
client = LookerSDK::Client.new
test_proxy = "http://my-test-proxy.com:232"
client.configure do |config|
config.proxy = test_proxy
end
client.instance_variable_get(:'@proxy').must_equal test_proxy
end
end

describe "config tests" do
Expand Down