From 92573df026b969f088a31b9943821941157d564c Mon Sep 17 00:00:00 2001 From: Corinne Kunze Date: Fri, 14 Sep 2018 11:33:36 -0700 Subject: [PATCH] Ability to use http proxy with StreamClient --- lib/looker-sdk/client.rb | 8 ++++++-- streaming.md | 26 +++++++++++++++++--------- test/looker/test_client.rb | 9 +++++++++ 3 files changed, 32 insertions(+), 11 deletions(-) diff --git a/lib/looker-sdk/client.rb b/lib/looker-sdk/client.rb index 49726cc..43c1ebd 100644 --- a/lib/looker-sdk/client.rb +++ b/lib/looker-sdk/client.rb @@ -351,11 +351,15 @@ def build_response(connection, request) OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE, } - # TODO: figure out how/if to support proxies # TODO: figure out how to test this comprehensively progress = nil - Net::HTTP.start(uri.host, uri.port, connect_opts) do |http| + http_klass = if connection.proxy.present? + Net::HTTP::Proxy(connection.proxy.uri.host, connection.proxy.uri.port) + else + Net::HTTP + end + http_klass.start(uri.host, uri.port, connect_opts) do |http| http.open_timeout = connection.options.open_timeout rescue 30 http.read_timeout = connection.options.timeout rescue 60 diff --git a/streaming.md b/streaming.md index 7f27d65..0b7e966 100644 --- a/streaming.md +++ b/streaming.md @@ -2,23 +2,23 @@ #### Beta Feature - Experimental! -This SDK makes it easy to fetch a response from a Looker API and hydrate it into a Ruby object.This convenience is great for working with configuration and administrative data. However, when the response is gigabytes of row data, pulling it all into memory doesn't work so well - you can't begin processing the data until after it has all downloaded, for example, and chewing up tons of memory will put a serious strain on the entire system - even crash it. +This SDK makes it easy to fetch a response from a Looker API and hydrate it into a Ruby object.This convenience is great for working with configuration and administrative data. However, when the response is gigabytes of row data, pulling it all into memory doesn't work so well - you can't begin processing the data until after it has all downloaded, for example, and chewing up tons of memory will put a serious strain on the entire system - even crash it. -One solution to all this is to use streaming downloads to process the data in chunks as it is downloaded. Streaming requires a little more code to set up but the benefits can be significant. +One solution to all this is to use streaming downloads to process the data in chunks as it is downloaded. Streaming requires a little more code to set up but the benefits can be significant. To use streaming downloads with the Looker SDK, simply add a block to an SDK call. The block will be called with chunks of data as they are downloaded instead of the method returning a complete result. For example: -```ruby +```ruby def run_look_to_file(look_id, filename, format, opts = {}) File.open(filename, 'w') do |file| sdk.run_look(look_id, format, opts) do |data, progress| file.write(data) puts "Wrote #{data.length} bytes of #{progress.length} total" - end - end -end + end + end +end run_look_to_file(38, 'out.csv', 'csv', limit: 10000) ``` @@ -40,7 +40,7 @@ You can also abort a streaming download by calling `progress.stop` within the bl else process_data(data) end - end + end ``` @@ -53,7 +53,15 @@ These caveats can be mitigated by knowing the structure of the data being stream #### A Tale of Two Stacks -The Looker Ruby SDK is built on top of Sawyer, and Sawyer sits on top of Faraday. Faraday does not support HTTP Streaming, so to do our streaming stuff we have to bypass Faraday and talk directly to the Net:HTTP stack. Our streaming implementation gathers connection settings from the Faraday stack so there's no additional config required for http streaming. +The Looker Ruby SDK is built on top of Sawyer, and Sawyer sits on top of Faraday. Faraday does not support HTTP Streaming, so to do our streaming stuff we have to bypass Faraday and talk directly to the Net:HTTP stack. Our streaming implementation gathers connection settings from the Faraday stack so there's no additional config required for http streaming. -Streaming downloads have not been tested with proxy connections. We assume attempting to stream across an http proxy will not work. +#### Proxy Connections +In order to stream with a proxy connection, you can define your proxy url in your client configs like so: +``` +LookerSDK::Client.new( + client_id: ENV['CLIENT_ID'], + client_secret: ENV['CLIENT_SECRET'], + proxy: ENV['HTTP_PROXY'] +) +``` diff --git a/test/looker/test_client.rb b/test/looker/test_client.rb index ea839e0..1f4bfb4 100644 --- a/test/looker/test_client.rb +++ b/test/looker/test_client.rb @@ -92,6 +92,15 @@ client.client_secret.wont_be_nil end end + + it "sets proxy with module methods" do + client = LookerSDK::Client.new + test_proxy = "http://my-test-proxy.com:232" + client.configure do |config| + config.proxy = test_proxy + end + client.instance_variable_get(:'@proxy').must_equal test_proxy + end end describe "config tests" do