Option to add proxy

[nkpanda]

Is there any option to forward the traffic via any app proxy? This will help in easy edit of json requests and understanding of full http requests.

[jhump]

grpcurl does not implement the HTTP transport or gRPC protocol details. Instead, it relies on the grpc/grpc-go project for that.

Looking through proxy-related issues in that project (such as grpc/grpc-go#1446), it appears that a proxy can be used by simply setting environment variables. I will investigate and report back how to set that up.

However, that linked issue above does state some limitations with the support of proxies that will apply here.

[bbuckland]

@jhump Is there any update here? Trying to get grpcurl to proxy traffic through tools like Charles or Proxyman.

[jhump]

It looks like the Go runtime for gRPC, which is what grpcurl uses under its hood, supports environment variables in a similar fashion as curl and python: HTTP_PROXY, HTTPS_PROXY, and NO_PROXY can inform the program how to route a request through a proxy.

So I think the following might work. Please try it out:

HTTP_PROXY=127.0.0.1:8080 grpcurl ...

[terinjokes]

@jhump I've tried this with HTTP_PROXY and HTTPS_PROXY and neither changed the behavior of grpcurl. I can see with the output of ss -ntp | grep grpcurl that it continued to connect directly instead of using my SSH tunnel.

[Z5eyhS0uubejR0SVmX2O]

@jhump I've tried this with HTTP_PROXY and HTTPS_PROXY and neither changed the behavior of grpcurl. I can see with the output of ss -ntp | grep grpcurl that it continued to connect directly instead of using my SSH tunnel.

Same here:

$ docker run --rm -e HTTP_PROXY=127.0.0.1:1234 fullstorydev/grpcurl api.grpc.me:443 list
grpc.reflection.v1alpha.ServerReflection
listennotes.v1.Podcasts

$ docker run --rm -e HTTPS_PROXY=127.0.0.1:1234 fullstorydev/grpcurl api.grpc.me:443 list
grpc.reflection.v1alpha.ServerReflection
listennotes.v1.Podcasts

$ docker run --rm -e HTTPS_PROXY=127.0.0.1:1234 -e GRPC_GO_LOG_VERBOSITY_LEVEL=99 -e GRPC_GO_LOG_SEVERITY_LEVEL=info fullstorydev/
grpcurl api.grpc.me:443 list
INFO: 2021/06/04 20:00:29 [core] parsed scheme: ""
INFO: 2021/06/04 20:00:29 [core] scheme "" not registered, fallback to default scheme
INFO: 2021/06/04 20:00:29 [core] ccResolverWrapper: sending update to cc: {[{api.grpc.me:443  <nil> 0 <nil>}] <nil> <nil>}
INFO: 2021/06/04 20:00:29 [core] ClientConn switching balancer to "pick_first"
INFO: 2021/06/04 20:00:29 [core] Channel switches to new LB policy "pick_first"
INFO: 2021/06/04 20:00:29 [core] Subchannel Connectivity change to CONNECTING
INFO: 2021/06/04 20:00:29 [core] Subchannel picks a new address "api.grpc.me:443" to connect
INFO: 2021/06/04 20:00:29 [core] pickfirstBalancer: UpdateSubConnState: 0xc0003504d0, {CONNECTING <nil>}
INFO: 2021/06/04 20:00:29 [core] Channel Connectivity change to CONNECTING
INFO: 2021/06/04 20:00:29 [core] Subchannel Connectivity change to READY
INFO: 2021/06/04 20:00:29 [core] pickfirstBalancer: UpdateSubConnState: 0xc0003504d0, {READY <nil>}
INFO: 2021/06/04 20:00:29 [core] Channel Connectivity change to READY
grpc.reflection.v1alpha.ServerReflection
listennotes.v1.Podcasts
INFO: 2021/06/04 20:00:29 [core] Channel Connectivity change to SHUTDOWN
INFO: 2021/06/04 20:00:29 [core] Subchannel Connectivity change to SHUTDOWN

[Z5eyhS0uubejR0SVmX2O]

The proxy settings are overridden by the customized dialer:

	dialer := func(ctx context.Context, address string) (net.Conn, error) {
		// NB: We *could* handle the TLS handshake ourselves, in the custom
		// dialer (instead of customizing both the dialer and the credentials).
		// But that requires using WithInsecure dial option (so that the gRPC
		// library doesn't *also* try to do a handshake). And that would mean
		// that the library would send the wrong ":scheme" metaheader to
		// servers: it would send "http" instead of "https" because it is
		// unaware that TLS is actually in use.
		conn, err := (&net.Dialer{}).DialContext(ctx, network, address)
		if err != nil {
			writeResult(err)
		}
		return conn, err
	}

It looks like this is the recommended path to take for proxy setting administration according to grpc/grpc-go#1446
It just wasn't coded to support proxies.

[jhump]

@terinjokes and @Z5eyhS0uubejR0SVmX2O, thanks for trying it out. I'm sorry that didn't work.

I'll have to do more research to understand how to correctly do the proxying using a custom dialer and then figure out a way to test.

If you have the inclination to investigate and then implement and test a fix, I'd happily accept a pull request.

[joanbono]

Hi! Is there any option to implement this or it is not planned atm?

[krugi]

Is there a plan to implement it?

[s4ke]

It would be great if this were supported. There already seems to exist a suitable PR?

[nmische]

I have submitted two potential fixes for this issue based on how grpc-go supports proxies.

The first fix removes the custom grpcurl dialer such that the upstream grpc-go dialer, which supports proxies, is used. PR #480 implements this approach. This seems to be the simplest approach, however I'm not sure why the custom dialer was implemented in the first place. I could be missing something important here....

If a custom dialer is required, another approach may be to port the grpc-go proxy dialer over to grpcurl. (Unfortunately this dialer cannot be imported and used directly because it is in an internal package.) PR #481 implements this approach, however, one major disadvantage I see with this approach is that the ported code needs to be kept up-to-date with the upstream.

[nmische]

@jhump or @dragonsinth: Just checking in to see if there is any feedback on #480 or #481? Thanks!

[dragonsinth]

#480 seems better to me, but I'd love to hear from @jhump

[ringerc]

Ideally this would respect the http_proxy and https_proxy env-vars for use with tools like kubectl socks5-proxy

[nmische]

Ideally this would respect the http_proxy and https_proxy env-vars for use with tools like kubectl socks5-proxy

Both #480 and #481 leverage the standard library's http.ProxyFromEnvironment (https://pkg.go.dev/net/http#ProxyFromEnvironment), so either would support those environment variables (as well as NO_PROXY).

[dragonsinth]

Another contestant
#487

[dragonsinth]

@jhump #480 seems the most straightforward to me.. it's just removing some complexity where we are using a custom dialer, to allow grpc to use the default dialer. I'm inclined to go in that direction, unless you can think of some reason that would be bad?

[nmische]

@dragonsinth, is there any possibility of moving forward without input from @jhump? I've been using #480 for several weeks and have not encountered any issues... Granted, I have not done anything too advanced, such as mTLS, but #480 should be fairly easy to revert if it does prove to cause any issues.

[LinPr]

well, I have a suggestion，since the description of grpcurl project is "Like cURL, but for gRPC", I think not only the environment variables , but also the the command line flags like curl -x, --proxy [protocol://]host[:port] Use this proxy need to be supported. Because usually the envrionment variable is more like a global configuration, it may have side impacts on some other apps running in the same session，but the command flag is only for one shot.

[dragonsinth]

Aight, let's just go with the "delete custom dialer and let gRPC do its thing" and see how that does. Thanks for all the contributions and discussion everyone.

[dragonsinth]

#480 merged

[nmische]

@dragonsinth: Thanks for all your help with this one. Will you be cutting a new release?

[dragonsinth]

@dragonsinth: Thanks for all your help with this one. Will you be cutting a new release?

Eventually, be we don't have extreme urgency around it since people can always go install grpcurl pretty easily.

[nmische]

Sounds good.

FWIW, I have use cases where I would like to install the pre-built binaries from https://github.com/fullstorydev/grpcurl/releases in environments without golang.