Notice: grpc-go will stop sending RPCs before HTTP/2 SETTINGS frame is received #64

dfawley · 2018-10-26T20:33:11Z

This is a notice that grpc-go intends to change in a way that we know will break the way cmux currently works by default. This will bring grpc-go in line with grpc-java's behavior, and C/wrapped languages will be following suit as well. Details and justification for the change can be found in grpc/grpc#17006. grpc-go's migration plan is proposed in grpc/grpc-go#2406. Please feel free to comment in the appropriate PR/issue for questions or concerns about this. Apologies in advance for the breaking change.

This removes RequireHandshakeHybrid support and changes the default behavior to RequireHandshakeOn. Dial calls will now block and wait for a successful handshake before proceeding. Users relying on the old hybrid behavior (cmux users) should consult soheilhy/cmux#64. Also, several tests have been updated to take this into consideration by sending settings frames.

due to soheilhy/cmux#64 fix #820 fix #840 fix #841

This removes RequireHandshakeHybrid support and changes the default behavior to RequireHandshakeOn. Dial calls will now block and wait for a successful handshake before proceeding. Users relying on the old hybrid behavior (cmux users) should consult soheilhy/cmux#64. Also, several tests have been updated to take this into consideration by sending settings frames.

Iulian7 · 2019-02-08T23:26:31Z

Any ideas how to match grpc-go requests then?

dfawley · 2019-02-08T23:29:20Z

@lulian7 - IIUC you should be able to use the workaround suggested here for Java gRPC clients: https://github.com/soheilhy/cmux#limitations

Iulian7 · 2019-02-08T23:33:39Z

@dfawley Works. I missed that.

This removes RequireHandshakeHybrid support and changes the default behavior to RequireHandshakeOn. Dial calls will now block and wait for a successful handshake before proceeding. Users relying on the old hybrid behavior (cmux users) should consult soheilhy/cmux#64. Also, several tests have been updated to take this into consideration by sending settings frames.

johanbrandhorst · 2019-05-13T21:46:52Z

Testing with gRPC-Go 1.19.1 I can't get this to work even with

grpcL := mux.MatchWithWriters(cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc"))

Has anything changed in 1.19 to make this even harder? I tried debugging by sending non-matched entries to another listener and printing the output and this is what I got:

���c+��]
80�l

or as a byte slice:

[22 3 1 0 217 1 0 0 213 3 3 86 90 234 191 121 47 47 94 124 149 27 35 62]

I'm not sure what format this is. Any ideas? @menghanl @dfawley

menghanl · 2019-05-13T22:05:55Z

@johanbrandhorst No, this behavior didn't change in 1.19.
I also don't recognize the bytes...

Does it work with 1.18? Is the client a simple client?

johanbrandhorst · 2019-05-13T22:24:07Z

I haven't tried 1.18 yet, but yes it is a simple grpc-go client. I'll try 1.18 and below.

johanbrandhorst · 2019-05-21T20:57:41Z

Hm, everything is working as expected when serving gRPC without TLS. I guess this is a false alarm. What's the best way to use cmux and still encrypt the connection?

johanbrandhorst · 2019-05-21T21:27:15Z

For future reference;

I got the following working:

Use tls.Listen to create the listener.
Use (*grpc.Server).Serve without grpc.Creds.
Use grpc.WithTransportCredentials as usual in the client.

This was carried over from some ancient grpc-gateway example code[1]; and we're not exposing the GRPC API that way right now. Using (*grpc.Server).ServeHTTP has some issues: 1. it's known to be slow, very slow[2] 2. it's seems to be not working with go-grpc >= 1.19 [3] Furthermore, we're exposing the proper GRPC server (the native go-grpc one) on a different port. If decide to (finally!) expose our GRPC API, that's the thing we should expose. So, since we don't need this multiplexing business, we shouldn't keep the code around. This is the cleanup. [1]: https://github.com/philips/grpc-gateway-example/ [2]: grpc/grpc-go#586 [3]: soheilhy/cmux#64 (comment) Signed-off-by: Stephan Renatus <[email protected]>

* automate-gateway: cleanup GRPC/HTTP multiplexing cruft This was carried over from some ancient grpc-gateway example code[1]; and we're not exposing the GRPC API that way right now. Using (*grpc.Server).ServeHTTP has some issues: 1. it's known to be slow, very slow[2] 2. it's seems to be not working with go-grpc >= 1.19 [3] Furthermore, we're exposing the proper GRPC server (the native go-grpc one) on a different port. If decide to (finally!) expose our GRPC API, that's the thing we should expose. So, since we don't need this multiplexing business, we shouldn't keep the code around. This is the cleanup. [1]: https://github.com/philips/grpc-gateway-example/ [2]: grpc/grpc-go#586 [3]: soheilhy/cmux#64 (comment) * [integration] automate-gateway: fix tests to use GRPC port So apparently these HAD used the multiplexing code I've deleted. Well, they don't have to. Changed the hardcoded port to automate-gateway's default grpc port. Signed-off-by: Stephan Renatus <[email protected]>

glerchundi · 2019-07-01T08:04:43Z

@johanbrandhorst I finally took some time to sketch a code snippet to make our use case work. It did, but I don't know if it's going to work after reading this commit (as we didn't upgraded to that grpc-go version yet). I want to expose grpc-gateway + pure grpc in the same port without security; It's a microservice never exposed directly to the outside so it's ok. But I don't want to commit to a solution that cannot be extended in the future by adopting an approach like BeyondCorp. This means that our microservice should be able to be exposed publicly by, obviously, using TLS. The mentioned code allows addition TLS over the whole cmux but I don't know if you encountered further problems with this approach?

johanbrandhorst · 2019-07-01T08:18:51Z

There should be no reason you can't do TLS termination before any packets reach the service.

glerchundi · 2019-07-01T20:57:13Z

Correct, thanks for confirming. Although we found another issue with the combo grpc-gateway, grpc, port-sharing, cmux and persistent connections kept by a reverse proxy (envoy in this case). It seems that cmux is not very useful if we're behind a proxy which keeps persistent connection without doing any differentiation between the underlying protocol being used.

Take this sequence as an example:
As stated by cmux, it matches the connection when it's accepted, so it's possible this to happen:

Someone makes a gRPC call
Envoy creates a new http connection
cmux sees it's a http2 connection and the content-type is gRPC it then creates a new connection against grpc.Serve
grpc.Serve replies to the gRPC request succesfully. While Envoy decides not to close the fresh connection for performance reasons
Someone makes a REST call
Envoy decides to use the same connection created before
As the connection was previously matched grpc.Serve is called again and it fails as it's not a gRPC call

WDYT @johanbrandhorst?

PS.: I don't really know which is the right forum for this question as there are lots of little components taking part in the overall problem...

johanbrandhorst · 2019-07-01T21:04:01Z

This is probably not the place, no. Maybe raise an issue against Envoy?

glerchundi · 2019-07-02T06:35:53Z

This is probably not the place, no. Maybe raise an issue against Envoy?

Already asked in Envoy and it seems like nothing could be done there as it keeps one and only connection pool for everything no matter it's plain or gRPC.

UPDATE: It can be done but cannot because we're using Contour to manage Envoy.

@mpuncel wrote:

Envoy uses the same connection pool for gRPC and non-gRPC. You could probably accomplish what you want with 2 different clusters and set up a routing rule that matches on gRPC to go to 1 and everything else goes to the other

johanbrandhorst · 2019-07-02T07:10:47Z

There's a question to be asked about why you're using cmux behind a proxy like envoy anyway. Envoy can do the traffic splitting for you, so why not just remove the cmux?

glerchundi · 2019-07-02T08:35:44Z

This is probably not the place, no. Maybe raise an issue against Envoy?

There are several reasons:

We want to have one and only one container for both (gRPC and gRPC-Gateway) as it simplifies everything from management/operation to understanding of how the microservice works by the team.
The operational cost for having multiple ports is bigger than having only one.
Although Envoy supports multiplexing connections based on more advanced L7 rules (like header content based ones), ~~Heptio~~ VMware Contour doesn't support it yet. Which means that currently it's not even possible.

Did you have this problem previously @johanbrandhorst? How do you currently operate services that publish both gRPC + REST in your environment?

Thanks!

johanbrandhorst · 2019-07-02T08:37:54Z

I think we can move this discussion to the #grpc channel on Gophers slack. We've taken up this issue for long enough.

breaking change: grpc/grpc-go#2406 workaround described in: - soheilhy/cmux#64 - https://github.com/soheilhy/cmux#limitations

breaking change: grpc/grpc-go#2406 workaround described in: - soheilhy/cmux#64 - https://github.com/soheilhy/cmux#limitations Signed-off-by: Christian Weichel <[email protected]>

* Add unit test for gRPC over cmux Signed-off-by: Yuri Shkuro <[email protected]> * Fix tests Signed-off-by: Yuri Shkuro <[email protected]> * Fix gRPC query service cmux breaking change: grpc/grpc-go#2406 workaround described in: - soheilhy/cmux#64 - https://github.com/soheilhy/cmux#limitations Signed-off-by: Christian Weichel <[email protected]> * Fix asertions Signed-off-by: Yuri Shkuro <[email protected]> * Use DialContext Signed-off-by: Yuri Shkuro <[email protected]> * Clean-up timeouts Signed-off-by: Yuri Shkuro <[email protected]>

ghost · 2019-10-07T13:36:05Z

TLS: net/http uses a type assertion to identify TLS connections; since cmux's lookahead-implementing connection wraps the underlying TLS connection, this type assertion fails. Because of that, you can serve HTTPS using cmux but http.Request.TLS would not be set in your handlers.

do I understand this correctly if I assume that I can still serve http2 and http1(grpc server+ grpc gateway server) via cmux on the same port and have TLS for the clients and I only have to enable tls on the listener that I provide to cmux and disable tls for the servers the cmux is routing to? ie. I still can serve secure grpc and grpc gateway via cmux since both servers are linked to the cmux only via memory and not network, hence no security issue with tls "termination" on cmux level?

johanbrandhorst · 2019-10-08T06:43:37Z

That is correct, TLS is terminated by cmux, but it's still in-memory, so it's about the same level of security.

delwaterman · 2021-07-07T13:15:11Z

Ran into this issue. Can we at least update the documentation for this?

gRPC-Go has been waiting for the settings frame for a while now (soheilhy#64), and having the old handler in the example is confusing users (soheilhy#67). Use the new handler in the example and remove the "Limitations" from the README.

* Add unit test for gRPC over cmux Signed-off-by: Yuri Shkuro <[email protected]> * Fix tests Signed-off-by: Yuri Shkuro <[email protected]> * Fix gRPC query service cmux breaking change: grpc/grpc-go#2406 workaround described in: - soheilhy/cmux#64 - https://github.com/soheilhy/cmux#limitations Signed-off-by: Christian Weichel <[email protected]> * Fix asertions Signed-off-by: Yuri Shkuro <[email protected]> * Use DialContext Signed-off-by: Yuri Shkuro <[email protected]> * Clean-up timeouts Signed-off-by: Yuri Shkuro <[email protected]>

jeanbza mentioned this issue Dec 20, 2018

transport: remove RequireHandshakeHybrid support grpc/grpc-go#2529

Merged

chrislusf added a commit to seaweedfs/seaweedfs that referenced this issue Jan 18, 2019

master add separate grpc port

67e2ea7

due to soheilhy/cmux#64 fix #820 fix #840 fix #841

This was referenced Mar 12, 2019

all SubConns are in TransientFailure, latest connection error: <nil> grpc/grpc-go#2636

Closed

README: Update limitations for gRPC clients #66

Open

duhruh mentioned this issue May 17, 2019

[FEATURE] Move to go modules grafeas/grafeas#297

Closed

srenatus mentioned this issue May 22, 2019

automate-gateway: cleanup GRPC/HTTP multiplexing cruft chef/automate#402

Merged

duhruh mentioned this issue May 22, 2019

Proposal: Move to go modules grafeas/grafeas#301

Closed

mohammed90 mentioned this issue Jul 2, 2019

gRPC Support keratin/authn-server#83

Closed

9 tasks

csweichel pushed a commit to csweichel/jaeger that referenced this issue Aug 23, 2019

Fix gRPC query service cmux

8c812fb

breaking change: grpc/grpc-go#2406 workaround described in: - soheilhy/cmux#64 - https://github.com/soheilhy/cmux#limitations

csweichel mentioned this issue Aug 23, 2019

Fix gRPC query service cmux jaegertracing/jaeger#1756

Closed

july2993 mentioned this issue Sep 27, 2019

upgrade grpc to 1.23 pingcap/tidb-binlog#756

Merged

menghanl mentioned this issue Dec 4, 2019

rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: timed out waiting for server handshake grpc/grpc-go#3215

Closed

mpictor mentioned this issue Feb 26, 2020

grpc-http-same-port example works (due to vendor/) but no longer matches upstream drgarcia1986/grpc-hw#3

Open

johanbrandhorst mentioned this issue Mar 31, 2020

how to do tls auth in grpc+grpc-gateway grpc-ecosystem/grpc-gateway#727

Closed

kevincantu mentioned this issue Oct 13, 2020

otel-collector requires HTTP/2 TLS passthrough from Envoy / Contour: should it? open-telemetry/opentelemetry-collector#1916

Closed

tonglil mentioned this issue Mar 19, 2021

Verify whether mTLS works open-telemetry/opentelemetry-collector#2732

Closed

johanbrandhorst mentioned this issue Jan 10, 2022

Use new gRPC matcher for README example #90

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Notice: grpc-go will stop sending RPCs before HTTP/2 SETTINGS frame is received #64

Notice: grpc-go will stop sending RPCs before HTTP/2 SETTINGS frame is received #64

dfawley commented Oct 26, 2018

Iulian7 commented Feb 8, 2019

dfawley commented Feb 8, 2019

Iulian7 commented Feb 8, 2019

johanbrandhorst commented May 13, 2019

menghanl commented May 13, 2019

johanbrandhorst commented May 13, 2019

johanbrandhorst commented May 21, 2019

johanbrandhorst commented May 21, 2019 •

edited

Loading

glerchundi commented Jul 1, 2019

johanbrandhorst commented Jul 1, 2019

glerchundi commented Jul 1, 2019 •

edited

Loading

johanbrandhorst commented Jul 1, 2019

glerchundi commented Jul 2, 2019 •

edited

Loading

johanbrandhorst commented Jul 2, 2019

glerchundi commented Jul 2, 2019

johanbrandhorst commented Jul 2, 2019

ghost commented Oct 7, 2019

johanbrandhorst commented Oct 8, 2019

delwaterman commented Jul 7, 2021

Notice: grpc-go will stop sending RPCs before HTTP/2 SETTINGS frame is received #64

Notice: grpc-go will stop sending RPCs before HTTP/2 SETTINGS frame is received #64

Comments

dfawley commented Oct 26, 2018

Iulian7 commented Feb 8, 2019

dfawley commented Feb 8, 2019

Iulian7 commented Feb 8, 2019

johanbrandhorst commented May 13, 2019

menghanl commented May 13, 2019

johanbrandhorst commented May 13, 2019

johanbrandhorst commented May 21, 2019

johanbrandhorst commented May 21, 2019 • edited Loading

glerchundi commented Jul 1, 2019

johanbrandhorst commented Jul 1, 2019

glerchundi commented Jul 1, 2019 • edited Loading

johanbrandhorst commented Jul 1, 2019

glerchundi commented Jul 2, 2019 • edited Loading

johanbrandhorst commented Jul 2, 2019

glerchundi commented Jul 2, 2019

johanbrandhorst commented Jul 2, 2019

ghost commented Oct 7, 2019

johanbrandhorst commented Oct 8, 2019

delwaterman commented Jul 7, 2021

johanbrandhorst commented May 21, 2019 •

edited

Loading

glerchundi commented Jul 1, 2019 •

edited

Loading

glerchundi commented Jul 2, 2019 •

edited

Loading