Skip to content

grpc: Fix cardinality violations in non-client streaming RPCs. #8385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions server.go
Original file line number Diff line number Diff line change
Expand Up @@ -1598,6 +1598,7 @@ func (s *Server) processStreamingRPC(ctx context.Context, stream *transport.Serv
s: stream,
p: &parser{r: stream, bufferPool: s.opts.bufferPool},
codec: s.getCodec(stream.ContentSubtype()),
desc: sd,
maxReceiveMessageSize: s.opts.maxReceiveMessageSize,
maxSendMessageSize: s.opts.maxSendMessageSize,
trInfo: trInfo,
Expand Down
18 changes: 17 additions & 1 deletion stream.go
Original file line number Diff line number Diff line change
Expand Up @@ -1580,6 +1580,7 @@
s *transport.ServerStream
p *parser
codec baseCodec
desc *StreamDesc

compressorV0 Compressor
compressorV1 encoding.Compressor
Expand Down Expand Up @@ -1774,6 +1775,9 @@
binlog.Log(ss.ctx, chc)
}
}
if !ss.desc.ClientStreams {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not immediately clear to me how this conditions indicates RecvMsg is called twice. From my understanding, the handling of cardinality violations in the server stream should be similar to the one in client stream. The check here should be for 0 requests being received for a non-client-streaming request.

grpc-go/stream.go

Lines 1142 to 1144 in e5de1e2

if !cs.desc.ServerStreams {
return status.Error(codes.Internal, "cardinality violation: received no response message from non-streaming RPC")
}

For > 1 requests, there should be a block near the end of this method similar to the following:

grpc-go/stream.go

Lines 1167 to 1178 in e5de1e2

if cs.desc.ServerStreams {
// Subsequent messages should be received by subsequent RecvMsg calls.
return nil
}
// Special handling for non-server-stream rpcs.
// This recv expects EOF or errors, so we don't collect inPayload.
if err := recv(a.parser, cs.codec, a.transportStream, a.decompressorV0, m, *cs.callInfo.maxReceiveMessageSize, nil, a.decompressorV1, false); err == io.EOF {
return a.transportStream.Status().Err() // non-server streaming Recv returns nil on success
} else if err != nil {
return toRPCErr(err)
}
return status.Error(codes.Internal, "cardinality violation: expected <EOF> for non server-streaming RPCs, but received another message")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any issue if we try to follow the same pattern as client streams here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In server-streaming, client can't send more than one message. So the following code will be useless for server.recvmsg().

grpc-go/stream.go

Lines 1167 to 1178 in e5de1e2

if cs.desc.ServerStreams {
// Subsequent messages should be received by subsequent RecvMsg calls.
return nil
}
// Special handling for non-server-stream rpcs.
// This recv expects EOF or errors, so we don't collect inPayload.
if err := recv(a.parser, cs.codec, a.transportStream, a.decompressorV0, m, *cs.callInfo.maxReceiveMessageSize, nil, a.decompressorV1, false); err == io.EOF {
return a.transportStream.Status().Err() // non-server streaming Recv returns nil on success
} else if err != nil {
return toRPCErr(err)
}
return status.Error(codes.Internal, "cardinality violation: expected <EOF> for non server-streaming RPCs, but received another message")

The check here should be for 0 requests being received for a non-client-streaming request.

grpc-go/stream.go

Lines 1142 to 1144 in e5de1e2

if !cs.desc.ServerStreams {
return status.Error(codes.Internal, "cardinality violation: received no response message from non-streaming RPC")
}

Right, in client.recvMsg() this check is triggered in case of 0 response from server.
But in server-stream, it is not possible to have 0 request. So the following case will only be triggered when server.recvmsg() is called more than once and receives an io.EOF.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In server-streaming, client can't send more than one message.

What is stopping a client from doing this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clientstream maintains a variable named sentlast which will be set to true when client sends a msg in case of non client-streaming RPCs.

grpc-go/stream.go

Lines 905 to 910 in e5de1e2

if cs.sentLast {
return status.Errorf(codes.Internal, "SendMsg called after CloseSend")
}
if !cs.desc.ClientStreams {
cs.sentLast = true
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed earlier, can we have the client (incorrectly) create a bidi stream by manipulating the strem desc?

Ideally, a well behaving client should never cause cardinality violations. The reason for adding these checks is to fail the RPCs with good error codes and messages when a client misbehaves.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the logic in server.recvmsg() to call recv() twice in case of non-client-stream RPCs. Also added a test with misbehaving client.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a behavior change? Users that call RecvMsg would previously get io.EOF and now they'll get a cardinality violation?

return status.Error(codes.Internal, "cardinality violation: received no request message from non-client-stream RPC")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit:

Suggested change
return status.Error(codes.Internal, "cardinality violation: received no request message from non-client-stream RPC")
return status.Error(codes.Internal, "cardinality violation: received no request message from non-client-streaming RPC")

}
return err
}
if err == io.ErrUnexpectedEOF {
Expand All @@ -1800,7 +1804,19 @@
binlog.Log(ss.ctx, cm)
}
}
return nil

if ss.desc.ClientStreams {
// Subsequent messages should be received by subsequent RecvMsg calls.
return nil
}
// Special handling for non-client-stream rpcs.
// This recv expects EOF or errors, so we don't collect inPayload.
if err := recv(ss.p, ss.codec, ss.s, ss.decompressorV0, m, ss.maxReceiveMessageSize, nil, ss.decompressorV1, true); err == io.EOF {
return nil
} else if err != nil {
return err
}

Check warning on line 1818 in stream.go

View check run for this annotation

Codecov / codecov/patch

stream.go#L1817-L1818

Added lines #L1817 - L1818 were not covered by tests
return status.Error(codes.Internal, "cardinality violation: expected <EOF> for non client-streaming RPCs, but received another message")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about something more like the first message:

Suggested change
return status.Error(codes.Internal, "cardinality violation: expected <EOF> for non client-streaming RPCs, but received another message")
return status.Error(codes.Internal, "cardinality violation: received multiple request messages for non-client-streaming RPC")

}

// MethodFromServerStream returns the method string for the input stream.
Expand Down
339 changes: 339 additions & 0 deletions test/end2end_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3740,6 +3740,345 @@ func (s) TestClientStreaming_ReturnErrorAfterSendAndClose(t *testing.T) {
}
}

// Tests the behavior for server-side streaming when server calls RecvMsg twice.
// Second call to RecvMsg should fail with Internal error.
func (s) TestServerStreaming_ServerCallRecvMsgTwice(t *testing.T) {
lis, err := testutils.LocalTCPListener()
if err != nil {
t.Fatal(err)
}
defer lis.Close()

ss := stubserver.StubServer{
StreamingOutputCallF: func(_ *testpb.StreamingOutputCallRequest, stream testgrpc.TestService_StreamingOutputCallServer) error {
// This is second call to RecvMsg(), the initial call having been performed by the server handler.
if err := stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
return nil
},
}
if err := ss.Start(nil); err != nil {
t.Fatal("Error starting server:", err)
}
defer ss.Stop()

ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()

stream, err := ss.Client.StreamingOutputCall(ctx, &testpb.StreamingOutputCallRequest{})
if err != nil {
t.Fatalf(".StreamingOutputCall(_) = _, %v, want <nil>", err)
}

if err := stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
}

// Tests the behavior for server-side streaming when client calls SendMsg twice.
// Second call to SendMsg should fail with Internal error.
func (s) TestServerStreaming_ClientCallSendMsgTwice(t *testing.T) {
lis, err := testutils.LocalTCPListener()
if err != nil {
t.Fatal(err)
}
defer lis.Close()

s := grpc.NewServer()
serviceDesc := grpc.ServiceDesc{
ServiceName: "grpc.testing.TestService",
HandlerType: (*any)(nil),
Methods: []grpc.MethodDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "ServerStreaming",
Handler: func(_ any, _ grpc.ServerStream) error {
return nil
},
ClientStreams: false,
ServerStreams: true,
},
},
}
s.RegisterService(&serviceDesc, &testServer{})
go s.Serve(lis)
defer s.Stop()

ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()
cc, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("grpc.NewClient(%q) failed unexpectedly: %v", lis.Addr(), err)
}
defer cc.Close()

desc := &grpc.StreamDesc{
StreamName: "ServerStreaming",
ServerStreams: true,
ClientStreams: false,
}

stream, err := cc.NewStream(ctx, desc, "/grpc.testing.TestService/ServerStreaming")
if err != nil {
t.Fatalf("cc.NewStream() failed unexpectedly: %v", err)
}

if err := stream.SendMsg(&testpb.Empty{}); err != nil {
t.Errorf("stream.SendMsg() = %v, want <nil>", err)
}

if err := stream.SendMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.SendMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
Comment on lines +3831 to +3833
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we confirm that this cancels the RPC, too, as observed from the server? If the stream ends on the client side it must perform a RST_STREAM with the server.

}

// Tests the behavior for unary RPC when server calls RecvMsg twice. Second call
// to RecvMsg should fail with Internal error.
func (s) TestUnaryRPC_ServerCallRecvMsgTwice(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be combined with TestServerStreaming_ServerCallRecvMsgTwice into a single table drive test? They seem to be exactly the same except for a couple of lines. Similarly for the ClientCallSendMsgTwice tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bump on this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test are different now.
PTAL.

lis, err := testutils.LocalTCPListener()
if err != nil {
t.Fatal(err)
}
defer lis.Close()

s := grpc.NewServer()
serviceDesc := grpc.ServiceDesc{
ServiceName: "grpc.testing.TestService",
HandlerType: (*any)(nil),
Methods: []grpc.MethodDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "UnaryCall",
Handler: func(_ any, stream grpc.ServerStream) error {
err := stream.RecvMsg(&testpb.Empty{})
if err != nil {
t.Errorf("stream.RecvMsg() = %v, want <nil>", err)
}

if err = stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
return nil
},
ClientStreams: false,
ServerStreams: false,
},
},
}
s.RegisterService(&serviceDesc, &testServer{})
go s.Serve(lis)
defer s.Stop()

ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()
cc, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("grpc.NewClient(%q) failed unexpectedly: %v", lis.Addr(), err)
}
defer cc.Close()

desc := &grpc.StreamDesc{
StreamName: "UnaryCall",
ServerStreams: false,
ClientStreams: false,
}

stream, err := cc.NewStream(ctx, desc, "/grpc.testing.TestService/UnaryCall")
if err != nil {
t.Fatalf("cc.NewStream() failed unexpectedly: %v", err)
}

if err := stream.SendMsg(&testpb.Empty{}); err != nil {
t.Errorf("stream.SendMsg() = %v, want <nil>", err)
}

if err := stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
}

// Tests the behavior for unary RPC when client calls SendMsg twice. Second call
// to SendMsg should fail with Internal error.
func (s) TestUnaryRPC_ClientCallSendMsgTwice(t *testing.T) {
lis, err := testutils.LocalTCPListener()
if err != nil {
t.Fatal(err)
}
defer lis.Close()

s := grpc.NewServer()
serviceDesc := grpc.ServiceDesc{
ServiceName: "grpc.testing.TestService",
HandlerType: (*any)(nil),
Methods: []grpc.MethodDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "UnaryCall",
Handler: func(_ any, _ grpc.ServerStream) error {
return nil
},
ClientStreams: false,
ServerStreams: false,
},
},
}
s.RegisterService(&serviceDesc, &testServer{})
go s.Serve(lis)
defer s.Stop()

ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()
cc, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("grpc.NewClient(%q) failed unexpectedly: %v", lis.Addr(), err)
}
defer cc.Close()

desc := &grpc.StreamDesc{
StreamName: "UnaryCall",
ServerStreams: false,
ClientStreams: false,
}

stream, err := cc.NewStream(ctx, desc, "/grpc.testing.TestService/UnaryCall")
if err != nil {
t.Fatalf("cc.NewStream() failed unexpectedly: %v", err)
}

if err := stream.SendMsg(&testpb.Empty{}); err != nil {
t.Errorf("stream.SendMsg() = %v, want <nil>", err)
}

if err := stream.SendMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.SendMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
}

// Tests the behavior for server-side streaming RPC when client misbehaves as Bidi-streaming
// and sends multiple nessages.
func (s) TestServerStreaming_ClientSendsMultipleMessages(t *testing.T) {
lis, err := testutils.LocalTCPListener()
if err != nil {
t.Fatal(err)
}
defer lis.Close()

s := grpc.NewServer()
serviceDesc := grpc.ServiceDesc{
ServiceName: "grpc.testing.TestService",
HandlerType: (*any)(nil),
Methods: []grpc.MethodDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "ServerStreaming",
Handler: func(_ any, stream grpc.ServerStream) error {
if err = stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
return nil
},
ClientStreams: false,
ServerStreams: true,
},
},
}
s.RegisterService(&serviceDesc, &testServer{})
go s.Serve(lis)
defer s.Stop()

ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()
cc, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("grpc.NewClient(%q) failed unexpectedly: %v", lis.Addr(), err)
}
defer cc.Close()

// Making the client bi-di to bypass the client side checks that stop a non-streaming client
// from sending multiple messages.
desc := &grpc.StreamDesc{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment describing the need to use a bi-di streaming client for this test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the comment here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had added a comment before test.
Now added a comment here as well.

StreamName: "ServerStreaming",
ServerStreams: true,
ClientStreams: true,
}

stream, err := cc.NewStream(ctx, desc, "/grpc.testing.TestService/ServerStreaming")
if err != nil {
t.Fatalf("cc.NewStream() failed unexpectedly: %v", err)
}

if err := stream.SendMsg(&testpb.Empty{}); err != nil {
t.Errorf("stream.SendMsg() = %v, want <nil>", err)
}

if err := stream.SendMsg(&testpb.Empty{}); err != nil {
t.Errorf("stream.SendMsg() = %v, want <nil>", err)
}

if err := stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
}

// Tests the behavior for server-side streaming RPC when client sends zero request message.
func (s) TestServerStreaming_ClientSendsZeroRequest(t *testing.T) {
lis, err := testutils.LocalTCPListener()
if err != nil {
t.Fatal(err)
}
defer lis.Close()

s := grpc.NewServer()
serviceDesc := grpc.ServiceDesc{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we don't need to change the stream descriptors on the server side for this test. If so, please use a stubserver and override one of the handlers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StreamingOutputCall have predefined calls to sendmsg() and recvmsg().
This test is to check the case when client sends no msg.

ServiceName: "grpc.testing.TestService",
HandlerType: (*any)(nil),
Methods: []grpc.MethodDesc{},
Streams: []grpc.StreamDesc{
{
StreamName: "ServerStreaming",
Handler: func(_ any, stream grpc.ServerStream) error {
if err = stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
return nil
},
ClientStreams: false,
ServerStreams: true,
},
},
}
s.RegisterService(&serviceDesc, &testServer{})
go s.Serve(lis)
defer s.Stop()

ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout)
defer cancel()
cc, err := grpc.NewClient(lis.Addr().String(), grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
t.Fatalf("grpc.NewClient(%q) failed unexpectedly: %v", lis.Addr(), err)
}
defer cc.Close()

desc := &grpc.StreamDesc{
StreamName: "ServerStreaming",
ServerStreams: true,
ClientStreams: false,
}

stream, err := cc.NewStream(ctx, desc, "/grpc.testing.TestService/ServerStreaming")
if err != nil {
t.Fatalf("cc.NewStream() failed unexpectedly: %v", err)
}

if err := stream.CloseSend(); err != nil {
t.Errorf("stream.CloseSend() = %v, want <nil>", err)
}
if err := stream.RecvMsg(&testpb.Empty{}); status.Code(err) != codes.Internal {
t.Errorf("stream.RecvMsg() = %v, want error %v", status.Code(err), codes.Internal)
}
}

// Tests that a client receives a cardinality violation error for client-streaming
// RPCs if the server call SendMsg multiple times.
func (s) TestClientStreaming_ServerHandlerSendMsgAfterSendMsg(t *testing.T) {
Expand Down