pubsub: Allow Receivers to manually pause/resume message streaming

[coxley]

Is your feature request related to a problem? Please describe.

There are two main methods of flow control today:

• ReceiveSettings.MaxOutstandingBytes
• ReceiveSettings.MaxOutstandingMessages

These are sufficient for many workloads, maybe even most. But we have workloads where the message count itself isn't a great indicator of how much the system can handle.

Two examples of this:

• Each event requires updating external state.
  • The amount of work to do depends on information in the payload, but not related to it's byte size.
• Each event is internally serialized by some key from the payload.
  • In this case, it's more expensive to handle 20k distinct keys vs. 1000 each with 20 messages. In the latter case, messages are spending a lot of time waiting their turn.

Disabling outstanding messages works, but then we have zero control. I'd love to provide my own back pressure.

Describe the solution you'd like

I'm not sure what the most ergonomic way in the current design would be. pubsub.Pause(ctx) from a receiving function feels too hidden; though convenient. Giving a callback that is called before pulling other messages down to see how many more it can tolerate could work. Or even a simple "yes, I'm ready for more" or "no, I'm still busy".

Describe alternatives you've considered

• Cancelling the context passed to sub.Receive when wanting to pause, starting it up again when wanting to resume.
  • The docs don't make it clear whether that's a "safe" thing to do if you want to keep in-flight messages going. Especially with the disclaimers of not calling msg.Ack()/msg.Nack() outside of receive.
• Issuing msg.Nack() to signal being too busy
  • This feels way too wasteful, especially since redeliveries aren't free as in beer. We pay for delivered bytes, whether they were seen before or not.

I suppose we could switch to apiv1.SubscriberClient.Pull. The main downside there is we have a lot of prior-art internally around the streaming SDK. Changing our main interface for getting messages would involve a lot more "specialness" about these few systems. I guess that's more of a "me" problem, but still worth pointing out I guess. :)

Additional context

Without having some manual say into flow control, it's difficult to do capacity management on systems that aren't simple stream in -> batch out. This would be a very appreciated feature!

[hongalex]

Thanks for filing an issue. When we make new changes to the Pub/Sub client libraries, we need to make sure we keep parity across languages, so I'll bring this up to the team to see what they think.

One thing I would add to your description is that you could build upon https://pkg.go.dev/cloud.google.com/go/pubsub/apiv1#SubscriberClient.StreamingPull rather than Pull. It still has the same problems as you mentioned (using our lower level library is a lot more complex than calling Receive).

Cancelling the context passed to sub.Receive when wanting to pause, starting it up again when wanting to resume.
The docs don't make it clear whether that's a "safe" thing to do if you want to keep in-flight messages going. Especially with the disclaimers of not calling msg.Ack()/msg.Nack() outside of receive.

So when cancelling sub.Receive, we currently only offer a full graceful shutdown option, which means all messages will need to be ack/nack/expired before that function returns. In this sense, it is "safe" to use for not wanting to pull more messages in, as you will never experience data loss.

[coxley]

@hongalex: Awesome, thanks! I'll take a look at StreamingPull if we need to go that way sooner.