add flush request in pipeline

[zenkovev]

Hello, there is a suggestion to add the ability to send a Flush message in Pipeline mode. This feature is available in Pipeline mode in the libpq library, in the PostgreSQL documentation this method is proposed for executing dependent commands, as an alternative to transaction mode for some cases. As for my need, I want to create a test scenario for such case using the library. Also, this feature may close issue #2025.

[jackc]

I think adding the ability to send a Flush is a good feature. But GetResultsNotCheckSync is a pretty awkward addition to the API. It would be really nice if that was not needed. If I understand correctly, the reason for this new method is because you may have sent messages that will not trigger a ReadyForQuery such as a Parse or Describe but you still want to read their response. But GetResults is designed to return nil, nil when the pipeline is empty.

Would it be reasonable to track the expected responses from non-ReadyForQuery triggering messages and fold this behavior into GetResults?

[zenkovev]

Yes, I agree that GetResultsNotCheckSync is not a good feature for the api. Yes, it is needed to receive messages after Flush without sending Sync, since in this mode only Sync triggers ReadyForQuery. From the point of view of the protocol, there are such cases of tracking other messages:

• Parse -> ParseComplete (or ErrorResponse)
• Bind -> BindComplete (or ErrorResponse)
• Execute -> CommandComplete, PortalSuspended (or EmptyQueryResponse, ErrorResponse)
• Sync -> ReadyForQuery (or ErrorResponse)
• Describe -> RowDescription, NoData (or ErrorResponse)
• Close -> CloseComplete (or ErrorResponse)

Also, this adds another level: we only need to track messages before Flush or Sync. In this case, it is possible that the public Flush as a function should be combined with the Flush as a message, so as not to overcomplicate the internal logic. As a possible solution, two counters can be added waiting for messages in the buffer and waiting for sent messages. This variant, at least when considered, looks like a working.

[jackc]

This does get tricky. What we're trying to do is is stay synced with the server without using a Sync point. But the difficulty of doing that is part of why Sync points exist.

But perhaps there is an approach that isn't too complicated. The pipeline system doesn't operate at the protocol message level. It operates at a slightly higher level. For example, there is no way to only send a Bind message. That is sent by SendQueryParams and SendQueryPrepared. Likewise, GetResults does not operate at the level of protocol messages. It returns a *ResultReader, *StatementDescription, or *PipelineSync.

Perhaps just as expectedReadyForQueryCount corresponds with *PipelineSync we should have expected* for ResultReader and StatementDescription. Errors might complicate this, but since errors abort everything until the next Sync, it might be as simple as zeroing the other counters. Except it's possible to have multiple Syncs and Flushes pending...

Yuck... this is really hard. I think we would actually need to have a record of everything sent so we could match it with what we expect to receive. That's more complicated than I'd like. But the alternative seems to be something like what you did originally: a way of getting results that doesn't know if there are any results pending or not.

🤷

[zenkovev]

I thought about this question, as a result, I can suggest the following construction. We can add a queue, the elements of which will contain three fields: the type of message (in the sense of the user, i.e. Prepare, QueryParams, QueryPrepared, etc), a flag indicating whether message was sent to the server, a flag indicating whether there was a Flush or Sync as a message after (that is, we can expect a response from the server). Then, it seems, it will be readable and transparent in terms of processing logic.

[jackc]

Yeah, I think that would work.

[zenkovev]

Well, then I will try to realize this variant after a while.

[zenkovev]

It turned out to be a bit more complicated than I expected, but the result seems acceptable.

Some notes:

1. In MultiResultReader field pipeline was added when the pipeline was added to the library, was not used and is unlikely to be used, so in order not to complicate the logic, I removed it.
2. I added the SendPipelineSync method, similar to the libpq library, so that the implicit transactional mode described in the documentation can be implemented.

[jackc]

Should these be public?

[zenkovev]

Yes, I think they should be private. Fixed.

[jackc]

Thanks!