fix(c/driver/postgresql): fix ingest with multiple batches #1393
Conversation
|
CC @WillAyd do the COPY changes here look reasonable? |
| PQerrorMessage(conn)); | ||
| return ADBC_STATUS_IO; | ||
| } | ||
| if (rows_affected) *rows_affected += array->length; |
There was a problem hiding this comment.
What we might want to do (certainly fine as a follow up) is send this data to the server at the end of each loop iteration using PQputCopyData:
https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND
There was a problem hiding this comment.
Sorry, what do you mean? We're still calling PQputCopyData for each batch, we're just not calling PQputCopyEnd until we've written all the batches
There was a problem hiding this comment.
(We could also be smarter here and flush to the server at a certain amount of data instead of each batch to smooth things out between large/small batches)
There was a problem hiding this comment.
Ah perfect. Ignore what I said then - I just missed it because I didn't see it in the diff
The COPY writer was ending the COPY command after each batch, so any dataset with more than one batch would fail. Instead, write the header once and don't end the command until we've written all batches.
Fixes #1310.