You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We currently filter transactions incoming on the replication stream based on the snapshot's xmin. We need to (also) use xmax and xip_list to filter correctly. All three values are returned by PG's pg_current_snapshot() function.
xmin is the lowest active transaction ID, there can be txids > xmin that have committed and so would appear in the snapshot. This, with our buffering of the replocation stream before fetfhing the snapshot, results in a condition where we can potentually have duplicate inserts between the snapshot and the shape log.
The current filtering logic should be corrected by an additional clause: any transaction coming from the replication stream for which xmin < xid and xid < xmax and xid not in xip_list needs to be skipped. In pseudocode form:
# Transaction already part of the snapshot because it had committed before# we started taking the snapshot.defphandle_txn(%Transaction{xid: xid},%{snapshot_xmin: xmin})whenxid<xmin,do: <skiptxn># Transaction commited sometime between xmin and the moment when we started# taking the snapshot.defphandle_txn(%Transaction{xid: xid},%{snapshot_xmax: xmax,snapshot_xip_list: xip_list})whenxid<xmaxandxidnot inxip_list,do: <skiptxn># Transaction was either active at the time of taking the snapshot:# xid == xmin or xid in xip_list# OR# it committed after the snapshot had completed:# xid >= xmaxdefphandle_txn(%Transaction{xid: xid},_),do: <addtxntothelog>
Once we have stored in the log the first transaction whose xid ≥ xmax, we can mark the snapshot such that the comparisons can be skipped for all future transactions. At this point, we are sure that any new transaction arriving on the replication stream is not part of the snapshot.
Skipping further comparisons will absolve us of the need to worry about the number of wraparounds of 32-bit transaction IDs.
The text was updated successfully, but these errors were encountered:
@samwillis has had a brilliant insight:
We currently filter transactions incoming on the replication stream based on the snapshot's
xmin
. We need to (also) usexmax
andxip_list
to filter correctly. All three values are returned by PG'spg_current_snapshot()
function.xmin is the lowest active transaction ID, there can be txids > xmin that have committed and so would appear in the snapshot. This, with our buffering of the replocation stream before fetfhing the snapshot, results in a condition where we can potentually have duplicate inserts between the snapshot and the shape log.
The current filtering logic should be corrected by an additional clause: any transaction coming from the replication stream for which
xmin < xid and xid < xmax and xid not in xip_list
needs to be skipped. In pseudocode form:Once we have stored in the log the first transaction whose
xid
≥xmax
, we can mark the snapshot such that the comparisons can be skipped for all future transactions. At this point, we are sure that any new transaction arriving on the replication stream is not part of the snapshot.Skipping further comparisons will absolve us of the need to worry about the number of wraparounds of 32-bit transaction IDs.
The text was updated successfully, but these errors were encountered: