Skip to content

Commit

Permalink
source-postgres - Streams not in the CDC publication still have a cur…
Browse files Browse the repository at this point in the history
…sor and PK
  • Loading branch information
evantahler committed May 16, 2024
1 parent 73a44d0 commit e55a291
Show file tree
Hide file tree
Showing 3 changed files with 83 additions and 84 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ data:
connectorSubtype: database
connectorType: source
definitionId: decd338e-5647-4c0b-adf4-da0e75f5a750
dockerImageTag: 3.4.4
dockerImageTag: 3.4.5
dockerRepository: airbyte/source-postgres
documentationUrl: https://docs.airbyte.com/integrations/sources/postgres
githubIssueLabel: source-postgres
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,14 +92,12 @@ public static AirbyteStream addCdcMetadataColumns(final AirbyteStream stream) {
/**
* Modifies streams that are NOT present in the publication to be full-refresh only streams. Users
* should be able to replicate these streams, just not in incremental mode as they have no
* associated publication.
* associated publication. Previously, we also setSourceDefinedCursor(false) and setSourceDefinedPrimaryKey(List.of()) for streams that are in the catalog but not in the CDC publication, but now that full refresh streams can be resumable, we should include this information.
*/
public static AirbyteStream setFullRefreshForNonPublicationStreams(final AirbyteStream stream,
final Set<AirbyteStreamNameNamespacePair> publicizedTablesInCdc) {
if (!publicizedTablesInCdc.contains(new AirbyteStreamNameNamespacePair(stream.getName(), stream.getNamespace()))) {
stream.setSupportedSyncModes(List.of(SyncMode.FULL_REFRESH));
stream.setSourceDefinedCursor(false);
stream.setSourceDefinedPrimaryKey(List.of());
}
return stream;
}
Expand Down
Loading

0 comments on commit e55a291

Please sign in to comment.