Release 0.8.2.Final

Release 0.8.2.Final

Notable changes

• Debezium upgraded to 3.0.7.Final
• Quarkus and other dependency versions are aligned to upstream Debezium
• Various code improvements added around: python package, java testing and "schema change topic" handling

What's Changed

• Upgrade debezium to 3.x and Simplify dependencies by @ismailsimsek in #455
• Use iceberg-bom dependency and Fix jandex plugin version by @ismailsimsek in #456
• Cache maven packages by @ismailsimsek in #457
• Switch caching to actions/setup-java to cache maven by @ismailsimsek in #458
• Update dist pom, align with Debezium by @ismailsimsek in #461
• corrected directory path by @SiddiqueAhmad in #462
• Bump io.debezium:debezium-build-parent from 3.0.5.Final to 3.0.6.Final by @dependabot in #465
• Bump version.debezium from 3.0.5.Final to 3.0.6.Final by @dependabot in #464
• Update documentation and java version in Installation section by @ismailsimsek in #467
• Simplify Minio container test resource by @ismailsimsek in #468
• Add test using jdbc catalog by @ismailsimsek in #469
• Add test using rest catalog by @ismailsimsek in #470
• Add informative exception message for deduplication by @ismailsimsek in #471
• Improve python lib, add pyproject.toml by @ismailsimsek in #477
• Bump actions/stale from 9.0.0 to 9.1.0 by @dependabot in #478
• Bump io.minio:minio from 8.5.14 to 8.5.17 by @dependabot in #476
• Bump io.debezium:debezium-build-parent from 3.0.6.Final to 3.0.7.Final by @dependabot in #475
• Bump version.debezium from 3.0.6.Final to 3.0.7.Final by @dependabot in #474
• Improve schema change topic handling by @ismailsimsek in #480

New Contributors

• @SiddiqueAhmad made their first contribution in #462

Full Changelog: 0.8.1.Final...0.8.2.Final

Release latest

Release latest

Release 0.8.1.Final

Release 0.8.1.Final

What's Changed

• Bump version.debezium from 2.7.3.Final to 2.7.4.Final by @dependabot in #448
• Revert Quarkus to a Debezium compatible version by @panga in #447
• Align groovy dependency with Debezium. Update It to 4.0.17 and switch to new artifact org.apache.groovy by @ismailsimsek in #449
• Upgrade jackson from 2.14.2 to 2.16.2, align with Debezium by @ismailsimsek in #451
• Upgrade iceberg from 1.7.0 to 1.7.1 by @ismailsimsek in #452

New Contributors

• @panga made their first contribution in #447

Full Changelog: 0.8.0.Final...0.8.1.Final

Release 0.8.0.Final

Release 0.8.0.Final

Release 0.7.0.Final

• Debezium: 2.7.2.Final
• Apache Iceberg: 1.6.1
• Quarkus: 3.13.3

Breaking Changes

Equality deletes:

• Now Equality delete files are containing key fields only and they are written with record projection. Now delete files only includes key files, previously delete files were including full delete record.

Notable changes

• Various code improvement added. Now code base is more closely aligned with upstream iceberg-kafka-connect
• Now size of the files generated is defined write.target-file-size-bytes config, previously Long.MAX_VALUE was used
• Iceberg upgraded from 1.6.0 to 1.6.1
• Debezium upgraded from 2.7.0.Final to 2.7.2.Final
• Quarkus upgraded from 3.13.0 to 3.13.3

What's Changed

• Bump version.quarkus from 3.13.0 to 3.13.2 by @dependabot in #400
• Bump io.smallrye:jandex-maven-plugin from 3.2.1 to 3.2.2 by @dependabot in #399
• Bump version.debezium from 2.7.0.Final to 2.7.1.Final by @dependabot in #398
• Bump version.testcontainers from 1.20.0 to 1.20.1 by @dependabot in #396
• Add code improvements by @ismailsimsek in #409
• Improve code, Improve class and method namings by @ismailsimsek in #410
• Improve deduplication code by @ismailsimsek in #411
• Use write.target-file-size-bytes config, previously Long.MAX_VALUE was used by @ismailsimsek in #412
• Fix, Create GenericAppenderFactory respecting to identifier fields. And create it with table properties by @ismailsimsek in #413
• Explicitly set operationId as UUID when creating OutputFileFactory by @ismailsimsek in #414
• Use Key fields and Record Projection for deletes by @ismailsimsek in #408
• Rename variable equalityFieldIds to identifierFieldIds by @ismailsimsek in #415
• Bump org.awaitility:awaitility from 4.2.1 to 4.2.2 by @dependabot in #406
• Bump org.postgresql:postgresql from 42.7.3 to 42.7.4 by @dependabot in #407
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.1 to 3.5.0 by @dependabot in #405
• In case of exception, call writer.abort() to cleanup by @ismailsimsek in #417
• Upgrade debezium and Quarkus to latest minor version by @ismailsimsek in #418

Full Changelog: 0.6.0.Final...0.7.0.Final

Release 0.6.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.13.0

Notable changes

• new config added to control casing of iceberg table names. To change tables names uppercase or lowercase folloing configuration properties could be used, see documentation. debezium.sink.iceberg.destination-uppercase-table-names=true or debezium.sink.iceberg.destination-lowercase-table-names=true
• Quarkus upgraded from 3.12.x to 3.13.x, See Quarkus release notes here

What's Changed

• Add option to change case of destination/iceberg table names by @ismailsimsek in #389
• Bump version.quarkus from 3.12.3 to 3.13.0 by @dependabot in #392
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.5 to 3.3.1 by @dependabot in #395
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.0 to 3.3.1 by @dependabot in #394
• Bump io.smallrye:jandex-maven-plugin from 3.2.0 to 3.2.1 by @dependabot in #393

Full Changelog: 0.5.0.Final...0.6.0.Final

Release 0.5.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.12.3

What's Changed

• Bump io.minio:minio from 8.5.7 to 8.5.11 by @dependabot in #372
• Bump org.codehaus.groovy:groovy-bom from 3.0.21 to 3.0.22 by @dependabot in #374
• Bump com.squareup.okio:okio from 3.4.0 to 3.9.0 by @dependabot in #376
• Upgrade quarkus from 3.10.0 to 3.10.2 by @ismailsimsek in #382
• Bump version.testcontainers from 1.19.8 to 1.20.0 by @dependabot in #387
• Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 by @dependabot in #385

Full Changelog: 0.5.0.Beta...0.5.0.Final

Release 0.5.0.Beta

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.5.2
• Quarkus: 3.10.0

Breaking Changes

Debezium version upgraded from 2.5.x to 2.7.x:

• For Debezium 2.6 Breaking Changes please see release notes
• For Debezium 2.7 Breaking Changes please see release notes

IcebergSchemaHistory wrong table location corrected:

IcebergSchemaHistory class was using wrong table namespace {catalog-name}.{table-name} now this is corrected to use {table-namespace}.{table-name}
If you were using different catalog-name please follow following steps for migration

1. stop debezium server
2. copy the IcebergSchemaHistory table to new location: {table-namespace}.{table-name}
3. start debezium server

Configuration change:

Configuration option debezium.sink.iceberg.upsert-op-column, defaultValue = __op is removed.

Notable changes

• Quarkus upgraded from 3.6.x to 3.10.x, See Quarkus release notes here
• Java release target upgraded from 11 to 17
• Full support for debezium nested events added. Now its possible to run consumer without event flattening.
• Debezium event processing improved, unnecessary schema de-serialization is avoided.

What's Changed

• Upgrade java to 17 by @ismailsimsek in #318
• Upgrade debezium to 2.7.0.Alpha1 by @ismailsimsek in #317
• Bump version.quarkus from 3.6.9 to 3.10.0 by @dependabot in #313
• Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
• Minor, Improve naming of state tables used in tests by @ismailsimsek in #322
• Add mongodb flattening settings to example config by @ismailsimsek in #324
• Fix reading config values in debezium storage classes by @ismailsimsek in #327
• Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328
• Update codeql-analysis.yml to use java 17 by @ismailsimsek in #329
• Upgrade spark to 4.0.x version and enable Mysql tests by @ismailsimsek in #331
• Remove DynamicBatchSizeWait class by @ismailsimsek in #332
• Enforce maven version is greater than 3.8.6 by @ismailsimsek in #334
• Don't partition tables on append mode. by @ismailsimsek in #340
• Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 by @dependabot in #338
• Bump org.postgresql:postgresql from 42.7.2 to 42.7.3 by @dependabot in #337
• Bump version.testcontainers from 1.19.7 to 1.19.8 by @dependabot in #336
• Improve IcebergChangeEvent class, Optimize event and schema deserialization by @ismailsimsek in #343
• Fail deduplication if event key is null by @ismailsimsek in #344
• Improve naming, rename JsonSchema to ChangeEventSchema by @ismailsimsek in #345
• Improve code and schema conversion tests by @ismailsimsek in #346
• Add more tests on schema conversion by @ismailsimsek in #348
• Improve conversion of debezium schema to iceberg schema by @ismailsimsek in #347
• Revert "Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 (#338)" by @ismailsimsek in #349
• Improve setting identifier fields by @ismailsimsek in #350
• Allow nested fields to be set as identifier by @ismailsimsek in #351
• Lower minimum maven version enforcement to 3.6.3 by @ismailsimsek in #353
• Optimize test logs, decrease verbose logs by @ismailsimsek in #354
• Check if configuration contains event flattening by @ismailsimsek in #352
• Test Field Addition is working, automatic schema expansion by @ismailsimsek in #355
• Test consumer without event flattening, Add flag to control identifier-field creation by @ismailsimsek in #356
• Document new features and improve documentation by @ismailsimsek in #357
• Remove stage repo used for spark 4.0.0-preview1 dependency by @ismailsimsek in #358
• Update README.md by @ismailsimsek in #359
• Update dependabot to weekly interval by @ismailsimsek in #360
• Bump docker/build-push-action from 5 to 6 by @dependabot in #361
• Improve handling delete events and logging message by @ismailsimsek in #366
• Improve documentation, schema change behaviour by @ismailsimsek in #367
• Bump com.squareup.okhttp3:okhttp from 4.11.0 to 4.12.0 by @dependabot in #364
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.5 to 3.3.0 by @dependabot in #363
• Bump version.debezium from 2.7.0.Alpha1 to 2.7.0.Final by @dependabot in #371

Full Changelog: 0.4.0.Final...0.5.0.Beta

Release 0.4.1.Final

This release includes fix to debezium storage classes {IcebergSchemaHistory, IcebergOffsetBackingStore}. Now both classes can be used by icebergevents consumer too.

• Debezium: 2.7.0.Alpha1
• Apache Iceberg: 1.5.2
• Quarkus: 3.10.0

Breaking Changes

IcebergSchemaHistory class was using wrong table name {catalog-name}.{table-name} now this is corrected to {table-namespace}.{table-name}
If you were using different catalog-name previously please follow following steps for migration

1. stop debezium server
2. move the table to new location: {table-namespace}.{table-name}
3. start the debezium server

What's Changed

• Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
• Fix reading config values in debezium storage classes by @ismailsimsek in #327
• Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328

Release 0.4.0.Final

Release 0.4.0.Final

What's Changed

• Fix multiple log4j logger bindings by @ismailsimsek in #289
• Fix __source_ts_ms values with 0 conversion and enable mongodb test by @ismailsimsek in #292
• Remove maven stage repository by @ismailsimsek in #293
• Test upsert with tables without PK by @ismailsimsek in #295
• Bump softprops/action-gh-release from 1 to 2 by @dependabot in #301
• Add example setup using docker-compose by @ismailsimsek in #302
• Add iceberg-hive-metastore jar to distribution by @ismailsimsek in #308
• Update docker-compose example by @ismailsimsek in #309
• Bump io.smallrye:jandex-maven-plugin from 3.1.6 to 3.1.7 by @dependabot in #300
• Bump version.testcontainers from 1.19.6 to 1.19.7 by @dependabot in #299
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.2 to 3.2.5 by @dependabot in #297
• Bump org.awaitility:awaitility from 4.2.0 to 4.2.1 by @dependabot in #296
• Bump dev-drprasad/delete-tag-and-release from 1.0 to 1.1 by @dependabot in #310
• Bump org.apache.maven.plugins:maven-assembly-plugin from 3.6.0 to 3.7.1 by @dependabot in #311
• Update debezium version to 2.5.4.Final by @ismailsimsek in #315
• Upgrade iceberg to 1.5.2 by @ismailsimsek in #316

Full Changelog: 0.3.0.Final...0.4.0.Final