v4.7.0

Enhancements

• Added support for Spark 3.2.1 by @kevinwallimann in #260
• Remove ingestor.spark.app.name by @kevinwallimann in #268

Bugfixes

• Create deterministic plan by @kevinwallimann in #266

Full Changelog: v4.6.0...v4.7.0

v4.6.0

Enhancements

• Preserve Avro schema by @kevinwallimann in #257

Bugfixes

• Fix deduplicator by @kevinwallimann in #263

Internal tasks

• Upgrade to ABRiS 5.0.0 by @kevinwallimann in #244
• Bump commons-configuration2 from 2.4 to 2.7 in /parent-conf by @dependabot in #199
• Bump log4j-api from 2.15.0 to 2.16.0 in /parent-conf by @dependabot in #249
• Bump log4j-core from 2.15.0 to 2.16.0 in /parent-conf by @dependabot in #250
• Bump log4j-api from 2.16.0 to 2.17.0 in /parent-conf by @dependabot in #251
• Bump log4j-core from 2.16.0 to 2.17.0 in /parent-conf by @dependabot in #252
• Bump log4j-api from 2.17.0 to 2.17.1 in /parent-conf by @dependabot in #255
• Bump log4j-core from 2.17.0 to 2.17.1 in /parent-conf by @dependabot in #254
• Feature/253 remove log4j2 by @kevinwallimann in #256
• Add --no-transfer-progress to build yml by @kevinwallimann in #258
• Upgrade za.co.absa.commons from v0.0.12 to v1.0.0 by @jozefbakus in #261

Full Changelog: v4.5.2...v4.6.0

v4.5.2

Internal Tasks

• Bump log4j-api from 2.13.3 to 2.15.0 in /parent-conf by @dependabot in #245
• Bump log4j-core from 2.13.3 to 2.15.0 in /parent-conf by @dependabot in #246

Full Changelog: v4.5.1...v4.5.2

v4.5.1

Enhancements

• Upgrade to Spark v3.1.2 (#237)

Bugfixes

• DeduplicateKafkaSinkTransformer should extract fields from consumer record in the innermost loop (#241)
• User-defined options should always be added at end (#146)

Note: Release v4.5.0 was skipped due to a mishap during its release process

v4.4.1

Bugfixes

• #232 Fix Deduplicator config

v4.4.0

Enhancements

• #225 Allow S3 checkpoint folder
• #227 Register nullable field with default value null in ConfluentAvroEncodingTransformer

v4.3.0

Enhancements

• #185 Add optional arguments for schema registry client
• #218 Add password file argument for schema registry

Internal tasks

• #220 Upgrade to ABRiS 4.2.0

v4.2.2

Bugfixes

• #205 A couple of tiny fixes for MongoDB Writer
• #207 Fix project build fails on Windows
• #203 Set starting offsets earliest when checkpoint folder is empty
• #212 Improve logging for deduplicator transformer
• #215 Add switch to preserve previous behaviour of non-nullable columns

Internal tasks

• #209 Create CODEOWNERS file

v4.2.1

Enhancements

• #179 Add copy transformer
• #181 Add option to keep columns in AvroDecodingTransformer
• #177 Add idempotent Kafka Writer across application restarts
• #186 Add builtin writer to a MongoDB database
• #193 Add support for Scala 2.12
• #196 Add support for Spark 3
• #197 Add support for processAllAvailable again

Bugfixes

• #195 Fix regressions in archetype

Internal tasks

• #192 Upgrade testcontainers
• #191 Make spark dependencies provided
• #200 Add missing license headers, add rat plugin
• #202 Use original license

v4.1.0

Enhancements

• #163 Create Parquet Stream Reader
• #170 Add renaming transformer
• #176 Upgrade to ABRiS v4.0.1

Bugfixes

• #165 Boolean config parameters should be interpreted consistently