v4.0.0

Breaking changes

• Support for multiple transformers (#82)
• Move CheckpointOffsetManager logic to Reader and Writer (#130)
• Express ConfluentAvroKafkaDecoder as transformer (#128)
• Refactor ParquetPartitioningStreamWriter to transformer (#118)
• Extract encoding part of KafkaStreamWriter to transformer component (#129)
• Remove support for processAllAvailable (#149)

Enhancements

• Retain and map global to local config in abstract factories (#153)

v3.3.0

Enhancements

• ParquetStreamWriter shouldn't write if metadata is inconsistent (#132)

Bugfixes

• Preserve nullability from Avro to Catalyst Schema (#137)
• Fix inconsistencies between CommandLineIngestorDriver and PropertiesIngestorDriver (#120)

v3.2.2

Enhancements

• Publish all modules (#107)

v3.1.0

Enhancements

• Add KafkaWriter (#108)
• Add generic partitioning option to ParquetStreamWriter (#116)
• Publish key from kafka source as key to kafka sink / HyperdriveContext (#114)
• Add Trigger.ProcessingTime to Writers (#84)

Bugfixes

• List delimiter does not work for CommandLineIngestor bug (#119)

v3.0.0

Breaking changes

• Property manager.checkpoint.base.location should contain complete path to checkpoint-location (#85)
• Configuration property keys of components should be accessible via reflection (#83)
• StreamWriter should not require destination as constructor argument (#88)
• OffsetManager should not expect topic as argument (#87)

v2.0.0

Breaking changes

• Remove AllNullableParquetStreamWriter (#46)
• Use writer.parquet.options.key=value instead of writer.parquet.extra.conf.1=key=value for extra configuration (#73)

Enhancements

• Add ParquetPartitioningStreamWriter (#52)
• Add property reader.option.kafka.startingOffsets to KafkaStreamReader (#55)
• Upgrade ABRiS to v3.1.1 (#80)
• Upgrade Spark to v2.4.3 (#45)

Bugfixes

• Delete destination directory when exception occurred during ingestion bug (#50)
• Fix NoClassDefFoundError (#62)
• Add default error message for configuration utils (#76)