New policy type and file reader. AK 2.7.0

Included in this release:

• Upgrade to Apache Kafka 2.7.0 and all project dependencies.
• New policy for ingesting files from S3 from event notifications (via AWS SQS or SNS).
• New file reader for ingesting other sort of binary files.

Upgrade to AK 2.6.0. New file readers & cleanup.

Included in this release:

• Upgrade to Apache Kafka 2.6.0 and all project dependencies.
• New Cobol/EBCDIC file reader.
• Added cleanup strategies into policies.
• Jackson-based file reader refactor: new XML and YAML file readers included.
• Fixed #53.

SFTP, ORC files & batching support

Included in this release:

• Support for SFTP file system.
• New file reader for ORC files.
• Batching support at policy level.
• Batching support in file readers.
• Performance improvements to avoid unnecessary seeking in files.
• Upgrade dependencies.

New major release

A major release with the following features:

• Upgrade to Kafka 2.5.0 and Confluent 5.5.0.
• Support for Google Cloud Storage, Azure Blob Storage & Azure Data Lake Store file systems.
• New policy based on cron expressions.
• Improved HDFS file watcher policy. It now allows reconnections.
• Removed old Delimited text file reader.
• New JSON file reader.
• New delimited file readers: CSV, TSV and FixedWidth file readers (based on Univocity parser).
• Allow setting a schema in delimited file readers.
• Enabled reading compressed files for text-based file reader (bzip2 and gzip).
• Test refactor and migrated to JUnit 5.
• Docker files.

Upgrade Hadoop version & new file reader

Features in this release:

• Upgrade Hadoop version to 2.9.0: now there is support for a new file system (SFTP).
• New agnostic file reader. Now, you don't have to specify a file reader to ingest one file type but this agnostic file reader to ingest any kind of file type available in the connector.
• Set a default value when using the DelimitedTextFileReader when the row is malformed.

File reader enhancements

Some additional configuration options in file readers:

• Text file reader encoding.
• Delimiter text file reader encoding.
• Avro file reader schema.
• Parquet file reader schema and projection.

Fix library dependencies

v0.1.1.1

Fix library dependencies

Configurable file readers

Enabled the option of configuring output field names when delivering records from the connector.

This configuration applies to:

• Text file reader.
• Sequence file reader.

First version

Initial release for Confluent 3.1.1.

Included in this release:

• Policies
  • Simple policy.
  • Sleepy policy.
  • HDFS file watch policy.
• File readers
  • Avro file reader.
  • Parquet file reader.
  • SequenceFile file reader.
  • Text file reader.
  • Delimited text file reader.
• Unit and integration tests (local FS and HDFS).
• Offset support when processing records.
• Documentation.