v14.0.0

Breaking

• remove AutoRuleCorpusTester
• removes the option to use synchronous bulk or parallel_bulk operation in favor of parallel_bulk in opensearch_output
• reimplement error handling by introducing the option to configure an error output
  • if no error output is configured, failed event will be dropped

Features

• adds health check endpoint to metrics on path /health
• changes helm chart to use new readiness check
• adds healthcheck_timeout option to all components to tweak the timeout of healthchecks
• adds desired_cluster_status option to opensearch output to signal healthy cluster status
• initially run health checks on setup for every configured component
• make imagePullPolicy configurable for helm chart deployments
• it is now possible to use Lucene compliant Filter Expressions
• make terminationGracePeriodSeconds configurable in helm chart values
• adds ability to configure error output
• adds option default_op_type to opensearch_output connector to set the default operation for indexing documents (default: index)
• adds option max_chunk_bytes to opensearch_output connector to set the maximum size of the request in bytes (default: 100MB)
• adds option error_backlog_size to logprep configuration to configure the queue size of the error queue
• the opensearch default index is now only used for processed events, errors will be written to the error output, if configured

Improvements

• remove AutoRuleCorpusTester
• adds support for rust extension development
• adds prebuilt wheels for architectures x86_64 on manylinux and musllinux based linux platforms to releases
• add manual how to use local images with minikube example setup to documentation
• move Configuration to top level of documentation
• add CONTRIBUTING file
• sets the default for flush_timeout and send_timeout in kafka_output connector to 0 seconds
• changed python base image for logprep to bitnami/python in cause of better CVE governance

Bugfix

• ensure logprep.abc.Component.Config is immutable and can be applied multiple times
• remove lost callback reassign behavior from kafka_input connector
• remove manual commit option from kafka_input connector
• pin mysql-connector-python to >=9.1.0 to accommodate for CVE-2024-21272 and update MySQLConnector to work with the new version

Details

• Handle UnicodeDecodeError in kafka input by @ppcad in #663
• make configuration immutable by @ekneg54 in #661
• Remove autorulecorpustester by @ekneg54 in #665
• add rust toolchain by @ekneg54 in #662
• add readiness checks by @ekneg54 in #652
• Fix release pipeline broken for arm images by @ekneg54 in #670
• Fix automatic docs generation by @ekneg54 in #671
• Add CONTRIBUTING file by @ekneg54 in #664
• add minor improvements by @ekneg54 in #676
• Fix exporter restart by @ekneg54 in #677
• Ensure kafka output flushes queue on shutdown by @ekneg54 in #679
• change base image to bitnami/python by @ekneg54 in #680
• Remove unneeded dependencies by @P4sca1 in #682
• Remove broken kafka implementations by @ekneg54 in #678
• Fix docker image and avoid implicit setuptools runtime dependency by @P4sca1 in #683
• test container image before push by @ekneg54 in #684
• make terminationGracePeriodSeconds configurable. by @ekneg54 in #686
• Scan images using Trivy by @P4sca1 in #685
• Dev autoruletester refactoring by @Malutthias in #594
• 629 no support for the mysql connector python library in version 900 by @ppcad in #688
• Add Lucene compliant regex filter expression by @djkhl in #675
• add abiltity to configure error output by @ekneg54 in #668
• prepare release 14 by @ekneg54 in #689

New Contributors

• @P4sca1 made their first contribution in #682
• @Malutthias made their first contribution in #594

Full Changelog: v13.1.2...v14.0.0