CHANGELOG.asciidoc

// Use these for links to issue and pulls. Note issues and pulls redirect one to
// each other on Github, so don't worry too much on using the right prefix.
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

[[release-notes-9.0.0-beta1]]
=== Beats version 9.0.0-beta1
https://github.com/elastic/beats/compare/v8.17.2\...v9.0.0-beta1[View commits]

==== Breaking changes

*Affecting all Beats*

- Set default Kafka version to 2.1.0 in Kafka output and Filebeat. {pull}41662[41662]
- Replace default Ubuntu-based images with UBI-minimal-based ones. {pull}42150[42150]
- removed support for a single `-` to precede multi-letter command line arguments.  Use `--` instead. {issue}42117[42117] {pull}42209[42209]

*Filebeat*

- Filebeat fails to start if there is any input with a duplicated ID. It logs the duplicated IDs and the offending inputs configurations. {pull}41731[41731]
- Filestream inputs with duplicated IDs will fail to start. An error is logged showing the ID and the full input configuration. {issue}41938[41938] {pull}41954[41954]
- Filestream inputs can define `allow_deprecated_id_duplication: true` to run keep the previous behaviour of running inputs with duplicated IDs. {issue}41938[41938] {pull}41954[41954]
- The Filestream input only starts to ingest a file when it is >= 1024 bytes in size. This happens because the fingerprint is the default file identity now. To restore the previous behaviour, set `file_identity.native: ~` and `prospector.scanner.fingerprint.enabled: false`. {issue}40197[40197] {pull}41762[41762]
- Filebeat fails to start when its configuration contains usage of the deprecated `log` or `container` inputs. However, they can still be used when `allow_deprecated_use: true` is set in their configuration. {pull}42295[42295]

*Osquerybeat*

- Upgrade osquery version to 5.13.1. {pull}40849[40849]

*Packetbeat*

- Use base-16 for reporting `serial_number` value in TLS fields in line with the ECS recommendation. {pull}41542[41542]

*Winlogbeat*

- Default to use raw API and delete older XML implementation. {pull}42275[42275]

==== Bugfixes

*Auditbeat*

- hasher: Add a cached hasher for upcoming backend. {pull}41952[41952]
- Split common tty definitions. {pull}42004[42004]

*Filebeat*

- Redact authorization headers in HTTPJSON debug logs. {pull}41920[41920]
- Further rate limiting fix in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41977[41977]
- The `_id` generation process for S3 events has been updated to incorporate the LastModified field. This enhancement ensures that the `_id` is unique. {pull}42078[42078]
- Fix truncation of bodies in request tracing by limiting bodies to 10% of the maximum file size. {pull}42327[42327]
- [Journald] Fixes handling of `journalctl` restart. A known symptom was broken multiline messages when there was a restart of journalctl while aggregating the lines. {issue}41331[41331] {pull}42595[42595]

*Metricbeat*

- Fix bug where Metricbeat unintentionally triggers Windows ASR. {pull}42177[42177]
- Remove `hostname` field from ZooKeeper's `mntr` data stream. {pull}41887[41887]

*Packetbeat*

- Properly marshal nested structs in ECS fields, fixing issues with mixed cases in field names. {pull}42116[42116]

==== Added

*Auditbeat*

- Improve logging in system/socket. {pull}41571[41571]

*Filebeat*

- Added out of the box support for Amazon EventBridge notifications over SQS to S3 input. {pull}40006[40006]
- Update CEL mito extensions to v1.16.0. {pull}41727[41727]
- Filebeat's registry is now added to the Elastic-Agent diagnostics bundle. {issue}33238[33238] {pull}41795[41795]
- Add `unifiedlogs` input for MacOS. {pull}41791[41791]
- Add evaluation state dump debugging option to CEL input. {pull}41335[41335]
- Rate limiting operability improvements in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41977[41977]
- Rate limiting fault tolerance improvements in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42094[42094]
- Introduce ignore older and start timestamp filters for AWS S3 input. {pull}41804[41804]
- Journald input now can report its status to Elastic-Agent. {issue}39791[39791] {pull}42462[42462]
- Publish events progressively in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42567[42567]
- Journald `include_matches.match` now accepts `+` to represent a logical disjunction (OR). {issue}40185[40185] {pull}42517[42517]
- The journald input is now generally available. {pull}42107[42107]

*Heartbeat*

- Add support for RFC7231 methods to HTTP monitors. {pull}41975[41975]

*Metricbeat*

- Add `use_kubeadm` config option in kubernetes module in order to toggle kubeadm-config API requests. {pull}40086[40086]
- Preserve queries for debugging when `merge_results: true` in SQL module. {pull}42271[42271]
- Collect more fields from ES node/stats metrics and only those that are necessary. {pull}42421[42421]

*Metricbeat*
- Add benchmark module. {pull}41801[41801]

*Osquerybeat*

- Increase maximum query timeout to 24 hours. {pull}42356[42356]

*Winlogbeat*

- Properly set events `UserData` when experimental API is used. {pull}41525[41525]
- Include XML is respected for experimental API. {pull}41525[41525]
- Forwarded events use renderedtext info for experimental API. {pull}41525[41525]
- Language setting is respected for experimental API. {pull}41525[41525]
- Language setting also added to decode XML wineventlog processor. {pull}41525[41525]
- Format embedded messages in the experimental API. {pull}41525[41525]
- Make the experimental API GA and rename it to winlogbeat-raw. {issue}39580[39580] {pull}41770[41770]
- Remove 22 clause limitation. {issue}35047[35047] {pull}42187[42187]
- Add handling for recoverable publisher disabled errors. {issue}35316[35316] {pull}42187[42187]

*Functionbeat*

- Remove Functionbeat binaries from CI pipelines. {issue}40745[40745] {pull}41506[41506]


[[release-notes-8.17.3]]
=== Beats version 8.17.3
https://github.com/elastic/beats/compare/v8.17.2\...v8.17.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Restored event Meta fields in the Elasticsearch output's error logs. {pull}42559[42559]

*Filebeat*

- [Journald] Fixes handling of `journalctl` restart. A known symptom was broken multiline messages when there was a restart of journalctl while aggregating the lines. {issue}41331[41331] {pull}42595[42595]
- Fix entityanalytics activedirectory provider full sync use before initialization bug. {pull}42682[42682]
- In the `http_endpoint` input, fix the check for a missing HMAC HTTP header. {pull}42756[42756]

*Metricbeat*

- Fixed panic caused by uninitialized meraki device wifi0 and wifi1 struct pointers in the device WiFi data fetching. {issue}42745[42745] {pull}42746[42746]
- Only fetch cluster-level index stats summary. {issue}36019[36019] {pull}42901[42901]
- Fixed an issue in Metricbeat's Windows module where data collection would fail if the data was unavailable. {issue}42802[42802] {pull}42803[42803]

*Winlogbeat*

- Sync missing changes in modules pipelines. {pull}42619[42619]

==== Added

*Affecting all Beats*

- Update Go version to 1.22.12. {pull}42681[42681]

*Filebeat*

- Introduce ignore older and start timestamp filters for AWS S3 input. {pull}41804[41804]
- Publish events progressively in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42567[42567]

*Metricbeat*

- Log every 401 response from Kubernetes API Server. {pull}42714[42714]
- Collect more fields from ES node/stats metrics and only those that are necessary. {pull}42421[42421]


[[release-notes-8.17.2]]
=== Beats version 8.17.2
https://github.com/elastic/beats/compare/v8.17.1\...v8.17.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix incorrect cloud provider identification in `add_cloud_metadata` processor using provider priority mechanism. {pull}41636[41636]
- Prevent panic if Libbeat processors are loaded more than once. {issue}41475[41475] {pull}41857[51857]
- Allow network condition to handle field values that are arrays of IP addresses. {pull}41918[41918]
- Fix a bug where log files are rotated on startup when interval is configured and rotateonstartup is disabled. {issue}41894[41894] {pull}41895[41895]
- Fix setting unique registry for non Beat receivers. {issue}42288[42288] {pull}42292[42292]
- The Kafka output now drops events when there is an authorization error. {issue}42343[42343] {pull}42401[42401]
- Fix autodiscovery memory leak related to metadata of start events. {pull}41748[41748]
- All standard queue metrics are now included in metrics monitoring, including: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`. {pull}42439[42439]
- The following output latency metrics are now included in metrics monitoring: `output.latency.{count, max, median, p99}`. {pull}42439[42439]

*Filebeat*

- Updated websocket retry error code list to allow more scenarios to be retried which could have been missed previously. {pull}42218[42218]
- In the `streaming` input, prevent panics on shutdown with a null check and apply a consistent namespace to contextual data in debug logs. {pull}42315[42315]
- Remove erroneous status reporting to Elastic-Agent from the Filestream input. {pull}42435[42435]

*Metricbeat*

- Continue collecting metrics even if the Cisco Meraki `getDeviceLicenses` operation fails. {pull}42397[42397]
- Fixed errors in the `elasticsearch.index` metricset when index settings are missing. {issue}42424[42424] {pull}42426[42426]

==== Added

*Metricbeat*

- Collect .NET CLR (IIS) Memory, Exceptions and LocksAndThreads metrics {pull}41929[41929]
- Add support of additional `collstats` metrics in mongodb module. {pull}42171[42171]
- Update beat module with apm-server tail sampling monitoring metrics fields {pull}42569[42569]


[[release-notes-8.17.1]]
=== Beats version 8.17.1
https://github.com/elastic/beats/compare/v8.17.0\...v8.17.1[View commits]

==== Breaking changes

*Affecting all Beats*

- Default Docker base image was reverted to Ubuntu 20.04 due to incompatability issues with glibc. {pull}42144[42144]
- Fix templates and docs to use correct `--` version of command line arguments. {issue}42038[42038] {pull}42060[42060]

==== Bugfixes

*Filebeat*

- Fix streaming input handling of invalid or empty websocket messages. {pull}42036[42036]
- Fix awss3 document ID construction when using the CSV decoder. {pull}42019[42019]
- Improve S3 object size metric calculation to support situations where Content-Length is not available. {pull}41755[41755]
- Fix Netflow Template Sharing configuration handling. {pull}42080[42080]

*Metricbeat*

- Do not report non-existant 0 values for RSS metrics in docker/memory. {pull}41449[41449]
- Don't skip first bucket value in GCP metrics metricset for distribution type metrics. {pull}41822[41822]
- [K8s Integration] Enhance HTTP authentication in case of token updates for Apiserver, Controllermanager and Scheduler metricsets. {issue}41910[41910] {pull}42016[42016]
- Fixed `creation_date` scientific notation output in the `elasticsearch.index` metricset. {pull}42053[42053]

==== Added

*Affecting all Beats*

- Update Go version to 1.22.10. {pull}42095[42095]
- Add regex pattern matching to add_kubernetes_metadata processor. {pull}41903[41903]

*Filebeat*

- Added support for retry configuration in GCS input. {issue}11580[11580] {pull}41862[41862]
- Added default values in the streaming input for websocket retries and put a cap on retry wait time to be lesser than equal to the maximum defined wait time. {pull}42012[42012]
- Added OAuth2 support with auto token refresh for websocket streaming input. {issue}41989[41989] {pull}42212[42212]
- Added infinite & blanket retry options to websockets and improved logging and retry logic. {pull}42225[42225]

*Metricbeat*

- Added `tier_preference`, `creation_date` and `version` fields to the `elasticsearch.index` metricset. {pull}41944[41944]
- Add `use_performance_counters` to collect CPU metrics using performance counters on Windows for `system/cpu` and `system/core`. {pull}41965[41965]

==== Deprecated

*Filebeat*

- Removed `bucket_timeout` config option for GCS input and replaced bucket context with parent program context. {issue}41107[41107] {pull}41970[41970]


[[release-notes-8.17.0]]
=== Beats version 8.17.0
https://github.com/elastic/beats/compare/v8.16.1\...v8.17.0[View commits]

==== Known issue

- Standalone Beats docker image will not start if `-e` option is not added {issue}42038[42038].

==== Breaking changes

*Affecting all Beats*

- Drop support for Debian 10 and upgrade statically linked glibc from 2.28 to 2.31. {pull}41402[41402]

==== Bugfixes

*Affecting all Beats*

- Ensure Elasticsearch output can always recover from network errors. {pull}40794[40794]
- Add `translate_ldap_attribute` processor. {pull}41472[41472]
- Remove unnecessary debug logs during idle connection teardown. {issue}40824[40824]
- Remove unnecessary reload for Elastic Agent managed beats when APM tracing config changes from nil to nil. {pull}41794[41794]

*Auditbeat*

- auditd: Use ECS `event.type: end` instead of `stop` for SERVICE_STOP, DAEMON_ABORT, and DAEMON_END messages. {pull}41558[41558]
- auditd: Update syscall names for Linux 6.11. {pull}41558[41558]
- hasher: Geneneral improvements and fixes. {pull}41863[41863]

*Filebeat*

- Fix double encoding of client_secret in the Entity Analytics input's Azure Active Directory provider. {pull}41393[41393]
- Add support for Access Points in the `aws-s3` input. {pull}41495[41495]
- Fix the "No such input type exist: 'salesforce'" error on the Windows/AIX platform. {pull}41664[41664]
- Fix handling of http_endpoint request exceeding memory limits. {issue}41764[41764] {pull}41765[41765]
- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]

*Metricbeat*

- Log Cisco Meraki `getDevicePerformanceScores` errors without stopping metrics collection. {pull}41622[41622]
- Fix incorrect handling of types in SQL module. {issue}40090[40090] {pull}41607[41607]

*Winlogbeat*

- Fix message handling in the experimental API. {issue}19338[19338] {pull}41730[41730]

==== Added

*Affecting all Beats*

- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]
- Add `uppercase` processor. {issue}22254[22254] {pull}41535[41535]
- Replace `compress/gzip` with https://github.com/klauspost/compress/gzip library for gzip compression. {pull}41584[41584]

*Auditbeat*

- Split module/system/process into common and provider bits. {pull}41868[41868]

*Filebeat*

- Improved Azure Blob Storage input documentation. {pull}41252[41252]
- Make ETW input GA. {pull}41389[41389]
- Added input metrics to GCS input. {issue}36640[36640] {pull}41505[41505]
- Add support for Okta entity analytics provider to collect role and factor data for users. {pull}41460[41460]
- Add support for Journald in the System module. {pull}41555[41555]
- Improve S3 polling mode states registry when using list prefix option. {pull}41869[41869]
- AWS S3 input registry cleanup for untracked s3 objects. {pull}41694[41694]
- The environment variable `BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true` enables internal logs tracer for the azure-eventhub input. {issue}41931[41931] {pull}41932[41932]

*Libbeat*

- Enrich events with EC2 tags in add_cloud_metadata processor. {pull}41477[41477]

*Metricbeat*

- Add `id` field to all the vSphere metricsets. {pull}41097[41097]
- Bump aerospike-client-go to version v7.7.1 and add support for basic auth in Aerospike module. {pull}41233[41233]
- Add support for region/zone for Vertex AI service in GCP module. {pull}41551[41551]
- Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]

*Winlogbeat*

- Add handling for missing `EvtVarType`s in experimental API. {issue}19337[19337] {pull}41418[41418]
- Implement exclusion range support for event_id. {issue}38623[38623] {pull}41639[41639]


[[release-notes-8.16.5]]
=== Beats version 8.16.5
https://github.com/elastic/beats/compare/v8.16.4\...v8.16.5[View commits]

==== Bugfixes

*Filebeat*

- [Journald] Fixes handling of `journalctl` restart. A known symptom was broken multiline messages when there was a restart of journalctl while aggregating the lines. {issue}41331[41331] {pull}42595[42595]

*Winlogbeat*

- Sync missing changes in modules pipelines. {pull}42619[42619]

==== Added

*Affecting all Beats*

- Update Go version to 1.22.12. {pull}42681[42681]

*Filebeat*

- Introduce ignore older and start timestamp filters for AWS S3 input. {pull}41804[41804]

*Metricbeat*

- Log every 401 response from Kubernetes API Server. {pull}42714[42714]


[[release-notes-8.16.4]]
=== Beats version 8.16.4
https://github.com/elastic/beats/compare/v8.16.3\...v8.16.4[View commits]

==== Bugfixes

*Filebeat*

- Updated websocket retry error code list to allow more scenarios to be retried which could have been missed previously. {pull}42218[42218]
- In the `streaming` input, prevent panics on shutdown with a null check and apply a consistent namespace to contextual data in debug logs. {pull}42315[42315]
- Remove erroneous status reporting to Elastic-Agent from the Filestream input. {pull}42435[42435]

==== Added

*Filebeat*

- Add metrics for number of events and pages published by HTTPJSON input. {issue}42340[42340] {pull}42442[42442]

*Metricbeat*

- Update beat module with apm-server tail sampling monitoring metrics fields. {pull}42569[42569]


[[release-notes-8.16.3]]
=== Beats version 8.16.3
https://github.com/elastic/beats/compare/v8.16.2\...v8.16.3[View commits]

==== Breaking changes

*Affecting all Beats*

- Default Docker base image was reverted to Ubuntu 20.04 due to incompatability issues with glibc. {pull}42144[42144]

==== Bugfixes

*Filebeat*

- Fix streaming input handling of invalid or empty websocket messages. {pull}42036[42036]
- Fix awss3 document ID construction when using the CSV decoder. {pull}42019[42019]
- Fix Netflow Template Sharing configuration handling. {pull}42080[42080]

*Metricbeat*

- [K8s Integration] Enhance HTTP authentication in case of token updates for Apiserver, Controllermanager and Scheduler metricsets. {issue}41910[41910] {pull}42016[42016]

==== Added

*Affecting all Beats*

- When running under Elastic-Agent Kafka output allows dynamic topic in `topic` field. {pull}40415[40415]
- The script processor has a new configuration option that only uses the cached javascript sessions and prevents the creation of new javascript sessions.
- Update Go version to 1.22.10. {pull}42095[42095]
- Reduce memory consumption of k8s autodiscovery and the add_kubernetes_metadata processor when Deployment metadata is enabled
- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]
- Add `uppercase` processor. {issue}22254[22254] {pull}41535[41535]
- Replace `compress/gzip` with https://github.com/klauspost/compress/gzip library for gzip compression. {pull}41584[41584]
- Add regex pattern matching to add_kubernetes_metadata processor. {pull}41903[41903]

*Filebeat*

- Added OAuth2 support with auto token refresh for websocket streaming input. {issue}41989[41989] {pull}42212[42212]
- Added infinite & blanket retry options to websockets and improved logging and retry logic. {pull}42225[42225]

*Metricbeat*

- Add `use_performance_counters` to collect CPU metrics using performance counters on Windows for `system/cpu` and `system/core` {pull}41965[41965]


[[release-notes-8.16.2]]
=== Beats version 8.16.2
https://github.com/elastic/beats/compare/v8.16.1\...v8.16.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Remove unnecessary reload for Elastic Agent managed beats when apm tracing config changes from nil to nil. {pull}41794[41794]
- Fix incorrect cloud provider identification in add_cloud_metadata processor using provider priority mechanism. {pull}41636[41636]

*Auditbeat*

- auditd: Use ECS `event.type: end` instead of `stop` for SERVICE_STOP, DAEMON_ABORT, and DAEMON_END messages. {pull}41558[41558]
- auditd: Update syscall names for Linux 6.11. {pull}41558[41558]
- hasher: Geneneral improvements and fixes. {pull}41863[41863]

*Filebeat*

- Fix missing key in streaming input logging. {pull}41600[41600]
- Fix the "No such input type exist: 'salesforce'" error on the Windows/AIX platform. {pull}41664[41664]
- Add support for Access Points in the `aws-s3` input. {pull}41495[41495]
- Improve S3 object size metric calculation to support situations where Content-Length is not available. {pull}41755[41755]
- Fix handling of http_endpoint request exceeding memory limits. {issue}41764[41764] {pull}41765[41765]
- Rate limiting fixes in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41583[41583]
- Fixes filestream logging the error "filestream input with ID 'ID' already exists, this will lead to data duplication[...]" on Kubernetes when using autodiscover. {pull}41585[41585]

*Metricbeat*

- Fix incorrect handling of types in SQL module. {issue}40090[40090] {pull}41607[41607]

==== Added

*Affecting all Beats*

- In this release we've introduced an image based on the hardened https://wolfi.dev/[Wolfi] image to provide secure containers to our self-managed customers, help with compliance regulations, and improve our supply chain security posture.

*Auditbeat*

- Split module/system/process into common and provider bits. {pull}41868[41868]

*Filebeat*

- Improve S3 polling mode states registry when using list prefix option. {pull}41869[41869]
- Add support for SSL and Proxy configurations for websoket type in streaming input. {pull}41934[41934]
- AWS S3 input registry cleanup for untracked s3 objects. {pull}41694[41694]
- The environment variable `BEATS_AZURE_EVENTHUB_INPUT_TRACING_ENABLED: true` enables internal logs tracer for the azure-eventhub input. {issue}41931[41931] {pull}41932[41932]
- Added default values in the streaming input for websocket retries and put a cap on retry wait time to be lesser than equal to the maximum defined wait time. {pull}42012[42012]

*Metricbeat*

- Add `id` field to all the vSphere metricsets. {pull}41097[41097]
- Add support for region/zone for Vertex AI service in GCP module {pull}41551[41551]


[[release-notes-8.16.1]]
=== Beats version 8.16.1
https://github.com/elastic/beats/compare/v8.16.0\...v8.16.1[View commits]

==== Breaking changes

*Packetbeat*

- Expire source port mappings. {pull}41581[41581]

==== Bugfixes

*Filebeat*

- Fix AWS region in aws-s3 input S3 polling mode.  {pull}41572[41572]


[[release-notes-8.16.0]]
=== Beats version 8.16.0
https://github.com/elastic/beats/compare/v8.15.4\...v8.16.0[View commits]

==== Known issues

*Filebeat*

- The AWS S3 input polling mode is not working when the S3 bucket is not in the `us-east-1` default region. This also impacts all AWS integrations and any custom AWS log integration which uses the `aws-s3` input polling mode. When using Filebeat, please add a `default_region` configuration with the region of the S3 bucket. For example:
+
["source","yaml"]
----
filebeat.inputs:
- type: aws-s3
  enabled: true
  credential_profile_name: elastic-observability
  default_region: us-east-2
  number_of_workers: 5
  bucket_arn: 'arn:aws:s3:::test1'
----

==== Breaking changes

*Affecting all Beats*

- Fix FQDN being lowercased when used as `host.hostname`. {issue}39993[39993]
- Beats won't log start up information when running under the Elastic Agent. {pull}40390[40390]
- Filebeat now needs `dup3`, `faccessat2`, `prctl` and `setrlimit` syscalls to run the journald input. If this input is not being used, the syscalls are not needed. All Beats have those syscalls allowed now because the default seccomp policy is global to all Beats. {pull}40061[40061]
- Beats will rate limit the logs about errors when indexing events on Elasticsearch, logging a summary every 10s. The logs sent to the event log is unchanged. {issue}40157[40157]

*Filebeat*

- Filebeat, when running with Elastic-Agent, reports status for Filestream input. {pull}40121[40121]
- Added support for hyphens in extension keys in `decode_cef` Filebeat processor. {pull}40427[40427]
- Journald: removed configuration options `include_matches.or`, `include_matches.and`, `backoff`, `max_backoff`, `cursor_seek_fallback`. {pull}40061[40061]
- Journald: `include_matches.match` now behaves in the same way as matchers in `journalctl`. Users should carefully update their input configuration. {pull}40061[40061]
- Journald: `seek` and `since` behaviour have been simplified, if there is a cursor (state) `seek` and `since` are ignored and the cursor is used. {pull}40061[40061]
- Redis: Added replication role as a field to submitted slowlogs.
- Added `container.image.name` to `journald` Filebeat input's Docker-specific translated fields. {pull}40450[40450]
- Remove deprecated awscloudwatch field from Filebeat. {pull}41089[41089]
- The performance of ingesting SQS data with the S3 input has improved by up to 60x for queues with many small events. `max_number_of_messages` config for SQS mode is now ignored, as the new design no longer needs a manual cap on messages. Instead, use `number_of_workers` to scale ingestion rate in both S3 and SQS modes. The increased efficiency may increase network bandwidth consumption, which can be throttled by lowering `number_of_workers`. It may also increase number of events stored in memory, which can be throttled by lowering the configured size of the internal queue. {pull}40699[40699]

*Metricbeat*

- Add support for specifying a custom endpoint for GCP service clients. {issue}40848[40848] {pull}40918[40918]

==== Bugfixes

*Auditbeat*

- Request status from a separate socket to avoid data congestion. {pull}41207[41207]

*Filebeat*

- Fix crashes in the journald input. {pull}40061[40061]
- Fix long filepaths in diagnostics exceeding max path limits on Windows. {pull}40909[40909]
- Fix a bug in Salesforce input to only handle responses with 200 status code. {pull}41015[41015]
- Fixed failed job handling and removed false-positive error logs in the GCS input. {pull}41142[41142]
- Bump github.com/elastic/go-sfdc dependency used by x-pack/filebeat/input/salesforce. {pull}41192[41192]
- Journald input now can read events from all boots {issue}41083[41083] {pull}41244[41244]
- Fix errors in SQS host resolution in the `aws-s3` input when using custom (non-AWS) endpoints. {pull}41504[41504]

*Metricbeat*

- Add GCP 'instance_id' resource label in ECS cloud fields. {issue}40033[40033] {pull}40062[40062]
- Remove excessive info-level logs in cgroups setup. {pull}40491[40491]
- Fix http server helper SSL config. {pull}39405[39405]

==== Added

*Filebeat*

- Implement Elastic Agent status and health reporting for Netflow Filebeat input. {pull}40080[40080]
- Add SSL and username support for Redis input, now the input includes support for Redis 6.0+. {pull}40111[40111]
- Add scaling up support for Netflow input. {issue}37761[37761] {pull}40122[40122]
- Update CEL mito extensions to v1.15.0. {pull}40294[40294]
- Improve logging in Okta Entity Analytics provider. {issue}40106[40106] {pull}40347[40347]
- Document `winlog` input. {issue}40074[40074] {pull}40462[40462]
- Added retry logic to websocket connections in the streaming input. {issue}40271[40271] {pull}40601[40601]
- Disable event normalization for netflow input. {pull}40635[40635]
- Allow attribute selection in the Active Directory entity analytics provider. {issue}40482[40482] {pull}40662[40662]
- Improve error quality when CEL program does not correctly return an events array. {pull}40580[40580]
- Added support for Microsoft Entra ID RBAC authentication. {issue}40434[40434] {pull}40879[40879]
- Add `use_kubeadm` config option for filebeat (both filbeat.input and autodiscovery) in order to toggle kubeadm-config api requests. {pull}40301[40301]
- Make HTTP library function inclusion non-conditional in CEL input. {pull}40912[40912]
- Add support for Crowdstrike streaming API to the streaming input. {issue}40264[40264] {pull}40838[40838]
- Add support to CEL for reading host environment variables. {issue}40762[40762] {pull}40779[40779]
- Add CSV decoder to awss3 input. {pull}40896[40896]
- Change request trace logging to include headers instead of complete request. {pull}41072[41072]
- Improved GCS input documentation. {pull}41143[41143]
- Add CSV decoding capacity to azureblobstorage input. {pull}40978[40978]
- Add CSV decoding capacity to gcs input. {pull}40979[40979]
- Add support to source AWS cloudwatch logs from linked accounts. {pull}41188[41188]
- Jounrald input now supports filtering by facilities. {pull}41061[41061]
- Add support to include AWS cloudwatch linked accounts when using log_group_name_prefix to define log group names. {pull}41206[41206]

*Heartbeat*

- Add journey duration to synthetics browser events. {pull}40230[40230]

*Metricbeat*

- Add new metrics fot datastore and minor changes to overall vSphere metrics. {pull}40766[40766]
- Add new metricset datastorecluster for vSphere module. {pull}40634[40634] {pull}40694[40694]
- Add AWS Cloudwatch capability to retrieve tags from AWS/ApiGateway resources. {pull}40755[40755]
- Add new metrics for the vSphere Virtualmachine metricset. {pull}40485[40485]
- Add `metrics_count` to Prometheus module if `metrics_count: true` is set. {pull}40411[40411]


[[release-notes-8.15.5]]
=== Beats version 8.15.5
https://github.com/elastic/beats/compare/v8.15.4\...v8.15.5[View commits]

==== Breaking changes

*Packetbeat*

- Expire source port mappings. {pull}41581[41581]

==== Bugfixes

*Affecting all Beats*

- Fix metrics not being ingested, due to "Limit of total fields [10000] has been exceeded while adding new fields [...]". The total fields limit has been increased to 12500. No significant performance impact on Elasticsearch is anticipated. {pull}41640[41640]

*Filebeat*

- Fix AWS region in `aws-s3` input S3 polling mode.  {pull}41572[41572]


[[release-notes-8.15.4]]
=== Beats version 8.15.4
https://github.com/elastic/beats/compare/v8.15.3\...v8.15.4[View commits]

==== Breaking changes

*Osquerybeat*

- Disable `allow_unsafe` osquery configuration. {pull}40130[40130]

==== Bugfixes

*Affecting all Beats*

- Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use. {pull}41356[41356]

*Filebeat*

- Log bad handshake details when websocket connection fails. {pull}41300[41300]
- Improve modification time handling for entities and entity deletion logic in the Active Directory entityanalytics input. {pull}41179[41179]
- Fix double encoding of `client_secret` in the Entity Analytics input's Azure Active Directory provider. {pull}41393[41393]
- The azure-eventhub input now correctly reports its status to the Elastic Agent on fatal errors. {pull}41469[41469]

*Metricbeat*

- Fix Kubernetes metadata sometimes not being present after startup. {pull}41216[41216]

*Winlogbeat*

- Fix truncated windows event log message. {pull}41327[41327]

==== Added

*Affecting all Beats*

- Replace Ubuntu 20.04 with 24.04 for Docker base images. {issue}40743[40743] {pull}40942[40942]
- Reduce memory consumption of k8s autodiscovery and the `add_kubernetes_metadata` processor when Deployment metadata is enabled.

*Heartbeat*

- Add monitor status reporter under managed mode. {pull}41077[41077]

*Metricbeat*

- Only watch metadata for ReplicaSets in metricbeat k8s module. {pull}41289[41289]


[[release-notes-8.15.3]]
=== Beats version 8.15.3
https://github.com/elastic/beats/compare/v8.15.2\...v8.15.3[View commits]

==== Known issues

*Affecting all Beats*

- Memory usage is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. {issue}41355[41355]

==== Breaking changes

*Filebeat*

- Change `log.file.path` field in awscloudwatch input to nested object. {pull}41099[41099]

==== Bugfixes

*Affecting all Beats*

- Allow port number 0 in the community ID flowhash processor. {pull}40259[40259]
- The journald input now restarts if there is an error/crash. {issue}32782[32782] {pull}40558[40558]

*Filebeat*

- Fix replace processor handling of zero string replacement validation. {pull}40751[40751]
- Add backup and delete for AWS S3 polling mode feature back. {pull}41071[41071]

*Metricbeat*

- Use namespace for GetListMetrics when it exists in AWS. {pull}41022[41022]

*Packetbeat*

- Fix upload of bundled ingest pipelines on Windows. {pull}41110[41110]

==== Added

*Affecting all Beats*

- Update Go version to 1.22.8. {pull}41139[41139]
- Add kafka compression support for ZSTD.

*Metricbeat*

- Restore `docker.network.in.*` and `docker.network.out.*` fields in docker module. {pull}40968[40968]


[[release-notes-8.15.2]]
=== Beats version 8.15.2
https://github.com/elastic/beats/compare/v8.15.0\...v8.15.2[View commits]

==== Known issues

*Affecting all Beats*

- Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file. {issue}41118[41118]

==== Known issues

*Affecting all Beats*

- Memory usage is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. {issue}41355[41355]

==== Bugfixes

*Affecting all Beats*

- Fix bug that prevented the Elasticsearch output from recovering from an interrupted connection. {issue}40705[40705] {pull}40769[40796]

*Metricbeat*

- Add GCP organization and project details to ECS cloud fields. {pull}40461[40461]

[[release-notes-8.15.1]]
=== Beats version 8.15.1
https://github.com/elastic/beats/compare/v8.15.0\...v8.15.1[View commits]

==== Known issues

*Affecting all Beats*

- Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file.
- Beats stop publishing data after a network error unless restarted. Avoid upgrading to 8.15.1. Affected Beats log `Get \"https://${ELASTICSEARCH_HOST}:443\": context canceled` repeatedly. {issue}40705{40705}
- Memory usage is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. {issue}41355[41355]

==== Bugfixes

*Affecting all Beats*

- Aborts all active connections for Elasticsearch output. {pull}40572[40572]
- Closes beat Publisher on beat stop and by the Agent manager. {pull}40572[40572]
- Fix handling of escaped brackets in syslog structured data. {issue}40445[40445] {pull}40446[40446]

*Auditbeat*

- Fix segfaults that may happen if user runs multiple instances of the package metricset {pull}40525[40525]
- Fix incorrect definition of struct utmp for arm64 {pull}40541[40541]

*Filebeat*

- Relax requirements in Okta entity analytics provider user and device profile data shape. {pull}40359[40359]
- Fix bug in Okta entity analytics rate limit logic. {issue}40106[40106] {pull}40267[40267]
- Fix order of configuration for EntraID entity analytics provider. {pull}40487[40487]
- Ensure Entra ID request bodies are not truncated and trace logs are rotated before 100MB. {pull}40494[40494]
- The Elasticsearch output now correctly logs the event fields to the event log file {issue}40509[40509] {pull}40512[40512]
- Fix the "No such input type exist: 'azure-eventhub'" error on the Windows platform {issue}40608[40608] {pull}40609[40609]
- awss3 input: Fix handling of SQS notifications that don't contain a region. {pull}40628[40628]
- Fix credential handling when workload identity is being used in GCS input. {issue}39977[39977] {pull}40663[40663]
- Fix high IO and handling of a corrupted registry log file. {pull}35893[35893]
- Fix filestream's registry GC: registry entries will never be removed if clean_inactive is set to "-1". {pull}40258[40258]

*Metricbeat*

- Fix first HTTP 401 error when fetching metrics from the Kubelet API caused by a token update {pull}40636[40636]
- Fix needlessly verbose logging in cgroups setup {issue}40620[40620]

==== Added

*Filebeat*

- Enable file ingestion to report detailed status to Elastic Agent {pull}40075[40075]
- Added `ignore_empty_values` flag in `decode_cef` Filebeat processor. {pull}40268[40268]

*Metricbeat*

- Added back `elasticsearch.node.stats.jvm.mem.pools.*` to the `node_stats` metricset {pull}40571[40571]

[[release-notes-8.15.0]]
=== Beats version 8.15.0
https://github.com/elastic/beats/compare/v8.14.3\...v8.15.0[View commits]

==== Known issues

*Affecting all Beats*

- Beats Docker images do not log to stderr by default. The workaround is to pass the CLI flag `-e` or to set `logging.to_stderr: true` in the configuration file. {issue}41118[41118]

*Filebeat*

- The Azure EventHub input in Filebeat is not found when running on Windows. Please refrain from upgrading to 8.15. See {issue}40608[40608] for details.
- Memory usage is not correctly limited by the number of events actively in the memory queue, but rather the maximum size of the memory queue regardless of usage. {issue}41355[41355]
- The AWS S3 input polling mode is not working when the S3 bucket is not in the `us-east-1` default region. This also impacts all AWS integrations and any custom AWS log integration which uses the `aws-s3` input polling mode. When using Filebeat, please add a `default_region` configuration with the region of the S3 bucket. For example:
+
["source","yaml"]
----
filebeat.inputs:
- type: aws-s3
  enabled: true
  credential_profile_name: elastic-observability
  default_region: us-east-2
  number_of_workers: 5
  bucket_arn: 'arn:aws:s3:::test1'
----

==== Breaking changes

*Filebeat*

- Tag events that come from a filestream in "take over" mode. {pull}39828[39828]
- Fix filestream's registry garbage collection: registry entries will never be removed if `clean_inactive` is set to "-1". {pull}40258[40258]

*Metricbeat*

- Remove fallback to the node limit for the `kubernetes.pod.cpu.usage.limit.pct` and `kubernetes.pod.memory.usage.limit.pct` metrics calculation.
- Add support for Kibana status metricset in v8 format. {pull}40275[40275]

*Osquerybeat*

- Add action responses data stream, allowing Osquerybeat to post action results directly to Elasticsearch. {pull}39143[39143]

==== Bugfixes

*Affecting all Beats*

- Rename the field "apache2.module.error" to "apache.module.error" in Apache error visualization. {issue}39480[39480] {pull}39481[39481]
- Validate config of the `replace` processor. {pull}40047[40047]

*Filebeat*

- Fix for Google Workspace duplicate events issue by adding canonical sorting over fingerprint keys array to maintain key order. {pull}40055[40055] {issue}39859[39859]
- Prevent panic in CEL and salesforce inputs when `github.com/hashicorp/go-retryablehttp` exceeds maximum retries. {pull}40144[40144]
- Update CEL mito extensions to v1.13.1. {pull}40307[40307]
- Fix bug in CEL input rate limit logic. {issue}40106[40106] {pull}40270[40270]

*Metricbeat*

- Set GCP metrics config period to the default (60s) when the value is below the minimum allowed period. {issue}30434[30434] {pull}40020[40020]
- Fix statistic methods for metrics collected for SQS. {pull}40207[40207]
- Update beat module with apm-server monitoring metrics fields. {pull}40127[40127]
- Fix Azure Monitor metric timespan to restore Storage Account PT1H metrics. {issue}40376[40376] {pull}40367[40367]

==== Added

*Affecting all Beats*

- Update Go version to 1.22.5. {pull}40082[40082]
- Introduce log message for not supported annotations for Hints based autodiscover. {pull}38213[38213]
- Add persistent volume claim name to volume if available. {pull}38839[38839]
- Raw events are now logged to a different file, this prevents potentially sensitive information from leaking into log files. {pull}38767[38767]
- Websocket input: Added runtime URL modification support based on state and cursor values. {issue}39858[39858] {pull}39997[39997]

*Auditbeat*

- Reduce data size for `add_session_metadata` processor by removing unneeded fields. {pull}39500[39500]
- Enrich process events with user and group names, with `add_session_metadata` processor. {pull}39537[39537]

*Filebeat*

- Ensure all responses sent by HTTP Endpoint are HTML-escaped. {pull}39329[39329]
- Improve logging of request and response with request trace logging in error conditions. {pull}39455[39455]
- Implement Elastic Agent status and health reporting for CEL Filebeat input. {pull}39209[39209]
- Add HTTP metrics to CEL input. {issue}39501[39501] {pull}39503[39503]
- Add default user-agent to CEL HTTP requests. {issue}39502[39502] {pull}39587[39587]
- Improve reindexing support in security module pipelines. {issue}38224[38224] {pull}39588[39588]
- Make HTTP Endpoint input GA. {issue}38979[38979] {pull}39410[39410]
- Add support for base64-encoded HMAC headers to HTTP Endpoint. {pull}39655[39655]
- Add user group membership support to Okta entity analytics provider. {issue}39814[39814] {pull}39815[39815]
- Add request trace support for Okta and EntraID entity analytics providers. {pull}39821[39821]
- Allow elision of set and append failure logging. {issue}34544[34544] {pull}39929[39929]
- Add ability to remove request trace logs from CEL input. {pull}39969[39969]
- Add ability to remove request trace logs from HTTPJSON input. {pull}40003[40003]
- Update CEL mito extensions version to v1.13.0 {pull}40035[40035]
- Add Jamf entity analytics provider. {pull}39996[39996]
- Add ability to remove request trace logs from `http_endpoint` input. {pull}40005[40005]
- Add ability to remove request trace logs from `entityanalytics` input. {pull}40004[40004]
- Relax constraint on Base DN in entity analytics Active Directory provider. {pull}40054[40054]
- Enhance input state reporting for CEL evaluations that return a single error object in events. {pull}40083[40083]
- Allow absent credentials when using GCS with Application Default Credentials. {issue}39977[39977] {pull}40072[40072]
- Allow cross-region bucket configuration in S3 input. {issue}22161[22161] {pull}40309[40309]

*Metricbeat*

- Support `schema_name` for MySQL performance metricset. {pull}38363[38363]
- Add `last_terminated_timestamp` metric in Kubernetes module. {pull}39200[39200] {issue}3802[3802]
- Add `pod.status.ready_time` and `pod.status.reason` metrics in Kubernetes module. {pull}39316[39316]
- Add "Buffer cache hit ratio base" to calculate "Buffer cache hit ratio" for performance metrics. {pull}40022[40022]
- Add support of Graphite series 1.1.0+ tagging extension for statsd module. {pull}39619[39619]


[[release-notes-8.14.3]]
=== Beats version 8.14.3
https://github.com/elastic/beats/compare/v8.14.2\...v8.14.3[View commits]

==== Known Issues

*Filebeat*
 - Filestream input will resend files that have been inactive for 30min or more. Workaround: set `clean_inactive` to a very high value, like 5 years: `clean_inactive: 43800h0m0s`. {issue}40178[40178]

==== Bugfixes

*Filebeat*
- Fix handling of deeply nested numeric values in HTTP Endpoint CEL programs. {pull}40115[40115]

*Metricbeat*

- Fix `namespace` filter option on metricset `state_namespace` enricher. {pull}39934[39934]
- Fix `namespace` filter option at Kubernetes provider level. {pull}39881[39881]
- Fix handling of access errors when reading process metrics. {pull}39627[39627]
- Fix behavior of cgroups path discovery when monitoring the host system from within a container. {pull}39627[39627]
- Fix issue where Beats may report incorrect metrics for its own process when running inside a container. {pull}39627[39627]
- Fix for MySQL/Performance - Query failure for MySQL versions below v8.0.1, for performance metric `quantile_95`. {pull}38710[38710]
- Fix Prometheus helper text parser to store each metric family type. {pull}39743[39743]
- Normalize AWS RDS CPU utilization values before making the metadata API call. {pull}39664[39664]
- Fix behavior of `pagetypeinfo` metrics. {pull}39985[39985]
- Fix query logic for temp and non-temp tablespaces in Oracle module. {issue}38051[38051] {pull}39787[39787]
- Fix missing metrics from CloudWatch when include_linked_accounts set to false. {issue}40071[40071] {pull}40135[40135]

==== Added

*Affecting all Beats*

- Update Go version to 1.21.12. {pull}40114[40114]


[[release-notes-8.14.2]]
=== Beats version 8.14.2
https://github.com/elastic/beats/compare/v8.14.1\...v8.14.2[View commits]

==== Known Issues

*Filebeat*
 - Filestream input will resend files that have been inactive for 30min or more. Workaround: set `clean_inactive` to a very high value, like 5 years: `clean_inactive: 43800h0m0s`. {issue}40178[40178]

==== Breaking changes

*Filebeat*

- Fix high I/O and handling of a corrupted registry log file. {pull}35893[35893]

==== Bugfixes

*Filebeat*
- Fix request trace filename handling in `http_endpoint` input. {pull}39410[39410]
- Fix filestream not correctly tracking the offset of a file when using the `include_message` parser. {pull}39873[39873] {issue}39653[39653]
- Upgrade `github.com/hashicorp/go-retryablehttp` to mitigate CVE-2024-6104. {pull}40036[40036]

==== Added

*Affecting all Beats*

- Update Go version to 1.21.11. {pull}39851[39851]

*Filebeat*

- Fix handling of infinite rate values in CEL rate limit handling logic. {pull}39940[39940]

*Heartbeat*

- Upgrade Node version to the latest LTS v18.20.3. {pull}40038[40038]

*Winlogbeat*

- Add ERROR_INVALID_PARAMETER to the list of recoverable errors. {pull}39781[39781]


[[release-notes-8.14.1]]
=== Beats version 8.14.1
https://github.com/elastic/beats/compare/v8.14.0\...v8.14.1[View commits]

==== Known Issues

*Filebeat*
 - Filestream input will resend files that have been inactive for 30min or more. Workaround: set `clean_inactive` to a very high value, like 5 years: `clean_inactive: 43800h0m0s`. {issue}40178[40178]

==== Bugfixes

*Heartbeat*

- Fix import of browser plugin for Agentbeat. {pull}39818[39818]


[[release-notes-8.14.0]]
=== Beats version 8.14.0
https://github.com/elastic/beats/compare/v8.13.4\...v8.14.0[View commits]

==== Known Issues

*Filebeat*
 - Filestream input will resend files that have been inactive for 30min or more. Workaround: set `clean_inactive` to a very high value, like 5 years: `clean_inactive: 43800h0m0s`. {issue}40178[40178]

==== Breaking changes

*Filebeat*

- Removed deprecated ZScaler from Beats. Use the https://docs.elastic.co/integrations/zscaler_zia[Zscaler Internet Access] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Tomcat from Beats. Use the https://docs.elastic.co/integrations/apache_tomcat[Apache Tomcat] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Squid from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated SonicWall from Beats. Use the https://docs.elastic.co/integrations/sonicwall[SonicWall Firewall] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Snort from Beats. Use the https://docs.elastic.co/integrations/snort[Snort] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Radware from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated Proofpoint from Beats. Use the https://docs.elastic.co/integrations/proofpoint_tap[Proofpoint TAP] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Netscout from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated Microsoft DHCP from Beats. Use the https://docs.elastic.co/integrations/microsoft_dhcp[Microsoft DHCP] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Juniper Junos from Beats. Use the https://docs.elastic.co/integrations/juniper_srx[Juniper SRX] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Juniper Netscreen from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated Infoblox from Beats. Use the https://docs.elastic.co/integrations/infoblox_nios[Infoblox NIOS] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Impreva from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated Fortinet Client Endpoint from Beats. Use the https://docs.elastic.co/integrations/fortinet_forticlient[Fortinet FortiClient Logs] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Fortinet Fortimail from Beats. Use the https://docs.elastic.co/integrations/fortinet_fortimail[Fortinet FortiMail] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Fortinet Fortimanager from Beats. Use the https://docs.elastic.co/integrations/fortinet_fortimanager[Fortinet FortiManager Logs] Elastic integration instead. {pull}38037[38037]
- Removed deprecated F5 from Beats. Use the https://docs.elastic.co/integrations/f5_bigip[F5 BIG-IP] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Cylance from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated Cisco Meraki from Beats. Use the https://docs.elastic.co/integrations/cisco_meraki[Cisco Meraki] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Cisco Nexus from Beats. Use the https://docs.elastic.co/integrations/cisco_nexus[Cisco Nexus] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Bluecoat from Beats. See {filebeat-ref}/migrate-from-deprecated-module.html[Migrate from a deprecated module] for migration options. {pull}38037[38037]
- Removed deprecated Barracuda from Beats. Use the https://docs.elastic.co/integrations/barracuda[Barracuda Web Application Firewall] Elastic integration instead. {pull}38037[38037]
- Removed deprecated Sophos UTM from Beats. Use the https://docs.elastic.co/integrations/sophos[Sophos] Elastic integration instead. {pull}38037[38037]
- Introduce input/netmetrics and refactor netflow input metrics. {pull}38055[38055]
- Update Salesforce module to use new Salesforce input. {pull}37509[37509]

*Heartbeat*

- Fix monitor state loader to not wait extra seconds for the last attempt. {pull}39621[39621]

==== Bugfixes

*Auditbeat*
- Set field types to correctly match ECS in sessionmd processor. {issue}38955[38955] {pull}38994[38994]
- Fix failing to enrich process events in sessionmd processor. {issue}38955[38955] {pull}39173[39173] {pull}39243[39243]
- Fix seccomp policy of FIM kprobes backend on arm64. {pull}39759[39759]

*Filebeat*
- Fix handling of endpoint for custom domains and ensure region, default_region, and region parsed from queue_url are applied in the order specified in the documentation for the awss3 input. {pull}39709[39709]
- Prevent HTTPJSON holding response bodies between executions. {issue}35219[35219] {pull}38116[38116]
- Fix the incorrect values generated by the uri_parts processor. {pull}38216[38216]
- Rename `activity_guid` to `activity_id` in ETW input events to suit other Windows inputs. {pull}38530[38530]
- Add missing provider registration and fix published entity for Active Directory entityanalytics provider. {pull}38645[38645]
- Fix handling of un-parsed JSON in O365 module. {issue}37800[37800] {pull}38709[38709]
- Fix filestream's registry GC: registry entries are now removed from the in-memory and disk store when they're older than the set TTL. {issue}36761[36761] {pull}38488[38488]
- Fix handling of truncated files in Filestream {issue}38070[38070] {pull}38416[38416]
- Fix panic when more than 32767 pipeline clients are active. {issue}38197[38197] {pull}38556[38556]
- Fix a bug in CloudWatch task allocation that could skip some logs. {issue}38918[38918] {pull}38953[38953]
- Prevent GCP Pub/Sub input blockage by increasing default value of `max_outstanding_messages`. {issue}35029[35029] {pull}38985[38985]
- entity-analytics input: Improve structured logging. {pull}38990[38990]
- Upgrade `azure-event-hubs-go` and `azure-storage-blob-go` dependencies. {pull}38861[38861]
- Fix concurrency/error handling bugs in the AWS S3 input that could drop data and prevent ingestion of large buckets. {pull}39131[39131]
- Fix EntraID query handling. {issue}39419[39419] {pull}39420[39420]
- Expand ID patterns in request trace logger for HTTP Endpoint. {pull}39656[39656]
- Fix awscloudwarch input: set startTime to `0` for the first iteration of retrieving log events from CloudWatch. {pull}40079[40079]

*Heartbeat*

- Redact synthexec cmd output. {pull}39535[39535]

*Metricbeat*

- RabbitMQ/queue - Change the mapping type of `rabbitmq.queue.consumers.utilisation.pct` to `scaled_float` from `long` because the values fall within the range of `[0.0, 1.0]`. Previously, conversion to integer resulted in reporting either `0` or `1`.
- Fix timeout caused by the retrival of which indices are hidden. {pull}39165[39165]

*Winlogbeat*

- Fix error handling in perfmon metrics. {issue}38140[38140] {pull}39404[39404]

==== Added

*Affecting all Beats*

- Update Go version to 1.21.10. {pull}39467[39467]
- Enable early event encoding in the Elasticsearch output, improving CPU and memory use. {pull}38572[38572]

*Auditbeat*

- Add `add_session_metadata` processor, which enables session viewer on Auditbeat data. {pull}37640[37640]
- Add procfs backend to the `add_session_metadata` processor. {pull}38799[38799]
- Add `process.entity_id`, `process.group.name` and `process.group.id` in `add_process_metadata` processor. Make FIM module with Kprobes backend to always add an appropriately configured `add_process_metadata` processor to enrich file events. {pull}38776[38776]

*Filebeat*

- Add Saved Object name field to Kibana audit logs. {pull}38307[38307]
- Add Salesforce input. {pull}37331[37331]
- Add logging for cache processor file reads and writes. {pull}38052[38052]
- Support VPC endpoint for aws-s3 input SQS queue url. {pull}38189[38189]
- Add support for complex event objects in the HTTP Endpoint input. {issue}37910[37910] {pull}38193[38193]
- Parse more fields from Elasticsearch slowlogs. {pull}38295[38295]
- Update CEL mito extensions to v1.10.0 to add keys/values helper. {pull}38504[38504]
- Add support for Active Directory an entity analytics provider. {pull}37919[37919]
- Add AWS AWSHealth metricset. {pull}38370[38370]
- Add debugging breadcrumb to logs when writing request trace log. {pull}38636[38636]
- Add benchmark input and discard output. {pull}37437[37437]

*Libbeat*

- Add support for Linux capabilities in `add_process_metadata`. {pull}38252[38252]

*Metricbeat*

- Add support for `shards_stats.total_count` in Elasticsearch Monitoring data. {pull}38891[38891]
- Add SSL support to MySQL module. {pull}37997[37997]
- Add SSL support for Aerospike module. {pull}38126[38126]

*Winlogbeat*

- Use fixed size buffer at first pass for event parsing, improving throughput. {issue}39530[39530] {pull}39544[39544]

==== Deprecated

*Filebeat*

- Deprecate `syslog` input in favor of `syslog` processor. {issue}37555[37555] {pull}38277[38277]
- Deprecate `o365audit` input in favor of `CEL` input. {issue}37719[37719] {pull}38922[38922]


[[release-notes-8.13.4]]
=== Beats version 8.13.4
https://github.com/elastic/beats/compare/v8.13.3\...v8.13.4[View commits]

==== Bugfixes

*Auditbeat*

- Prevent scenario of losing children-related file events in a directory for recursive fsnotify backend of auditbeat file integrity module. {pull}39133[39133]
- Allow extra syscalls by auditbeat required in FIM with kprobes back-end. {pull}39361[39361]
- Fix losing events in FIM for MacOS X by allowing always to walk an added directory to monitor. {pull}39362[39362]

*Metricbeat*

- Fix Azure Monitor support for multiple aggregation types. {issue}39192[39192] {pull}39204[39204]


[[release-notes-8.13.3]]
=== Beats version 8.13.3
https://github.com/elastic/beats/compare/v8.13.2\...v8.13.3[View commits]

==== Breaking changes

*Metricbeat*
- Setting period for counter cache for Prometheus `remote_write` to at least to 60 seconds. {pull}38553[38553]

==== Bugfixes

*Affecting all Beats*
- Change cache processor documentation from `write_period` to `write_interval`. {pull}38561[38561]
- Fix cache processor expiries heap cleanup on partial file writes. {pull}38561[38561]
- Fix cache processor expiries infinite growth when large a large TTL is used and recurring keys are cached. {pull}38561[38561]
- Fix parsing of RFC 3164 process IDs in syslog processor. {issue}38947[38947] {pull}38982[38982]

*Filebeat*

- Fix indexing failures by re-enabling event normalisation in netflow input. {issue}38703[38703] {pull}38780[38780]
- Fix config validation for CEL and HTTPJSON inputs when using password grant authentication and `client.id` or `client.secret` are not present. {pull}38962[38962]
- Updated Websocket input title to align with existing inputs. {pull}39006[39006]
- [threatintel] MISP splitting fix for empty responses. {issue}38739[38739] {pull}38917[38917]
- Restore netflow input on Windows. {pull}39024[39024]

==== Added

*Affecting all Beats*

- Update Go version to 1.21.9. {pull}38727[38727]
- The environment variable `BEATS_ADD_CLOUD_METADATA_PROVIDERS` overrides configured/default `add_cloud_metadata` providers. {pull}38669[38669]

*Auditbeat*

- Add process data to file events (Linux only, eBPF backend). {pull}38199[38199]
- Add container id to file events (Linux only, eBPF backend). {pull}38328[38328]

*Metricbeat*

- Add new fields to configure the lease duration, retry and renew when using leader elector with Kubernetes autodiscover. {pull}38471[38471]


[[release-notes-8.13.2]]
=== Beats version 8.13.2
https://github.com/elastic/beats/compare/v8.13.1\...v8.13.2[View commits]

==== Bugfixes

*Heartbeat*

- Fix reference yaml format. {pull}38584[38584]


[[release-notes-8.13.1]]
=== Beats version 8.13.1
https://github.com/elastic/beats/compare/v8.13.0\...v8.13.1[View commits]

==== Bugfixes

*Filebeat*

- Fix Filebeat GCS input panic. {pull}38407[38407]

*Heartbeat*

- Reset prctl dumpable flag after cap drop. {pull}38269[38269]


[[release-notes-8.13.0]]
=== Beats version 8.13.0
https://github.com/elastic/beats/compare/v8.12.2\...v8.13.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Upgrade Go version to 1.21.8. Removes support for Windows 8.1. See https://tip.golang.org/doc/go1.21#windows. {pull}38209[38209]
- The behavior of `queue.mem.flush.min_events` has been simplified. It now serves as a simple maximum on the size of all event batches. There are no longer performance implications in its relationship to `bulk_max_size`. {pull}37795[37795]

*Auditbeat*

- Add opt-in `KProbes` backend for file_integrity module. {pull}37796[37796]

*Filebeat*

- Convert netflow input to API v2 and disable event normalisation. {pull}37901[37901]

*Winlogbeat*

- Add "keystore.path" configuration settings to $workdir\data\{{.BeatName}}.keystore. {issue}12315[12315] {pull}37237[37237]

==== Bugfixes

*Affecting all Beats*
- Support Elastic Agent control protocol chunking support. {pull}37343[37343]
- Upgrade elastic-agent-libs to v0.7.5. Removes obsolete "Treating the CommonName field on X.509 certificates as a host name..." deprecation warning for 8.0. {pull}37755[37755]
- Fix the paths in the .cmd script added to the path by the Windows MSI to point to the new C:\Program Files installation location. https://github.com/elastic/elastic-stack-installers/pull/238
- Upgrade elastic-agent-system-metrics to v0.9.2. Skips permissions errors when reading /proc/pid/io. {pull}38234[38234].

*Filebeat*

- Fix a race condition that could crash Filebeat with a "negative WaitGroup counter" error. {pull}38094[38094]
- Fix "failed processing S3 event for object key" error on aws-s3 input when key contains the "+" character. {issue}38012[38012] {pull}38125[38125]
- Fix duplicated addition of regexp extension in CEL input. {pull}38181[38181]
- Fix HTTPJSON handling of empty object bodies in POST requests. {issue}33961[33961] {pull}38290[38290]
- Fix PEM key validation for CEL and HTTPJSON inputs. {pull}38405[38405]

*Heartbeat*

- Adjust State loader to only retry when response code status is 5xx. {pull}37981[37981]

*Metricbeat*

- Fix Azure Monitor 429 error by causing Metricbeat to retry the request again. {pull}38294[38294]
- Fix fields not being parsed correctly in postgresql/database. {issue}25301[25301] {pull}37720[37720]

==== Added

*Affecting all Beats*

- Ignore Kubernetes node and namespace update events that do not change pod metadata. {issue}37338[37338] {pull}37431[37431]
- Enhance add_cloud_metadata processor with `orchestrator.cluster.name`, `orchestrator.cluster.id` and `azure.resourcegroup.name` when running inside an AKS cluster. {issue}33081[33081] {pull}37685[37685]
- Upgrade go-sysinfo from 1.12.0 to 1.13.1. {pull}37996[37996]
- Make `range` condition work with numeric values as strings. {pull}38080[38080]
- Allow users to configure number of output workers (for outputs that support workers) with either `worker` or `workers`. {pull}38257[38257]
- Kafka output now validates the `topics` and `topic` configuration values. {pull}38058[38058]

*Auditbeat*

- Add Linux capabilities to processes in the system/process. {pull}37453[37453]
- Add opt-in eBPF backend for file_integrity module. {pull}37223[37223]

*Filebeat*

- Update SQL input documentation regarding Oracle DSNs {pull}37590[37590]
- Add support for complete URL replacement in HTTPJSON chain steps. {pull}37486[37486]
- Add support for user-defined query selection in EntraID entity analytics provider. {pull}37653[37653]
- Update CEL extensions library to v1.8.0 to provide runtime error location reporting. {issue}37304[37304] {pull}37718[37718]
- Add request trace logging for chained API requests. {issue}37551[36551] {pull}37682[37682]
- Add support for PEM-based Okta auth in HTTPJSON. {pull}37772[37772]
- Prevent complete loss of long request trace data. {issue}37826[37826] {pull}37836[37836]
- Added experimental version of the Websocket Input. {pull}37774[37774]
- Add support for PEM-based Okta auth in CEL. {pull}37813[37813]
- Add ETW input. {pull}36915[36915]
- Update CEL mito extensions to v1.9.0 to add keys/values helper. {pull}37971[37971]
- Add parseDateInTZ value template for the HTTPJSON input. {pull}37738[37738]
- Improve rate limit handling by HTTPJSON. {issue}36207[36207] {pull}38161[38161] {pull}38237[38237]

*Libbeat*
- Add watcher that can be used to monitor Linux kernel events. {pull}37833[37833]
- Added support for ETW reader. {pull}36914[36914]

*Heartbeat*
- Upgrade github.com/elastic/go-elasticsearch/v8 to v8.12.0. {pull}37673[37673]

*Metricbeat*

- Fix containerd metrics grouping for TSDB. {pull}37537[37537]

*Packetbeat*

- Bump Windows Npcap version to v1.79. {pull}37733[37733]
- Add support for pipeline loading. {pull}37291[37291]


[[release-notes-8.12.2]]
=== Beats version 8.12.2
https://github.com/elastic/beats/compare/v8.12.1\...v8.12.2[View commits]

==== Bugfixes

*Filebeat*

- [threatintel] MISP pagination fixes. {pull}37898[37898]
- Fix file handle leak when handling errors in filestream. {pull}37973[37973]

*Packetbeat*

- Fix interface device parsing for packetbeat protocols. {pull}37946[37946]

==== Added

*Metricbeat*

- Update `getOpTimestamp` in `replstatus` to fix sort and temp files generation issue in MongoDB. {pull}37688[37688]


[[release-notes-8.12.1]]
=== Beats version 8.12.1
https://github.com/elastic/beats/compare/v8.12.0\...v8.12.1[View commits]

==== Known Issues

*Affecting all Beats*

Performance regression in AWS S3 inputs using SQS notification.

In 8.12 the default memory queue flush interval was raised from 1 second to 10 seconds. In many configurations this improves performance because it allows the output to batch more events per round trip, which improves efficiency. However, the SQS input has an extra bottleneck that interacts badly with the new value. For more details see {issue}37754[37754].

If you are using the Elasticsearch output, and your output configuration uses a performance preset, switch it to `preset: latency`. If you use no preset or use `preset: custom`, then set `queue.mem.flush.timeout: 1s` in your queue or output configuration.

==== Breaking changes

*Affecting all Beats*

- add_cloud_metadata processor: `huawei` provider is now treated as `openstack`. Huawei cloud runs on OpenStack
platform, and when viewed from a metadata API standpoint, it is impossible to differentiate it from OpenStack. If you
know that your deployments run on Huawei Cloud exclusively, and you wish to have `cloud.provider` value as `huawei`,
you can achieve this by overwriting the value using an `add_fields` processor. {pull}35184[35184]

==== Bugfixes

*Affecting all Beats*

- aws: Add credential caching for `AssumeRole` session tokens. {issue}37787[37787]
- Lower logging level to debug when attempting to configure beats with unknown fields from autodiscovered events/environments. {pull}[37816][37816]

*Filebeat*

- Fix nil pointer dereference in the httpjson input. {pull}37591[37591]
- Fix TCP/UDP metric queue length parsing base. {pull}37714[37714]
- Fix m365_defender cursor value and query building. {pull}37116[37116]
- Update github.com/lestrrat-go/jwx dependency. {pull}37799[37799]

*Heartbeat*

- Fix setuid root when running under cgroups v2. {pull}37794[37794]

*Metricbeat*

- Fix Azure Resource Metrics missing metrics (min and max aggregations) after upgrade to 8.11.3. {issue}37642[37642] {pull}37643[37643]

==== Added

*Filebeat*

- Relax TCP/UDP metric polling expectations to improve metric collection. {pull}37714[37714]


[[release-notes-8.12.0]]
=== Beats version 8.12.0
https://github.com/elastic/beats/compare/v8.11.4\...v8.12.0[View commits]

==== Known Issues

*Affecting all Beats*

Performance regression in AWS S3 inputs using SQS notification.

In 8.12 the default memory queue flush interval was raised from 1 second to 10 seconds. In many configurations this improves performance because it allows the output to batch more events per round trip, which improves efficiency. However, the SQS input has an extra bottleneck that interacts badly with the new value. For more details see {issue}37754[37754].

If you are using the Elasticsearch output, and your output configuration uses a performance preset, switch it to `preset: latency`. If you use no preset or use `preset: custom`, then set `queue.mem.flush.timeout: 1s` in your queue or output configuration.

If you are not using the Elasticsearch output, set `queue.mem.flush.timeout: 1s` in your queue or output configuration.

==== Breaking changes

*Affecting all Beats*

- Windows MSI installers now store configuration in C:\Program Files instead of C:\ProgramData. https://github.com/elastic/elastic-stack-installers/pull/209

*Heartbeat*

- Decrease the ES default timeout to 10 for the load monitor state requests.
- Windows MSI installers now store configuration in C:\Program Files instead of C:\ProgramData. https://github.com/elastic/elastic-stack-installers/pull/209

*Osquerybeat*

- Upgrade to osquery 5.10.2. {pull}37115[37115]

==== Bugfixes

*Filebeat*

- Add validation to the `http_endpoint` config for empty URL. {pull}36816[36816] {issue}36772[36772]
- Fix merging of array fields (processors, paths, parsers) in configurations generated from hints and default config. {issue}36838[36838] {pull}36857[36857]

==== Added

*Affecting all Beats*

- Allow `queue` configuration settings to be set under the output. {issue}35615[35615] {pull}36788[36788]
- Raise up logging level to warning when attempting to configure {beats} with unknown fields from autodiscovered events/environments.
- Elasticsearch output now supports `idle_connection_timeout`. {issue}35616[35615] {pull}36843[36843]
- Upgrade to Go 1.20.12. {pull}37350[37350]
- The Elasticsearch output can now configure performance presets with the `preset` configuration field. {pull}37259[37259]
- Upgrade `elastic-agent-system-metrics` to v0.9.1. See https://github.com/elastic/elastic-agent-system-metrics/releases/tag/v0.9.1. {pull}37353[37353]
- Upgrade to elastic-agent-libs v0.7.3 and golang.org/x/crypto v0.17.0. {pull}37544[37544]

*Auditbeat*

- Add `ignore_errors` option to audit module. {issue}15768[15768] {pull}36851[36851]
- Fix copy arguments for strict aligned architectures. {pull}36976[36976]

*Filebeat*

- Allow http_endpoint input to receive PUT and PATCH requests. {pull}36734[36734]
- Avoid unwanted publication of Azure entity records. {pull}36753[36753]
- Avoid unwanted publication of Okta entity records. {pull}36770[36770]
- Add support for Digest Authentication to CEL input. {issue}35514[35514] {pull}36932[36932]
- Use filestream input with `file_identity.fingerprint` as default for hints autodiscover. {issue}35984[35984] {pull}36950[36950]
- Add network processor in addition to interface based direction resolution. {pull}37023[37023]
- Make CEL input log current transaction ID when request tracing is turned on. {pull}37065[37065]
- Make Azure Blob Storage input GA and update docs accordingly. {pull}37128[37128]
- Add request trace logging to http_endpoint input. {issue}36951[36951] {pull}36957[36957]
- Make GCS input GA and update docs accordingly. {pull}37127[37127]
- Suppress and log max HTTP request retry errors in CEL input. {pull}37160[37160]
- Prevent CEL input from re-entering the eval loop when an evaluation failed. {pull}37161[37161]
- Update CEL extensions library to v1.7.0. {pull}37172[37172]

*Auditbeat*

- Upgrade go-libaudit to v2.4.0. {issue}36776[36776] {pull}36964[36964]
- Add a `/inputs/` route to the HTTP monitoring endpoint that exposes metrics for each dataset instance. {pull}36971[36971]

*Heartbeat*
- Capture and log the individual connection metrics for all the lightweight monitors.

*Metricbeat*

- Add metrics grouping by dimensions and time to Azure app insights. {pull}36634[36634]
- Align on the algorithm used to transform Prometheus histograms into Elasticsearch histograms. {pull}36647[36647]
- Enhance GCP billing with detailed tables identification, additional fields, and optimized data handling. {pull}36902[36902]
- Add a `/inputs/` route to the HTTP monitoring endpoint that exposes metrics for each metricset instance. {pull}36971[36971]
- Add Linux IO metrics to system/process. {pull}37213[37213]
- Add new memory/cgroup metrics to Kibana module. {pull}37232[37232]

*Packetbeat*

- Add metrics for TCP flags. {issue}36992[36992] {pull}36975[36975]

*Winlogbeat*

- Make ingest pipeline routing robust to letter case of channel names for forwarded events. {issue}36670[36670] {pull}36899[36899]
- Document minimum permissions required for local user account. {issue}15773[15773] {pull}37176[37176]

==== Deprecated

*Filebeat*

- Deprecate rsa2elk Filebeat modules. {issue}36125[36125] {pull}36887[36887]


[[release-notes-8.11.4]]
=== Beats version 8.11.4
https://github.com/elastic/beats/compare/v8.11.3\...v8.11.4[View commits]

==== Bugfixes

*Heartbeat*

- Added fix for formatting the logs from stateloader properly. {pull}37369[37369]
- Remove duplicated syscall from ARM seccomp profile. {pull}37440[37440]

*Metricbeat*

- Nest the `region` and `availability_zone` ECS fields within the cloud field. {pull}37015[37015]
- Fix CPU and memory metrics collection from privileged process on Windows. {issue}17314[17314]{pull}37027[37027]
- Add memory hard limit from container metadata and remove usage percentage in AWS Fargate. {pull}37194[37194]
- Ignore parser errors from unsupported metrics types on Prometheus client and continue parsing until EOF is reached. {pull}37383[37383]
- Fix the reference time rounding on Azure Metrics. {issue}37204[37204] {pull}37365[37365]

==== Added

*Packetbeat*

- Bump Windows Npcap version to v1.78. {issue}37300[37300] {pull}37370[37370]


[[release-notes-8.11.3]]
=== Beats version 8.11.3
https://github.com/elastic/beats/compare/v8.11.2\...v8.11.3[View commits]

The 8.11.3 patch release contains a fix for a potential security vulnerability. Please see our link:https://discuss.elastic.co/c/announcements/security-announcements/31[security advisory for more details].

==== Breaking changes

*Affecting all Beats*

- `queue.mem.events` is changing from `4096` to `3200`.
- `queue.mem.flush.min_events` is changing from `2048` to `1600`.
- `queue.mem.flush.timeout` is changing from `1s` to `10s`.
- `output.elasticsearch.bulk_max_size` is changing from `50` to `1600`.
- `output.elasticsearch.idle_connection_timeout` is changing from `60s` to `3s`.


[[release-notes-8.11.2]]
=== Beats version 8.11.2
https://github.com/elastic/beats/compare/v8.11.1\...v8.11.2[View commits]

==== Breaking changes

*Affecting all Beats*

- Avoid logging fields values when handling Elasticsearch output errors except at the debug log level. The debug log level must now be used to see detailed errors, for example mapping errors and their cause. {pull}37229[37229]

==== Bugfixes

*Affecting all Beats*
- Fix memqueue producer blocking indefinitely even after being cancelled. {issue}22813[22813] {pull}37077[37077]

*Auditbeat*

- Fix documentation regarding socket type selection. {issue}37174[37174] {pull}37175[37175]
- Fix guess trigger for system/socket credentials on newer kernels. {issue}36905[36905] {pull}37136[37136]

*Filebeat*

- Do not error when Okta API returns no data. {pull}37092[37092]
- Fix request body close behaviour in HTTP_Endpoint when handling GZIP compressed content. {pull}37091[37091]
- Make CEL input `now` global evaluate to a time in UTC. {pull}37159[37159]

*Heartbeat*

- Fix mapping errors with invalid URLs. {pull}37255[37255]

*Metricbeat*

- Fix memory leak on Windows {issue}37142[37142] {pull}37171[37171]
- Enhance Azure Metrics metricset with refined grouping logic and resolved duplication issues for TSDB compatibility. {pull}36823[36823]
- Fix unintended skip in metric collection on Azure Monitor. {issue}37204[37204] {pull}37203[37203]
- Fix the "api-version query parameter (?api-version=) is required for all requests" error in Azure Billing. {pull}37158[37158]
- add_cloud_metadata: fix the `orchestrator` metadata for the aws cloud provider

*Winlogbeat*

- Fix dashboards under Kibana 8.x. {issue}37080[37080] {pull}37085[37085]

==== Added

*Affecting all Beats*

- Upgrade to Go 1.20.11. {pull}37123[37123]


[[release-notes-8.11.1]]
=== Beats version 8.11.1
https://github.com/elastic/beats/compare/v8.11.0\...v8.11.1[View commits]

IMPORTANT: Due to a memory leak issue, Windows users running {agent} are recommended to avoid upgrading to this release and instead use the 8.11.2 release in which the issue is resolved. If you've already upgraded to version 8.11.0 or 8.11.1, we recommend upgrading to 8.11.2 as soon as possible.

==== Added

*Affecting all Beats*

- Setting environmental variable `ELASTIC_NETINFO:false` will disable the `netinfo.enabled` option of add_host_metadata processor.


[[release-notes-8.11.0]]
=== Beats version 8.11.0
https://github.com/elastic/beats/compare/v8.10.4\...v8.11.0[View commits]


IMPORTANT: Due to a memory leak issue, Windows users running {agent} are recommended to avoid upgrading to this release and instead use the 8.11.2 release in which the issue is resolved. If you've already upgraded to version 8.11.0 or 8.11.1, we recommend upgrading to 8.11.2 as soon as possible.

==== Breaking changes

*Affecting all Beats*

- The Elasticsearch output now enables compression by default. This decreases network data usage by an average of 70-80%, in exchange for 20-25% increased CPU use and ~10% increased ingestion time. The previous default can be restored by setting the flag `compression_level: 0` under `output.elasticsearch`. {pull}36681[36681]
- The `elastic-agent-autodiscover` library is updated to version 0.6.4, disabling metadata for deployment and cronjob. Pods that will be created from deployments or cronjobs will not have the extra metadata field for `kubernetes.deployment` or `kubernetes.cronjob`, respectively. {pull}36879[36879]

*Filebeat*

- Switch types of `log.file.device`, `log.file.inode`, `log.file.idxhi`, `log.file.idxlo` and `log.file.vol` fields to strings to better align with ECS and integrations. {pull}36697[36697]

*Metricbeat*

- The System module now collects the number of threads per process. The elastic-agent-system-metrics was updated to v0.7.0 as this version collects the number of threads.

==== Bugfixes

*Affecting all Beats*

- Upgrade `elastic-agent-libs` to v0.6.0, allowing a Beat running as a Windows service to receive more than one change request. {pull}36896[36896]

*Filebeat*

- Added a fix for the Crowdstrike pipeline handling of process arrays. {pull}36496[36496]
- Fix handling of response errors in HTTPJSON and CEL request trace logging. {pull}36956[36956]

*Heartbeat*

- Fix retries to trigger on a down monitor with no previous state. {pull}36842[36842]
- Bump NodeJS minor version to 18.18.2. {pull}36961[36961]
- Fix monitor duration calculation with retries. {pull}36900[36900]

*Metricbeat*

- Fix Azure Monitor empty metricnamespace. {pull}36295[36295]
- Fix GCP compute metadata. {pull}36338[36338]
- Add missing 'TransactionType' dimension for Azure Storage Account. {pull}36413[36413]
- Add log error when statsd server fails to start. {pull}36477[36477]
- Fix CassandraConnectionClosures metric configuration. {pull}34742[34742]
- Fix event mapping implementation for statsd module. {pull}36925[36925]

*Winlogbeat*

- Fix User Account Control Attributes Table values for Security module. {issue}36999[36999] {pull}37009[37009]

==== Added

*Affecting all Beats*

- Upgrade to Go 1.20.10. {pull}36846[36846]
- Add support for forward lookups (`A`, `AAAA`, and `TXT`) in DNS processor. {issue}11416[11416] {pull}36394[36394]
- Mark `syslog` processor as GA, improve docs about how processor handles syslog messages. {issue}36416[36416] {pull}36417[36417]
- Add support for AWS external IDs. {issue}36321[36321] {pull}36322[36322]
- Disable `netinfo.enabled` option of `add-host-metadata processor`to enhance `host.ip` and `host.mac`. {pull}36506[36506]
- {Beats} will now connect to older {es} instances by default. {pull}36884[36884]
- Upgrade golang/x/net to v0.17.0. Updates the publicsuffix table used by the registered_domain processor. {pull}36969[36969]

*Filebeat*

- Reduce HTTPJSON metrics allocations. {pull}36282[36282]
- Add support for a simplified input configuraton when running under {agent}. {pull}36390[36390]
- Make HTTPJSON response body decoding errors more informative. {pull}36481[36481]
- Allow fine-grained control of entity analytics API requests for Okta provider. {issue}36440[36440] {pull}36492[36492]
- Add support for expanding `journald.process.capabilities` into the human-readable effective capabilities in the ECS `process.thread.capabilities.effective` field. {issue}36454[36454] {pull}36470[36470]
- Allow fine-grained control of entity analytics API requests for AzureAD provider. {issue}36440[36440] {pull}36441[36441]
- For request tracer logging in CEL and httpjson the request and response body are no longer included in `event.original`. The body is still present in `http.{request,response}.body.content`. {pull}36531[36531]
- Add support for Okta OAuth2 provider in the CEL input. {issue}36336[36336] {pull}36521[36521]
- Improve error logging in HTTPJSON input. {pull}36529[36529]
- Disable warning message about ingest pipeline loading when running under Elastic Agent. {pull}36659[36659]
- Add input metrics to http_endpoint input. {issue}36402[36402] {pull}36427[36427]
- Remove Event Normalization from GCP PubSub Input. {pull}36716[36716]
- Add support for new features & removed partial save mechanism in the Azure Blob Storage input. {issue}35126[35126] {pull}36690[36690]
- Improve template evaluation logging for HTTPJSON input. {pull}36668[36668]
- Add CEL partial value debug function. {pull}36652[36652]
- Add support for new features and remove partial save mechanism in the GCS input. {issue}35847[35847] {pull}36713[36713]
- Add cache processor. {pull}36786[36786]

*Packetbeat*

- Bump Windows Npcap version to v1.76. {issue}36539[36539] {pull}36549[36549]


[[release-notes-8.10.4]]
=== Beats version 8.10.4
https://github.com/elastic/beats/compare/v8.10.3\...v8.10.4[View commits]


[[release-notes-8.10.3]]
=== Beats version 8.10.3
https://github.com/elastic/beats/compare/v8.10.2\...v8.10.3[View commits]

==== Bugfixes

*Affecting all Beats*
- syslog processor - Fix the ability to use `when` conditions on the processor. {issue}36762[36762]

*Packetbeat*

- Fix default cache size calculation. {pull}36723[36723]

==== Added

*Filebeat*

- Update mito CEL extension library to v1.6.0. {pull}36651[36651]
- Re-use buffers to optimise memory allocation in fingerprint mode of filestream {pull}36736[36736]

*Packetbeat*

- Improve efficiency of sniffers by deduplicating interface configurations. {issue}36574[36574] {pull}36576[36576]


[[release-notes-8.10.2]]
=== Beats version 8.10.2
https://github.com/elastic/beats/compare/v8.10.1\...v8.10.2[View commits]

==== Bugfixes

*Packetbeat*

- Prevent panic when more than one interface is configured in Fleet. {issue}36574[36574] {pull}36575[36575]

==== Added

*Affecting all Beats*

- Upgrade Go to 1.20.8 {pull}36597[36597]


[[release-notes-8.10.1]]
=== Beats version 8.10.1
https://github.com/elastic/beats/compare/v8.10.0\...v8.10.1[View commits]

==== Bugfixes

*Filebeat*

- Revert error introduced in {pull}35734[35734] when symlinks can't be resolved in filestream. {pull}36557[36557]
- Fix ignoring external input configuration in `take_over: true` mode {issue}36378[36378] {pull}36395[36395]


[[release-notes-8.10.0]]
=== Beats version 8.10.0
https://github.com/elastic/beats/compare/v8.9.2\...v8.10.0[View commits]

==== Bugfixes

*Affecting all Beats*
- Improve StreamBuf append to improve performance when reading long lines from files. {pull}35928[35928]
- Eliminate cloning of event in deepUpdate {pull}35945[35945]
- Fix ndjson parser to store JSON fields correctly under `target` {issue}29395[29395]
- Add default cgroup regex for `add_process_metadata` processor {pull}36484[36484] {issue}32961[32961]
- Fix environment capture by `add_process_metadata` processor. {issue}36469[36469] {pull}36471[36471]
- Fix status reporting to {agent} when output configuration is invalid running under Elastic-Agent {pull}35719[35719]

*Filebeat*

- Fix error message formatting from filestream input. {pull}35658[35658]
- Fixed concurrency and flaky tests issue in Azure Blob Storage input. {issue}35983[35983] {pull}36124[36124]
- Filter out duplicate paths resolved from matching globs. {issue}36253[36253] {pull}36256[36256]
- Remove 'onFilteredOut' and 'onDroppedOnPublish' callback logs {issue}36299[36299] {pull}36399[36399]
- Ensure winlog input retains metric collection when handling recoverable errors. {issue}36479[36479] {pull}36483[36483]

*Metricbeat*

- Fix the gap in fetching forecast API metrics at the end of each month for Azure billing module  {pull}36142[36142]
- Add option in SQL module to execute queries for all databases. {pull}35688[35688]
- Add support for api_key authentication in elasticsearch module  {pull}36274[36274]
- Add remaining dimensions for Azure storage account to make them available for TSDB enablement. {pull}36331[36331]

*Packetbeat*

- Fix panic in HTTP protocol parsing when host header has empty host part. {issue}36497[36497] {issue}36518[36518]

*Winlogbeat*

- Ensure event loggers retains metric collection when handling recoverable errors. {issue}36479[36479] {pull}36483[36483]
- Fix the ability to use filtering features (e.g. `ignore_older`, `event_id`, `provider`, `level`) while reading `.evtx` files. {issue}16826[16826] {pull}36173[36173]

==== Added

*Affecting all Beats*

- When running under Elastic-Agent the status is now reported per Unit instead of the whole Beat {issue}35874[35874] {pull}36183[36183]
- Mark `translate_sid` processor is GA. {issue}36279[36279] {pull}36280[36280]
- Upgrade Go to 1.20.7 {pull}36241[36241]

*Auditbeat*

- Add support for `security.selinux` and `system.posix_acl_access` extended attributes to FIM. {issue}36265[36265] {pull}36310[36310]

*Filebeat*

- Adding filename details from zip to response for httpjson {issue}33952[33952] {pull}34044[34044]
- Allow specifying since when to read journald entries. {pull}35408[35408]
- Under elastic-agent the input metrics will now be included in agent diagnostics dumps. {pull}35798[35798]
- Improve CEL input performance. {pull}35915[35915]
- Added support for min/max template functions in httpjson input. {issue}36094[36094] {pull}36036[36036]
- Add `clean_session` configuration setting for MQTT input.  {pull}35806[35806]
- Add fingerprint mode for the filestream scanner and new file identity based on it {issue}34419[34419] {pull}35734[35734]
- Add file system metadata to events ingested via filestream {issue}35801[35801] {pull}36065[36065]
- Add support for localstack based input integration testing {pull}35727[35727]
- Allow parsing bytes in and bytes out as long integer in CEF processor. {issue}36100[36100] {pull}36108[36108]
- Add support for registered owners and users to AzureAD entity analytics provider. {pull}36092[36092]
- Added support for Okta OAuth2 provider in the httpjson input. {pull}36273[36273]
- Add support of the interval parameter in Salesforce setupaudittrail-rest fileset. {issue}35917[35917] {pull}35938[35938]
- Add device handling to Okta input package for entity analytics. {pull}36049[36049]
- Add setup option `--force-enable-module-filesets`, that will act as if all filesets have been enabled in a module during setup. {issue}30915[30915] {pull}36286[36286]
- [Azure] Add input metrics to the azure-eventhub input. {pull}35739[35739]

*Metricbeat*

- Add support for multiple regions in GCP {pull}32964[32964]
- Add kubernetes.deployment.status.* fields for Kubernetes module {pull}35999[35999]

*Packetbeat*

- Under elastic-agent the input metrics will now be included in agent diagnostics dumps. {pull}35798[35798]
- Add support for multiple regions in GCP {pull}32964[32964]

*Winlogbeat*

- Under elastic-agent the input metrics will now be included in agent diagnostics dumps. {pull}35798[35798]

==== Deprecated

*Heartbeat*

- Deprecate aws_elb autodiscover provider. {pull}36191[36191]


[[release-notes-8.9.2]]
=== Beats version 8.9.2
https://github.com/elastic/beats/compare/v8.9.1\...v8.9.2[View commits]

==== Bugfixes

*Filebeat*

- Fix panic when redact option is not provided to CEL input. {issue}36387[36387] {pull}36388[36388]
- Update mito CEL extension library to v1.5.0. {pull}36146[36146]

==== Added

*Metricbeat*

- Add Azure resource tags support to Azure Billing module {pull}36428[36428]


[[release-notes-8.9.1]]
=== Beats version 8.9.1
https://github.com/elastic/beats/compare/v8.9.0\...v8.9.1[View commits]

==== Bugfixes

*Auditbeat*

- auditd: Expanded the bitmask applied to ECS file.mode so that the SUID, SGID, and sticky bits can be represented. {pull}36294[36294]

*Filebeat*

- Fix panic when SQS input metrics getter is invoked. {pull}36101[36101] {issue}36077[36077]
- Make CEL input's `now` global variable static for evaluation lifetime. {pull}36107[36107]
- Fix handling of TCP/UDP address resolution during metric initialization. {issue}35064[35064] {pull}36287[36287]
- Fix handling of Juniper SRX structured data when there is no leading Junos element. {issue}36270[36270] {pull}36308[36308]
- Remove erroneous error log in GCPPubSub input. {pull}36296[36296]
- Fix Filebeat Cisco module with missing escape character. {issue}36325[36325] {pull}36326[36326]

*Heartbeat*

- Enable heartbeat-wide publish timeout setting with run_once. {pull}35721[35721]
- Added default timezone UTC to heartbeat docker images to fix synthetics journeys navigation errors. {pull}36193[36193]

*Packetbeat*

- Fix handling of Npcap installation options from Fleet. {pull}35541[35541] {pull}35935[35935]

*Winlogbeat*

- Fix powershell details regexp to prevent excessive backtracking when processing command invocations. {pull}36178[36178]

==== Added

*Affecting all Beats*

- Upgrade Go to 1.19.12 {pull}36246[36246]
- Add warning message to SysV init scripts for RPM-based systems that lack `/etc/rc.d/init.d/functions`. {issue}35708[35708] {pull}36188[36188]

*Filebeat*

- Add support for endpoint resolver in AWS config {pull}36208[36208]


[[release-notes-8.9.0]]
=== Beats version 8.9.0
https://github.com/elastic/beats/compare/v8.8.2\...v8.9.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Do not print context cancelled error message when running under agent {pull}36006[36006]
- Fix recovering from invalid output configuration when running under Elastic-Agent {pull}36016[36016]

*Filebeat*

- Add support in s3 input for JSON with array of objects. {pull}35475[35475]
- RFC5424 syslog timestamps with offset 'Z' will be treated as UTC rather than using the default timezone. {pull}35360[35360]
- Fixed a minor code error in the GCS input scheduler where a config value was being used directly instead of the source struct. {pull}35729[35729]
- Fix CEL input JSON marshalling of nested objects. {issue}35763[35763] {pull}35774[35774]
- Fix metric collection in GCPPubSub input. {pull}35773[35773]
- Fix end point deregistration in http_endpoint input. {issue}35899[35899] {pull}35903[35903]
- Fix duplicate ID panic in filestream metrics. {issue}35964[35964] {pull}35972[35972]
- Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. {pull}35996[35996]
- Fix handling of NUL-terminated log lines in Fortinet Firewall module. {issue}36026[36026] {pull}36027[36027]
- Make redact field configuration recommended in CEL input and log warning if missing. {pull}36008[36008]
- Fix handling of region name configuration in awss3 input {pull}36034[36034]

*Heartbeat*

- Update gval version. {pull}35636[35636]
- Filter dev flags for ui monitors inside synthetics_args. {pull}35788[35788]
- Fix temp dir running out of space with project monitors. {issue}35843[35843]

*Metricbeat*

- Fix no error logs displayed in CloudWatch EC2, RDS and SQS metadata {issue}34985[34985] {pull}35035[35035]
- Remove Beta warning from IIS application_pool metricset {pull}35480[35480]
- Improve documentation for ActiveMQ module {issue}35113[35113] {pull}35558[35558]
- Resolve statsd module's prematurely halting of metrics parsing upon encountering an invalid packet. {pull}35075[35075]

*Packetbeat*

- Fix double channel close panic when reloading. {pull}35324[35324]

*Winlogbeat*

- Prevent panic on closing iterators on empty channels in experimental API. {issue}33966[33966] {pull}35423[35423]

==== Added

*Affecting all Beats*

- Add Hetzner Cloud as a provider for `add_cloud_metadata`. {pull}35456[35456]
- Upgrade version of elastic-agent-autodiscover to v0.6.1 for improved memory consumption on k8s. {pull}35483[35483]
- Added `orchestrator.cluster.id` and `orchestrator.cluster.name` fields to the add_cloud_metadata processor, AWS cloud provider. {pull}35182[35182]
- Lowercase reported hostnames per Elastic Common Schema (ECS) guidelines for the host.name field. Upgraded github.com/elastic/go-sysinfo to 1.11.0. {pull}35652[35652]

*Filebeat*

- Added support for decoding apache parquet files in awss3 input. {issue}34662[34662] {pull}35578[35578]
- Add support for CRC validation in Filebeat's HTTP endpoint input. {pull}35204[35204]
- Add support for CRC validation in Zoom module. {pull}35604[35604]
- Add execution budget to CEL input. {pull}35409[35409]
- Add XML decoding support to HTTPJSON. {issue}34438[34438] {pull}35235[35235]
- Add delegated account support when using Google ADC in `httpjson` input. {pull}35507[35507]
- Add metrics for filestream input. {pull}35529[35529]
- Add support for collecting `httpjson` metrics. {pull}35392[35392]
- Add XML decoding support to CEL. {issue}34438[34438] {pull}35372[35372]
- Mark CEL input as GA. {pull}35559[35559]
- Add metrics for gcp-pubsub input. {pull}35614[35614]
- Allow non-AWS endpoints for awss3 input. {issue}35496[35496] {pull}35520[35520]
- Add Okta input package for entity analytics. {pull}35611[35611]
- Expose harvester metrics from filestream input {pull}35835[35835] {issue}33771[33771]
- Add device support for Azure AD entity analytics. {pull}35807[35807]

*Libbeat*
- Added support for Apache Parquet file reader. {issue}34662[34662] {pull}35183[35183]

*Metricbeat*

- Add GCP Redis metadata {pull}33701[33701]
- Migrate Azure Billing, Monitor, and Storage metricsets to the newer SDK. {pull}33585[33585]
- Add support for float64 values parsing for statsd metrics of counter type. {pull}35099[35099]

*Packetbeat*

- Added `packetbeat.interfaces.fanout_group` to allow a Packetbeat sniffer to join an AF_PACKET fanout group. {issue}35451[35451] {pull}35453[35453]

*Winlogbeat*

- Set `host.os.type` and `host.os.family` to "windows" if not already set. {pull}35435[35435]
- Handle empty DNS answer data in QueryResults for the Sysmon Pipeline {pull}35207[35207]

[[release-notes-8.8.2]]
=== Beats version 8.8.2
https://github.com/elastic/beats/compare/v8.8.1\...v8.8.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Make sure k8s watchers are closed when closing k8s meta processor. {pull}35630[35630]
- Upgraded Apache Arrow library used in `x-pack/libbeat/reader/parquet` from v11 to v12.0.1 in order to fix cross-compilation issues. {pull}35640[35640]
- Fix panic when the disk queue's MaxRetryInterval configuration is specified, but RetryInterval is not. {pull}35820[35820]

*Filebeat*

- Fix syslog message parsing for fortinet.firewall to take into account quoted values. {pull}35522[35522]
- [Filebeat GCS input] Fixed an issue where `bucket_timeout` was being applied to the entire bucket poll interval and not individual bucket object read operations. Fixed a map write concurrency issue arising from data races when using a high number of workers. Fixed the flaky tests that were present in the GCS test suit. {pull}35605[35605]
- Fix handling of IPv6 unspecified addresses in TCP input. {issue}35064[35064] {pull}35637[35637]
- Improve error reporting and fix IPv6 handling of TCP and UDP metric collection. {pull}35772[35772]
- Fix input reload on autodiscover. {issue}34388[34388] {pull}35645[35645]

*Heartbeat*

- Fix serialization of processors when running diagnostics. {pull}35698[35698]

*Metricbeat*

- Fix calculation of the `host.cpu.usage` metric for EC2. {pull}35717[35717]

==== Added

*Affecting all Beats*

- Update Go version to 1.19.10. {pull}35751[35751]

*Filebeat*

- [GCS] Added scheduler debug logs and improved the context passing mechanism by removing them from struct params and passing them as function arguments. {pull}35674[35674]

*Packetbeat*

- Add AF_PACKET metrics. {issue}35428[35428] {pull}35489[35489]

==== Deprecated

*Heartbeat*

- Removed support for `zip_url` and `local` browser sources. {pull}35429[35429]

[[release-notes-8.8.1]]
=== Beats version 8.8.1
https://github.com/elastic/beats/compare/v8.8.0\...v8.8.1[View commits]

==== Bugfixes

*Affecting all Beats*

- 'add_cloud_metadata' processor: add `cloud.region` field for GCE cloud provider
- 'add_cloud_metadata' processor: update Azure metadata API version to get missing `cloud.account.id` field

*Filebeat*

- Fix "Can only start an input when all related states are finished" error when running under Elastic Agent {pull}35250[35250] {issue}33653[33653]
- [system] Sync system/auth dataset with system integration 1.29.0. {pull}35581[35581]
- Fix filestream false positive log error "filestream input with ID 'xyz' already exists" {issue}31767[31767]
- Fix error when trying to use `include_message` parser {issue}35440[35440]

==== Added

*Filebeat*

- Add sanitization capabilities to azure-eventhub input {pull}34874[34874]

*Auditbeat*
- Migration of system/package module storage from gob encoding to flatbuffer encoding in bolt db. {pull}34817[34817]

*Metricbeat*

- Support collecting metrics from both the monitoring account and linked accounts from AWS CloudWatch. {pull}35540[35540]
- Add new parameter `include_linked_accounts` to enable/disable metrics collection from multiple linked AWS Accounts {pull}35648[35648]


[[release-notes-8.8.0]]
=== Beats version 8.8.0
https://github.com/elastic/beats/compare/v8.7.1...v8.8.0[View commits]


==== Bugfixes

*Affecting all Beats*
- Fix race condition when stopping runners {pull}32433[32433]
- Fix concurrent map writes when system/process code called from reporter code {pull}32491[32491]
- The Elasticsearch output now splits large requests instead of dropping them when it receives a StatusRequestEntityTooLarge error. {pull}34911[34911]
- In cases where the matcher detects a non-string type in a match statement, report the error as a debug statement, and not a warning statement. {pull}35119[35119]
- `add_cloud_metadata` processor: Add `cloud.region` field for GCE cloud provider.
- `add_cloud_metadata` processor: Update Azure metadata API version to get missing `cloud.account.id` field.

*Filebeat*
- [GCS Input] Added missing locks for safe concurrency. {pull}34914[34914]
- Fix the `ignore_inactive` option being ignored in Filebeat's filestream input. {pull}34770[34770]
- Add input instance ID to request trace filename for httpjson and cel inputs. {pull}35024[35024]
- Sanitize filenames for request tracer in httpjson input. {pull}35143[35143]
- Sanitize filenames for request tracer in cel input. {pull}35154[35154]
- Fix the grok expression outputs of log files. {pull}35221[35221]
- Move repeated Windows event channel not found errors in winlog input to debug level.  {issue}35314[35314] {pull}35317[35317]
- Fix crash when processing forwarded logs missing a message. {issue}34705[34705] {pull}34865[34865]
- Fix crash when loading azurewebstorage cursor with no partially processed data. {pull}35433[35433]

*Heartbeat*

- Fix panics when parsing when HTTP URL is not parseable. {pull}34702[34702]
- Fix broken state ID location naming. {pull}35336[35336]
- Fix project monitor temp directories permission to include group access. {pull}35398[35398]
- Fix output pipeline exit on `run_once`. {pull}35376[35376]
- Fix formatting issue with socket trace timeout. {pull}35434[35434]

*Metricbeat*

- Make generic SQL GA. {pull}34637[34637]
- Collect missing `remote_cluster` in Elasticsearch CCR metricset. {pull}34957[34957]
- Add context with timeout in AWS API calls. {pull}35425[35425]

*Osquerybeat*

- Adds the `elastic_file_analysis` table to the Osquery extension for macOS builds. {pull}35056[35056]

*Packetbeat*

- Fix BPF filter setting not being applied to sniffers. {issue}35363[35363] {pull}35484[35484]

*Winlogbeat*

- Move repeated channel not found errors to debug level.  {issue}35314[35314] {pull}35317[35317]
- Fix panic due to misrepresented buffer use. {pull}35437[35437]
- Allow program termination when attempting to open an absent channel. {pull}35474[35474]

==== Added

*Filebeat*

- Add metric `sqs_messages_waiting_gauge` for aws-s3 input. {pull}34488[34488]
- Add support for Okta debug attributes, `risk_reasons`, `risk_behaviors` and `factor`. {issue}33677[33677] {pull}34508[34508]
- Add `nginx.ingress_controller.upstream.ip` to `related.ip` {issue}34645[34645] {pull}34672[34672]
- Include NAT and firewall IPs in `related.ip` in Fortinet Firewall module. {issue}34640[34640] {pull}34673[34673]
- Add UNIX socket log parsing for NGINX `ingress_controller`. {pull}34732[34732]
- Add metric `sqs_worker_utilization` for aws-s3 input. {pull}34793[34793]
- Register MIME handlers for CSV types in CEL input. {pull}34934[34934]
- Add MySQL authentication message parsing and `related.ip` and `related.user` fields. {pull}34810[34810]
- Mention `mito` CEL tool in CEL input docs. {pull}34959[34959]
- Add nginx ingress_controller parsing if one of upstreams fails to return response. {pull}34787[34787]
- Allow neflow v9 and ipfix templates to be shared between source addresses. {pull}35036[35036]
- Add support for collecting IPv6 metrics. {pull}35123[35123]
- Add Oracle authentication messages parsing {pull}35127[35127]

*Heartbeat*
- Add status to monitor run log report.
- Remov Beta label for browser monitors. {pull}35424[35424].

*Metricbeat*

- Add GCP Carbon Footprint metricbeat data. {pull}34820[34820]
- Add event loop utilization metric to Kibana module. {pull}35020[35020]

*Winlogbeat*

- Add `event.category` and `event.type` to Sysmon module for EventIDs 8, 9, 19, 20, 27, 28, 255. {pull}35193[35193]

[[release-notes-8.7.1]]
=== Beats version 8.7.1
https://github.com/elastic/beats/compare/v8.7.0\...v8.7.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix Beats started by agent not respecting the allow_older_versions: true configuration flag {issue}34227[34227] {pull}34964[34964]
- Beats started by agent now configure one global instance of each default add_x_metadata processor instead of creating one processor per input. Fixes a significant performance degradation for the aws-s3 input. {issue}35000[35000] {pull}35031[35031]

*Filebeat*

- [GCS Input] Added missing locks for safe concurrency {pull}34914[34914]
- Add input instance id to request trace filename for httpjson and cel inputs {pull}35037[35037]
- Fix panic in TCP and UDP inputs on Linux when collecting socket metrics from OS. {issue}35064[35064]
- Correctly collect TCP and UDP metrics for unspecified address values. {pull}35111[35111]
- Fix base for UDP and TCP queue metrics and UDP drops metric. {pull}35123[35123]
- Sanitize filenames for request tracer in httpjson and cel inputs. {pull}35143[35143]
- Fix handling of MySQL audit logs with strict JSON parser. {issue}35158[35158] {pull}35160[35160]
- decode_cef processor: Fix ECS output by making `observer.ip` into an array of strings instead of string. {issue}35140[35140] {pull}35149[35149]
- Fix accidental error overwrite in defer statement in entityanalytics Azure AD input. {issue}35153[35153] {pull}35169[35169]

*Heartbeat*

- Fix issue using projects in airgapped environments by disabling npm audit. {pull}34936[34936]

*Packetbeat*

- Fix documentation for `flows.period` related to flow reporting. {pull}35009[35009]

==== Added

*Affecting all Beats*

- Allow users to enable features via configuration, starting with the FQDN reporting feature. {issue}1070[1070] {pull}34456[34456]

*Winlogbeat*

- Add note in documentation about 21 event ID clause limit {issue}35048[35048] {pull}35049[35049]

[[release-notes-8.7.0]]
=== Beats version 8.7.0
https://github.com/elastic/beats/compare/v8.6.2\...v8.7.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix dropped events when monitor a beat under the agent and send its `Host info` log entry. {pull}34599[34599]
- Fix panics when a processor is closed twice {pull}34647[34647]
- Update elastic-agent-system-metrics to v0.4.6 to allow builds on mips platforms. {pull}34674[34674]

*Filebeat*
- [Auditbeat System Package] Added support for Apple Silicon chips. {pull}34433[34433]
with the ecs field name `container`. {pull}34403[34403]
automatic splitting at root level, if root level element is an array. {pull}34155[34155]
- Prevent Elasticsearch from spewing log warnings about redundant wildcard when setting up ingest pipelines. {issue}34249[34249] {pull}34550[34550]
- Gracefully handle Windows event channel not found errors in winlog input. {issue}30201[30201] {pull}34605[34605]
- Fix the issue of `cometd` input worker getting closed in case of a network connection issue and an EOF error. {issue}34326[34326] {pull}34327[34327]
- Fix for httpjson first_response object throwing false positive errors by making it a flag based object {issue}34747[34747] {pull}34748[34748]
- Fix errors and panics due to re-used processors {pull}34761[34761]
- Add missing Basic Authentication support to CEL input {issue}34609[34609] {pull}34689[34689]

*Heartbeat*

- Fix integration hashing to prevent reloading all when updated. {pull}34697[34697]
- Fix release of job limit semaphore when context is cancelled. {pull}34697[34697]
with the ecs field name `container`. {pull}34403[34403]
automatic splitting at root level, if root level element is an array. {pull}34155[34155]
- Fix broken mapping for state.ends field. {pull}34891[34891]

*Filebeat*

- Allow the `misp` fileset in the Filebeat `threatintel` module to ignore CIDR ranges for an IP field. {issue}29949[29949] {pull}34195[34195]
- Remove incorrect reference to CEL ext extensions package. {issue}34610[34610] {pull}34620[34620]
- Fix handling of RFC5988 links' relation parameters by `getRFC5988Link` in HTTPJSON. {issue}34603[34603] {pull}34622[34622]
- Drop empty API response events for Microsoft module. {issue}34786[34786] {pull}34893[34893]

*Metricbeat*

- Fix kafka dashboard field names {pull}33555[33555]

*Winlogbeat*

- Fix handling of event data with keys containing dots. {issue}34345[34345] {pull}34549[34549]
- Gracefully handle channel not found errors. {issue}30201[30201] {pull}34605[34605]
- Clarify query term limits warning and remove link to missing Microsoft doc page. {pull}34715[34715]
- Improve documentation for event_logs.name configuration. {pull}34931[34931]

*Functionbeat*

- Fix Kinesis events timestamp to use timestamp of the event record instead of when the record was processed {pull}33593[33593]

==== Added

*Filebeat*

- Add backup to bucket and delete functionality for the `aws-s3` input. {issue}30696[30696] {pull}33559[33559]
- Add support for polling system UDP stats for UDP input metrics. {pull}34070[34070]
- Add support for recognizing the log level in Elasticsearch JVM logs {pull}34159[34159]
- Add new Entity Analytics input with Azure Active Directory support. {pull}34305[34305]
- Added metric `sqs_lag_time` for aws-s3 input. {pull}34306[34306]
- Add metrics for TCP packet processing. {pull}34333[34333]
- Add metrics for unix socket packet processing. {pull}34335[34335]
- Add beta `take over` mode for `filestream` for simple migration from `log` inputs {pull}34292[34292]
- Add pagination support for Salesforce module. {issue}34057[34057] {pull}34065[34065]
- Allow users to redact sensitive data from CEL input debug logs. {pull}34302[34302]
- Add support for new Rabbitmq timestamp format for logs {pull}34211[34211]
- Allow user configuration of timezone offset in Cisco ASA and FTD modules. {pull}34436[34436]
- Allow user configuration of timezone offset in Checkpoint module. {pull}34472[34472]
- Fill okta.request.ip_chain.* as a flattened object in Okta module. {pull}34621[34621]
- Fixed GCS log format issues. {pull}34659[34659]
- Add Basic Authentication support on constructed requests to CEL input {issue}34609[34609] {pull}34689[34689]
- Add string manipulation extensions to CEL input {issue}34610[34610] {pull}34689[34689]
- Improve CEL input documentation {pull}34831[34831]
- Add metrics documentation for CEL and AWS CloudWatch inputs. {issue}34887[34887] {pull}34889[34889]
- Metrics hosted by the HTTP monitoring endpoint for the `aws-cloudwatch`, `aws-s3`, `cel`, and `lumberjack` inputs are now available under `/inputs/` instead of `/dataset`.

*Heartbeat*

- Users can now configure max scheduler job limits per monitor type via env var. {pull}34307[34307]

- Remove host and port matching restrictions on hint-generated monitors. {pull}34376[34376]

*Metricbeat*

- Remove GCP Compute metadata cache {pull}33655[33655]
- Add GCP Redis regions support {pull}33728[33728]
- Changed cloudwatch module to call ListMetrics API only once per region, instead of per AWS namespace {pull}34055[34055]
- Add beta ingest_pipeline metricset to Elasticsearch module for ingest pipeline monitoring {pull}34012[34012]
- Handle duplicated TYPE line for prometheus metrics {issue}18813[18813] {pull}33865[33865]

*Packetbeat*

- Reduce logging level for ENOENT to WARN when mapping sockets to processes. {issue}33793[33793] {pull}33854[33854]
- Add metrics for TCP and UDP packet processing. {pull}33833[33833] {pull}34353[34353]
- Allow user to prevent Npcap library installation on Windows. {issue}34420[34420] {pull}34428[34428]
- Add metrics documentation for TCP and UDP protocols. {issue}34887[34887] {pull}34889[34889]

*Winlogbeat*

- Add metrics for log event processing. {pull}33922[33922]
- Add metrics documentation for event processing. {issue}34887[34887] {pull}34889[34889]
- Added processing for Windows Event ID's 4797, 5379, 5380, 5381, and 5382 for the Security Ingest Pipeline {issue}34293[34293] {pull}34294[34294]
- Added processing for Windows Event ID's 5140 and 5145 for the Security Ingest Pipeline {pull}34352[34352]

[[release-notes-8.6.2]]
=== Beats version 8.6.2
https://github.com/elastic/beats/compare/v8.6.1\...v8.6.2[View commits]


==== Bugfixes

*Affecting all Beats*

- Fix lockfile logic, retry locking {pull}34194[34194]
- Log errors from the Elastic Agent V2 client errors channel. Avoids blocking when error occurs communicating with the Elastic Agent. {pull}34392[34392]
- Only log publish event messages in trace log level under elastic-agent. {pull}34391[34391]
- Fix issue where updating a single Elastic Agent configuration unit results in other units being turned off. {pull}34504[34504]

*Auditbeat*

*Filebeat*
- [Azure blob storage] Changed logger field name from `container` to `container_name` so that it does not clash
- [GCS] Added support for more mime types & introduced offset tracking via cursor state. Also added support for
- [httpsjon] Improved error handling during pagination with chaining & split processor {pull}34127[34127]
- Fix EOF on single line not producing any event. {issue}30436[30436] {pull}33568[33568]
- Fix handling of error in states in direct aws-s3 listing input {issue}33513[33513] {pull}33722[33722]
- Fix `httpjson` input page number initialization and documentation. {pull}33400[33400]
- Fix reporting of `filebeat.events.active` in log events such that the current value is always reported instead of the difference from the last value. {pull}33597[33597]
- Fix splitting array of strings/arrays in httpjson input {issue}30345[30345] {pull}33609[33609]
- Fix Google workspace pagination and document ID generation. {pull}33666[33666]
- Update mito CEL extension library to v0.0.0-20221207004749-2f0f2875e464 {pull}33974[33974]
- [azure-eventhub input] Switch the run EPH run mode to non-blocking {pull}34075[34075]
- Fixing system tests not returning expected content encoding for azure blob storage input. {pull}34412[34412]
- [Azure Logs] Fix authentication_processing_details parsing in sign-in logs. {issue}34330[34330] {pull}34478[34478]

*Heartbeat*
- Fix broken zip URL monitors. NOTE: Zip URL Monitors will be removed in version 8.7 and replaced with project monitors. {pull}33723[33723]
- Fix bug where states.duration_ms was incorrect type. {pull}33563[33563]
- Fix handling of long UDP messages in UDP input. {issue}33836[33836] {pull}33837[33837]
- Fix beat capabilities on Docker image. {pull}33584[33584]
- Fix serialization of state duration to avoid scientific notation. {pull}34280[34280]
- Enable nodejs engine strict validation when bundling synthetics. {pull}34470[34470]

*Metricbeat*

- Fix metrics split through different events and metadata not matching for aws cloudwatch. {pull}34483[34483]
- Fix metadata enricher with correct container ids for pods with multiple containers in container metricset. Align `kubernetes.container.id` and `container.id` fields for state_container metricset. {pull}34516[34516]

*Packetbeat*

- Fixed a race condition in Packetbeat that could cause crashes or instability {pull}34514[34514]


==== Added


*Filebeat*

- Added support for HTTP destination override to Google Cloud Storage input. {pull}34413[34413]


[[release-notes-8.6.1]]
=== Beats version 8.6.1
https://github.com/elastic/beats/compare/v8.6.0\...v8.6.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix unnecessarily restarting every time a new configuration was received from the Elastic Agent. {pull}34346[34346]

*Filebeat*
- [google_workspace] Fix pagination and cursor value update. {pull}34274[34274]
- Fix handling of quoted values in auditd module. {issue}22587[22587] {pull}34069[34069]
- The `close.on_state_change.inactive` default value is now set to 5 minutes, matching the documentation.
- Fix handling of null or empty arrays during split with keep parent option. {pull}34322[34322]

*Metricbeat*

- Support Oracle-specific connection strings in SQL module {issue}32089[32089] {pull}32293[32293]
- Remove deprecated metrics from controller manager, scheduler and proxy {pull}34161[34161]

*Osquerybeat*

- Fix data_stream configuration, enforce the default values used before 8.6.0. {pull}34246[34246]

*Winlogbeat*

- Corrects issue with security events with source IP of "LOCAL" or "Unknown" failing to ingest {issue}19627[19627] {pull}34295[34295]

==== Added

*Filebeat*

- Allow user configuration of keep-alive behaviour for HTTPJSON and CEL inputs. {issue}33951[33951] {pull}34014[34014]
- [GCS] Added support for more mime types & introduced offset tracking via cursor state. Also added support for automatic splitting at root level, if root level element is an array. {pull}34155[34155]

[[release-notes-8.6.0]]
=== Beats version 8.6.0
https://github.com/elastic/beats/compare/v8.5.3\...v8.6.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix Windows service install/uninstall when Win32_Service returns error, add logic to wait until the Windows Service is stopped before proceeding. {pull}33322[33322]
- Support for multiline zookeeper logs. {issue}2496[2496]
- Allow `clock_nanosleep` in the default seccomp profiles for amd64 and 386. Newer versions of glibc (e.g. 2.31) require it. {issue}33792[33792]
- Disable lockfile when running under elastic-agent. {pull}33988[33988]

*Filebeat*

- [httpsjon] Improved error handling during pagination with chaining & split processor. {pull}34127[34127]
- [Azure blob storage] Added support for more mime types & introduced offset tracking via cursor state. {pull}33981[33981]
- Fix handling of error in states in direct aws-s3 listing input. {issue}33513[33513] {pull}33722[33722]
- Fix PANW handling of messages with event.original already set. {issue}33829[33829] {pull}33830[33830]
- Rename identity as identity_name when the value is a string in Azure Platform Logs. {pull}33654[33654]
- Fix 'requires pointer' error while getting cursor metadata. {pull}33956[33956]
- Fix input cancellation handling when HTTP client does not support contexts. {issue}33962[33962] {pull}33968[33968]
- Update mito CEL extension library to v0.0.0-20221207004749-2f0f2875e464. {pull}33974[33974]
- Fix CEL result deserialisation when evaluation fails. {issue}33992[33992] {pull}33996[33996]
- Fix handling of non-200/non-429 status codes. {issue}33999[33999] {pull}34002[34002]
- [azure-eventhub input] Switch the run EPH run mode to non-blocking. {pull}34075[34075]

*Heartbeat*
- Fix browser monitor summary reporting as up when monitor is down. {issue}33374[33374] {pull}33819[33819]

*Packetbeat*

- Fix panic on memcache transaction with no request or response. {issue}33852[33852] {pull}33853[33853]
- Fix termination logic. {pull}33979[33979]

==== Added

*Affecting all Beats*

- Add `http.pprof` config options for enabling block and mutex profiling. {issue}33572[33572] {pull}33576[33576]
- Add `add_formatted_index` processor that allows the resulting index for an event to be changed based on content from the event. {pull}33800[33800]
- deps: Updated to github.com/elastic/go-sysinfo v1.9.0. {pull}33864[33864]
- Fix panic due to close of already closed channel during shutdown. {pull}33971[33971]

*Auditbeat*

- Add file parser processor to file_integrity module. {pull}28802[28802]
- Improve documentation for symlink handling behaviour in file integrity module. {pull}33430[33430]
- Ensure file integrity module watch paths are absolute. {pull}33430[33430]

*Filebeat*

- Add `text/csv` decoder to `httpjson` input. {pull}28564[28564]
- Update `aws-s3` input to connect to non AWS S3 buckets. {issue}28222[28222] {pull}28234[28234]
- Add support for '/var/log/pods/' path for add_kubernetes_metadata processor with `resource_type: pod`. {pull}28868[28868]
- Add documentation for add_kubernetes_metadata processors `log_path` matcher. {pull}28868[28868]
- Add support for parsers on journald input. {pull}29070[29070]
- Add support in httpjson input for oAuth2ProviderDefault of password grant_type. {pull}29087[29087]
- threatintel module: Add new Recorded Future integration. {pull}30030[30030]
- Allow iptables module to parse ulogd v2 TOS field in logs. {pull}32126[32126]
- Fix handling of invalid UserIP and LocalIP values. {pull}32896[32896]
- Allow http_endpoint instances to share ports. {issue}32578[32578] {pull}33377[33377]
- Improve httpjson documentation for split processor. {pull}33473[33473]
- Added separation of transform context object inside httpjson. Introduced new clause `.parent_last_response.*`. {pull}33499[33499]
- Cloud Foundry input uses server-side filtering when retrieving logs. {pull}33456[33456]
- Add `parse_aws_vpc_flow_log` processor. {pull}33656[33656]
- Update `aws.vpcflow` dataset in AWS module have a configurable log `format` and to produce ECS 8.x fields. {pull}33699[33699]
- Modified `aws-s3` input to reduce mutex contention when multiple SQS message are being processed concurrently. {pull}33658[33658]
- Disable "event normalization" processing for the aws-s3 input to reduce allocations. {pull}33673[33673]
- Add Common Expression Language input. {pull}31233[31233]
- Add support for http+unix and http+npipe schemes in httpjson input. {issue}33571[33571] {pull}33610[33610]
- Add support for http+unix and http+npipe schemes in cel input. {issue}33571[33571] {pull}33712[33712]
- Add `decode_duration`, `move_fields` processors. {pull}31301[31301]
- Add metrics for UDP packet processing. {pull}33870[33870]
- Convert UDP input to v2 input. {pull}33930[33930]
- Improve collection of risk information from Okta debug data. {issue}33677[33677] {pull}34030[34030]
- Adding filename details from zip to response for httpjson. {issue}33952[33952] {pull}34044[34044]

*Heartbeat*

- Upgrade node to 18.12.0.

*Metricbeat*

- Add Data Granularity option to AWS module to allow for for fewer API calls of longer periods and keep small intervals. {issue}33133[33133] {pull}33166[33166]
- Update README file on how to run Metricbeat on Kubernetes. {pull}33308[33308]
- Add per-thread metrics to system_summary. {pull}33614[33614]
- Add GCP CloudSQL metadata. {pull}33066[33066]
- Add support for multiple regions in GCP. {pull}32964[32964]
- Add namespace metadata to all namespaced kubernetes resources. {pull}33763[33763]

*Packetbeat*

- Add fragmented IPv4 packet reassembly. {issue}33012[33012] {pull}33296[33296]

[[release-notes-8.5.3]]
=== Beats version 8.5.3
https://github.com/elastic/beats/compare/v8.5.2\...v8.5.3[View commits]

==== Breaking changes

*Filebeat*

- Fixed error spam from `add_kubernetes_metadata` processor when running on AKS. {pull}33697[33697]

==== Bugfixes

*Heartbeat*

- Fix broken zip URL monitors. NOTE: Zip URL Monitors will be removed in version 8.7 and replaced with project monitors. {pull}33723[33723]
- Fix handling of long UDP messages in UDP input. {issue}33836[33836] {pull}33837[33837]


[[release-notes-8.5.2]]
=== Beats version 8.5.2
https://github.com/elastic/beats/compare/v8.5.1\...v8.5.2[View commits]

[[release-notes-8.5.1]]
=== Beats version 8.5.1
https://github.com/elastic/beats/compare/v8.5.0\...v8.5.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Re-enable build optimizations to reduce binary size and improve performance. {pull}33620[33620]
- Expand fields in `decode_json_fields` if target is set. {issue}31712[31712] {pull}32010[32010]
- Fix in AWS related services initialisation relying on custom endpoint resolver. {issue}32888[32888] {pull}32921[32921]
- Keep `orchestrator.cluster.name` if `kubeconfig` is not returned in GKE metadata. {pull}33418[33418]

*Filebeat*

- Fix `httpjson` input page number initialization and documentation. {pull}33400[33400]
- Add handling of AAA operations for Cisco ASA module. {issue}32257[32257] {pull}32789[32789]
- Fix gc.log always shipped even if gc fileset is disabled. {issue}30995[30995]
- Fix handling of empty array in httpjson input. {pull}32001[32001]
- Fix reporting of `filebeat.events.active` in log events such that the current value is always reported instead of the difference from the last value. {pull}33597[33597]
- Fix splitting array of strings/arrays in httpjson input. {issue}30345[30345] {pull}33609[33609]
- Fix Google workspace pagination and document ID generation. {pull}33666[33666]

*Heartbeat*
- Fix bug affecting let's encrypt and other users of cross-signed certs, where cert expiration was incorrectly calculated. {issue}33215[33215]
- Fix broken disable feature for kibana configured monitors. {pull}33293[33293]
- Fix states client support for output options. {pull}33405[33405]
- Fix states client reloader under managed mode. {pull}33405[33405]
- Fix bug where states.duration_ms was incorrect type. {pull}33563[33563]

==== Bugfixes

*Heartbeat*
- Fix bug affecting let's encrypt and other users of cross-signed certs, where cert expiration was incorrectly calculated. {issue}33215[33215]
- Fix broken disable feature for kibana configured monitors. {pull}33293[33293]
- Fix states client support for output options. {pull}33405[33405]
- Fix states client reloader under managed mode. {pull}33405[33405]

*Metricbeat*

- Add tags to events based on parsed identifier. {pull}33472[33472]
- Skip over unsupported filesystems in the system.filesystem metricset instead of failing immediately. Fix debug statement in system.fsstat metricset. {pull}33646[33646]

==== Added

*Affecting all Beats*

- Beats will now attempt to recover if a lockfile has not been removed. {pull}[33169]

[[release-notes-8.5.0]]
=== Beats version 8.5.0
https://github.com/elastic/beats/compare/v8.4.3\...v8.5.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Upgrade to Go 1.18. Certificates signed with SHA-1 are now rejected. See the Go 1.18 https://tip.golang.org/doc/go1.18#sha1[release notes] for details. {pull}32493[32493]
- Fix formatting of MAC hardware addresses populated by the add_host_metadata processor. {issue}32264[32264] {pull}32265[32265]

==== Bugfixes

*Affecting all Beats*

- Fix metric namespacing for self-monitoring to correct some process incorrectly reading as zero. {pull}32336[32336]

*Auditbeat*

- Fix rendering of MAC addresses to conform to ECS. {issue}32621[32621] {pull}32622[32622]

*Filebeat*

- Fix rendering of MAC addresses to conform to ECS. {issue}32621[32621] {pull}32622[32622]
- Import dashboards from CEF integration. {pull}32766[32766]
- Fix how to handle IPv6 addresses in the fileset `nginx/ingress_controller` for Filebeat. {pull}32989[32989]
- Fix requestID parsing in AWS cloudtrail fileset. {pull}33143[33143]
- Fix input metrics not being unregistered when an input closes. This led to panics when configuration was reloaded for the aws-s3, aws-cloudwatch, and lumberjack inputs. {pull}33259[33259]
- Add handling of AAA operations for Cisco ASA module. {issue}32257[32257] {pull}32789[32789]
- Fix gc.log always shipped even if gc fileset is disabled {issue}30995[30995]
- Fix handling of Cisco 302020 messages in ASA and FTD modules. {pull}33089[33089]

*Heartbeat*
- Fix bug affecting Let's Encrypt and other users of cross-signed certs, where cert expiration was incorrectly calculated. {issue}33215[33215]
- Fix broken disable feature for Kibana-configured monitors. {pull}33293[33293]

*Metricbeat*

- Fix GCP storage field naming {pull}32806[32806]
- In `module/windows/perfmon`, changed collection method of the second counter value required to create a displayable value {pull}32305[32305]
- Change max query size for GetMetricData API to 500 and add RecentlyActive for ListMetrics API call. {pull}33105[33105]
- Add GCP CloudSQL region filter. {pull}32943[32943]
- Fix Logstash cgroup mappings. {pull}33131[33131]
- Remove unused `elasticsearch.node_stats.indices.bulk.avg_time.bytes` mapping. {pull}33263[33263]

*Packetbeat*

- Fix formatting of debug logs. {pull}32698[32698]
- Fix rendering of MAC addresses to conform to ECS. {issue}32621[32621] {pull}32622[32622]

*Winlogbeat*

- Reduce severity of message salvage failure logging. {pull}32697[32697]

==== Added

*Filebeat*
- Import dashboard from Fortinet Fortigate firewall integration. {issue}19810[19810] {pull}33003[33003]

*Heartbeat*
- Add new states field for internal use by new synthetics app. {pull}30632[30632]

*Packetbeat*
- Add option to allow sniffer to change device when default route changes. {issue}31905[31905] {pull}32681[32681]
- Add option to allow sniffing multiple interface devices. {issue}31905[31905] {pull}32933[32933]
- Bump Windows Npcap version to v1.71. {issue}33164[33164] {pull}33172[33172]

==== Deprecated

*Heartbeat*
- Deprecate `zip_url` and local monitor options. {pull}33123[33123]

[[release-notes-8.4.3]]
=== Beats version 8.4.3
https://github.com/elastic/beats/compare/v8.4.2\...v8.4.3[View commits]

==== Bugfixes

*Filebeat*

- Fix handling of Cisco 302020 messages in ASA and FTD modules. {pull}33089[33089]
- Fix handling of empty array in httpjson input. {pull}32001[32001]

==== Added

*Affecting all Beats*

- Added `--enable-all-filesets` to the `setup` command to simplify loading all ingest pipelines. {issue}30916[30916] {pull}33114[33114]

[[release-notes-8.4.2]]
=== Beats version 8.4.2
https://github.com/elastic/beats/compare/v8.4.1\...v8.4.2[View commits]

==== Bugfixes

*Affecting all Beats*
- Fix in AWS related services initialisation relying on custom endpoint resolver. {issue}32888[32888] {pull}32921[32921]

*Filebeat*

- Fix for pagination at root level not working when used with chaining. {pull}32722[32722]

*Metricbeat*

- Fix and improve AWS metric period calculation to avoid zero-length intervals. {pull}32724[32724]
- Add missing cluster metadata to k8s module metricsets. {pull}32979[32979] {pull}33032[33032]

==== Added

*Metricbeat*

- Allow filtering on AWS tags by more than 1 value per key. {pull}32775[32775]
- Azure Billing: switch to Cost Management API for forecast data. {pull}32589[32589]

[[release-notes-8.4.1]]
=== Beats version 8.4.1
https://github.com/elastic/beats/compare/v8.4.0\...v8.4.1[View commits]

==== Known Issue

// tag::credentials-error[]

*Filebeat*

Filebeat agents configured to read from AWS inputs may return an error similar to the following:

[source,shell]
----
sqs ReceiveMessage failed: operation error SQS: ReceiveMessage, https response
error StatusCode: 403, RequestID: cb57783a-505f-5099-9160-23b8eea8ddbb,
api error SignatureDoesNotMatch: Credential should be scoped to a valid region.
----

This error was introduced by a breaking change in the AWS library.

IMPORTANT: This issue also affects FIPS-enabled endpoints. **If you rely on FIPS,
do not upgrade until version 8.4.2 of the {stack} is available.** The workaround
documented here will not resolve this problem.

Suggested resolution: In the Filebeat configuration, if an AWS input or module
configuration sets `endpoint` to a non empty string, set it to an empty string
instead. Also make sure the default AWS region is set in an environment
variable, credentials or instance profile, or in the `default_region` setting in
the configuration. For example:

[source,yaml]
----
filebeat.inputs:
- type: aws-s3
...
  endpoint: "" <1>
  default_region: us-east-1
----
<1> You can set this value to an empty string or remove the configuration setting.

Or for modules:

[source,yaml]
----
s3access:
    enabled: false
...
    var.endpoint: "" <1>
    var.default_region: us-east-1
----
<1> You can set this value to an empty string or remove the configuration setting

// end::credentials-error[]

==== Bugfixes

*Auditbeat*

- Fixes a bug with the auditd module where data is corrupted because it was not copied before the byte slice was reused. {issue}32818[32818] {pull}32823[32823]

*Filebeat*

- Update `cloud.region` parsing in cloudtrail fileset. {pull}32763[32763]
- Fix file.path field in cloudtrail fileset to use json.digestS3Object. {pull}32759[32759]
- Fix not parsing as json when `json` and `ndjson` content types have charset information in `aws-s3` input {pull}32767[32767]

==== Added

*Filebeat*

- httpjson input: Add `toJSON` helper function to template context. {pull}32472[32472]

[[release-notes-8.4.0]]
=== Beats version 8.4.0
https://github.com/elastic/beats/compare/v8.3.3\...v8.4.0[View commits]

==== Known Issues

*Auditbeat*

Auditbeat/auditd integration will send malformed data and may crash in version 8.4.0. {issue}32818[32818]

Suggested resolution:
Do not start Auditbeat or auditd integration on Elastic Agent at version 8.4.0. Skip 8.4.0 and upgrade directly to 8.4.1.

This issue is resolved in 8.4.1 and later.

//include credentials error from 8.4.1 section
include::CHANGELOG.asciidoc[tag=credentials-error]

==== Breaking changes

*Heartbeat*
- Browser monitors (beta) now write to the `synthetics-*` index prefix. {pull}32064[32064]
- Setting a custom index for a given monitor is now deprecated. Streams are preferred. {pull}32064[32064]
- Browser monitors now default to a max concurrency of two. {pull}32564[32564]

==== Bugfixes

*Affecting all Beats*

- Fix namespacing for agent self-monitoring, CPU no longer reports as zero. {pull}32336[32336]
- Expand fields in `decode_json_fields` if target is set. {issue}31712[31712] {pull}32010[32010]

*Auditbeat*

- auditd module: Fix parsing of audit rules where arguments are quoted (like file paths containing spaces). {pull}32421[32421]
- auditd module: Fix minimum AuditStatus length so that library can support kernels from 2.6.32. {pull}32421[32421]
- system/socket: Reduce memory usage of the dataset. {issue}32191[32191] {pull}32192[32192]

*Filebeat*

- Fix counter for number of events published in `httpjson` input. {pull}31993[31993]
- Fix handling of Checkpoint event for R81. {issue}32380[32380] {pull}32458[32458]
- gcp-pubsub input: Restart Pub/Sub client on all errors. {issue}32550[32550] {pull}32712[32712]

*Heartbeat*

- Send targetted error message for unexpected synthetics exits. {pull}31936[31936]
- Reduced memory usage slightly for browser monitors. {pull}32317[32317]
- Automatically kill zombie-ish node processes. {pull}32393[32393]
- Added timeout for browser monitors. {pull}32434[32434]
- Fix bug with browser jobs that had missing check groups or sent empty events. {pull}32542[32542]

*Metricbeat*

- Update Kubernetes apiserver metricset to not collect deprecated metrics and fix dashboard. {pull}31973[31973]
- Check for nil metadata in GCP. {pull}32281[32281]
- Update Kubernetes controllermanager metricset to not collect deprecated metrics and fix dashboard. {pull}32037[32037]
- Fix ARN parsing for Cloudwatch resource names with leading slashes. {pull}32358[32358]
- Fix an infinite loop in AWS billing metricset. {pull}32626[32626]
- Add missing metrics in AWS Transit Gateway module {pull}32617[32617]
- Replace internal expiring cache used by the Kubernetes module with in-memory dictionary. {pull}32539[32539]
- Oracle Module: Refactor module to use existing host parsers instead of doing its own parsing of hosts. {issue}31611[31611] {pull}31692[#31692]
- Oracle Module: Correctly handle special characters in the connection string. {issue}24609[24609] {pull}31368[#31368]

*Winlogbeat*

- Powershell: Fix processing of parameter details. {pull}31833[31833]
- Security: Fix processing of sidlist, access list and access mask. {pull}31833[31833]
- Fix fatal invalid memory write on Windows 11. {issue}32469[32469] {pull}32519[32519]
- Fix handling of event formatting when no metadata is available on Windows 11. {issue}32468[32468] {pull}32519[32519]

==== Added

*Affecting all Beats*

- Improve performance of disk queue by coalescing writes. {pull}31935[31935]

*Auditbeat*

- Add `immutable` option to the auditd module. {issue}8352[8352] {pull}32381[32381]

*Filebeat*

- Add `auth.oauth2.google.jwt_json` option to `httpjson` input. {pull}31750[31750]
- Add authentication fields to RabbitMQ module documents. {issue}31159[31159] {pull}31680[31680]
- Add template helper function for decoding hexadecimal strings. {pull}31886[31886]
- Add new `parser` called `include_message` to filter based on message contents. {issue}31794[31794] {pull}32094[32094]
- Extend list of mapped record types in o365 Audit module. {pull}32217[32217]
- Add references for CRI-O configuration in input-container and in our Kubernetes manifests. {issue}32149[32149] {pull}32151[32151]
- httpjson input: Add `replaceAll` helper function to template context. {pull}32365[32365]
- Optimize grok patterns in system.auth module pipeline. {pull}32360[32360]
- Checkpoint module: add authentication operation outcome enrichment. {issue}32230[32230] {pull}32431[32431]
- Add documentation for decode_xml_wineventlog processor field mappings.  {pull}32456[32456]

*Metricbeat*

- Oracle Module: New sysmetric metricset. {issue}30946[30946] {pull}31462[#31462]
- AWS Fargate: Added support for DesiredStatus and KnownStatus. {issue}32077[32077] {pull}32342[#32342]
- Enable Generic SQL merge metrics to a single event for sql_queries using new flag. {pull}32394[32394]
- Add distribution type metrics for GCP. {pull}32170[32170]

*Packetbeat*

- Add support for specifying default route interface sniffing. {issue}31905[31905] {pull}31950[31950]
- Add support for TCP transport to the SIP protocol. {issue}28166[28166] {pull}32346[32346]

[[release-notes-8.3.3]]
=== Beats version 8.3.3
https://github.com/elastic/beats/compare/v8.3.2\...v8.3.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix OS name reported by add_host_metadata on Windows 11. {issue}30833[30833] {pull}32259[32259]
- Fix race condition when reloading running inputs. {pull}32309[32309]

*Filebeat*

- Fix Cisco AMP rate limit and pagination. {pull}32030[32030]
- Fix wrong state ID in states registry for awss3 s3 direct input. {pull}32164[32164]
- cisco/asa: fix handling of user names when there are Security Group Tags present. {issue}32009[32009] {pull}32196[32196]

*Metricbeat*

- Update elasticsearch node_stats metricset to use keyword for cgroup memory instead of long. {pull}32197[32197]

==== Added

*Metricbeat*

- Azure Billing: upgrade Usage Details API to version 2019-10-01. {pull}31970[31970]

[[release-notes-8.3.2]]
=== Beats version 8.3.2
https://github.com/elastic/beats/compare/v8.3.1\...v8.3.2[View commits]

==== Bugfixes

*Filebeat*
- Fix handling of stale log message handling in the winlog input {issue}32168[32168] {pull}32176[32176]

*Heartbeat*

- Fix regression where we write a dotted (non-nested) key `event.type`. {pull}32097[32097]

*Metricbeat*

- Fix for unintentionally reporting the libbeat "handles" metrics under "system" instead of "beat" as they were in previous releases. https://github.com/elastic/elastic-agent-system-metrics/pull/37[system-metrics#37]

*Winlogbeat*

- Fix handling of stale log message handling {issue}32168[32168] {pull}32176[32176]

==== Added

*Metricbeat*

- Differentiate between actual idle CPU states and an uninterruptible disk sleep. https://github.com/elastic/elastic-agent-system-metrics/pull/32[system-metrics#32]

[[release-notes-8.3.1]]
=== Beats version 8.3.1
https://github.com/elastic/beats/compare/v8.3.0\...v8.3.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Improve syslog parser/processor error handling. {issue}31246[31246] {pull}31798[31798]

*Auditbeat*

- Fix handling of audit status messages for modern kernels with reduced audit message feature support. {issue}31616[31616] {pull}32141[32141]

*Filebeat*

- Fix handling and mapping of syslog priority, facility and severity values in Cisco module. {pull}32025[32025]
- Fix deduplication in Google workspace module by changing fingerprint processor target field from `@metadata.id` to `@metadata._id`. {pull}31898[31898]
- Adding a fix for threatintel module where MISP was paginating forever. {pull}31784[31784]
- Fix http_endpoint input TLS handshake failures. {pull}32105[32105]
- Fix last write pagination commit checkpoint on `aws-s3` input for s3 direct polling when using the same bucket and different list prefixes. {pull}31776[31776]

[[release-notes-8.3.0]]
=== Beats version 8.3.0
https://github.com/elastic/beats/compare/v8.2.3\...v8.3.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Allow loading secrets that contain commas from the keystore {pull}31694[31694].

*Auditbeat*

- Fix audit status collection on kernels prior to version 5.9. {issue}31616[31616] {pull}31710[31710]

*Filebeat*

- Do not emit error log when filestream reader reaches EOF and `close.reader.on_eof` is enabled. {pull}31109[31109]
- sophos.xg: Update module to handle new log fields. {issue}31038[31038] {pull}31388[31388]
- Fix MISP documentation for `var.filters` config option. {pull}31434[31434]
- Fix type mapping of client.as.number in okta module. {pull}31676[31676]
- If a file is ignored by `filestream` because of ignore_older settings, when it is updated, only the new lines are shipped to the output. {issue}31924[31924] {pull}31972[31972]

*Heartbeat*

- Fix unintentional use of no-op logger. {pull}31543[31543]

*Metricbeat*

- make `system/filesystem` code sensitive to `hostfs` and migrate libraries to `elastic-agent-opts` {pull}31001[31001]
- Fix kubernetes module's internal cache expiration issue. This avoid metrics like `kubernetes.container.cpu.usage.limit.pct` from not being populated. {pull}31785[31785]
- add missing HealthyHostCount and UnHealthyHostCount for application ELB. {pull}31853[31853]

*Winlogbeat*

- Sysmon: Drop fields with "-" value (unset) {pull}31556[31556]

==== Added

*Affecting all Beats*

- Add new config option `timestamp.precision` to configure timestamps. {pull}31682[31682]

*Auditbeat*

- Add `backlog_wait_time_actual` to the output of the `auditbeat auditd show-status` command. {pull}31535[31535]

*Filebeat*

- Support SASL/SCRAM authentication in the Kafka input. {pull}31167[31167]
- checkpoint module: Add `network.transport` derived from IANA number. {pull}31076[31076]
- Add URL Encode template function for httpjson input. {pull}30962[30962]
- Add `application/zip` decoder to the `httpsjon` input. {issue}31282[31282] {pull}31304[31304]
- Default value of `filebeat.registry.flush` increased from 0s to 1s. CPU and disk I/O usage are reduced because the registry is not written to disk for each ingested log line. {issue}30279[30279]
- Cisco ASA/FTD: Add support for messages 434001 and 434003. {pull}31533[31533]
- Change threatintel module from beta to GA. {pull}31693[31693]
- Add template helper function for hashing strings. {issue}31613[31613] {pull}31630[31630]
- Add extended okta.debug_context.debug_data handling. {pull}31676[31676]
- Add a new `salesforce` module to collect data from salesforce. {pull}31486[31486]

*Auditbeat*

- auditd: Updated the go-libaudit library version to v2.3.0. This refreshes the syscall names for Linux and adds ECS categorizations for more audit anomaly events. {pull}31519[31519]

*Filebeat*

- http_endpoint input: Add support for requests with `Content-Encoding: gzip`. {issue}31005[31005]

*Heartbeat*

- Add support for `pushed` browser monitor source from the synthetics agent. {pull}31428[31428]
- Add ARM64 seccomp profile. {issue}31285[31285] {pull}31422[31422]
- Add new `playwright_options` config for browser monitors. {issue}28197[28196] {pull}31737[31737]

*Metricbeat*

- Add new Kubernetes module dashboards {pull}31591[31591]
- system/core: add cpuinfo information for Linux hosts {pull}31643[31643]

*Winlogbeat*

- Add parent process ID to new process creation events. {issue}29237[29237] {pull}31102[31102]
- Sysmon: Support for Sysmon Registry non-QWORD/DWORD events. {pull}31556[31556]

==== Deprecated

*Heartbeat*
- Bump node.js version for synthetics to 16.15.0. {pull}31675[31675]

[[release-notes-8.2.3]]
=== Beats version 8.2.3
https://github.com/elastic/beats/compare/v8.2.2\...v8.2.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix Windows service timeouts when the "TCP/IP NetBIOS Helper" service is disabled. {issue}31810[31810] {pull}31835[31835]

*Filebeat*

- Fix last write pagination commit checkpoint on `aws-s3` input for s3 direct polling when using the same bucket and different list prefixes. {pull}31776[31776]

[[release-notes-8.2.2]]
=== Beats version 8.2.2
https://github.com/elastic/beats/compare/v8.2.1\...v8.2.2[View commits]

==== Bugfixes

*Auditbeat*

- Remove invalid "network_traffic" term from event.category. {pull}31674[31674]

*Filebeat*

- Remove invalid "network_traffic" term from event.category. {pull}31674[31674]

[[release-notes-8.2.1]]
=== Beats version 8.2.1
https://github.com/elastic/beats/compare/v8.2.0\...v8.2.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix group write permissions on runtime directories. {pull}30869[30869]
- Store syslog version as string. {pull}31446[31446]
- Accept XML that declares non-UTF-8 encoding to allow decode_xml and decode_xml_wineventlog decoding of incorrectly annotated documents. {issue}31395[31395] {pull}31546[31546]

*Filebeat*

- Netflow: replace invalid field value. {pull}31295[31295]
- google_workspace: Fix pagination to prevent skipped events when more than one page is present. {pull}31372[31372]
- cisco: Fix umbrella dns logs populating destination.ip instead of source.nat.ip. {pull}31454[31454]
- Duplicate awscloudwatch.* fields to aws.cloudwatch.* in aws-cloudwatch input. {pull}31488[31488]
- aws-s3 input: Stop SQS keep-alive routine on InvalidParameterValue error. {issue}30675[30675] {pull}31499[31499]
- Supporting the double digit date parsing in ingest pipeline for oracle logs. {pull}31514[31514]
- Fix handling of code_sign data in ThreatIntel Malwarebazaar. {issue}29972[29972] {pull}31552[31552]
- Remove invalid term from event.outcome in the cisco asa and ftd modules. {pull}31628[31628]

*Heartbeat*

- Restrict setuid to containerized environments. {pull}30869[30869]

*Metricbeat*

- Improve handling of disabled commands in Zookeeper Metricbeat module. {pull}31013[#31013]

*Packetbeat*

- Use /proc/<pid>/comm for linux process names where possible. {pull}31527[31527]
- Move "protocol" term from event.category to event.type in SIP events. {pull}31599[31599]

*Winlogbeat*

- Fix resource handle leak during event log enrichment. {pull}31504[31504]
- Fix winlogbeat.registry_flush being ignored. {issue}31666[31666] {pull}31669[31669]

==== Added

*Affecting all Beats*

- Update to Go 1.17.10 {issue}31636[31636]
- Add support for nanosecond precision timestamps. {issue}15871[15871] {pull}31553[31553]

*Filebeat*

- Add `storage_account_container` configuration option to Azure logs. {pull}31279[31279]
- Sanitize the Azure storage account container names with underscores (_). {pull}31384[31384]
- Add missing docs for the `delegated_account` option in the `httpjson` input. {pull}31498[31498]

*Metricbeat*

- Extend documentation about `orchestrator.cluster` fields {pull}30518[30518]
- Generic SQL code reorganization, with support for raw metrics and query lists {pull}31568[31568]
- Add metadata for missing k8s resources/metricsets {pull}31590[31590]
- Fix `include_top_n` fields in system/process {pull}31595[31595]

[[release-notes-8.2.0]]
=== Beats version 8.2.0
https://github.com/elastic/beats/compare/v8.1.3\...v8.2.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Fix mapping of parent process information provided by add_process_metadata. {issue}29874[29874] {pull}30727[30727]

==== Bugfixes

*Filebeat*

- m365_defender: Fix processing when alerts.entities is an empty list. {issue}31223[31223] {pull}31227[31227]
- Prevent filestream from rereading whole files if they are rotated using rename. {pull}31268[31268]

*Heartbeat*

- Heartbeat now successfully runs synthetic monitors on ARM processors. {pull}31114[31114]

*Metricbeat*

- Add back missing metrics to system/linux. {pull}30774[30774]
- GCP metrics query instances with aggregatedList API to improve efficiency. {pull}30154[#30153]
- Fix Jolokia module to print URI for one of the debug logs. {pull}30943[#30943]
- Handle docker reporting different capitalization for disk usage metrics. {pull}30978[#30978]

*Winlogbeat*

- Fix routing for PowerShell events. {issue}31287[31287] {pull}31291[31291]
- Fix missing annotation of event.module. {issue}31330[31330] {pull}31331[31331]

==== Added

*Affecting all Beats*

- Add support for port mapping in docker hints. {pull}31243[31243]
- Relax timestamp syntax for RFC3164 syslog to allow leading zero on day. {issue}16824[16824] {pull}31254[31254]

*Filebeat*

- Add extraction of `related.hosts` to Microsoft 365 Defender ingest pipeline {issue}29859[29859] {pull}29863[29863]
- Improve recovery from corrupted registries. {issue}25135[25135] {pull}30994[30994]
- Add support in httpjson input for chain calls. {pull}29816[29816]

*Auditbeat*

- Include config file (`auditbeat.elastic-agent.yml`) in tar.gz and zip packages for use with Elastic Agent.

*Metricbeat*

- Add `kubernetes.container.status.last.reason` metric {pull}30306[30306]
- Fix overflow in `iostat` metrics {pull}30679[30679]
- Add `commandstats` field to Redis module {pull}29662[29662]
- Add `kubernetes.volume.fs.inodes.pct` field. {pull}30785[30785]
- Improve Kubernetes dashboard. {pull}30913[30913]
- Populate new container ECS fields in Docker module. {pull}30399[30399]
- Populate new container ECS fields in Kubernetes module. {pull}30181[30181]
- Populate ecs container fields in Containerd module. {pull}31025[31025]
- Add new metricset shovel to RabbitMQ module {pull}31534[31534]

*Winlogbeat*

- Improve the error message when the registry file content is invalid. {pull}30543[30543]

[[release-notes-8.1.3]]
=== Beats version 8.1.3
https://github.com/elastic/beats/compare/v8.1.2\...v8.1.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Load data stream during setup, so users do not need extra permissions during publishing. {issue}30647[30647] {pull}31048[31048]
- Add ecs container fields {pull}31020[31020]
- Fix docs reference for syslog processor {pull}31087[31087]
- Fix AWS config initialization issue when using a role {issue}30999[30999] {pull}31014[31014]

*Filebeat*

- Prevent logic race on clearing data during request in httpjson. {pull}30730[30730]
- Prevents filestream inputs from being stuck while being created. {pull}31240[31240]

*Filebeat*

- Recover CEF extensions from messages with invalid/incomplete headers. {issue}30757[30757] {pull}30938[30938]
- Fix panic in filestream input when `copy_truncate` log rotation strategy is used {issue}29024[29024] {pull}31041[31041]
- Fix Azure signinlogs authentication_requirement_policies field type and several missing fields. {pull}31062[31062]
- Cyberark PAS: Fix error ingesting events with a single entry in the CAProperties field. {pull}31094[31094]
- Fix Azure activitylogs identity field type and several missing fields. {pull}31170[31170]
- checkpoint: Fix ingest error when a message contains trailing spaces {pull}31197[31197]

*Metricbeat*

- Fix delay in perfmon counters collection {issue}30686[30686] {pull}30861[#30861]

*Packetbeat*

- Prevent panic when querying Npcap version on Windows. {issue}30820[30820] {pull}30821[30821] {pull}30837[30837]

*Winlogbeat*

- Fix evtx parsing failures. {issue}30621[30621] {pull}30942[30942]

==== Added

*Affecting all Beats*

- Add FIPS configuration option for all AWS API calls. {pull}28899[28899]
- Add support for kafka message headers. {pull}29940[29940]
- Add support for non-unique Kafka headers for output messages. {pull}30369[30369]
- Add syslog parser and processor. {issue}30139[30139] {pull}30541[30541]
- Add action_input_type for the .fleet-actions-results {pull}30562[30562]
- Add cronjob metadata by default {pull}30637[30637]
- New option `setup.template.json.data_stream` is added to indicate if the JSON index template is a data stream. {pull}31048[31048]

*Winlogbeat*

- Retry EvtSubscribe from start if fails with strict mode. {issue}29793[29793] {pull}30155[30155]


[[release-notes-8.1.2]]
=== Beats version 8.1.2
https://github.com/elastic/beats/compare/v8.1.1\...v8.1.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Add more descriptive error logs for common connection failures in Kafka inputs / outputs. {pull}30776[30776]

*Filebeat*

- Fix add_kubernetes_metadata matcher: support rotated logs when `resource_type: pod` {pull}30720[30720]

*Packetbeat*

- packetbeat: Fix timeout error in af_packet capture mode. {issue}30731[30731] {issue}30822[30822] {pull}30882[30882]

==== Added

*Heartbeat*

- Generate summary documents for journeys which exit successfully, but do not emit `journey/end` events {pull}30825[30825]

[[release-notes-8.1.1]]
=== Beats version 8.1.1
https://github.com/elastic/beats/compare/v8.1.0\...v8.1.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fixes Beats crashing when glibc >= 2.35 is used {issue}30576[30576]
- Log errors when parsing and applying config blocks and if the input is disabled. {pull}30534[30534]
- Wildcard fields no longer have a default ignore_above setting of 1024. {issue}30096[30096] {pull}30668[30668]
- Ensure that the Reloadable part of beats are initialized before the Manager is started. {issue}30533[30533]
- Ignore bugfix version when running version compatibility check against Elasticsearch. {pull}30746[30746]

*Filebeat*

- Fix compatibility with ECS by renaming `source` log key to `source_file` {issue}30667[30667]
- Report the starting offset of the line in `log.offset` when using `filestream` instead of the end to be ECS compliant. {pull}30445[30445]
- Update documentation for accessing `last_response.url.params` in httpjson input. {pull}30739[30739]

*Packetbeat*

- Prevent panic when querying Npcap version on Windows. {issue}30820[30820] {pull}30821[30821]

[[release-notes-8.1.0]]
=== Beats version 8.1.0
https://github.com/elastic/beats/compare/v8.0.1\...v8.1.0[View commits]

==== Breaking changes

*Filebeat*

- Remove Recorded Future fileset integration from threatintel module. {pull}30564[30564]

==== Bugfixes

*Auditbeat*

- auditd: Add error.message to events when processing fails. {pull}30009[30009]
- Fix handling of execve call events which have no argument. {issue}30585[30585] {pull}30586[30586]

*Filebeat*

- Fix ECS version string in threatintel to be consistent with other modules and add event.timezone. {issue}30499[30499] {pull}30570[30570]
- Add default paths value to MySQL Enterprise module to prevent issues with pipeline installations {pull}30598[30598]

*Winlogbeat*

- Add provider names to Security pipeline conditional check in routing pipeline. {issue}27288[27288] {pull}29781[29781]

*Functionbeat*

- Pass AWS region configuration correctly. {issue}28520[28520] {pull}30238[30238]

==== Added

*Affecting all Beats*

- Name all k8s workqueue. {pull}28085[28085]
- Discover changes in Kubernetes nodes metadata as soon as they happen. {pull}23139[23139]
- Update k8s library {pull}29394[29394]
- Add support for latest k8s versions v1.23 and v1.22 {pull}29575[29575]
- Add `script` processor to all beats {issue}29269[29269] {pull}29752[29752]
- Only connect to Elasticsearch instances with the same version or newer. {pull}29683[29683]
- Move umask from code to service files. {pull}29708[29708]
- Add metadata change support for some processors {pull}30183[30183]

*Auditbeat*

- system/socket: Add process.entity_id capture for socket events. {issue}30230[30230] {pull}30231[30231]

*Filebeat*

- Add support for filtering in journald input with `unit`, `kernel`, `identifiers` and `include_matches`. {pull}29294[29294]
- Add new `userAgent` and `beatInfo` template functions for httpjson input {pull}29528[29528]
- Add pipeline in FB's supported hints. {pull}30212[30212]

*Metricbeat*

- Add `add_resource_metadata` configuration to Kubernetes module. {pull}29133[29133]
- Add `containerd` module with `cpu`, `memory`, `blkio` metricsets. {pull}29247[29247]
- Add `container.id` and `container.runtime` ECS fields in container metricset. {pull}29560[29560]
- Add `memory.workingset.limit.pct` field in Kubernetes container/pod metricset. {pull}29547[29547]
- Add k8s metadata in state_cronjob metricset. {pull}29572[29572]
- Add `xpack.enabled` support for Enterprise Search module. {pull}29871[29871]
- Add gcp firestore metricset. {pull}29918[29918]
- Remove strict parsing on RabbitMQ module {pull}30090[30090]

*Packetbeat*

- Add automated OEM Npcap installation handling. {pull}29112[29112] {pull}30438[30438] {pull}30493[30493]
- Add support for capturing TLS random number and OCSP status request details. {issue}29962[29962] {pull}30102[30102]

[[release-notes-8.0.1]]
=== Beats version 8.0.1
https://github.com/elastic/beats/compare/v8.0.0\...v8.0.1[View commits]

==== Bugfixes

*Filebeat*

- tcp/unix input: Stop accepting connections after socket is closed. {pull}29712[29712]
- Fix using log_group_name_prefix in aws-cloudwatch input. {pull}29695[29695]
- Fix multiple instances of the same module configured within `filebeat.modules` in filebeat.yml. {issue}29649[29649] {pull}29952[29952]
- aws-s3: fix race condition in states used by s3-poller. {issue}30123[30123] {pull}30131[30131]

*Filebeat*
- Fix broken Kafka input {issue}29746[29746] {pull}30277[30277]
- cisco module: Fix change the broke ASA and FTD configs that used `var.input: syslog`. {pull}30072[30072]
- aws-s3: fix race condition in states used by s3-poller. {issue}30123[30123] {pull}30131[30131]

*Heartbeat*
- Fix missing mapping for `service.name`. {pull}30324[30324]

*Winlogbeat*

- Fix run loop when reading from evtx file {pull}30006[30006]


[[release-notes-8.0.0]]
=== Beats version 8.0.0
https://github.com/elastic/beats/compare/v7.17.0\...v8.0.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Remove the deprecated `xpack.monitoring.*` settings. Going forward only `monitoring.*` settings may be used. {issue}9424[9424] {pull}18608[18608]
- Remove deprecated/undocumented `IncludeCreatorMetadata` setting from kubernetes metadata config options. {pull}28006[28006]
- Remove deprecated fields from kubernetes module. {pull}28046[28046]
- Remove deprecated config option `aws_partition`. {pull}28120[28120]
- Improve stats API by adding host metadata. {pull}27963[27963]
- Libbeat: logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. {issue}15544[15544] {pull}28573[28573]
- Remove `auto` from the available options of `setup.ilm.enabled` and set the default value to `true`. {pull}28671[28671]
- add_process_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
- add_docker_metadata processor: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]
- Use data streams instead of indices for storing events from Beats. {pull}28450[28450]
- Remove option `setup.template.type` and always load composable template with data streams. {pull}28450[28450]
- Remove several ILM options (`rollover_alias` and `pattern`) as data streams do not require index aliases. {pull}28450[28450]
- Populate index template's `default_fields` setting with ECS fields only. {pull}28596[28596] {issue}28215[28215]
- Remove deprecated `--template` and `--ilm-policy` flags. Use `--index-management` instead. {pull}28870[28870]
- Remove `logging.files.suffix` option, and default to datetime endings in log file names. The format of the new name is `{beatname}-{date}(-n)?.ndjson`. Example log file names from oldest to newest: `filebeat-20200101.ndjson`, `filebeat-20200101-1.ndjson`, `filebeat-20200101-2.ndjson`. {pull}28927[28927]
- Align kubernetes configuration settings. {pull}29908[29908]
- Change log file extension for Beats and Elastic Agent to `.ndjson`. If you are collecting the logs, you must change the path configuration to `/path/to/logs/{beatname}*.ndjson` to avoid any issues. {pull}28927[28927]
- Remove legacy support for SSLv3. {pull}30071[30071]

*Filebeat*

- Add `while_pattern` type to multiline reader. {pull}19662[19662]
- auditd dataset: Use `process.args` to store program arguments instead of `auditd.log.aNNN` fields. {pull}29601[29601]
- Remove deprecated old `awscloudwatch` input name. {pull}29844[29844]
- Remove `docker` input. Please use `filestream` input with `container` parser or `container` input. {pull}28817[28817]

*Metricbeat*

- Remove network and diskio metrics from ec2 metricset. {pull}28316[28316]
- Rename `read/write_io.ops_per_sec` to `read/write.iops` in rds metricset. {pull}28350[28350]
- system/process metricset: Replace usage of deprecated `process.ppid` field with `process.parent.pid`. {pull}28620[28620]

*Packetbeat*

- `event.category` no longer contains the value `network_traffic` because this is not a valid ECS event category value. {pull}20556[20556]
- Remove deprecated TLS fields in favor of `tls.server.x509` and `tls.client.x509` ECS fields. {pull}28487[28487]
- HTTP: The field `http.request.method` will maintain its original case. {pull}28620[28620]

*Winlogbeat*

- Remove top level `hash` property from sysmon events. {pull}20653[20653]
- Move module processing from local Javascript processor to ingest node. {issue}29184[29184] {pull}29435[29435]

==== Bugfixes

*Auditbeat*

- libbeat/processors/add_process_metadata: Fix memory leak in process cache. {issue}24890[24890] {pull}29717[29717]

*Filebeat*

- Fix using `log_group_name_prefix` in `aws-cloudwatch` input. {pull}29695[29695]

*Heartbeat*

- Add fonts to support more types of characters for multiple languages. {pull}29861[29861]

*Metricbeat*

- Extract correct index property in `kibana.stats` metricset. {pull}29622[29622]
- Fixed bug with `elasticsearch/cluster_stats` metricset not recording license expiration date correctly. {pull}29711[29711]

*Packetbeat*

- Prevent incorrect use of AMQP protocol parsing from causing silent failure. {pull}29017[29017]
- Fix error handling in MongoDB protocol parsing. {pull}29017[29017]
- Redis: fix incorrectly handle with two-words redis command. {issue}14872[14872] {pull}14873[14873]
- Unify gopacket dependencies. {pull}29167[29167]

==== Added

*Affecting all Beats*

- Add config option `rotate_on_startup` to file output. {issue}19150[19150] {pull}19347[19347]
- Update to ECS 8.0 fields. {pull}28620[28620]
- Support custom analyzers in `fields.yml`. {issue}28540[28540] {pull}28926[28926]
- Support self-signed certificates on outputs. {pull}29229[29229]
- Add FIPS configuration option for all AWS API calls. {pull}[28899]
- Warn users when connecting to older versions of Elasticsearch instances. {pull}29723[29723]
- `add_fields` processor is now able to set metadata in events. {pull}30092[30092]

*Auditbeat*

- system/process: Prevent hashing files in other mnt namespaces. {issue}25777[25777] {issue}29678[29678] {pull}29786[29786]

*Metricbeat*

- Add preliminary AIX support. {pull}27954[27954]
- Add option to skip older k8s events. {pull}29396[29396]
- Add `elasticsearch.cluster.id` field to Beat and Kibana modules. {pull}29577[29577]
- Add `elasticsearch.cluster.id` field to Logstash module. {pull}29625[29625]

*Winlogbeat*

- Add support for sysmon event ID 26; `FileDeleteDetected`. {issue}26280[26280] {pull}29957[29957]

*Elastic Log Driver*

- Fixed docs for hosts. {pull}23644[23644]

[[release-notes-7.17.0]]
=== Beats version 7.17.0
https://github.com/elastic/beats/compare/v7.16.3\...v7.17.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Change Docker base image from CentOS 7 to Ubuntu 20.04 {pull}29681[29681]

==== Bugfixes

*Affecting all Beats*

- Enrich kubernetes metadata with node annotations. {pull}29605[29605]

*Auditbeat*

- system/socket: Fix startup errors on newer 5.x kernels due to missing _do_fork function. {issue}29607[29607] {pull}29744[29744]
- system/package: Fix parsing of Installed-Size field of DEB packages. {issue}16661[16661] {pull}17188[17188]
- system module: Fix panic during initialisation when /proc/stat can't be read. {pull}17569[17569]
- system/package: Fix an error that can occur while trying to persist package metadata. {issue}18536[18536] {pull}18887[18887]
- system/socket: Fix bugs leading to wrong process being attributed to flows. {pull}29166[29166] {issue}17165[17165]
- system/socket: Fix process name and arg truncation for long names, paths and args lists. {issue}24667[24667] {pull}29410[29410]

*Filebeat*

- aws-s3: Stop trying to increase SQS message visibility after ReceiptHandleIsInvalid errors. {pull}29480[29480]
- Fix handling of IPv6 addresses in netflow flow events. {issue}19210[19210] {pull}29383[29383]
- Fix `sophos` KV splitting and syslog header handling {issue}24237[24237] {pull}29331[29331]
- Undo deletion of endpoint config from cloudtrail fileset in {pull}29415[29415]. {pull}29450[29450]
- Make Cisco ASA and FTD modules conform to the ECS definition for event.outcome and event.type. {issue}29581[29581] {pull}29698[29698]
- ibmmq: Fixed `@timestamp` not being populated with correct values. {pull}29773[29773]
- aws-s3: Improve gzip detection to avoid false negatives. {issue}29968[29968]
- decode_cef: Fix panic when recovering from invalid CEF extensions that contain escape characters. {issue}30010[30010]

*Heartbeat*

- Fix race condition in http monitors using `mode:all` that can cause crashes. {pull}29697[pull]
- Fix broken ICMP availability check that prevented heartbeat from starting in rare cases. {pull}29413[pull]
- Fix broken macOS ICMP python e2e test. {pull}29900[29900]
- Only add monitor.status to browser events when summary. {pull}29460[29460]
- Also add summary to journeys for which the synthetics runner crashes. {pull}29606[29606]
- Update size of ICMP packets to adhere to standard min size. {pull}29948[29948]

*Metricbeat*

- Use xpack.enabled on SM modules to write into .monitoring indices when using Metricbeat standalone {pull}28365[28365]
- Fix in rename processor to ingest metrics for `write.iops` to proper field instead of `write_iops` in rds metricset. {pull}28960[28960]
- Enhance filter check in kubernetes event metricset. {pull}29470[29470]
- Fix gcp metrics metricset apply aligner to all metric_types {pull}29514[29513]
- Fixed GCP GKE Overview dashboard {pull}29913[29913]
- Remove overriding of index pattern on the Kubernetes overview dashboard. {pull}29676[29676]

==== Added

*Affecting all Beats*

- SASL/SCRAM in the Kafka output is no longer beta. {pull}29126[29126]
- Add job.name in pods controlled by Jobs {pull}28954[28954]

*Heartbeat*

- More errors are now visible in ES with new logic failing monitors later to ease debugging. {pull}29413[pull]

*Winlogbeat*

- Add support for custom XML queries {issue}1054[1054] {pull}29330[29330]

==== Deprecated

==== Known Issue


[[release-notes-7.16.3]]
=== Beats version 7.16.3
https://github.com/elastic/beats/compare/v7.16.2\...v7.16.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Fields of type `match_only_text` (i.e. `message`) and `wildcard` were missing from the template's `default_field` list. {issue}29633[29633] {pull}29634[29634]

[[release-notes-7.16.2]]
=== Beats version 7.16.2
https://github.com/elastic/beats/compare/v7.16.1\...v7.16.2[View commits]

==== Bugfixes

*Filebeat*

- Resolve issue with @timestamp for `defender_atp`. {pull}28272[28272]
- Fix handling of escaped newlines in the `decode_cef` processor. {issue}16995[16995] {pull}29268[29268]

==== Added

*Filebeat*

- Update Cisco module to enable TCP input. {issue}26118[26118] {issue}28821[28821] {pull}26159[26159]

*Winlogbeat*

- Add configuration option for registry file flush timeout {issue}29001[29001] {pull}29053[29053]

[[release-notes-7.16.1]]
=== Beats version 7.16.1
https://github.com/elastic/beats/compare/v7.16.0\...v7.16.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Overwrite index name in index template correctly. {issue}28571[28571] {pull}29299[29299]

==== Added

*Filebeat*

- Add elapsed time information to `aws-s3` input errors and log messages. {pull}29328[29328]

[[release-notes-7.16.0]]
=== Beats version 7.16.0
https://github.com/elastic/beats/compare/v7.15.2...v7.16.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Load index templates v2 (composable index templates) by default when talking to ES 7.16 or ES 8.x. Please note that you cannot load templates into Elasticsearch 7.8 or older with this default. To load templates to these ES version, set `setup.template.type` back to `legacy`. {pull}28538[28538]
- Previously, RE2 and thus Golang had a bug where `(|a)*` matched more characters than `(|a)+`. To stay consistent with PCRE, the bug was fixed. Configurations that rely on the old, buggy behaviour has to be adjusted. See more about Golang bug: https://github.com/golang/go/issues/46123 {pull}27543[27543]
- Remove Journalbeat. Use `journald` input of Filebeat instead. {pull}29131[29131]

*Heartbeat*

- Change behavior in case of duplicate monitor IDs in configs to be last monitor wins. {pull}29041[29041]

*Metricbeat*

- Align fields to Beats naming conventions in GCP module. {issue}27231[27231] {pull}27974[27974]

*Functionbeat*

- Support for Google Cloud Functions have been removed, as it has been in Beta for a long time and been broken for a few releases. Please use other tools provided by Elastic to fetch data from GCP (e.g. Filebeat).

==== Bugfixes

*Affecting all Beats*

- Fix discovery of Nomad allocations with multiple events during startup. {pull}28700[28700]
- Fix the wrong beat name on monitoring and state endpoint {issue}27755[27755]
- Skip configuration checks in autodiscover for configurations that are already running {pull}29048[29048]
- Fix `decode_json_processor` to always respect `add_error_key` {pull}29107[29107]
- Fix `add_labels` flattening of array values. {pull}29211[29211]
- Skip `add_kubernetes_metadata` processor when Kubernetes metadata are already present {pull}27689[27689]

*Auditbeat*

- Fix handling of root and relative paths {issue}24430[24430] {pull}28354[28354]
- Fix handling of long file names on Windows. {issue}25334[25334] {pull}28517[28517]
- System/socket dataset: Fix uninstallation of return kprobes. {issue}28608[28608] {pull}28609[28609]
- Fix auditbeat tracing struct decoding. {pull}28580[28580]

*Filebeat*

- Update indentation for azure filebeat configuration. {pull}26604[26604]
- Tolerate faults when Windows Event Log session is interrupted {issue}27947[27947] {pull}28191[28191]
- Add support for username in Cisco ASA security negotiation logs {pull}26975[26975]
- Relax time parsing and capture group and session type in Cisco ASA module {issue}24710[24710] {pull}28325[28325]
- Correctly track bytes read when max_bytes is exceeded. {issue}28317[28317] {pull}28352[28352]
- Fix parsing of apache log levels including numbers. {pull}28717[28717]
- Upgrade `azure-eventhub` SDK reference, contains potential checkpoint fixes. {pull}28919[28919]
- Revert usageDetails api version to 2019-01-01. {pull}28995[28995]
- Fix in `aws-s3` input regarding provider discovery through endpoint {pull}28963[28963]
- Fix `threatintel.misp` filters configuration. {issue}27970[27970]
- Fix opening files on Windows in filestream so open files can be deleted. {issue}29113[29113] {pull}29180[29180]

*Heartbeat*

- Fix broken seccomp filtering and improve security via `setcap` and `setuid` when running as root on linux in containers. {pull}27878[27878]
- Log browser `zip_url` download failures as `warn` instead of as `info`. {pull}28440[28440]
- Properly locate base stream in fleet configs. {pull}28455[28455]
- Stop logging params values. {pull}28774[28774]
- Remove accidentally included `cups` library in Docker images. {pull}28853[pull]
- Fix broken monitors with newer versions of image relying on `dup3`. {pull}28938[pull

*Metricbeat*

- `beat` module respects `basepath` config option. {pull}28162[28162]
- Fix list_docker.go {pull}28374[28374]
- Fix RDS metadata in Cloudwatch metricset. {pull}29106[29106]
- Errors should be thrown as errors. Metricsets inside metricbeat will now throw errors as the `error` log level. {pull}27804[27804]

*Winlogbeat*

- Tolerate faults when Windows Event Log session is interrupted {issue}27947[27947] {pull}28191[28191]
- Add ECS 1.9 new users fields {pull}26509[26509]
- Don't split hyphenated tokens {pull}28483[28483]
- Correctly handle AccessMask if it is an integer or list of masks. {pull}29016[29016]

==== Added

*Affecting all Beats*

- Allow non-padded base64 data to be decoded by `decode_base64_field` {pull}27311[27311], {issue}27021[27021]
- The Kafka support library Sarama has been updated to 1.29.1. {pull}27717[27717]
- Kafka is now supported up to version 2.8.0. {pull}27720[27720]
- Add Huawei Cloud provider to add_cloud_metadata. {pull}27607[27607]
- Add default seccomp policy for linux arm64. {pull}27955[27955]
- Add cluster level add_kubernetes_metadata support for centralized enrichment {pull}24621[24621]
- Update cloud.google.com/go library. {pull}28229[28229]
- Add additional metadata to the root HTTP endpoint. {pull}28265[28265]
- Upgrade k8s.io/client-go library. {pull}28228[28228]
- Update ECS to 1.12.0. {pull}27770[27770]
- Fields mapped as `match_only_text` will automatically fallback to a `text` mapping when using Elasticsearch versions that do not support `match_only_text`. {pull}27770[27770]
- Do not load ML jobs to Elasticsearch 8.x from new Beats 7.x releases. {pull}27771[27771]
- Update kubernetes scheduler and controllermanager endpoints in elastic-agent-standalone-kubernetes.yaml with secure ports {pull}28675[28675]
- Add default seccomp policy for Linux arm64. {pull}27955[27955]
- Add `http.pprof.enabled` option to libbeat to allow http/pprof endpoints on the socket that libbeat creates for metrics. {issue}21965[21965]
- Enable IMDSv2 support for `add_cloud_metadata` processor on AWS. {issue}22101[22101] {pull}28285[28285]

*Filebeat*

- Add `timezone` config option to the `decode_cef` processor. {issue}27232[27232] {pull}27727[27727]
- Add `timezone` config option to the `syslog` input. {pull}27727[27727]
- Added support for parsing syslog dates containing a leading 0 (e.g. `Sep 01`) rather than a space. {pull}27775[27775]
- Add base64 Encode functionality to `httpjson` input. {pull}27681[27681]
- Add `join` and `sprintf` functions to `httpjson` input. {pull}27735[27735]
- Improve memory usage of line reader of `log` and `filestream` input. {pull}27782[27782]
- Add `ignore_empty_value` flag to `httpjson` `split` processor. {pull}27880[27880]
- Add support for passing a prefix on S3 bucket list mode for AWS-S3 input {pull}28252[28252] {issue}27965[27965]
- Update Cisco ASA/FTD ingest pipeline grok/dissect patterns for multiple message IDs. {issue}26869[26869] {pull}26879[26879]
- Add write access to `url.value` from `request.transforms` in `httpjson` input. {pull}27937[27937]
- Add Base64 encoded HMAC and UUID template functions to `httpjson` input {pull}27873[27873]
- Release checkpoint module as GA. {pull}27814[27814]
- Make aws-cloudwatch input GA. {pull}28161[28161]
- Move processing to ingest node for AWS vpcflow fileset. {pull}28168[28168]
- Release zoom module as GA. {pull}28106[28106]
- Add support for secondary object attribute handling in ThreatIntel MISP module {pull}28124[28124]
- Azure signinlogs - Add support for ManagedIdentitySignInLogs, NonInteractiveUserSignInLogs, and ServicePrincipalSignInLogs. {issue}23653[23653]
- Add `base64Decode` and `base64DecodeNoPad` functions to `httpsjon` templates. {pull}28385[28385]
- Add 'early_limit' config option for Rate-Limiting `httpjson`. Default rate-limiting for Okta will start when remaining is `1`. {pull}28513[28513]
- Add latency config option for `aws-cloudwatch` input. {pull}28509[28509]
- Added proxy support to `threatintel/malwarebazaar`. {pull}28533[28533]
- Sophos UTM: Support logs containing hostname in Syslog header. {pull}28638[28638]
- Moving Oracle Filebeat module to GA. {pull}28754[28754]
- Add support in `aws-s3` input for S3 notification from SNS to SQS. {pull}28800[28800]
- Add support in `aws-s3` input for custom script parsing of S3 notifications. {pull}28946[28946]
- Improve error handling in `aws-s3` input for malformed S3 notifications. {issue}28828[28828] {pull}28946[28946]
- `filestream` and `log` inputs accept null (`\u0000`) as line terminator. {pull}28998[28998]

*Heartbeat*

- Support JSON expressions / validation of JSON arrays. {pull}28073[28073]
- Experimental `run once` mode. {pull}25972[25972]
- Add `keyword` multi-field mapping for `synthetics.step.name`. {pull}28452[28452]

*Metricbeat*

- Enable `journald` input type in Filebeat. {issue}7955[7955] {pull}27351[27351]
- Added a new beta `enterprisesearch` module for Elastic Enterprise Search {pull}27549[27549]
- Register additional name for `storage` metricset in the azure module. {pull}28447[28447]
- Update reference to gosigar pacakge for filesystem windows fix. {pull}28909[28909]
- Override `Host()` on statsd MetricSet {pull}29103[29103]
- Add Linux pressure metricset {pull}27355[27355]
- Add User-Agent header to HTTP requests. {issue}18160[18160] {pull}27509[27509]

*Functionbeat*

- Add support for AWS Kinesis record deaggregation {pull}28241[28241]

*Winlogbeat*

- Add support for event language selection from config file {pull}19818[19818]

==== Deprecated

*Affecting all Beats*

- Deprecate `setup.template.type`. In the future Beats will load data streams instead of regular indices.

*Filebeat*

- Deprecate `log` input in favour of `filestream` input. {pull}28623[28623]


[[release-notes-7.15.2]]
=== Beats version 7.15.2
https://github.com/elastic/beats/compare/v7.15.1\...v7.15.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Beats dashboards use custom index when `setup.dashboards.index` is set. {issue}21232[21232] {pull}27901[27901]
- Fix handling of float data types within processors. {issue}28279[28279] {pull}28280[28280]
- Allow `clone3` syscall in seccomp filters. {pull}28117[28117]
- Remove unnecessary escaping step in dashboard loading, so they can be displayed in Kibana. {pull}28395[28395]
- Fix AWS proxy_url config from url to string type.  {pull}28725[28725]
- Fix `fingerprint` processor to give it access to the `@timestamp` field. {issue}28683[28683]

*Filebeat*

- Fix initialization of http client in Cloudfoundry input. {issue}28271[28271] {pull}28277[28277]
- Fix aws-s3 input by checking if GetObject API call response content type exists. {pull}28457[28457]
- Set `url` as a pointer in the `httpjson` template context to ensure access to all methods. {pull}28695[28695]
- Fix `google_workspace` documentation links. {pull}28657[28657]

*Metricbeat*

- Divide RDS metric cpu.total.pct by 100. {pull}28456[28456]

*Packetbeat*

- Handle truncated DNS records more gracefully. {issue}21495[21495] {pull}28297[28297]
- Fix data stream name for network flows when running under Elastic Agent and Fleet. {pull}28408[28408]

[[release-notes-7.15.1]]
=== Beats version 7.15.1
https://github.com/elastic/beats/compare/v7.15.0\...v7.15.1[View commits]

==== Bugfixes

*Filebeat*

- Update Sophos xg module pipeline to deal with missing `date` and `time` fields. {pull}27834[27834]
- sophos/xg fileset: Add missing pipeline for System Health logs. {pull}27827[27827] {issue}27826[27826]

*Metricbeat*

- Add support for kube-state-metrics v2.0.0. {pull}27552[27552]

[[release-notes-7.15.0]]
=== Beats version 7.15.0
https://github.com/elastic/beats/compare/v7.14.2\...v7.15.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Loading Kibana assets (dashboards, index templates) rely on Saved Object API. So to provide a reliable service, Beats can only import and export dashboards using at least Kibana 7.15. {issue}20672[20672] {pull}27220[27220]

*Filebeat*

- Remove all alias fields pointing to ECS fields from modules. This affects the Suricata and Traefik modules. {issue}10535[10535] {pull}26627[26627]
- Fix Crowdstrike ingest pipeline that was creating flattened `process` fields. {issue}27622[27622] {pull}27623[27623]
- Rename `log.path` to `log.file.path` in filestream to be consistent with `log` input and ECS. {pull}27761[27761]

*Heartbeat*
- Remove long deprecated `watch_poll` functionality. {pull}27166[27166]
- Fix inconsistency in `event.dataset` values between heartbeat and fleet by always setting this value to the monitor type / fleet dataset. {pull}27535[27535]

*Metricbeat*

- Fix Elasticsearch jvm.gc.collectors.old being exposed as young {issue}19636[19636] {pull}26616[26616]

==== Bugfixes

*Affecting all Beats*

- Improve `perfmon` metricset performance. {pull}26886[26886]
- Preserve annotations in a kubernetes namespace metadata {pull}27045[27045]
- Fix build constraint that caused issues with doc builds. {pull}27381[27381]
- Do not try to load ILM policy if `check_exists` is `false`. {pull}27508[27508] {issue}26322[26322]
- Fix bug with cgroups hierarchy override path in cgroups {pull}27620[27620]
- Beat `setup kibana` command may use the elasticsearch API key defined in `output.elasticsearch.api_key`. {issue}24015[24015] {pull}27540[27540]
- Fix `decode_xml` handling of array merging when using `to_lower: true`. {pull}27922[27922]
- Separate namespaces for V1 and V2 controller paths {pull}27676[27676]
- Do not try to load ILM policy if `check_exists` is `false`. {pull}27508[27508] {issue}26322[26322]
- Kubernetes autodiscover fails in node scope if node name cannot be discovered {pull}26947[26947]

*Auditbeat*

- File Integrity Module: Honor `include_files` when doing initial scan. {issue}27273[27273] {pull}27722[27722]

*Filebeat*

- Update Filebeat compatibility function to remove processor description field on ES < 7.9.0 {pull}27774[27774]
- Make filestream events ECS compliant. {issue}27776[27776]

*Metricbeat*

- Allow metric prefix override per service in gcp module. {pull}26960[26960]
- Update metrics configuration and dashboards after changes in the Azure Monitor {pull}27520[27520]

*Winlogbeat*

- Fix an issue with message template caching in the `wineventlog-experimental` API implementation. {pull}26826[26826]

==== Added

*Affecting all Beats*

- Add proxy support for AWS functions. {pull}26832[26832]
- Added policies to the Elasticsearch output for non indexible events {pull}26952[26952]
- Add `logging.metrics.namespaces` config option to control what metric groups are reported in logs. {pull}25727[25727]
- Add sha256 digests to RPM packages. {issue}23670[23670]
- Add new 'offline' docker image for Elastic Agent. {pull}27052[27052]
- Add cgroups V2 support {pull}27242[27242]
- Update ECS field definitions to ECS 1.11.0. {pull}27107[27107]
- The disk queue is now GA. {pull}27515[27515]
- Add `daemonset.name` in pods controlled by DaemonSets {pull}26808[26808], {issue}25816[25816]

*Filebeat*

- Add new template functions and `value_type` parameter to `httpjson` transforms. {pull}26847[26847]
- Add support to merge registry updates in the filestream input across multiple ACKed batches in case of backpressure in the registry or disk. {pull}25976[25976]
- Add support to `decode_cef` for MAC addresses that do not contain separator characters. {issue}27050[27050] {pull}27109[27109]
- Add new `hmac` template function for httpjson input {pull}27168[27168]
- Update `tags` and `threatintel.indicator.provider` fields in `threatintel.anomali` ingest pipeline {issue}24746[24746] {pull}27141[27141]
- Move AWS module and filesets to GA. {pull}27428[27428]
- Update ecs.version to ECS 1.11.0. {pull}27107[27107]
- Add option for S3 input to work without SQS notification {issue}18205[18205] {pull}27332[27332]

*Metricbeat*

- Move openmetrics module to oss. {pull}26561[26561]
- Fix release state of kubernetes metricsets. {pull}26864[26864]
- Add `gke` metricset collection to `gcp` module {pull}26824[26824]
- Added `statsd.mappings` configuration for Statsd module {pull}26220[26220]
- Added Airflow lightweight module {pull}26220[26220]
- Add state_job metricset to Kubernetes module{pull}26479[26479]
- Bump AWS SDK version to v0.24.0 for WebIdentity authentication flow {issue}19393[19393] {pull}27126[27126]


[[release-notes-7.14.2]]
=== Beats version 7.14.2
https://github.com/elastic/beats/compare/v7.14.1\...v7.14.2[View commits]

==== Bugfixes

*Filebeat*

- Auditd module: Fix the top exec commands dashboard visualization. {pull}27638[27638]
- Store offset in `log.offset` field of events from the filestream input. {pull}27688[27688]
- Fix `httpjson` input rate limit processing and documentation. {pull}27739[27739]

[[release-notes-7.14.1]]
=== Beats version 7.14.1
https://github.com/elastic/beats/compare/v7.14.0\...v7.14.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Allow conditional processing in `decode_xml` and `decode_xml_wineventlog`. {pull}27159[27159]

*Filebeat*

- Convert the o365 module's `client.port` and `source.port` to numbers (from strings) in events. {pull}22939[22939]
- Fix the Snyk module to work with the new API changes. {pull}27358[27358]
- Fix a bug in `http_endpoint` that caused numbers encoded as strings. {issue}27382[27382] {pull}27480[27480]

*Metricbeat*

- Change `server_status_path` default setting to `nginx_status` for the `nginx` module. {pull}26642[26642]
- Change `startTime` and `endTime` of `GetMetricData` API in cloudwatch metricset to be only one collection period apart. {pull}27327[27327]
- Fix cloudwatch metricset collecting duplicate data points. {pull}27248[27248]
- Add percent formatters to system/process. {pull}27374[27374]
- Fix instance machineType reporting in compute metricset of GCP module. {pull}27363[27363]

==== Added

*Filebeat*

- Update Elasticsearch module's ingest pipeline for parsing new deprecation logs. {issue}26857[26857] {pull}26880[26880]

[[release-notes-7.14.0]]
=== Beats version 7.14.0
https://github.com/elastic/beats/compare/v7.13.4\...v7.14.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Removed beats central management {pull}25696[25696], {issue}23908[23908]
- MacOSX minimum supported version set to 10.14 {issue}24193[24193]

*Filebeat*

- Change logging in logs input to structure logging. Some log message formats have changed. {pull}25299[25299]
- All url.* fields apart from url.original in the Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules are now url unescaped due to using the Elasticsearch uri_parts processor. {pull}24699[24699]
- Change source field for `event.action` in `fortinet.firewall` module to `fortinet.firewall.action` instead of `fortinet.firewall.eventtype`. {pull}24816[24816]
- threatintel module: Changed the type of `threatintel.indicator.first_seen` from `keyword` to `date`. {pull}26765[26765]

*Heartbeat*

- Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. {pull}25808[25808]

*Metricbeat*

- Adjust host fields to adopt new names from 1.9.0 ECS. {pull}24312[24312]

==== Bugfixes

*Affecting all Beats*

- Omit full index template from errors that occur while loading the template. {pull}25743[25743]
- In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively.
- Fix encoding errors when using the disk queue on nested data with multi-byte characters {pull}26484[26484]

*Auditbeat*

- file_integrity: Create fsnotify watcher only when starting file_integrity module {pull}19505[19505]
- system/socket: Fix kprobe grouping to allow running more than one instance. {pull}20325[20325]
- system/socket: Fixed a crash due to concurrent map read and write. {issue}21192[21192] {pull}21690[21690]
- auditd: Fix an error condition causing a lot of `audit_send_reply` kernel threads being created. {pull}22673[22673]
- system/socket: Fixed start failure when run under config reloader. {issue}20851[20851] {pull}21693[21693]
- system/socket: Having some CPUs unavailable to Auditbeat could cause startup errors or event loss. {pull}22827[22827]

*Filebeat*

- Fix mapping of `fortinet.firewall.mem` as integer. {pull}19335[19335]
- Add `shared_credential_file` to cloudtrail config {issue}15652[15652] {pull}15656[15656]
- Fix integer overflow in S3 offsets when collecting very large files. {pull}22523[22523]
- Fix issue with m365_defender, when parsing incidents that has no alerts attached: {pull}25421[25421]
- Fix default config template values for paths on oracle module: {pull}26276[26276]
- Fix Elasticsearch compatibility for modules that use `copy_from` in `set` processors. {issue}26629[26629]
- Change type of max_bytes in all configs to be cfgtype.ByteSize {pull}26699[26699]
- Change `checkpoint.source_object` from Long to Keyword. {issue}25124[25124] {pull}25145[25145]
- Fix Nginx module pipelines. {issue}19088[19088] {pull}24699[24699]
- Fix incorrect field name appending to `related.hash` in `threatintel.abusechmalware` ingest pipeline. {issue}25151[25151] {pull}25674[25674]
- Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148]
- Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675]
- Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
- Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508]
- Fix Suricata metadata fields breaking visualizations, moved out of flattened datatype. {pull}26710[26710]
- Fix `httpjson` template data key for `url.params`. {pull}26848[26848]
- Cisco asa/ftd: Fix reversed usage of observer ingress and egress interfaces. {pull}26265[26265]
- Fix `aws.s3access` pipeline when remote IP is a `-`. {issue}26913[26913] {pull}26940[26940]
- Fix service name in aws-cloudwatch input from cloudwatchlogs to logs. {pull}27007[27007]

*Heartbeat*

- Add Context to otherwise ambiguous HTTP body read errors. {pull}25499[25499]

*Metricbeat*

- Major refactor of system/cpu and system/core metrics. {pull}25771[25771]
- Fix GCP Project ID being ingested as `cloud.account.id` in `gcp.billing` module {issue}26357[26357] {pull}26412[26412]
- Fix memory leak in SQL module when database is not available. {issue}25840[25840] {pull}26607[26607]
- Fix aws metric tags with resourcegroupstaggingapi paginator.  {issue}26385[26385] {pull}26443[26443]
- Fix quoting in GCP billing table name  {issue}26855[26855] {pull}26870[26870]
- Recover `service.address` field in vsphere module {issue}26902[26902] {pull}26904[26904]

*Winlogbeat*

- Fix `related.ip` field in renameCommonAuthFields {pull}24892[24892]

*Functionbeat*

- Expose region in AWS configuration so Functionbeat can deploy the Lambda in the correct place. {pull}26523[26523]

==== Added

*Affecting all Beats*

- Add support for defining explicitly named dynamic templates without path/type match criteria {pull}25422[25422]
- Improve ES output error insights. {pull}25825[25825]
- Add orchestrator.cluster.name/url fields as k8s metadata {pull}26056[26056]
- Libbeat: report beat version to monitoring. {pull}26214[26214]
- Ensure common proxy settings support in HTTP clients: `proxy_disabled`, `proxy_url`, `proxy_headers` and typical environment variables `HTTP_PROXY`, `HTTPS_PROXY`, `NOPROXY`. {pull}25219[25219]

*Filebeat*

- Update PanOS module to parse Global Protect & User ID logs. {issue}24722[24722] {issue}24724[24724] {pull}24927[24927]
- Add HMAC signature validation support for http_endpoint input. {pull}24918[24918]
- Add new grok pattern for iptables module for Ubiquiti UDM {issue}25615[25615] {pull}25616[25616]
- Add multiline support to aws-s3 input. {issue}25249[25249] {pull}25710[25710] {pull}25873[25873]
- Add monitoring metrics to the `aws-s3` input. {pull}25711[25711]
- Added `network.direction` fields to Zeek and Suricata modules using the `add_network_direction` processor {pull}24620[24620]
- Add Content-Type override to aws-s3 input. {issue}25697[25697] {pull}25772[25772]
- In Cisco Umbrella fileset add users from cisco.umbrella.identities to related.user. {pull}25776[25776]
- Add fingerprint processor to generate fixed ids for `google_workspace` events. {pull}25841[25841]
- Update PanOS module to parse HIP Match logs. {issue}24350[24350] {pull}25686[25686]
- Support MongoDB 4.4 in filebeat's MongoDB module. {issue}20501[20501] {pull}24774[24774]
- Enhance GCP module to populate orchestrator.* fields for GKE / K8S logs {pull}25368[25368]
- Add log_group_name_prefix config into aws-cloudwatch input. {pull}26187[26187]
- Move Filebeat azure module to GA. {pull}26114[26114] {pull}26168[26168]
- Make `filestream` input GA. {pull}26127[26127]
- http_endpoint: Support multiple documents in a single request by POSTing an array or NDJSON format. {pull}25764[25764]
- Add new `parser` to `filestream` input: `container`. {pull}26115[26115]
- Add support for ISO8601 timestamps in Zeek fileset {pull}25564[25564]
- Add possibility to include headers in resulting docs and preserve the original event in http_endpoint input {pull}26279[26279]
- Add `preserve_original_event` option to `o365audit` input. {pull}26273[26273]
- Add `log.flags` to events created by the `aws-s3` input. {pull}26267[26267]
- Add `include_s3_metadata` config option to the `aws-s3` input for including object metadata in events. {pull}26267[26267]
- RFC 5424 and UNIX socket support in the Syslog input are now GA {pull}26293[26293]
- Update grok patterns for HA Proxy module {issue}25827[25827] {pull}25835[25835]
- Update PanOS module's date processor formats to parse `strict_date_optional_time_nanos`. {issue}26033[26033] {pull}26158[26158]
- Update Okta module to parse additional fields to `okta.debug_context.debug_data`. {issue}25689[25689] {pull}25818[25818]
- Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350]

- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
- Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
- Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
- Update `fortinet` ingest pipelines. {issue}22136[22136] {issue}25254[25254] {pull}24816[24816]
- Release Filebeat Stack Monitoring modules as GA {pull}26226[26226]
- Use default add_locale for fortinet.firewall {issue}20300[20300] {pull}26524[26524]

*Heartbeat*

- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
- Add `proxy_headers` to HTTP monitor. {pull}25219[25219]
- Suppress too many bad message error logs when reading from corrupted journal for 5 seconds. {pull}26224[26224]
- Add `replicas.ready` field to state_statefulset in Kubernetes module {pull}26088[26088]

*Metricbeat*

- Refactor `state_*` metricsets to share response from endpoint. {pull}25640[25640]
- Add server id to zookeeper events. {pull}25550[25550]
- Add additional network metrics to docker/network {pull}25354[25354]
- Migrate ec2 metricsets to use cloudwatch input. {pull}25924[25924]
- Reduce number of requests done by kubernetes metricsets to kubelet. {pull}25782[25782]
- Migrate rds metricsets to use cloudwatch input. {pull}26077[26077]
- Migrate sqs metricsets to use cloudwatch input. {pull}26117[26117]
- Collect linked account information in AWS billing. {pull}26285[26285]
- Add total CPU to vSphere virtual machine metrics. {pull}26167[26167]
- Add AWS Kinesis metricset. {pull}25989[25989]
- Add Cluster filter on ECS Kubernetes overview dashboard and corresponding section on Kubernetes module documentation page. {pull}26919[26919]

*Packetbeat*

- Add `url.extension` to HTTP events {issue}25990[25990] {pull}25999[25999]

*Winlogbeat*

- Changed the log level of the "Successfully published events" message from `info` to `debug` to reduce verbosity of the `info` logging level. To track event log reader activity use the `published_events` metric. {pull}25617[25617]

==== Deprecated

*Filebeat*

- Deprecate the MISP module. The Threat Intel module should be used instead. {issue}25240[25240]


[[release-notes-7.13.4]]
=== Beats version 7.13.4
https://github.com/elastic/beats/compare/v7.13.3\...v7.13.4[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix `add_process_metadata` processor complaining about valid pid fields not being valid integers. {pull}26829[26829] {issue}26830[26830]

*Auditbeat*

- Do not close filestream harvester if an unexpected error is returned when `close.on_state_change.*` is enabled. {pull}26411[26411]

*Filebeat*

- Fix Elasticsearch compatibility for modules that use `type: ip` with `convert` processors. {issue}26629[26629] {pull}26676[26676]
- Fix Elasticsearch compatibility for modules that use the `network_direction` processor. {issue}26629[26629] {pull}26676[26676]
- Fix Elasticsearch compatibility for modules that use the `registered_domain` processor. {issue}26629[26629] {pull}26676[26676]


[[release-notes-7.13.3]]
=== Beats version 7.13.3
https://github.com/elastic/beats/compare/v7.13.2\...v7.13.3[View commits]

==== Bugfixes

*Filebeat*

- Fix bug in aws-s3 input where the end of gzipped log files might have been discarded. {pull}26260[26260]
- Clone value when copy fields in processors to avoid crash. {issue}19206[19206] {pull}20500[20500]
- Fix bug in `httpjson` that prevented `first_event` getting updated. {pull}26407[26407]
- Fix bug in the Syslog input that misparsed rfc5424 days starting with 0. {pull}26419[26419]

[[release-notes-7.13.2]]
=== Beats version 7.13.2
https://github.com/elastic/beats/compare/v7.13.1\...v7.13.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix ILM alias creation when write alias exists and initial index does not exist. {pull}26146[26146]
- Fix ILM setup log reporting that a policy or an alias was created, even though the creation of any resource was disabled. {issue}24046[24046] {pull}24480[24480]
- Fix ILM alias not being created if `setup.ilm.check_exists: false` and `setup.ilm.overwrite: true` has been configured. {pull}24480[24480]
- Allow cgroup self-monitoring to see alternate `hostfs` paths. {pull}24334[24334]
- Fix `make setup` instructions for a new Beat. {pull}24944[24944]
- Fix out-of-date FreeBSD vagrantbox. {pull}25652[25652]
- Fix handling of `file_selectors` in aws-s3 input. {pull}25792[25792]
- Include date separator in the filename prefix of `dateRotator` to make sure nothing gets purged accidentally. {pull}26176[26176]

*Auditbeat*

- auditd: Fix kernel deadlock when netlink congestion causes "no buffer space available" errors. {issue}26031[26031] {pull}26032[26032]

*Filebeat*

- o365: Avoid mapping exception for `Parameters` and `ExtendedProperties` fields of string type. {pull}26164[26164]

[[release-notes-7.13.1]]
=== Beats version 7.13.1
https://github.com/elastic/beats/compare/v7.13.0\...v7.13.1[View commits]

==== Bugfixes

*Auditbeat*

- Mitigate deadlock is aws-s3 input when SQS visibility timeout is exceeded. {issue}25750[25750]
- Fix httpjson cursor override with empty values by adding `ignore_empty_value` option. {pull}25802[25802]

*Filebeat*

- Improve inode reuse handling by removing state for removed files more eagerly from the internal state table in the logs inputs. {pull}25756[25756]

[[release-notes-7.13.0]]
=== Beats version 7.13.0
https://github.com/elastic/beats/compare/v7.12.1\...v7.13.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Use alias to report container image in k8s metadata. {pull}24380[24380]
- Set `cleanup_timeout` to zero by default in docker and kubernetes autodiscover in all beats except Filebeat where it is kept to 60 seconds. {pull}24681[24681]
- Update to ECS 1.9.0. {pull}24909[24909]

*Filebeat*

- Changes filebeat httpjson input's append transform to create a list even with only a single value{pull}25074[25074]
- Deprecated the cyberark module (replaced by cyberarkpas). {issue}25261[25261] {pull}25505[25505]

*Metricbeat*

- Store `cloudfoundry.container.cpu.pct` in decimal form and as `scaled_float`. {pull}24219[24219]
- Remove `index_stats.created` field from Elasticsearch/index Metricset {pull}25113[25113]

==== Bugfixes

*Affecting all Beats*

- Fix events being dropped if they contain a floating point value of NaN or Inf. {pull}25051[25051]
- Fix templates being overwritten if there was an error when check for the template existance. {pull}24332[24332]
- Add `expand_keys` to the list of permitted config fields for `decode_json_fields` {pull}24862[24862]
- Fix discovery of short-living and failing pods in Kubernetes autodiscover {issue}22718[22718] {pull}24742[24742]
- Fix panic when overwriting metadata {pull}24741[24741]
- Fix role_arn to work with access keys for AWS. {pull}25446[25446]
- Fix `community_id` processor so that ports greater than 65535 aren't valid. {pull}25409[25409]

*Auditbeat*

- Fix o365 module config when client_secret contains special characters. {issue}25058[25058]

*Filebeat*

- Fix date parsing in GSuite/login fileset. {issue}24694[24694]
- Improve Cisco ASA/FTD parsing of messages {pull}23766[23766]
  - Better support for identity FW messages.
  - Change network.bytes, source.bytes, and destination.bytes to long from integer since value can exceed integer capacity.
  - Add descriptions for various processors for easier pipeline editing in Kibana UI.
- Fix usage of unallowed ECS event.outcome values in Cisco ASA/FTD pipeline. {pull}24744[24744].
- Fix IPtables Pipeline and Ubiquiti dashboard. {issue}24878[24878] {pull}24928[24928]
- Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066]
- Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829]
- Fix o365 module config when client_secret contains special characters. {issue}25058[25058]
- Fix s3 input when there is a blank line in the log file. {pull}25357[25357]
- Remove space from field `sophos.xg.trans_src_ ip`. {issue}25154[25154] {pull}25250[25250]
- Fix `checkpoint.action_reason` when its a string, not a Long. {issue}25575[25575] {pull}25609[25609]
- Fix `fortinet.firewall.addr` when its a string, not an IP address. {issue}25585[25585] {pull}25608[25608]

*Metricbeat*

- Sort correctly the keys when accessing JMX through the Jolokia module {pull}25631[25631]
- Change lookup_fields from metricset.host to service.address {pull}15883[15883]
- Fix incorrect types of fields GetHits and Ops in NodeInterestingStats for Couchbase module in Metricbeat {issue}21021[21021] {pull}23287[23287]
- Fix GCP not able to request Cloudfunctions metrics if a region filter was set {pull}24218[24218]
- Fix type of `uwsgi.status.worker.rss` type. {pull}24468[24468]
- Accept text/plain type by default for prometheus client scraping. {pull}24622[24622]
- Use working set bytes to calculate the pod memory limit pct when memory usage is not reported (ie. Windows pods). {pull}25428[25428]
- Fix copy-paste error in libbeat docs. {pull}25448[25448]
- Fix azure billing dashboard. {pull}25554[25554]

*Winlogbeat*

- Change `event.code` and `winlog.event_id` from int to keyword. {pull}25176[25176]

==== Added

*Affecting all Beats*

- Add `wineventlog` schema to `decode_xml` processor. {issue}23910[23910] {pull}24726[24726]
- Add new ECS 1.9 field `cloud.service.name` to `add_cloud_metadata` processor. {pull}24993[24993]
- Libbeat: report queue capacity, output batch size, and output client count to monitoring. {pull}24700[24700]
- Add kubernetes.pod.ip field in kubernetes metadata. {pull}25037[25037]
- Discover changes in Kubernetes namespace metadata as soon as they happen. {pull}25117[25117]
- Add `decode_xml_wineventlog` processor. {issue}23910[23910] {pull}25115[25115]
- Add new setting `gc_percent` for tuning the garbage collector limits via configuration file. {pull}25394[25394]
- Add `unit` and `metric_type` properties to fields.yml for populating field metadata in Elasticsearch templates {pull}25419[25419]
- Add new option `suffix` to `logging.files` to control how log files are rotated. {pull}25464[25464]
- Validate that required functionality in Elasticsearch is available upon initial connection. {pull}25351[25351]

*Filebeat*

- Support X-Forwarder-For in IIS logs. {pull}19142[192142]
- Add support for logs generated by servers configured with `log_statement` and `log_duration` in PostgreSQL module. {pull}24607[24607]
- Added fifteen new message IDs to Cisco ASA/FTD pipeline. {pull}24744[24744]
- Added NTP fileset to Zeek module {pull}24224[24224]
- Add `proxy_url` config for httpjson v2 input. {issue}24615[24615] {pull}24662[24662]
- Change `okta.target` to `flattened` field type.  {issue}24354[24354] {pull}24636[24636]
- Added `http.request.id` to `nginx/ingress_controller` and `elasticsearch/audit`. {pull}24994[24994]
- Add `awsfargate` module to collect container logs from Amazon ECS on Fargate. {pull}25041[25041]
- New module `cyberarkpas` for CyberArk Privileged Access Security audit logs. {pull}24803[24803]
- Add `uri_parts` processor to Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules ingest pipelines. {issue}19088[19088] {pull}24699[24699]
- New module `zookeeper` for Zookeeper service and audit logs {issue}25061[25061] {pull}25128[25128]
- Add parsing for `haproxy.http.request.raw_request_line` field  {issue}25480[25480] {pull}25482[25482]
- Mark `filestream` input beta. {pull}25560[25560]
- Add User Agent Parser for Azure Sign In Logs Ingest Pipeline {pull}23201[23201]

*Heartbeat*

- Handle datastreams for fleet. {pull}24223[24223]
- Add --sandbox option for browser monitor. {pull}24172[24172]
- Support additional 'root' fields from synthetics. {pull}24770[24770]
- Browser zip_url source type. {pull}24714[24714]

*Metricbeat*

- Add support for Consul 1.9. {pull}24123[24123]
- Add support for defining metrics_filters for prometheus module in hints. {pull}24264[24264]
- Add support for PostgreSQL 10, 11, 12 and 13. {pull}24402[24402]
- Add support for SASL/SCRAM authentication to the Kafka module. {pull}24810[24810]

*Winlogbeat*

- Add support for sysmon v13 events 24 and 25. {issue}24217[24217] {pull}24945[24945]

[[release-notes-7.12.1]]
=== Beats version 7.12.1
https://github.com/elastic/beats/compare/v7.12.0\...v7.12.1[View commits]

==== Breaking changes

*Filebeat*

- Possible values for Netflow's locality fields (source.locality, destination.locality and flow.locality) are now `internal` and `external`, instead of `private` and `public`. {issue}24272[24272] {pull}24295[24295]

==== Bugfixes

*Affecting all Beats*

- Fix templates being overwritten if there was an error when check for the template existance. {pull}24332[24332]
- Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data {pull}17223[17223]
- Fix inode removal tracking code when files are replaced by files with the same name {pull}25002[25002]
- Fix `mage GenerateCustomBeat` instructions for a new beat {pull}17679[17679]
- Fix bug with annotations dedot config on k8s not used {pull}25111[25111]
- Fix negative Kafka partition bug {pull}25048[25048]

*Filebeat*

- Properly update offset in case of unparasable line. {pull}22685[22685]
- Fix Cisco ASA parser for message 722051. {pull}24410[24410]
- Fix `google_workspace` pagination. {pull}24668[24668]
- Fix netflow module ignoring detect_sequence_reset flag. {issue}24268[24268] {pull}24270[24270]
- Fix Cisco ASA parser for message 302022. {issue}24405[24405] {pull}24697[24697]
- Fix Cisco AMP `@metadata._id` calculation {issue}24717[24717] {pull}24718[24718]
- Fix date parsing in GSuite/login and Google Workspace/login filesets. {issue}24694[24694]
- Fix gcp/vpcflow module error where input type was defaulting to file. {pull}24719[24719]
- Improve PanOS parsing and ingest pipeline. {issue}22413[22413] {issue}22748[22748] {pull}24799[24799]
- Fix S3 input validation for non amazonaws.com domains. {issue}24420[24420] {pull}24861[24861]
- Fix google_workspace and okta modules pagination when next page template is empty. {pull}24967[24967]
- Fix gcp module field names to use gcp instead of googlecloud. {pull}25038[25038]

*Heartbeat*

- Fix panic when initialization of ICMP monitors fail twice. {pull}25073[25073]

*Metricbeat*

- Ignore unsupported derive types for filesystem metricset. {issue}22501[22501] {pull}24502[24502]


==== Added

*Filebeat*

- Updating field mappings for Cisco AMP module, fixing certain fields. {pull}24661[24661]
- Add support for upper case field names in Sophos XG module {pull}24693[24693]
- Add `fail_on_template_error` option for httpjson input. {pull}24784[24784]



[[release-notes-7.12.0]]
=== Beats version 7.12.0
https://github.com/elastic/beats/compare/v7.11.2\...v7.12.0[View commits]

==== Breaking changes

*Filebeat*

- Rename `s3` input to `aws-s3` input. {pull}23469[23469]

*Heartbeat*

- Refactor synthetics configuration to new syntax. {pull}23467[23467]

==== Bugfixes

*Affecting all Beats*

- Fix `nested` subfield handling in generated Elasticsearch templates. {issue}23178[23178] {pull}23183[23183]
- Fix CPU usage metrics on VMs with dynamic CPU config {pull}23154[23154]
- Allow configuring credential_profile_name and shared_credential_file when using role_arn. {pull}24174[24174]
- Fix panic with inline SSL when the certificate or key was smaller than 256 bytes. {issue}23820[23820] {pull}23858[23858]

*Auditbeat*

- system/login: Fixed offset reset on inode reuse. {pull}24414[24414]
- system/login: Add additional offset check for utmp files. {pull}24515[24515]

*Filebeat*

- CheckPoint Firewall module: Change event.severity JSON data type to a number because the field mapping is a `long`. {pull}23424[23424]
- Cisco IOS: Change icmp.type/code and igmp.type JSON data types to strings because the fields mappings are `keyword`. {pull}23424[23424]
- CrowdStrike Falcon: Change JSON field types to match the field mappings. {pull}23424[23424]
- Fortinet Firewall: Drop `fortinet.firewall.assignip` when the value is "N/A". {pull}23424[23424]
- Juniper SRX: Change JSON field types to match the field mappings. {pull}23424[23424]
- Suricata EVE: Convert `suricata.eve.flow_id` to string because the field is a keyword in the mapping. {pull}23424[23424]
- Zeek DNS: Ignore failures in data type conversions. And change `dns.id` JSON field to a string to match its `keyword` mapping. {pull}23424[23424]
- Update `filestream` reader offset when a line is skipped. {pull}23417[23417]
- Add check for empty values in azure module. {pull}24156[24156]
- Change the `event.created` in Netflow events to be the time the event was created by Filebeat
- Fix Zoom module parameters for basic auth and url path. {pull}23779[23779]
- Use rfc6587 framing for fortinet firewall and clientendpoint filesets when transferring over tcp. {pull}23837[23837]
- Fix httpjson input logging so it doesn't conflict with ECS. {pull}23972[23972]
- Fix Logstash module handling of logstash.log.log_event.action field. {issue}20709[20709]
- aws/s3access dataset was populating event.duration using the wrong unit. {pull}23920[23920]
- Zoom module pipeline failed to ingest some chat_channel events. {pull}23904[23904]
- Fix Netlow module issue with missing `internal_networks` config parameter. {issue}24094[24094] {pull}24110[24110]
- in httpjson input using encode_as "application/x-www-form-urlencoded" now sets Content-Type correctly {issue}24331[24331] {pull}24336[24336]
- Fix default `scope` in `add_nomad_metadata`. {issue}24559[24559]

*Metricbeat*

- Add stack monitoring section to elasticsearch module documentation {pull}#23286[23286]
- Fix ec2 metricset fields.yml and the integration test {pull}23726[23726]
- Unskip s3_request integration test. {pull}23887[23887]
- Add system.hostfs configuration option for system module. {pull}23831[23831]

==== Added

*Affecting all Beats*

- Honor kube event resysncs to handle missed watch events {pull}22668[22668]
- Add autodiscover provider and metadata processor for Nomad. {pull}14954[14954] {pull}23324[23324]
- Add `processors.rate_limit.n.dropped` monitoring counter metric for the `rate_limit` processor. {pull}23330[23330]
- Deprecate aws_partition config parameter for AWS, use endpoint instead. {pull}23539[23539]
- Update the baseline version of Sarama (Kafka support library) to 1.27.2. {pull}23595[23595]
- Add kubernetes.volume.fs.used.pct field. {pull}23564[23564]
- Add the `enable_krb5_fast` flag to the Kafka output to explicitly opt-in to FAST authentication. {pull}23629[23629]
- Added new decode_xml processor to libbeat that is available to all beat types. {pull}23678[23678]
- Add deployment name in pod's meta. {pull}23610[23610]
- Added ECS 1.8 `host.os.type` field to `add_host_metadata` processor. {pull}23513[23513]
- Add `selector` information in Kubernetes services' metadata. {pull}23730[23730]

*Auditbeat*

- Improve file_integrity monitoring when a file is created/deleted in quick succession. {issue}17347[17347] {pull}22170[22170]
- system/host: Add new ECS 1.8 field `os.type` in `host.os.type`. {pull}23513[23513]
- Update Auditbeat auditd module to ECS 1.8 {pull}23594[23594] {issue}23118[23118]

*Filebeat*

- Add parsing of tcp flags to AWS vpcflow fileset {issue}228020[22820] {pull}23157[23157]
- Added support for first_event context in Filebeat httpjson input {pull}23437[23437]
- Adding Threat Intel module {pull}21795[21795]
- Added username parsing from Cisco ASA message 302013. {pull}21196[21196]
- Added `encode_as` and `decode_as` options to httpjson along with pluggable encoders/decoders {pull}23478[23478]
- Added feature to modules to adapt Ingest Node pipelines for compatibility with older Elasticsearch versions by removing unsupported processors. {pull}23763[23763]
- Added support for Cisco AMP API as a new fileset. {pull}22768[22768]
- Added RFC6587 framing option for tcp and unix inputs {issue}23663[23663] {pull}23724[23724]
- Added `application/x-ndjson` as decode option for httpjson input {pull}23521[23521]
- Added `application/x-www-form-urlencoded` as encode option for httpjson input {pull}23521[23521]
- Move aws-s3 input to GA. {pull}23631[23631]
- Populate `source.mac` and `destination.mac` for Suricata EVE events. {issue}23706[23706] {pull}23721[23721]
- Added string splitting for httpjson input {pull}24022[24022]
- Added Signatures fileset to Zeek module {pull}23772[23772]
- Upgrade Cisco ASA/FTD/Umbrella to ECS 1.8.0. {pull}23819[23819]
- Add new ECS user and categories features to google_workspace/gsuite {issue}23118[23118] {pull}23709[23709]
- Move crowdstrike JS processor to ingest pipelines and upgrade to ECS 1.8.0 {issue}23118[23118] {pull}23875[23875]
- Update Filebeat auditd dataset to ECS 1.8.0. {pull}23723[23723] {issue}23118[23118]
- Updated microsoft defender_atp and m365_defender to ECS 1.8. {pull}23897[23897] {issue}23118[23118]
- Updated o365 module to ECS 1.8. {issue}23118[23118] {pull}23896[23896]
- Upgrade CEF module to ECS 1.8.0. {pull}23832[23832]
- Upgrade fortinet/firewall to ECS 1.8 {issue}23118[23118] {pull}23902[23902]
- Upgrade Zeek to ECS 1.8.0. {issue}23118[23118] {pull}23847[23847]
- Updated azure module to ECS 1.8. {issue}23118[23118] {pull}23927[23927]
- Update aws/s3access to ECS 1.8. {issue}23118[23118] {pull}23920[23920]
- Upgrade panw module to ECS 1.8 {issue}23118[23118] {pull}23931[23931]
- Updated aws/cloudtrail fileset to ECS 1.8. {issue}23118[23118] {pull}23911[23911]
- Upgrade juniper/srx to ECS 1.8.0. {issue}23118[23118] {pull}23936[23936]
- Update mysqlenterprise module to ECS 1.8. {issue}23118[23118] {pull}23978[23978]
- Upgrade sophos/xg fileset to ECS 1.8.0. {issue}23118[23118] {pull}23967[23967]
- Upgrade system/auth to ECS 1.8 {issue}23118[23118] {pull}23961[23961]
- Upgrade elasticsearch/audit to ECS 1.8 {issue}23118[23118] {pull}24000[24000]
- Upgrade okta to ECS 1.8.0 and move js processor to ingest pipeline {issue}23118[23118] {pull}23929[23929]
- Update zoom module to ECS 1.8. {pull}23904[23904] {issue}23118[23118]
- Add fileset to ingest PostgreSQL CSV logs. {pull}23334[23334]
- Add beta support for RFC 5424 to the Syslog input. {pull}23954[23954]

*Heartbeat*

- Bundle synthetics dependencies with Heartbeat docker image. {pull}23274[23274]

*Heartbeat*

- Update Journalbeat to ECS 1.8. {pull}23737[23737]

*Metricbeat*

- Enrich events of `state_service` metricset with Kubernetes services' metadata. {pull}23730[23730]
- Add support for Darwin/arm M1. {pull}24019[24019]
- Check fields are documented in AWS metricsets. {pull}23887[23887]
- Add container.image.name and containe.name ECS fields for state_container. {pull}23802[23802]
- Add support for the MemoryPressure, DiskPressure, OutOfDisk and PIDPressure status conditions in state_node. {pull}23905[23905]

*Packetbeat*

- Upgrade to ECS 1.8.0. {pull}23783[23783]
- Add `event.type: [connection]` to flow events and include `end` for final flows. {pull}24564[24564]

*Functionbeat*

- Provide more ways to set AWS credentials. {issue}12464[12464] {pull}23344[23344]
- Add support for multiple regions {pull}21065[21065]

*Heartbeat*

- Add support for script processor. {pull}23229[23229]

*Winlogbeat*

- Add Audit and Authentication Policy Change Events and related.ip information {pull}20684[20684]
- Add new ECS 1.8 improvements. {pull}23563[23563]
- Remove deprecated eventlogging API that was used for Windows XP/2003 and associated unused code. {pull}24463[24463]

==== Deprecated

*Affecting all Beats*

- Selecting `full` in `ssl.verification_mode` option will not treat CommonName field in x509 certificates as a hostname when Subject Alternative Name is not present from v8.0. Please update your certificates so it contains at least one DNSName instead of relying on CommonName in the new major version of Beats.


[[release-notes-7.11.2]]
=== Beats version 7.11.2
https://github.com/elastic/beats/compare/v7.11.1\...v7.11.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix issue discovering docker containers and metadata after reconnections {pull}24318[24318]

*Filebeat*

- Fix Okta default date formatting. {issue}24018[24018] {pull}24025[24025]
- Fix aws/vpcflow generating errors for empty logs or unidentified formats. {pull}24167[24167]
- Add `nodes` to filebeat-kubernetes.yaml ClusterRole. {issue}24051[24051] {pull}24052[24052]

*Metricbeat*

- Add check for iis/application_pool metricset for nil worker process id values. {issue}23605[23605] {pull}23647[23647]

[[release-notes-7.11.1]]
=== Beats version 7.11.1
https://github.com/elastic/beats/compare/v7.11.0\...v7.11.1[View commits]

==== Bugfixes

*Filebeat*

- Fix goroutines leak with some inputs in autodiscover. {pull}23722[23722]
- Fix various processing errors in the Suricata module. {pull}23236[23236]

*Elastic Logging Plugin*

- Fix out of date CLI flags on docs. {pull}23628[23628]

[[release-notes-7.11.0]]
=== Beats version 7.11.0
https://github.com/elastic/beats/compare/v7.10.2\...v7.11.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs. {pull}21179[21179]
- Update to ECS 1.7.0. {pull}22571[22571]
- Add support for SCRAM-SHA-512 and SCRAM-SHA-256 in Kafka output. {pull}12867[12867]

*Auditbeat*

- Use ECS 1.7 ingress/egress network directions instead of inbound/outbound for system/socket. {pull}22991[22991]
- Use ingress/egress instead of inbound/outbound for ECS 1.7 in auditd module. {pull}23000[23000]

*Filebeat*

- Add fileset to ingest Kibana's ECS audit logs. {pull}22696[22696]
- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095]
- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571]
- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]

*Heartbeat*
- Adds negative body match. {pull}20728[20728]

*Metricbeat*

- Change cloud.provider from googlecloud to gcp. {pull}21775[21775]
- Rename googlecloud module to gcp module. {pull}22246[22246]
- Use ingress/egress instead of inbound/outbound for system/socket metricset. {pull}22992[22992]
- Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. {pull}23335[23335]

*Packetbeat*

- Update how Packetbeat classifies network directionality to bring it in line with ECS 1.7 {pull}22996[22996]

*Winlogbeat*

- Use ECS 1.7 ingress/egress instead of inbound/outbound network.direction in sysmon. {pull}22997[22997]

==== Bugfixes

*Affecting all Beats*

- Fix memory leak and events duplication in docker autodiscover and add_docker_metadata. {pull}21851[21851]
- Fix duplicated pod events in kubernetes autodiscover for pods with init or ephemeral containers. {pull}22438[22438]
- Fix FileVersion contained in Windows exe files. {pull}22581[22581]
- Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure {issue}12211[12211], {pull}13387[13387]
- Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss` as gauges (rather than counters). {pull}22877[22877]
- Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service {pull}22874[22874]
- Fix reporting of cgroup metrics when running under Docker {pull}22879[22879]
- Fix typo in config docs {pull}23185[23185]
- Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419]
- Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484]

*Auditbeat*

- file_integrity: stop monitoring excluded paths {issue}21278[21278] {pull}21282[21282]
- Note incompatibility of system/socket on ARM. {pull}23381[23381]

*Filebeat*

- Fix Zeek dashboard reference to `zeek.ssl.server.name` field. {pull}21696[21696]
- Fix network.direction logic in zeek connection fileset. {pull}22967[22967]
- Fix aws s3 overview dashboard. {pull}23045[23045]
- Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072]
- Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966]
- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126]
- Simplify regex for organization custom prefix in AWS/CloudTrail fileset. {issue}23203[23203] {pull}23204[23204]
- Fix syslog header parsing in infoblox module. {issue}23272[23272] {pull}23273[23273]
- Fix concurrent modification exception in Suricata ingest node pipeline. {pull}23534[23534]
- Fix handling of ModifiedProperties field in Office 365. {pull}23777[23777]

*Heartbeat*

- Fixed missing `tls` fields when connecting to https via proxy. {issue}15797[15797] {pull}22190[22190]

*Metricbeat*

- Change Session ID type from int to string {pull}22359[22359]
- Fix filesystem types on Windows in filesystem metricset. {pull}22531[22531]
- Fix failiures caused by custom beat names with more than 15 characters {pull}22550[22550]
- Update NATS dashboards to leverage connection and route metricsets {pull}22646[22646]
- Fix rate metrics in Kafka broker metricset by using last minute rate instead of mean rate. {pull}22733[22733]
- Update config in `windows.yml` file. {issue}23027[23027]{pull}23327[23327]
- Fix metric grouping for windows/perfmon module {issue}23489[23489] {pull}23505[23505]

*Packetbeat*

- Fix SIP parser logic related to line length check. {pull}23411[23411]


*Winlogbeat*

- Protect against accessing an undefined variable in Security module. {pull}22937[22937]
- Add source.ip validation for event ID 4778 in the Security module. {issue}19627[19627]

==== Added

*Affecting all Beats*

- Add istiod metricset. {pull}21519[21519]
- Add support for OpenStack SSL metadata APIs in `add_cloud_metadata`. {pull}21590[21590]
- Add cloud.account.id for GCP into add_cloud_metadata processor. {pull}21776[21776]
- Add proxy metricset for istio module. {pull}21751[21751]
- Add kubernetes.node.hostname metadata of Kubernetes node. {pull}22189[22189]
- Enable always add_resource_metadata for Pods and Services of kubernetes autodiscovery. {pull}22189[22189]
- Add add_resource_metadata option setting (always enabled) for add_kubernetes_metadata setting. {pull}22189[22189]
- Add support for ephemeral containers in kubernetes autodiscover and `add_kubernetes_metadata`. {pull}22389[22389] {pull}22439[22439]
- Added support for wildcard fields and keyword fallback in beats setup commands. {pull}22521[22521]
- Fix polling node when it is not ready and monitor by hostname {pull}22666[22666]
- Add `expand_keys` option to `decode_json_fields` processor and `json` input, to recusively de-dot and expand json keys into hierarchical object structures {pull}22849[22849]
- Update k8s client and release k8s leader lock gracefully {pull}22919[22919]
- Improve event normalization performance {pull}22974[22974]
- Add tini as init system in docker images {pull}22137[22137]
- Added "detect_mime_type" processor for detecting mime types {pull}22940[22940]
- Added "add_network_direction" processor for determining perimeter-based network direction. {pull}23076[23076]
- Added new `rate_limit` processor for enforcing rate limits on event throughput. {pull}22883[22883]
- Allow node/namespace metadata to be disabled on kubernetes metagen and ensure add_kubernetes_metadata honors host {pull}23012[23012]
- Improve equals check. {pull}22778[22778]

*Auditbeat*

- Add several improvements for auditd module for improved ECS field mapping {pull}22647[22647]
- Add ECS 1.7 `configuration` categorization in certain events in auditd module. {pull}23000[23000]

*Filebeat*


- Adding support for Oracle Database Audit Logs {pull}21991[21991]
- Add max_number_of_messages config into s3 input. {pull}21993[21993]
- Add SSL option to checkpoint module {pull}19560[19560]
- Added support for MySQL Enterprise audit logs. {pull}22273[22273]
- Rename googlecloud module to gcp module. {pull}22214[22214]
- Rename awscloudwatch input to aws-cloudwatch. {pull}22228[22228]
- Rename google-pubsub input to gcp-pubsub. {pull}22213[22213]
- Copy tag names from MISP data into events. {pull}21664[21664]
- Added TLS JA3 fingerprint, certificate not_before/not_after, certificate SHA1 hash, and certificate subject fields to Zeek SSL dataset. {pull}21696[21696]
- Add platform logs in the azure filebeat module. {pull}22371[22371]
- Added `event.ingested` field to data from the Netflow module. {pull}22412[22412]
- Improve panw ECS url fields mapping. {pull}22481[22481]
- Improve Nats filebeat dashboard. {pull}22726[22726]
- Add support for UNIX datagram sockets in `unix` input. {issues}18632[18632] {pull}22699[22699]
- Add `http.request.mime_type` for Elasticsearch audit log fileset. {pull}22975[22975]
- Add new httpjson input features and mark old config ones for deprecation {pull}22320[22320]
- Add configuration option to set external and internal networks for panw panos fileset {pull}22998[22998]
- Add `subbdomain` fields for rsa2elk modules. {pull}23035[23035]
- Add subdomain enrichment for suricata/eve fileset. {pull}23011[23011]
- Add subdomain enrichment for zeek/dns fileset. {pull}23011[23011]
- Add `event.category` "configuration" to auditd module events. {pull}23010[23010]
- Add `event.category` "configuration" to gsuite module events. {pull}23010[23010]
- Add `event.category` "configuration" to o365 module events. {pull}23010[23010]
- Add `event.category` "configuration" to zoom module events. {pull}23010[23010]
- Add `network.direction` to auditd/log fileset. {pull}23041[23041]
- Add logic for external network.direction in sophos xg fileset {pull}22973[22973]
- Preserve AWS CloudTrail eventCategory in aws.cloudtrail.event_category. {issue}22776[22776] {pull}22805[22805]
- Add top_level_domain enrichment for suricata/eve fileset. {pull}23046[23046]
- Add top_level_domain enrichment for zeek/dns fileset. {pull}23046[23046]
- Add `observer.egress.zone` and `observer.ingress.zone` for cisco/asa and cisco/ftd filesets. {pull}23068[23068]
- Allow cisco/asa and cisco/ftd filesets to override network directionality based off of zones. {pull}23068[23068]
- Allow cef and checkpoint modules to override network directionality based off of zones {pull}23066[23066]
- Add `network.direction` to netflow/log fileset. {pull}23052[23052]
- Add the ability to override `network.direction` based on interfaces in Fortinet/firewall fileset. {pull}23072[23072]
- Add `network.direction` override by specifying `internal_networks` in gcp module. {pull}23081[23081]
- Migrate microsoft/defender_atp to httpjson v2 config {pull}23017[23017]
- Migrate microsoft/m365_defender to httpjson v2 config {pull}23018[23018]
- Migrate okta to httpjson v2 config {pull}23059[23059]
- Add support for Snyk Vulnerability and Audit API. {pull}22677[22677]
- Misp improvements: Migration to httpjson v2 config, pagination and deduplication ID {pull}23070[23070]
- Add Google Workspace module and mark Gsuite module as deprecated {pull}22950[22950]
- Mark m365 defender, defender atp, okta and google workspace modules as GA {pull}23113[23113]
- Added `alternative_host` option to google pubsub input {pull}23215[23215]

*Heartbeat*

- Add mime type detection for http responses. {pull}22976[22976]

*Metricbeat*

- Move s3_daily_storage and s3_request metricsets to use cloudwatch input. {pull}21703[21703]
- Duplicate system.process.cmdline field with process.command_line ECS field name. {pull}22325[22325]
- Add awsfargate module task_stats metricset to monitor AWS ECS Fargate. {pull}22034[22034]
- Add connection and route metricsets for nats metricbeat module to collect metrics per connection/route. {pull}22445[22445]
- Add unit file states to system/service {pull}22557[22557]
- `kibana` module: `stats` metricset no-longer collects usage-related data. {pull}22732[22732]
- Add more TCP states to Metricbeat system socket_summary. {pull}14347[14347]
- Add io.ops in fields exported by system.diskio. {pull}22066[22066]
- Adjust the Apache status fields in the fleet mode. {pull}22821[22821]
- Add AWS Fargate overview dashboard. {pull}22941[22941]
- Add process.state, process.cpu.pct, process.cpu.start_time and process.memory.pct. {pull}22845[22845]
- Move IIS module to GA and map fields. {issue}22609[22609] {pull}23024[23024]
- Apache: convert status.total_kbytes to status.total_bytes in fleet mode. {pull}23022[23022]
- Release MSSQL as GA {pull}23146[23146]

*Packetbeat*

- Add support for overriding the published index on a per-protocol/flow basis. {pull}22134[22134]
- Change build process for x-pack distribution {pull}21979[21979]
- Tuned the internal queue size to reduce the chances of events being dropped. {pull}22650[22650]
- Add support for "http.request.mime_type" and "http.response.mime_type". {pull}22940[22940]

*Winlogbeat*

- Add file.pe and process.pe fields to ProcessCreate & LoadImage events in Sysmon module. {issue}17335[17335] {pull}22217[22217]
- Add dns.question.subdomain fields for sysmon DNS events. {pull}22999[22999]
- Add additional event categorization for security and sysmon modules. {pull}22988[22988]
- Add dns.question.top_level_domain fields for sysmon DNS events. {pull}23046[23046]

*Elastic Log Driver*

- Add new winlogbeat security dashboard {pull}18775[18775]

==== Deprecated

*Filebeat*

- The experimental modules for Citrix Netscaler and Symantec Endpoint Protection have been removed.
  As we continue to expand our coverage of common security data sources, we may consider supporting
  Citrix Netscaler and Symantec Endpoint Protection in a future release. {issue}23129[23129] {pull}23130[23130]

==== Known Issue



[[release-notes-7.10.2]]
=== Beats version 7.10.2
https://github.com/elastic/beats/compare/v7.10.1\...v7.10.2[View commits]

==== Bugfixes

*Filebeat*

- Add JSON body check for SQS message. {pull}21727[21727]
- Fix cisco umbrella module config by adding input variable. {pull}22892[22892]
- Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277]

*Metricbeat*

- Change `vsphere.datastore.capacity.used.pct` value to betweeen 0 and 1. {pull}23148[23148]

[[release-notes-7.10.1]]
=== Beats version 7.10.1
https://github.com/elastic/beats/compare/v7.10.0\...v7.10.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix denial of service flaw where a remote attacker could cause the {beats} process to crash by presenting a specially malformed TLS public key. For more information, see https://discuss.elastic.co/t/beats-7-10-1-security-update/258160[Beats 7.10.1 Security Update].
- Fix index template loading when the new index format is selected. {issue}22482[22482] {pull}22682[22682]

*Auditbeat*

- auditd: Fix error condition that caused a lot of `audit_send_reply` kernel threads to be created. {pull}22673[22673]
- system/socket: Fix start failure when run under config reloader. {issue}20851[20851] {pull}21693[21693]
- system/socket: Fix startup error with some 5.x kernels. {issue}18755[18755] {pull}22787[22787]
- system/socket: Fix startup errors and event loss caused by some CPUs being unavailable to Auditbeat. {pull}22827[22827]

*Filebeat*

- Fix missing variable when loading aws pipelines. {pull}22645[22645]
- Fix parsing error by dropping `aws.vpcflow.pkt_srcaddr` and `aws.vpcflow.pkt_dstaddr` when equal to "-". {pull}22721[22721] {issue}22716[22716]

*Heartbeat*

- Replace the `service_name` monitor option with `service.name`, which is more correct. We will support the old option until 8.0. {pull}20330[20330]
- Fix problem where the `enabled: false` setting on monitors prevented Heartbeat from starting. {pull}22829[22829]

*Metricbeat*

- Stop generating NaN values from Cloud Foundry module to avoid errors in outputs. {pull}22634[22634]
- Fix Logstash module to no longer emit redundant events when `xpack.enabled: true` is set. {pull}22808[22808]

==== Added

*Filebeat*

- Add DNS response IP addresses to `related.ip` in Suricata module. {pull}22291[22291]

*Functionbeat*

- Add support for parallelization factor for kinesis. {pull}20727[20727]

[[release-notes-7.10.0]]
=== Beats version 7.10.0
https://github.com/elastic/beats/compare/v7.9.3\...v7.10.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Added `certificate` TLS verification mode to ignore server name mismatch. {issue}12283[12283] {pull}20293[20293]
- Remove redundant `cloudfoundry.*.timestamp` fields. This value is set in `@timestamp`. {pull}21175[21175]
- Allow embedding of CAs, Certificate of private keys for anything that supports TLS in outputs and inputs {pull}21179[21179]
- API address is a required setting in `add_cloudfoundry_metadata`. {pull}21759[21759]

*Auditbeat*

- Change network.direction values to ECS recommended values (inbound, outbound). {issue}12445[12445] {pull}20695[20695]
- Docker container needs to be explicitly run as user root for auditing. {pull}21202[21202]
- File integrity dataset no longer includes the leading dot in `file.extension` values (e.g. it will report "png" instead of ".png") to comply with ECS. {pull}21644[21644]

*Filebeat*

* Cisco {pull}18753[18753]
* CrowdStrike {pull}19132[19132]
* Fortinet {pull}19133[19133]
* iptables {pull}18756[18756]
* Checkpoint {pull}18754[18754]
* Netflow {pull}19087[19087]
* Zeek {pull}19113[19113] (`forwarded` tag is not included by default)
* Suricata {pull}19107[19107] (`forwarded` tag is not included by default)
* CoreDNS {pull}19134[19134] (`forwarded` tag is not included by default)
* Envoy Proxy {pull}19134[19134] (`forwarded` tag is not included by default)
- Move file metrics to dataset endpoint {pull}19977[19977]
- Fix PANW field spelling "veredict" to "verdict" on `event.action` {pull}18808[18808]
- Tracking session end reason in panw module. {pull}18705[18705]
- API address and shard ID are required settings in the Cloud Foundry input. {pull}21759[21759]

*Heartbeat*


*Journalbeat*



*Metricbeat*

- Remove "invalid zero" metrics on Windows and Darwin, don't report linux-only memory and disk I/O metrics when running under agent. {pull}21457[21457]
- API address and shard ID are required settings in the Cloud Foundry module. {pull}21759[21759]

*Packetbeat*


*Winlogbeat*


*Functionbeat*


==== Bugfixes

*Affecting all Beats*

- Remove unnecessary restarts of metricsets while using Node autodiscover {pull}19974[19974]
- [Metricbeat][Kubernetes] Change `cluster_ip` field from `ip` to `keyword`. {pull}20571[20571]
- [Autodiscover] Handle input-not-finished errors in config reload. {pull}20915[20915]
- Orderly close processors when processing pipelines are not needed anymore to release their resources. {pull}16349[16349]
- Fix parsing of expired licences. {issue}21112[21112] {pull}22180[22180]

*Auditbeat*

- auditd: Fix spelling of anomaly in `event.category`.
- auditd: Fix typo in `event.action` of `removed-user-role-from`. {pull}19300[19300]
- auditd: Fix typo in `event.action` of `used-suspicious-link`. {pull}19300[19300]

*Filebeat*

- Fix mapping of `fortinet.firewall.mem` as `integer`. {pull}19335[19335]
- Fix auditd module syscall table for ppc64 and ppc64le. {pull}20052[20052]
- Fix Filebeat OOMs on very long lines {issue}19500[19500], {pull}19552[19552]
- Ignore missing in Zeek module when dropping unecessary fields. {pull}19984[19984]
- Fix `event.outcome` logic for azure/siginlogs fileset {pull}20254[20254]
- Improve validation checks for Azure configuration {issue}20369[20369] {pull}20389[20389]
- Fix `event.kind` for system/syslog pipeline {issue}20365[20365] {pull}20390[20390]
- Fix `event.type` for zeek/ssl and duplicate `event.category` for zeek/connection {pull}20696[20696]
- Remove wrongly mapped `tls.client.server_name` from `fortinet/firewall` fileset. {pull}20983[20983]
- Handle multiple upstreams in ingress-controller. {pull}21215[21215]
- Provide backwards compatibility for the `append` processor when Elasticsearch is less than 7.10.0. {pull}21159[21159]
- Fix checkpoint module when logs contain time field. {pull}20567[20567]
- Fix syslog RFC 5424 parsing in the CheckPoint module. {pull}21854[21854]
- Fix incorrect connection state mapping in zeek connection pipeline. {pull}22151[22151] {issue}22149[22149]
- Fix for `field [source] not present as part of path [source.ip]` error in azure pipelines. {pull}22377[22377]
- Fix handing missing eventtime and assignip field being set to N/A for fortinet module. {pull}22361[22361]

*Heartbeat*

- Add support for new `service_name` option to all monitors. {pull}19932[19932].

*Journalbeat*


*Metricbeat*

- Add support for azure light metricset `app_stats`. {pull}20639[20639]
- Fix ec2 disk and network metrics to use Sum statistic method. {pull}20680[20680]
- Fix ec2 disk and network metrics to use Sum statistic method. {pull}20680[20680]
- Update fields.yml in the azure module, missing metrics field. {pull}20918[20918]
- Disable Kafka metricsets based on Jolokia by default. They require a different configuration. {pull}20989[20989]
- Fix timestamp handling in remote_write. {pull}21166[21166]
- Visualization title fixes in aws, azure and googlecloud compute dashboards. {pull}21098[21098]
- Fix retrieving resources by ID for the azure module. {pull}21711[21711] {issue}21707[21707]
- Use timestamp from CloudWatch API when creating events. {pull}21498[21498]
- Report the correct windows events for system/filesystem {pull}21758[21758]
- Fix regular expression in windows/permfon. {pull}22146[22146] {issue}21125[21125]
- Fix azure storage event format. {pull}21845[21845]
- Fix panic in kubernetes autodiscover related to keystores {issue}21843[21843] {pull}21880[21880]
- [Kubernetes] Remove redundant dockersock volume mount {pull}22009[22009]
- Revert change to report `process.memory.rss` as `process.memory.wss` on Windows. {pull}22055[22055]
- Add interval information to `monitor` metricset in azure. {pull}22152[22152]
- Remove `io.time` from windows {pull}22237[22237]
- Fix instance name in perfmon metricset. {issue}22218[22218] {pull}22261[22261]

*Packetbeat*

- Add "network" to `event.category` {issue}20364[20364] {pull}20392[20392]

*Winlogbeat*

- Fix invalid IP addresses in DNS query results from Sysmon data. {issue}18432[18432] {pull}18436[18436]
- Fix `event.outcome` in the security module for non-English languages. {issue}20079[20079] {pull}20564[20564]
- Fields from Winlogbeat modules were not being included in index templates and patterns. {pull}18983[18983]
- Protect against accessing undefined variables in Sysmon module. {issue}22219[22219] {pull}22236[22236]

*Functionbeat*

- Fix catchall bucket config errors by adding more validation. {issue}17572[17572] {pull}20887[20887]
- Fix Google Cloud Function configuration issue. {issue}20864[20864] {pull}22156[22156]

==== Added

*Affecting all Beats*

- Add minimum cache TTL for successful DNS responses. {pull}18986[18986]
- Add support for DNS over TLS for the `dns` processor. {pull}19321[19321]
- Add leader election for Kubernetes autodiscover. {pull}20281[20281]
- Add capability of enriching process metadata with container id also for non-privileged containers in `add_process_metadata` processor. {pull}19767[19767]
- Add `replace_fields` config option in `add_host_metadata` for replacing host fields. {pull}20490[20490] {issue}20464[20464]
- Add ingress controller dashboards. {pull}21052[21052]
- Added experimental `citrix` module. {pull}20820[20820]
- Added experimental `cyberark` module. {pull}20820[20820]
- Added experimental `proofpoint` module. {pull}20820[20820]
- Added experimental `snort` module. {pull}20820[20820]
- Added experimental `symantec` module. {pull}20820[20820]
- Added experimental dataset `barracuda/spamfirewall`. {pull}20820[20820]
- Added experimental dataset `cisco/meraki`. {pull}20820[20820]
- Added experimental dataset `f5/bigipafm`. {pull}20820[20820]
- Added experimental dataset `fortinet/fortimail`. {pull}20820[20820]
- Added experimental dataset `fortinet/fortimanager`. {pull}20820[20820]
- Added experimental dataset `juniper/netscreen`. {pull}20820[20820]
- Added experimental dataset `sophos/utm`. {pull}20820[20820]
- Add Cloud Foundry tags in related events. {pull}21177[21177]
- Cloud Foundry metadata is cached to disk. {pull}20775[20775]
- Add option to select the type of index template to load: `legacy`, `component`, `index`. {pull}21212[21212]
- Release `add_cloudfoundry_metadata` as GA. {pull}21525[21525]
- Added Kafka version 2.2 to the list of supported versions. {pull}22328[22328]

*Auditbeat*

- Add enrichment of auditd seccomp events with name of the architecture, syscall, and signal. {issue}14055[14055] {pull}19300[19300]

*Filebeat*


- Add support for reading auditd logs that are prefixed with `node=`. {pull}19659[19659]
- Add `event.ingested` to all Filebeat modules. {pull}20386[20386]
- Add `event.ingested` for Suricata module {pull}20220[20220]
- Add support for custom header and headersecret for filebeat `http_endpoint` input {pull}20435[20435]
- Convert `httpjson` to v2 input {pull}20226[20226]
- Add `event.ingested` to all Filebeat modules. {pull}20386[20386]
- Return error when log harvester tries to open a named pipe. {issue}18682[18682] {pull}20450[20450]
- Avoid goroutine leaks in Filebeat readers. {issue}19193[19193] {pull}20455[20455]
- Improve Zeek x509 module with `x509` ECS mappings {pull}20867[20867]
- Improve Zeek SSL module with `x509` ECS mappings {pull}20927[20927]
- Added new properties field support for `event.outcome` in azure module {pull}20998[20998]
- Improve Zeek Kerberos module with `x509` ECS mappings {pull}20958[20958]
- Improve Fortinet firewall module with `x509` ECS mappings {pull}20983[20983]
- Improve Santa module with `x509` ECS mappings {pull}20976[20976]
- Improve Suricata Eve module with `x509` ECS mappings {pull}20973[20973]
- Added new module for Zoom webhooks {pull}20414[20414]
- Add `type` and `sub_type` to panw `panos` fileset {pull}20912[20912]
- Always attempt community_id processor on zeek module {pull}21155[21155]
- Add `related.hosts` ecs field to all modules {pull}21160[21160]
- Keep cursor state between `httpjson` input restarts {pull}20751[20751]
- Convert aws s3 to v2 input {pull}20005[20005]
- Add support for additional fields from V2 ALB logs. {pull}21540[21540]
- Release Cloud Foundry input as GA. {pull}21525[21525]
- New Cisco Umbrella dataset {pull}21504[21504]
- New `juniper.srx` dataset for Juniper SRX logs. {pull}20017[20017]
- Adding support for Microsoft 365 Defender (Microsoft Threat Protection) {pull}21446[21446]
- Adding support for FIPS in s3 input {pull}21446[21446]
- Update Okta documentation for new stateful restarts. {pull}22091[22091]
- Use workers in `aws-s3` input to process SQS messages. {pull}27199[27199]

*Heartbeat*

- Add index and pipeline settings to monitor configurations. {pull}20610[20610]

*Journalbeat*


*Metricbeat*

- Add `state_statefulset` metricset to Metricbeat recommended configuration for k8s. {pull}17627[17627]
- Infer types in Prometheus remote_write. {pull}19944[19944]
- Add `cloud.instance.name` into aws ec2 metricset. {pull}20077[20077]
- Add host inventory metrics into aws ec2 metricset. {pull}20171[20171]
- Add `scope` setting for Elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. {issue}18539[18539] {pull}18547[18547]
- Add `state_daemonset` metricset for Kubernetes Metricbeat module {pull}20649[20649]
- Add host inventory metrics to googlecloud compute metricset. {pull}20391[20391]
- Add host inventory metrics to azure compute_vm metricset. {pull}20641[20641]
- Add host inventory metrics to system module. {pull}20415[20415]
- Add billing data collection from Cost Explorer into aws billing metricset. {pull}20527[20527] {issue}20103[20103]
- Migrate `compute_vm` metricset to a light one, map `cloud.instance.id` field. {pull}20889[20889]
- Request prometheus endpoints to be gzipped by default {pull}20766[20766]
- Add latency config parameter into aws module. {pull}20875[20875]
- Add `billing` metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738]
- Release all kubernetes `state` metricsets as GA {pull}20901[20901]
- Move `compute_vm_scaleset` to light metricset. {pull}21038[21038] {issue}20985[20985]
- Sanitize `event.host`. {pull}21022[21022]
- Add support for different Azure Cloud environments in the metricbeat azure module. {pull}21044[21044] {issue}20988[20988]
- Add overview and platform health dashboards to Cloud Foundry module. {pull}21124[21124]
- Release `lambda` metricset in aws module as GA. {issue}21251[21251] {pull}21255[21255]
- Add dashboard for `pubsub` metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137]
- Move Prometheus query & remote_write to GA. {pull}21507[21507]
- Map cloud data filed `cloud.account.id` to azure subscription.  {pull}21483[21483] {issue}21381[21381]
- Expand unsupported option from namespace to metrics in the azure module. {pull}21486[21486]

*Packetbeat*

- Add an example to packetbeat.yml of using the `forwarded` tag to disable
- Add 100-continue support {issue}15830[15830] {pull}19349[19349]
- Add initial SIP protocol support {pull}21221[21221]


*Functionbeat*


*Winlogbeat*


*Elastic Log Driver*
- Add support to change beat name, and support for Kibana Logs. {pull}20522[20522]

==== Deprecated


- N/A

[[release-notes-7.9.3]]
=== Beats version 7.9.3
https://github.com/elastic/beats/compare/v7.9.2\...v7.9.3[View commits]

==== Bugfixes

*Affecting all Beats*

- The `o365audit` input and `o365` module now recover from an authentication problem or other fatal errors, instead of terminating. {pull}21258[21258]

*Auditbeat*

- system/socket: Fix a crash due to concurrent map read and write. {issue}21192[21192] {pull}21690[21690]

*Filebeat*

- Add field limit check for AWS Cloudtrail flattened fields. {pull}21388[21388] {issue}21382[21382]

*Metricbeat*

- Fix `remote_write` flaky test. {pull}21173[21173]
- Fix panic in Kubernetes autodiscovery caused by storing stateless keystores. {issue}21843[21843] {pull}21880[21880]
- Remove redundant dockersock volume mount to avoid problems on Kubernetes deployments that do not use docker as the container runtime. {pull}22009[22009]


[[release-notes-7.9.2]]
=== Beats version 7.9.2
https://github.com/elastic/beats/compare/v7.9.1\...v7.9.2[View commits]

==== Breaking changes

*Affecting all Beats*

- Autodiscover doesn't generate any configuration when a variable is missing. Previously it generated an incomplete configuration. {pull}20898[20898]

==== Bugfixes

*Affecting all Beats*

- Explicitly detect missing variables in autodiscover configuration, log them at the debug level. {issue}20568[20568] {pull}20898[20898]
- Fix `libbeat.output.write.bytes` and `libbeat.output.read.bytes` metrics of the Elasticsearch output. {issue}20752[20752] {pull}21197[21197]

*Filebeat*

- Provide backwards compatibility for the `set` processor when Elasticsearch is less than 7.9.0. {pull}20908[20908]
- Fix an error updating file size being logged when EOF is reached. {pull}21048[21048]
- Fix error when processing AWS Cloudtrail Digest logs. {pull}21086[21086] {issue}20943[20943]

*Metricbeat*

- The Kibana collector applies backoff when errored at getting usage stats {pull}20772[20772]
- The `elasticsearch/index` metricset only requests wildcard expansion for hidden indices if the monitored Elasticsearch cluster supports it. {pull}20938[20938]
- Fix panic index out of range error when getting AWS account name. {pull}21101[21101] {issue}21095[21095]
- Handle missing counters in the application_pool metricset. {pull}21071[21071]

*Functionbeat*

- Do not need Google credentials if not required for the operation. {issue}17329[17329] {pull}21072[21072]
- Fix dependency issues of GCP functions. {issue}20830[20830] {pull}21070[21070]

==== Added

*Affecting all Beats*

- Add container ECS fields in kubernetes metadata. {pull}20984[20984]

[[release-notes-7.9.1]]
=== Beats version 7.9.1
https://github.com/elastic/beats/compare/v7.9.0\...v7.9.1[View commits]

==== Breaking changes

*Affecting all Beats*

- Removed experimental modules `citrix`, `kaspersky`, `rapid7` and `tenable`. {pull}20706[20706]

==== Bugfixes

*Affecting all Beats*

- Update replicaset group to apps/v1 {pull}15854[15854]
- Rename cloud.provider `az` value to `azure` inside the add_cloud_metadata processor. {pull}20689[20689]
- Add missing country_name geo field in `add_host_metadata` and `add_observer_metadata` processors. {issue}20796[20796] {pull}20811[20811]

*Filebeat*

- Fix long registry migration times. {pull}20717[20717] {issue}20705[20705]
- Fix event types and categories in auditd module to comply with ECS {pull}20652[20652]
- Update documentation in the azure module filebeat. {pull}20815[20815]

*Heartbeat*

- Stop rescheduling tasks of stopped monitors. {pull}20570[20570]

*Metricbeat*

- Updates vm_compute metricset with more info on guest metrics. {pull}20448[20448]
- Add fallback for PdhExpandWildCardPathW failing in perfmon metricset. {issue}20139[20139] {pull}20630[20630]
- Fix resource tags in aws cloudwatch metricset {issue}20326[20326]  {pull}20385[20385]
- Fill cloud.account.name with accountID if account alias doesn't exist. {pull}20736[20736]

*Winlogbeat*

- Fix duplicated field error when exporting index-pattern with migration.6_to_7.enabled. {issue}20521[20521] {pull}20540[20540]
- Fix `event.outcome` in the security module for non-English languages. {issue}20079[20079] {pull}20564[20564]

==== Added

*Affecting all Beats*

- Added support for more message types for Cisco ASA and FTD. {pull}20565[20565]

[[release-notes-7.9.0]]
=== Beats version 7.9.0
https://github.com/elastic/beats/compare/v7.8.1\...v7.9.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Ensure dynamic template names are unique for the same field. {pull}18849[18849]

*Filebeat*

- With the default configuration the cloud modules (AWS, Azure, Googlecloud, o365, Okta)
will no longer send the `host` field that contains information about the host Filebeat is
running on. This is because the `host` field specifies the host on which the event
happened. {issue}13920[13920] {pull}18223[18223]
- With the default configuration the following modules will no longer send the `host`
field. You can revert this change by configuring tags for the module and omitting
`forwarded` from the list.
* Cisco {pull}18753[18753]
* CrowdStrike {pull}19132[19132]
* Fortinet {pull}19133[19133]
* Iptables {pull}18756[18756]
* Checkpoint {pull}18754[18754]
* Netflow {pull}19087[19087]
* Zeek {pull}19113[19113] (`forwarded` tag is not included by default)
* Suricata {pull}19107[19107] (`forwarded` tag is not included by default)
* CoreDNS {pull}19134[19134] (`forwarded` tag is not included by default)
* Envoy Proxy {pull}19134[19134] (`forwarded` tag is not included by default)
* CEF module {issue}13920[13920] {pull}18223[18223]
* Palo Alto Networks module {issue}13920[13920] {pull}18223[18223]
- Okta module now requires objects instead of JSON strings for the `http_headers`, `http_request_body`, `pagination`, `rate_limit`, and `ssl` variables. {pull}18953[18953]
- Add oauth support for httpjson input. {issue}18415[18415] {pull}18892[18892]
- Add `split_events_by` option to httpjson input. {pull}19246[19246]
- Add `date_cursor` option to httpjson input. {pull}19483[19483]
- Add Gsuite module with SAML support. {pull}19329[19329]
- Add Gsuite User Accounts support. {pull}19329[19329]
- Add Gsuite Login audit support. {pull}19702[19702]
- Add Gsuite Admin support. {pull}19769[19769]
- Add Gsuite Drive support. {pull}19704[19704]
- Add Gsuite Groups support. {pull}19725[19725]

*Metricbeat*

- Move service config under metrics and simplify metric types. {pull}18691[18691]
- Fix ECS compliance of `user.id` field in system/users metricset. {pull}19019[19019]
- Rename googlecloud stackdriver metricset to metrics. {pull}19718[19718]

*Winlogbeat*

- Add PowerShell module. Support for event ID's: `400`, `403`, `600`, `800`, `4103`, `4014`, `4105`, `4106`. {issue}16262[16262] {pull}18526[18526]
- Fix PowerShell processing of downgraded engine events. {pull}18966[18966]
- Fix unprefixed fields in `fields.yml` for PowerShell module. {issue}18984[18984]

==== Bugfixes

*Affecting all Beats*

- Fix potential race condition in fingerprint processor. {pull}18738[18738]
- Add better handling for Kubernetes Update and Delete watcher events. {pull}18882[18882]
- Fix config reload metrics (`libbeat.config.module.start/stops/running`). {pull}19168[19168]
- Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed. {pull}18979[18979]
- Server-side TLS config now validates that certificate and key settings are both specified. {pull}19584[19584]

*Auditbeat*

- system/socket: Fix issue with dataset using 100% CPU and becoming unresponsive in some scenarios. {pull}19033[19033] {pull}19764[19764]

*Filebeat*

- Fix Kubernetes Watcher goroutine leaks when input config is invalid and `input.reload` is enabled. {issue}18629[18629] {pull}18630[18630]
- Okta module now sets the Elasticsearch `_id` field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. {pull}18953[18953]
- Fix `netflow` module to support 7 bytepad for IPFIX template. {issue}18098[18098]
- Fix improper nesting of session_issuer object in AWS cloudtrail fileset. {issue}18894[18894] {pull}18915[18915]
- Fix Cisco ASA 3020** and 106023 messages. {pull}17964[17964]
- Add missing `default_field: false` to AWS filesets fields.yml. {pull}19568[19568]
- Fix memory leak in tcp and unix input sources. {pull}19459[19459]
- Fix Cisco ASA dissect pattern for 313008 & 313009 messages. {pull}19149[19149]
- Fix bug with empty filter values in system/service. {pull}19812[19812]

*Metricbeat*

- Fix incorrect usage of hints builder when exposed port is a substring of the hint. {pull}19052[19052]
- Stop counterCache only when already started. {pull}19103[19103]
- Remove dedot for tag values in AWS module. {issue}19112[19112] {pull}19221[19221]
- Fix empty field name errors in the application pool metricset. {pull}19537[19537]
- Fix mapping of service start type in the service metricset of the Windows module. {pull}19551[19551]
- Fix config example in the perfmon configuration files. {pull}19539[19539]
- Fix k8s scheduler compatibility issue. {pull}19699[19699]
- Fix SQL module mapping NULL values as string. {pull}18955[18955] {issue}18898[18898]

*Packetbeat*

- Fix process monitoring when ipv6 is disabled under Linux. {issue}19941[19941] {pull}19945[19945]

==== Added

*Affecting all Beats*

- Add initial instrument of Beats with APM GO Agent. {pull}17938[17938]
- Add optional regex based cid extractor to `add_kubernetes_metadata` processor. {pull}17360[17360]
- Add k8s keystore backend. {pull}18096[18096]
- Change ownership of files in docker images so they can be used in secured environments. {pull}12905[12905]
- Upgrade k8s.io/client-go and k8s keystore tests. {pull}18817[18817]
- Add support for multiple sets of hints on autodiscover. {pull}18883[18883]
- Add a configurable delay between retries when app metadata cannot be retrieved by `add_cloudfoundry_metadata`. {pull}19181[19181]
- Add data type conversion in `dissect` processor for converting string values to other basic data types. {pull}18683[18683]
- Add the `ignore_failure` configuration option to the dissect processor. {pull}19464[19464]
- Add the `overwrite_keys` configuration option to the dissect processor. {pull}19464[19464]
- Add support to trim captured values in the dissect processor. {pull}19464[19464]
- Add the `max_cached_sessions` option to the script processor. {pull}19562[19562]

*Auditbeat*

- Add ECS categorization info for Auditd module. {pull}18596[18596]

*Filebeat*


- Add http_endpoint input. {pull}18298[18298]
- Add `observer.vendor`, `observer.product`, and `observer.type` to Palo Alto Networks module events. {pull}18223[18223]
- The `logstash` module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. {issue}9964[9964] {pull}18095[18095]
- Improve ECS categorization field mappings in CoreDNS module. {issue}16159[16159] {pull}18424[18424]
- Improve ECS categorization field mappings in Envoyproxy module. {issue}16161[16161] {pull}18395[18395]
- Improve ECS categorization field mappings in Cisco module. {issue}16028[16028] {pull}18537[18537]
- The s3 input can now automatically detect gzipped objects. {issue}18283[18283] {pull}18764[18764]
- Add geoip AS lookup and improve ECS categorization in AWS cloudtrail fileset. {issue}18644[18644] {pull}18958[18958]
- Add support for v1 consumer API in Cloud Foundry input and use it by default. {pull}19125[19125]
- Add new mode to multiline reader to aggregate constant number of lines. {pull}18352[18352]
- Explicitly set ECS version in all Filebeat modules. {pull}19198[19198]
- Add awscloudwatch input. {pull}19025[19025]
- Add automatic retries and exponential backoff to httpjson input. {pull}18956[18956]
- Change the Palo Alto Networks module to pass through (rather than drop) message types other than threat and traffic. {issue}16815[16815] {pull}19375[19375]
- Improve ECS categorization field mappings in Traefik module. {issue}16183[16183] {pull}19379[19379]
- Improve ECS categorization field mappings in Azure module. {issue}16155[16155] {pull}19376[19376]
- Add automatic retries and exponential backoff to httpjson input. {pull}18956[18956]
- Add text and flattened versions of fields with unknown subfields in AWS cloudtrail fileset. {issue}18866[18866] {pull}19121[19121]
- Add Microsoft Defender ATP Module. {issue}17997[17997] {pull}19197[19197]
- Add initial support for configurable file identity tracking. {pull}18748[18748]
- Add experimental dataset tomcat/log for Apache Tomcat logs. {pull}19713[19713]
- Add experimental dataset netscout/sightline for Netscout Arbor Sightline logs. {pull}19713[19713]
- Add experimental dataset barracuda/waf for Barracuda Web Application Firewall logs. {pull}19713[19713]
- Add experimental dataset f5/bigipapm for F5 Big-IP Access Policy Manager logs. {pull}19713[19713]
- Add experimental dataset bluecoat/director for Bluecoat Director logs. {pull}19713[19713]
- Add experimental dataset cisco/nexus for Cisco Nexus logs. {pull}19713[19713]
- Add experimental dataset citrix/virtualapps for Citrix Virtual Apps logs. {pull}19713[19713]
- Add experimental dataset cylance/protect for Cylance Protect logs. {pull}19713[19713]
- Add experimental dataset fortinet/clientendpoint for Fortinet FortiClient Endpoint Protection logs. {pull}19713[19713]
- Add experimental dataset imperva/securesphere for Imperva Secure Sphere logs. {pull}19713[19713]
- Add experimental dataset infoblox/nios for Infoblox Network Identity Operating System logs. {pull}19713[19713]
- Add experimental dataset juniper/junos for Juniper Junos OS logs. {pull}19713[19713]
- Add experimental dataset kaspersky/av for Kaspersky Anti-Virus logs. {pull}19713[19713]
- Add experimental dataset microsoft/dhcp for Microsoft DHCP Server logs. {pull}19713[19713]
- Add experimental dataset tenable/nessus_security for Tenable Nessus Security Scanner logs. {pull}19713[19713]
- Add experimental dataset rapid7/nexpose for Rapid7 Nexpose logs. {pull}19713[19713]
- Add experimental dataset radware/defensepro for Radware DefensePro logs. {pull}19713[19713]
- Add experimental dataset sonicwall/firewall for Sonicwall Firewalls logs. {pull}19713[19713]
- Add experimental dataset squid/log for Squid Proxy Server logs. {pull}19713[19713]
- Add experimental dataset zscaler/zia for Zscaler Internet Access logs. {pull}19713[19713]

*Heartbeat*

- Record HTTP response headers. {pull}18327[18327]

*Journalbeat*

- Added an `id` config option to inputs to allow running multiple inputs on the same journal. {pull}18467[18467]
- Add basic ECS categorization and `log.syslog` fields. {pull}19176[19176]

*Metricbeat*

- Add client address to events from http server module. {pull}18336[18336]
- Add new fields to HAProxy module. {issue}18523[18523]
- Add Tomcat overview dashboard. {pull}14026[14026]
- Accept prefix as metric_types config parameter in googlecloud stackdriver metricset. {pull}19345[19345]
- Add dashboards for googlecloud load balancing metricset. {pull}18369[18369]
- Add support for v1 consumer API in Cloud Foundry module and use it by default. {pull}19268[19268]
- Add support for named ports in autodiscover. {pull}19398[19398]
- Add param `aws_partition` to support aws-cn, aws-us-gov regions. {issue}18850[18850] {pull}19423[19423]
- Add support for wildcard `*` in dimension value of AWS CloudWatch metrics config. {issue}18050[18050] {pull}19660[19660]
- The `elasticsearch/index` metricset now collects metrics for hidden indices. {issue}18639[18639] {pull}18703[18703]
- Added `performance` and `query` metricsets to `mysql` module. {pull}18955[18955]
- The `elasticsearch-xpack/index` metricset now reports hidden indices as such. {issue}18639[18639] {pull}18706[18706]
- Adds support for app insights metrics in the Azure module. {issue}18570[18570] {pull}18940[18940]
- Added cache and connection_errors metrics to status metricset of MySQL module. {issue}16955[16955] {pull}19844[19844]
- Update MySQL dashboard with connection errors and cache metrics. {pull}19913[19913] {issue}16955[16955]

*Packetbeat*

- Add an example to packetbeat.yml of using the `forwarded` tag to disable `host` metadata fields when processing network data from network tap or mirror port. {pull}19209[19209]
- Add ECS fields for x509 certs, event categorization, and related IP info. {pull}19167[19167]

*Functionbeat*

- Add basic ECS categorization and `cloud` fields. {pull}19174[19174]

*Elastic Log Driver*

- Add support for `docker logs` command. {pull}19531[19531]

==== Deprecated

*Metricbeat*

- Deprecate tags config parameter in cloudwatch metricset. {pull}16733[16733]
- Deprecate tags.resource_type_filter config parameter and replace with resource_type. {pull}19688[19688]

[[release-notes-7.8.1]]
=== Beats version 7.8.1
https://github.com/elastic/beats/compare/v7.8.0\...v7.8.1[View commits]

==== Breaking changes

*Filebeat*

- Adds check on `<no value>` config option value for the azure input `resource_manager_endpoint`. {pull}18890[18890]

==== Bugfixes

*Affecting all Beats*

- The `monitoring.elasticsearch.api_key` value is correctly base64-encoded before being sent to the monitoring Elasticsearch cluster. {issue}18939[18939] {pull}18945[18945]
- Fix kafka topic setting not allowing upper case characters. {pull}18854[18854] {issue}18640[18640]
- Fix redis key setting not allowing upper case characters. {pull}18854[18854] {issue}18640[18640]

*Auditbeat*

- system/package: Fix librpm loading on Fedora 31/32. {pull}NNNN[NNNN]

*Filebeat*

- Fix date and timestamp formats for fortigate module {pull}19316[19316]
- Fix `googlecloud.audit` pipeline to only take in fields that are explicitly defined by the dataset. {issue}18465[18465] {pull}18472[18472]
- Fix a rate limit related issue in httpjson input for Okta module. {issue}18530[18530] {pull}18534[18534]
- Fix tls mapping in suricata module {issue}19492[19492] {pull}19494[19494]

*Metricbeat*

- Set tags correctly if the dimension value is ARN {issue}19111[19111] {pull}19433[19433]
- Fix bug incorrect parsing of float numbers as integers in Couchbase module {issue}18949[18949] {pull}19055[19055]
- Add missing info about the rest of the azure metricsets in the documentation. {pull}19601[19601]

==== Added

*Filebeat*

- Add support for timezone offsets and `Z` to decode_cef timestamp parser. {pull}19346[19346]

*Metricbeat*

- Update Couchbase to version 6.5 {issue}18595[18595] {pull}19055[19055]

[[release-notes-7.8.0]]
=== Beats version 7.8.0
https://github.com/elastic/beats/compare/v7.7.0\...v7.8.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Introduce APM instrumentation, which is active when running the beat with `ELASTIC_APM_ACTIVE=true`. {pull}17938[17938]

*Filebeat*

- Improve ECS field mappings in panw module.  `event.outcome` now only contains success or failure, as recommended by the {ecs-ref}/ecs-event.html[ECS specification]. {issue}16025[16025] {pull}17910[17910]
- Improve ECS categorization field mappings for nginx module. `http.request.referrer` is now lowercase, and it is only populated when nginx sets a value. {issue}16174[16174] {pull}17844[17844]
- Improve ECS field mappings in santa module. `hash.sha256` is moved to `process.hash.sha256`, and certificate fields are now under `santa.certificate`. {issue}16180[16180] {pull}17982[17982]

==== Bugfixes

*Affecting all Beats*

- Fix a bug in config reloading that could result in memory leaks or lost events when an output was rapidly reloaded multiple times. {issue}10491[10491] {pull}17381[17381]
- Fix panic when assigning a key to a `nil` value in an event. {pull}18143[18143]

*Heartbeat*

- Fix TCP TLS checks to properly validate hostnames. In previous 7.x versions, this only worked for IP SANs. {pull}17549[17549]

*Metricbeat*

- No longer send NaNs for memory metrics that don't exist on the platform being monitored. {pull}17400[17400]
- Add a switch to the driver definition on SQL module to use pretty names. {pull}17378[17378]

==== Added

*Affecting all Beats*

- Update supported versions of `redis` output. {pull}17198[17198]
- Add `replace` processor for replacing string values of fields. {pull}17342[17342]
- Add `urldecode` processor for decoding URL-encoded fields. {pull}17505[17505]
- Add support for AWS IAM `role_arn` in credentials config. {pull}17658[17658] {issue}12464[12464]
- Add Kerberos support to Elasticsearch output. {pull}17927[17927]
- Set `agent.name` to the hostname by default. {issue}16377[16377] {pull}18000[18000]
- Add keystore support for autodiscover static configurations. {pull}16306[16306]
- Add support for basic ECS logging. {pull}17974[17974]
- Add config example of how to skip the `add_host_metadata` processor when forwarding logs. {issue}13920[13920] {pull}18153[18153]
- Add backoff configuration options for the Kafka output. {issue}16777[16777] {pull}17808[17808]
- Add keystore support for autodiscover static configurations. {pull}16306[16306]
- Add Kerberos support to Elasticsearch output. {pull}17927[17927]
- Add support for fixed length extraction in `dissect` processor. {pull}17191[17191]

*Auditbeat*

- Add system module process dataset ECS categorization fields. {pull}18032[18032]
- Add system module user dataset ECS categorization fields. {pull}18035[18035]
- Add system module login dataset ECS categorization fields. {pull}18034[18034]
- Add system module package dataset ECS categorization fields. {pull}18033[18033]
- Add ECS categories for system module host dataset. {pull}18031[18031]
- Add system module socket dataset ECS categorization fields. {pull}18036[18036]
- Add file integrity module ECS categorization fields. {pull}18012[18012]
- Add `file.mime_type`, `file.extension`, and `file.drive_letter` for file integrity module. {pull}18012[18012]

*Filebeat*

- Add source field in k8s events. {pull}17209[17209]
- Add new `crowdstrike` module for ingesting Crowdstrike Falcon streaming API endpoint event data. {pull}16988[16988]
- Improve ECS categorization field mappings in mongodb module. {issue}16170[16170] {pull}17371[17371]
- Improve ECS categorization field mappings for mssql module. {issue}16171[16171] {pull}17376[17376]
- Improve ECS categorization field mappings for mysql module. {issue}16172[16172] {pull}17491[17491]
- Add new Checkpoint Syslog filebeat module. {pull}17682[17682]
- Add config option to select a different azure cloud env in the azure-eventhub input and azure module. {issue}17649[17649] {pull}17659[17659]
- Enhance `elasticsearch/server` fileset to handle ECS-compatible logs emitted by Elasticsearch. {issue}17715[17715] {pull}17714[17714]
- Add Unix stream socket support as an input source and a syslog input source. {pull}17492[17492]
- Improve ECS categorization field mappings in misp module. {issue}16026[16026] {pull}17344[17344]
- Enhance `elasticsearch/deprecation` fileset to handle ECS-compatible logs emitted by Elasticsearch. {issue}17715[17715] {pull}17728[17728]
- Make `decode_cef` processor GA. {pull}17944[17944]
- Add new Fortigate Syslog filebeat module. {pull}17890[17890]
- Improve ECS categorization field mappings in redis module. {issue}16179[16179] {pull}17918[17918]
- Improve ECS categorization field mappings in rabbitmq module. {issue}16178[16178] {pull}17916[17916]
- Improve ECS categorization field mappings in postgresql module. {issue}16177[16177] {pull}17914[17914]
- Improve ECS categorization field mappings for nginx module. {issue}16174[16174] {pull}17844[17844]
- Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. {pull}15668[15668]
- Improve ECS categorization field mappings for zeek module. {issue}16029[16029] {pull}17738[17738]
- Improve ECS categorization field mappings for netflow module. {issue}16135[16135] {pull}18108[18108]
- Add an input option `publisher_pipeline.disable_host` to disable `host.name` from being added to events by default. {pull}18159[18159]
- Improve ECS categorization field mappings in system module. {issue}16031[16031] {pull}18065[18065]
- Improve ECS categorization field mappings in osquery module. {issue}16176[16176] {pull}17881[17881]
- Add support for v10, v11 and v12 logs on Postgres {issue}13810[13810] {pull}17732[17732]
- Add dashboard for Google Cloud Audit and AWS CloudTrail. {pull}17379[17379]

*Heartbeat*

- Add additional ECS compatible fields for TLS information. {pull}17687[17687]

*Metricbeat*

- Refactor windows/perfmon metricset configuration options and event output. {pull}17596[17596]
- Add more detailed error messages, system tests and small refactoring to the service metricset in windows. {pull}17725[17725]
- Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[16471] {pull}17609[17609]
- Add Metricbeat IIS module dashboards. {pull}17966[17966]
- Add dashboard for the azure database account metricset. {pull}17901[17901]
- Allow partial region and zone name in googlecloud module config. {pull}17913[17913]
- Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. {issue}17141[17141] {pull}17719[17719]
- Move the perfmon metricset to GA. {issue}16608[16608] {pull}17879[17879]
- Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[16471] {pull}17609[17609]
- Add static mapping for metricsets under aws module. {pull}17614[17614] {pull}17650[17650]
- Add dashboard for googlecloud storage metricset. {pull}18172[18172]
- Collect new `bulk` indexing metrics from Elasticsearch when `xpack.enabled:true` is set. {issue}17977[17977] {pull}17992[17992]
- Remove requirement to connect as sysdba in Oracle module. {issue}15846[15846] {pull}18182[18182]
- Update MSSQL module to fix some SSPI authentication and add brackets to USE statements. {pull}17862[17862]

*Winlogbeat*

- Set `process.command_line` and `process.parent.command_line` from Sysmon Event ID 1. {pull}17327[17327]
- Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module. {pull}17517[17517]
- Add registry and code signature information and ECS categorization fields for sysmon module. {pull}18058[18058]

[[release-notes-7.7.1]]
=== Beats version 7.7.1
https://github.com/elastic/beats/compare/v7.7.0\...v7.7.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix `keystore add` command hanging on Windows. {issue}18649[18649] {pull}18654[18654]

*Filebeat*

- Unescape filenames in SQS messages to resolve file paths correctly. {pull}18370[18370]
- Improve failure handler for Cisco ASA and FTD pipelines to avoid mapping temporary fields. {issue}18391[18391] {pull}18392[18392]
- Fix `source.address` field not being set for the Nginx `ingress_controller` fileset. {pull}18511[18511]
- Fix Google Cloud `audit` fileset to only take in fields that are explicitly defined by the fileset. {issue}18465[18465] {pull}18472[18472]
- Fix rate limit related issue in the `httpjson` input for the Okta module. {issue}18530[18530] {pull}18534[18534]
- Fix Cisco ASA and FTD parsing errors caused by NAT fields that contain a hostname instead of an IP. {issue}14034[14034] {pull}18376[18376]
- Fix PANW module to use correct mappings for bytes and packets counters. {issue}18522[18522] {pull}18525[18525]
- Fix Office 365 ingest failures caused by IP addresses surrounded by square brackets. {issue}18587[18587] {pull}18591[18591]

*Metricbeat*

- Fix `tags_filter` setting to work correctly for the AWS `cloudwatch` metricset. {pull}18524[18524]

==== Added

*Filebeat*

- Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. {pull}15668[15668]
- Make `decode_cef` processor GA. {pull}17944[17944]

[[release-notes-7.7.0]]
=== Beats version 7.7.0
https://github.com/elastic/beats/compare/v7.6.2\...v7.7.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Environment variables can no longer reference other environment variables or objects. {pull}15937[15937]
- Change `aws_elb` autodiscovery provider field name from `elb_listener.*` to `aws.elb.*`. {issue}16219[16219] {pull}16402[16402]
- Remove support for using `add_docker_metadata` and `add_kubernetes_metadata` processors from the `script` processor. They can still be used as normal processors in the configuration. {issue}16349[16349] {pull}16514[16514]

==== Bugfixes

*Affecting all Beats*

- Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data. {pull}17223[17223]
- Fix `add_cloud_metadata` processor to better support modifying sub-fields with other processors. {pull}13808[13808]
- Fix panic in the Logstash output when trying to send events to closed connection. {pull}15568[15568]
- Fix logging target settings being ignored when Beats are started via systemd or docker. {issue}12024[12024] {pull}15422[15442]
- Fix issue where default go logger is not discarded when either * or stdout is selected. {issue}10251[10251] {pull}15708[15708]
- Remove superfluous use of `number_of_routing_shards` setting from the default template. {pull}16038[16038]
- Automatically convert index names to lowercase. {pull}16081[16081]
- Fix loading processor annotation hints, allowing the value to be a full configuration section. {pull}16348[16348]
- Add `ssl.ca_sha256` to the list of supported TLS options. This option allows you to check that a specific certificate is used as part of the verified chain. {issue}15717[15717]
- Fix `NewContainerMetadataEnricher` to use default config for kubernetes module. No longer requires the user to have `labels.dedot: true` in the configuration as it is now properly the default. {pull}16857[16857]
- Improve logging messages for the `add_kubernetes_metadata` processor. {pull}16866[16866]
- Fail to start if httpprof is used and it cannot be initialized. {pull}17028[17028]
- Fix concurrency issues in convert processor when used in the global context. {pull}17032[17032]
- Fix bug with `monitoring.cluster_uuid` setting not always being exposed via GET /state Beats API. {issue}16732[16732] {pull}17420[17420]
- Fix building on FreeBSD by removing build flags from `add_cloudfoundry_metadata` processor. {pull}17486[17486]

*Filebeat*

- Fix mapping error when zeek weird logs do not contain IP addresses. {pull}15906[15906]
- Fix merging of fileset inputs to replace paths and append processors. {pull}16450[16450]
- Fix Elasticsearch `_id` field set by S3 and Google Pub/Sub inputs. {pull}17026[17026]
- Fix various Cisco FTD parsing issues. {issue}16863[16863] {pull}16889[16889]
- Fix default index pattern in IBM MQ Filebeat dashboard. {pull}17146[17146]
- Fix a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. {issue}17242[17242] {pull}17243[17243]
- Fix MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. {issue}17086[17086] {pull}17156[17156]
- Fix `elasticsearch.audit` data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. {pull}17406[17406]
- Fix decoding errors caused by trailing spaces in CEF messages. {pull}17253[17253]
- Fix activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. {pull}17428[17428]

*Metricbeat*

- Change `lookup_fields` setting from `metricset.host` to `service.address`. {pull}15883[15883]
- Make `logstash-xpack` module once again have parity with internally-collected Logstash monitoring data. {pull}16198[16198]
- Improve metrics collection in the `system/service` metricset on older linux distributions. {pull}16902[16902]
- Use max in k8s apiserver dashboard aggregations. {pull}17018[17018]
- Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from `elasticsearch/ccr` metricset. {issue}16511[16511] {pull}17073[17073]
- Use max in k8s overview dashboard aggregations. {pull}17015[17015]
- Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. {issue}12435[12435] {pull}17272[17272]
- Fix missing Accept header for Prometheus and OpenMetrics module. {issue}16870[16870] {pull}17291[17291]
- Combine cloudwatch aggregated metrics into single event. {pull}17345[17345]
- Fix how we filter services by name in system/service. {pull}17400[17400]
- Fix problem where `cloudwatch` metricset was not collecting tags correctly. {issue}17419[17419] {pull}17424[17424]
- Check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. {pull}17418[17418]
- Fix `aws.s3.bucket.name` terms_field in s3 overview dashboard. {pull}17542[17542]
- Fix Unix socket path in memcached module. {pull}17512[17512]
- Fix vsphere VM dashboard host aggregation visualizations. {pull}17555[17555]

==== Added

*Affecting all Beats*

- Include network information by default when using the `add_host_metadata` or `add_observer_metadata` processor. {issue}15347[15347] {pull}16077[16077]
- Add `aws_ec2` provider for autodiscovery. {issue}12518[12518] {pull}14823[14823]
- Add support for multiple passwords in redis output. {issue}16058[16058] {pull}16206[16206]
- Add support for Histogram type in fields.yml. {pull}16570[16570]
- Windows .exe files now have embedded file version info. {issue}15232[15232]
- Remove experimental flag from `setup.template.append_fields`. {pull}16576[16576]
- Add `add_cloudfoundry_metadata` processor to annotate events with Cloud Foundry application data. {pull}16621[16621]
- Add `translate_sid` processor on Windows for converting Windows security identifier (SID) values to names. {issue}7451[7451] {pull}16013[16013]
- Add support for Kubernetes provider to recognize namespace level defaults. {pull}16321[16321]
- Add ability to enrich the `container.id` with the process id by using the `add_process_metadata` processor. {pull}15947[15947]
- Update RPM packages contained in Beat Docker images. {issue}17035[17035]
- Add Kerberos support to Kafka input and output. {pull}16781[16781]

*Auditbeat*

- Add examples to the kubernetes manifests to show how to
configure the auditd module and use processors to enrich events with metadata.
- In the kubernetes manifests, mount the data directory from the host, so data persist between executions in the same node. {pull}17429[17429]
- Log to stderr when using kubernetes manifests. {pull}17443[174443]
- Fix memory leak on when we miss socket close kprobe events. {pull}17500[17500]

*Filebeat*

- Add ECS tls fields to the smtp, rdp, and ssl filesets in the zeek module, and the s3access and elb filesets in the aws module. {issue}15757[15757] {pull}15935[15936]
- Add Nginx `ingress_controller` fileset. {pull}16197[16197]
- Add ECS tls and categorization fields to apache module. {issue}16032[16032] {pull}16121[16121]
- Add MQTT input. {issue}15602[15602] {pull}16204[16204]
- Improve ECS categorization, container, and process field mappings in auditd module. {issue}16153[16153] {pull}16280[16280]
- Add ECS categorization fields to activemq module. {issue}16151[16151] {pull}16201[16201]
- Improve ECS field mappings in aws module. {issue}16154[16154] {pull}16307[16307]
- Improve ECS categorization field mappings in googlecloud module. {issue}16030[16030] {pull}16500[16500]
- Add `cloudwatch` and `ec2` filesets to aws module. {issue}13716[13716] {pull}16579[16579]
- Improve ECS categorization field mappings in kibana module. {issue}16168[16168] {pull}16652[16652]
- Add `cloudfoundry` input to send events from Cloud Foundry. {pull}16586[16586]
- Improve ECS field mappings in haproxy module. {issue}16162[16162] {pull}16529[16529]
- Allow users to override pipeline ID in fileset input config. {issue}9531[9531] {pull}16561[16561]
- Improve ECS categorization field mappings in logstash module. {issue}16169[16169] {pull}16668[16668]
- Improve ECS categorization field mappings in iis module. {issue}16165[16165] {pull}16618[16618]
- Improve the `decode_cef` processor by reducing the number of memory allocations. {pull}16587[16587]
- Improve ECS categorization field mapping in kafka module. {issue}16167[16167] {pull}16645[16645]
- Improve ECS categorization field mapping in icinga module. {issue}16164[16164] {pull}16533[16533]
- Improve ECS categorization field mappings in ibmmq module. {issue}16163[16163] {pull}16532[16532]
- Add custom string mapping to CEF module to support Forcepoint NGFW. {issue}14663[14663] {pull}15910[15910]
- Add ECS fields to CEF module. {issue}16157[16157] {pull}16338[16338]
- Improve ECS categorization and host field mappings in elasticsearch module. {issue}16160[16160] {pull}16469[16469]
- Improve ECS categorization field mappings in suricata module. {issue}16181[16181] {pull}16843[16843]
- Release ActiveMQ module as GA. {issue}17047[17047] {pull}17049[17049]
- Improve ECS categorization field mappings in iptables module. {issue}16166[16166] {pull}16637[16637]
- Add pattern for Cisco ASA / FTD Message 734001. {issue}16212[16212] {pull}16612[16612]
- Add `o365audit` input type for consuming events from Office 365 Management Activity API. {issue}16196[16196] {pull}16244[16244]
- Add custom string mapping to CEF module to support Check Point devices. {issue}16041[16041] {pull}16907[16907]
- Add `o365` module for ingesting Office 365 management activity API events. {issue}16196[16196] {pull}16386[16386]
- Add Okta module. {pull}16362[16362]
- Improve AWS cloudtrail field mappings. {issue}16086[16086] {issue}16110[16110] {pull}17155[17155]
- Make the `azure-eventhub` input GA. {issue}15671[15671] {pull}17313[17313]
- Add `access_key_id`, `secret_access_key`, and `session_token` to the aws module config. {pull}17456[17456]

*Heartbeat*

- Allow a list of status codes for HTTP checks. {pull}15587[15587]

*Journalbeat*

- Improve parsing of `syslog.pid` in Journalbeat to strip the username when
present. {pull}16116[16116]

*Metricbeat*

- Add lambda metricset in aws module. {pull}15260[15260]
- Add DynamoDB AWS light module. {pull}15097[15097]
- Add IBM MQ light-weight module. {pull}15301[15301]
- Add mixer metricset for Istio Metricbeat module. {pull}15696[15696]
- Add mesh metricset for Istio Metricbeat module. {pull}15535[15535]
- Add pilot metricset for Istio Metricbeat module. {pull}15761[15761]
- Add galley metricset for Istio Metricbeat module. {pull}15857[15857]
- Add `key/value` mode for SQL module. {issue}15770[15770] {pull}15845[15845]
- Add support for Unix socket in Memcached module. {issue}13685[13685] {pull}15822[15822]
- Make the `system/cpu` metricset collect normalized CPU metrics by default. {issue}15618[15618] {pull}15729[15729]
- Add kubernetes storage class support via kube-state-metrics. {pull}16145[16145]
- Add `up` metric to prometheus metrics collected from host. {pull}15948[15948]
- Add citadel metricset for Istio Metricbeat module. {pull}15990[15990]
- Add support for processors in light modules. {issue}14740[14740] {pull}15923[15923]
- Add ability to collect AuroraDB metrics in rds metricset. {issue}14142[14142] {pull}16004[16004]
- Reuse connections in SQL module. {pull}16001[16001]
- Improve the `logstash` module (when `xpack.enabled` is set to `true`) to use the override `cluster_uuid` returned by Logstash APIs. {issue}15772[15772] {pull}15795[15795]
- Add region parameter in googlecloud module. {issue}15780[15780] {pull}16203[16203]
- Add `database_account` azure metricset. {issue}15758[15758]
- Add support for Dropwizard metrics 4.1. {pull}16332[16332]
- Add support for NATS 2.1. {pull}16317[16317]
- Add azure container metricset in order to monitor containers. {issue}15751[15751] {pull}16421[16421]
- Improve the `haproxy` module to support metrics exposed via HTTPS. {issue}14579[14579] {pull}16333[16333]
- Add filtering option for prometheus collector. {pull}16420[16420]
- Add metricsets based on Ceph Manager Daemon to the `ceph` module. {issue}7723[7723] {pull}16254[16254]
- Add Load Balancing metricset to GCP. {pull}15559[15559]
- Release `statsd` module as GA. {pull}16447[16447] {issue}14280[14280]
- Add collecting tags and tags_filter for rds metricset in aws module. {pull}16605[16605] {issue}16358[16358]
- Add OpenMetrics module. {pull}16596[16596]
- Add `redisenterprise` module. {pull}16482[16482] {issue}15269[15269]
- Add `cloudfoundry` module to send events from Cloud Foundry. {pull}16671[16671]
- Add system/users metricset as beta. {pull}16569[16569]
- Align fields to ECS and add more tests for the azure module. {issue}16024[16024] {pull}16754[16754]
- Add additional cgroup fields to docker/diskio. {pull}16638[16638]
- Add overview dashboard for googlecloud compute metricset. {issue}16534[16534] {pull}16819[16819]
- Add Prometheus remote write endpoint. {pull}16609[16609]
- Release STAN module as GA. {pull}16980[16980]
- Add query metricset for prometheus module. {pull}17104[17104]
- Release ActiveMQ module as GA. {issue}17047[17047] {pull}17049[17049]
- Add support for CouchDB v2. {issue}16352[16352] {pull}16455[16455]
- Add dashboards for the azure container metricsets. {pull}17194[17194]
- Separate the `vpc` metricset into three smaller metricsets: `vpn`, `transitgateway`, and `natgateway`. {pull}16892[16892]
- Use Elasticsearch histogram type to store Prometheus histograms. {pull}17061[17061]
- Allow to rate Prometheus counters when scraping them. {pull}17061[17061]
- Release the Oracle module as GA. {issue}14279[14279] {pull}16833[16833]
- Add Storage metricsets to GCP module. {pull}15598[15598]
- Release the vsphere module as GA. {issue}15798[15798] {pull}17119[17119]
- Add PubSub metricset to Google Cloud Platform module. {pull}15536[15536]
- Add dashboard for `redisenterprise` module. {pull}16752[16752]
- Add dashboard for VSphere host cluster and virtual machine. {pull}14135[14135]
- Add test for documented fields check for metricsets without a http input. {issue}17315[17315] {pull}17334[17334]
- Release the azure module as GA. {pull}17319[17319]
- In the kubernetes manifests, mount the data directory from the host, so data persist between executions in the same node. {pull}17429[17429]
- Release the CockroachDB module as GA. {pull}32527[32527]

*Packetbeat*

- Add `dns.question.subdomain` and `dns.question.top_level_domain` fields. {pull}14578[14578]
- Add `redact_headers` configuration option to allow HTTP request headers to be redacted whilst keeping the header field included in the Beat. {pull}15353[15353]
- Enable setting promiscuous mode automatically. {pull}11366[11366]

*Winlogbeat*

- Add Audit and Log Management, Computer Object Management, and Distribution Group related events to the Security module. {pull}15217[15217]
- Add experimental event log reader implementation that should be faster in most cases. {issue}6585[6585] {pull}16849[16849]

[[release-notes-7.6.2]]
=== Beats version 7.6.2
https://github.com/elastic/beats/compare/v7.6.1\...v7.6.2[View commits]

==== Breaking changes

*Affecting all Beats*

- Fix an issue that could cause redundant configuration reloads. {pull}16440[16440]
- Fix metadata enrichers to use default config for kubernetes module. {pull}17020[17020]

*Metricbeat*

- Make use of secure port when accessing Kubelet API {pull}16063[16063]

==== Bugfixes

*Affecting all Beats*

- Fix k8s metadata issue regarding node labels not shown up on root level of metadata. {pull}16834[16834]

*Filebeat*

- Ensure all zeek timestamps include millisecond precision. {issue}14599[14599] {pull}16766[16766]
- Fix issue where autodiscover hints default configuration was not being copied. {pull}16987[16987]

*Metricbeat*

- Convert increments of 100 nanoseconds/ticks to milliseconds for WriteTime and ReadTime in diskio metricset (Windows) for consistency. {issue}14233[14233]
- Fix diskio issue for windows 32 bit on disk_performance struct alignment. {issue}16680[16680]

==== Added

*Affecting all Beats*

- Add monitoring variable `libbeat.config.scans` to distinguish scans of the configuration directory from actual reloads of its contents. {pull}16440[16440]

*Winlogbeat*

- Add more DNS error codes to the Sysmon module. {issue}15685[15685]

[[release-notes-7.6.1]]
=== Beats version 7.6.1
https://github.com/elastic/beats/compare/v7.6.0\...v7.6.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix k8s pods labels broken schema. {pull}16480[16480]
- Fix k8s pods annotations broken schema. {pull}16554[16554]

*Filebeat*

- Fix a connection error in httpjson input. {pull}16123[16123]
- Fix mapping error for cloudtrail additionalEventData field {pull}16088[16088]
- Rewrite azure filebeat dashboards, due to changes in kibana. {pull}16466[16466]
- Adding the var definitions in azure manifest files, fix for errors when executing command setup. {issue}16270[16270] {pull}16468[16468]

*Heartbeat*

- Fix scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. {pull}16397[16397]

*Metricbeat*

- Avoid parsing errors returned from prometheus endpoints. {pull}15712[15712]
- Change sqs metricset to use average as statistic method. {pull}16438[16438]

*Functionbeat*

- Fix timeout option of GCP functions. {issue}16282[16282] {pull}16287[16287]

==== Added

*Winlogbeat*

- Made the event parser more lenient w.r.t. invalid event log definition version numbers. {issue}15838[15838]

[[release-notes-7.6.0]]
=== Beats version 7.6.0
https://github.com/elastic/beats/compare/v7.5.1\...v7.6.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Remove version information from default ILM policy for improved upgrade experience on custom policies. {pull}14745[14745]
- Running `setup` cmd respects `setup.ilm.overwrite` setting for improved support of custom policies. {pull}14741[14741]
- Cleanup the x-pack licenser code to use the new license endpoint and the new format. Replaces the url /_xpack/license with /_license. {pull}15091[15091]
- The document id fields has been renamed from @metadata.id to @metadata._id {pull}15859[15859]
- Two Beat instances with the same data path cannot be run concurrently. {pull}14069[14069]

*Filebeat*

- CEF extensions are now mapped to the data types defined in the CEF guide. {pull}14342[14342]

*Journalbeat*

- Remove broken dashboard. {pull}15288[15288]

*Metricbeat*

- Update cloudwatch metricset mapping for both metrics and dimensions. {pull}15245[15245]

*Packetbeat*

- TLS: Fields have been changed to adapt to ECS. {pull}15497[15497]
- TLS: The behavior of send_certificates and include_raw_certificates options has changed. {pull}15497[15497]

==== Bugfixes

*Affecting all Beats*

- Fix spooling to disk blocking infinitely if the lock file can not be acquired. {pull}15338[15338]
- Fix `metricbeat test output` with an ipv6 ES host in the output.hosts. {pull}15368[15368]
- Fix `convert` processor conversion of string to integer with leading zeros. {issue}15513[15513] {pull}15557[15557]
- Fix existing agent.*, ecs.version, and host.name fields getting overwritten by Beats if they are already present in the original event. {pull}14407[14407]
- Fix issue where TLS settings would be ignored when a forward proxy was in use. {pull}15516[$15516]
- Beats no longer attempts to load dashboards if they are unavailable. {pull}15802[15802]

*Auditbeat*

- system/socket: Fix compatibility issue with kernel 5.x. {pull}15771[15771]

*Filebeat*

- Fix a problem in Filebeat input httpjson where interval is not used as time.Duration. {pull}14728[14728]
- Fix SSL config in input.yml for Filebeat httpjson input in the MISP module. {pull}14767[14767]
- Check content-type when creating new reader in s3 input. {pull}15252[15252] {issue}15225[15225]
- Fix session reset detection and a crash in Netflow input. {pull}14904[14904]
- Handle errors in handleS3Objects function and add more debug messages for s3 input. {pull}15545[15545]
- netflow: Allow for options templates without scope fields. {pull}15449[15449]
- netflow: Fix bytes/packets counters on some devices (NSEL and Netstream). {pull}15449[15449]
- netflow: Fix compatibility with some Cisco devices by changing the field `class_id` from short to long. {pull}15449[15449]
- Fix dashboard for Cisco ASA Firewall. {issue}15420[15420] {pull}15553[15553]
- Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. {issue}15502[15502] {pull}15590[15590]
- Add shared_credential_file to cloudtrail config. {issue}15652[15652] {pull}15656[15656]
- Fix typos in zeek notice fileset config file. {issue}15764[15764] {pull}15765[15765]
- Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the `elasticsearch` module. {issue}15840[15840] {pull}15900[15900]
- Improve `elasticsearch/audit` fileset to handle timestamps correctly. {pull}15942[15942]

*Heartbeat*

- Fix excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639]

*Metricbeat*

- Fix regular expression to detect instance name in perfmon metricset. {issue}14273[14273] {pull}14666[14666]
- Fix `docker.container.size` fields values {issue}14979[14979] {pull}15224[15224]
- Make `kibana` module more resilient to Kibana unavailability. {issue}15258[15258] {pull}15270[15270]
- Fix panic exception with some unicode strings in perfmon metricset. {issue}15264[15264]
- Make `logstash` module more resilient to Logstash unavailability. {issue}15276[15276] {pull}15306[15306]
- Add username/password in Metricbeat autodiscover hints {pull}15349[15349]
- Add dedot for tags in ec2 metricset and cloudwatch metricset. {issue}15843[15843] {pull}15844[15844]
- Use RFC3339 format for timestamps collected using the SQL module. {pull}15847[15847]
- Add dedot for cloudwatch metric name. {issue}15916[15916] {pull}15917[15917]
- Fixed issue `logstash-xpack` module suddenly ceasing to monitor Logstash. {issue}15974[15974] {pull}16044[16044]

==== Added

*Affecting all Beats*

- Add a friendly log message when a request to docker has exceeded the deadline. {pull}15336[15336]
- GA the `script` processor. {pull}14325[14325]
- Add `fingerprint` processor. {issue}11173[11173] {pull}14205[14205]
- Add support for API keys in Elasticsearch outputs. {pull}14324[14324]
- Add consumer_lag in Kafka consumergroup metricset {pull}14822[14822]
- Make use of consumer_lag in Kafka dashboard {pull}14863[14863]
- Refactor kubernetes autodiscover to enable different resource based discovery {pull}14738[14738]
- Add `add_id` processor. {pull}14524[14524]
- Enable TLS 1.3 in all beats. {pull}12973[12973]
- Spooling to disk creates a lockfile on each platform. {pull}15338[15338]
- Enable DEP (Data Execution Protection) for Windows packages. {pull}15149[15149]
- Users can now specify `monitoring.cloud.*` to override `monitoring.elasticsearch.*` settings. {issue}14399[14399] {pull}15254[15254]
- Add support to kubernetes autodiscovery to add additional metadata from other source to events. {pull}14875[14875]
- Update to ECS 1.4.0. {pull}14844[14844]
- Add document_id setting to decode_json_fields processor. {pull}15859[15859]

*Filebeat*

- Add new fileset googlecloud/audit for ingesting Google Cloud Audit logs. {pull}15200[15200]
- Add dashboards to the CEF module (ported from the Logstash ArcSight module). {pull}14342[14342]
- Add expand_event_list_from_field support in s3 input for reading json format AWS logs. {issue}15357[15357] {pull}15370[15370]
- Add azure-eventhub input which will use the azure eventhub go sdk. {issue}14092[14092] {pull}14882[14882]
- Expose more metrics of harvesters (e.g. `read_offset`, `start_time`). {pull}13395[13395]
- Include log.source.address for unparseable syslog messages. {issue}13268[13268] {pull}15453[15453]
- Release aws elb fileset as GA. {pull}15426[15426] {issue}15380[15380]
- Integrate the azure-eventhub with filebeat azure module (replace the kafka input). {pull}15480[15480]
- Release aws s3access fileset to GA. {pull}15431[15431] {issue}15430[15430]
- Add cloudtrail fileset to AWS module. {issue}14657[14657] {pull}15227[15227]
- New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. {pull}14553[14553]
- google-pubsub input: ACK pub/sub message when acknowledged by publisher. {issue}13346[13346] {pull}14715[14715]
- Remove Beta label from google-pubsub input. {issue}13346[13346] {pull}14715[14715]
- Add dashboard for AWS ELB fileset. {pull}15804[15804]
- Set event.outcome field based on googlecloud audit log output. {pull}15731[15731]
- Add dashboard for AWS vpcflow fileset. {pull}16007[16007]

*Heartbeat*

*Metricbeat*

- Expand data for the `system/memory` metricset {pull}15492[15492]
- Add azure `storage` metricset in order to retrieve metric values for storage accounts. {issue}14548[14548] {pull}15342[15342]
- Add cost warnings for the azure module. {pull}15356[15356]
- Release elb module as GA. {pull}15485[15485]
- Add a `system/network_summary` metricset {pull}15196[15196]
- Allow Metricbeat's beat module to read monitoring information over a named pipe or unix domain socket. {pull}14558[14558]
- Enable script processor. {pull}14711[14711]
- Add STAN dashboard {pull}15654[15654]

*Functionbeat*

- Add monitoring info about triggered functions. {pull}14876[14876]
- Add Google Cloud Platform support. {pull}13598[13598]

[[release-notes-7.5.2]]
=== Beats version 7.5.2
https://github.com/elastic/beats/compare/v7.5.1\...v7.5.2[View commits]

==== Breaking changes

*Journalbeat*

- Remove broken dashboard. {pull}15288[15288]

==== Bugfixes

*Affecting all Beats*

- Fix `convert` processor conversion of string to integer with leading zeros. {issue}15513[15513] {pull}15557[15557]

*Filebeat*

- Check content-type when creating new reader in s3 input. {pull}15252[15252] {issue}15225[15225]
- Fix session reset detection and a crash in Netflow input. {pull}14904[14904]
- netflow: Allow for options templates without scope fields. {pull}15449[15449]
- netflow: Fix bytes/packets counters on some devices (NSEL and Netstream). {pull}15449[15449]
- netflow: Fix compatibility with some Cisco devices by changing the field `class_id` from short to long. {pull}15449[15449]
- Fix dashboard for Cisco ASA Firewall. {issue}15420[15420] {pull}15553[15553]

*Metricbeat*

- Fix regular expression to detect instance name in perfmon metricset. {issue}14273[14273] {pull}14666[14666]
- Fix `docker.container.size` fields values {issue}14979[14979] {pull}15224[15224]
- Make `kibana` module more resilient to Kibana unavailability. {issue}15258[15258] {pull}15270[15270]
- Fix panic exception with some unicode strings in perfmon metricset. {issue}15264[15264]
- Make `logstash` module more resilient to Logstash unavailability. {issue}15276[15276] {pull}15306[15306]

==== Added

*Affecting all Beats*

- Add a friendly log message when a request to docker has exceeded the deadline. {pull}15336[15336]

*Filebeat*

- Include log.source.address for unparseable syslog messages. {issue}13268[13268] {pull}15453[15453]

[[release-notes-7.5.1]]
=== Beats version 7.5.1
https://github.com/elastic/beats/compare/v7.5.0\...v7.5.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix `proxy_url` option in Elasticsearch output. {pull}14950[14950]
- Fix bug with potential concurrent reads and writes from event.Meta map by Kafka output. {issue}14542[14542] {pull}14568[14568]
- Fix license detection, when a beats successfully connect to Elasticsearch the detected license will be show in the log at info level. {pull}15834[15834]
- Fix the `parameters` option configured in the Elasticsearch output so the values are added to the query string on bulk request. {issue}18325[18325]

*Filebeat*

- Change iis url path grok pattern from URIPATH to NOTSPACE. {issue}12710[12710] {pull}13225[13225] {issue}7951[7951] {pull}13378[13378] {pull}14754[14754]
- Fix azure filesets test files. {issue}14185[14185] {pull}14235[14235]
- Update Logstash module's Grok patterns to support Logstash 7.4 logs. {pull}14743[14743]
- Remove references to non-existent Zeek `signatures` fileset. {pull}18878[18878]

*Metricbeat*

- Fix perfmon expanding counter path/adding counter to query when OS language is not english. {issue}14684[14684] {pull}14800[14800]
- Add extra check on `ignore_non_existent_counters` flag if the PdhExpandWildCardPathW returns no errors but does not expand the counter path successfully in windows/perfmon metricset. {pull}14797[14797]
- Fix rds metricset from reporting same values for different instances. {pull}14702[14702]
- Closing handler after verifying the registry key in diskio metricset. {issue}14683[14683] {pull}14759[14759]
- Fix docker network stats when multiple interfaces are configured. {issue}14586[14586] {pull}14825[14825]
- Fix ListMetrics pagination in aws module. {issue}14926[14926] {pull}14942[14942]
- Fix CPU count in docker/cpu in cases where no `online_cpus` are reported {pull}15070[15070]
- Add domain state to kvm module {pull}17673[17673]
- Fix Kubernetes Overview Dashboard to correctly display non 10s intervals for node usage {pull}19675[19675]

[[release-notes-7.5.0]]
=== Beats version 7.5.0
https://github.com/elastic/beats/compare/v7.4.1\...v7.5.0[View commits]

==== Breaking changes

*Affecting all Beats*

- By default, all Beats-created files and folders will have a umask of 0027 (on POSIX systems). {pull}14119[14119]

*Filebeat*

*Heartbeat*

- JSON/Regex checks against HTTP bodies will only consider the first 100MiB of the HTTP body to prevent excessive memory usage. {pull}14223[14223]

*Metricbeat*

==== Bugfixes

*Affecting all Beats*

- Disable `add_kubernetes_metadata` if no matchers found. {pull}13709[13709]
- Better wording for xpack beats when the _xpack endpoint is not reachable. {pull}13771[13771]
- Kubernetes watcher at `add_kubernetes_metadata` fails with StatefulSets {pull}13905[13905]
- Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS or Beats that accept connections over TLS and validate client certificates. {pull}14146[14146]
- Fix memory leak in kubernetes autodiscover provider and add_kubernetes_metadata processor happening when pods are terminated without sending a delete event. {pull}14259[14259]
- Fix kubernetes `metaGenerator.ResourceMetadata` when parent reference controller is nil {issue}14320[14320] {pull}14329[14329]

*Auditbeat*

- Socket dataset: Fix start errors when IPv6 is disabled on the kernel. {issue}13953[13953] {pull}13966[13966]

*Filebeat*

- Fix a denial of service flaw when parsing malformed DSA public keys in Go.
If {filebeat} is configured to accept incoming TLS connections with client
authentication enabled, a remote attacker could cause the Beat to stop
processing events. (CVE-2019-17596) See https://www.elastic.co/community/security/
- Fix timezone parsing of rabbitmq module ingest pipelines. {pull}13879[13879]
- Fix conditions and error checking of date processors in ingest pipelines that use `event.timezone` to parse dates. {pull}13883[13883]
- Fix timezone parsing of Cisco module ingest pipelines. {pull}13893[13893]
- Fix timezone parsing of logstash module ingest pipelines. {pull}13890[13890]
- Fix timezone parsing of iptables, mssql and panw module ingest pipelines. {pull}13926[13926]
- Fixed increased memory usage with large files when multiline pattern does not match. {issue}14068[14068]
- Fix azure fields names. {pull}14098[14098] {pull}14132[14132]
- Fix calculation of `network.bytes` and `network.packets` for bi-directional netflow events. {pull}14111[14111]
- Accept '-' as http.response.body.bytes in apache module. {pull}14137[14137]
- Fix timezone parsing of MySQL module ingest pipelines. {pull}14130[14130]
- Improve error message in s3 input when handleSQSMessage failed. {pull}14113[14113]
- Fix race condition in S3 input plugin. {pull}14359[14359]

*Heartbeat*

- Fix storage of HTTP bodies to work when JSON/Regex body checks are enabled. {pull}14223[14223]

*Metricbeat*

- Fix a denial of service flaw when parsing malformed DSA public keys in Go.
If {metricbeat} is configured to accept incoming TLS connections with client
authentication enabled, a remote attacker could cause the Beat to stop
processing events. (CVE-2019-17596) See https://www.elastic.co/community/security/
- PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function. {issue}12590[12590] {pull}12622[12622]
- Fix `docker.cpu.system.pct` calculation by using the reported number online cpus instead of the number of metrics per cpu. {pull}13691[13691]
- Change kubernetes.event.message to text {pull}13964[13964]
- Fix performance counter values for windows/perfmon metricset.{issue}14036[14036] {pull}14039[14039] {pull}14108[14108]
- Add FailOnRequired when applying schema and fix metric names in mongodb metrics metricset. {pull}14143[14143]
- Convert indexed ms-since-epoch timestamp fields in `elasticsearch/ml_job` metricset to ints from float64s. {issue}14220[14220] {pull}14222[14222]
- Fix ARN parsing function to work for ELB ARNs. {pull}14316[14316]
- Update azure configuration example. {issue}14224[14224]
- Limit some of the error messages to the logs only {issue}14317[14317] {pull}14327[14327]
- Fix cloudwatch metricset with names and dimensions in config. {issue}14376[14376] {pull}14391[14391]
- Fix marshaling of ms-since-epoch values in `elasticsearch/cluster_stats` metricset. {pull}14378[14378]

*Packetbeat*

- Fix parsing of the HTTP host header when it contains a port or an IPv6 address. {pull}14215[14215]


==== Added

*Affecting all Beats*

- Fail with error when autodiscover providers have no defined configs. {pull}13078[13078]
- Add autodetection mode for add_docker_metadata and enable it by default in included configuration files{pull}13374[13374]
- Add autodetection mode for add_kubernetes_metadata and enable it by default in included configuration files. {pull}13473[13473]
- Use less restrictive API to check if template exists. {pull}13847[13847]
- Do not check for alias when setup.ilm.check_exists is false. {pull}13848[13848]
- Add support for numeric time zone offsets in timestamp processor. {pull}13902[13902]
- Add condition to the config file template for add_kubernetes_metadata {pull}14056[14056]
- Marking Central Management deprecated. {pull}14018[14018]
- Add `keep_null` setting to allow Beats to publish null values in events. {issue}5522[5522] {pull}13928[13928]
- Add shared_credential_file option in aws related config for specifying credential file directory. {issue}14157[14157] {pull}14178[14178]
- Ensure that init containers are no longer tailed after they stop. {pull}14394[14394]
- Libbeat HTTP's Server can listen to a unix socket using the `unix:///tmp/hello.sock` syntax. {pull}13655[13655]
- Libbeat HTTP's Server can listen to a Windows named pipe using the `npipe:///hello` syntax. {pull}13655[13655]
- Adding new `Enterprise` license type to the licenser. {issue}14246[14246]
- Add endpoint config in AWS config to support using custom endpoint accessing AWS APIs. {issue}16245[16245] {pull}16263[16263]

*Auditbeat*

- Socket: Add DNS enrichment. {pull}14004[14004]

*Filebeat*

- Add support for virtual host in Apache access logs {pull}12778[12778]
- Update CoreDNS module to populate ECS DNS fields. {issue}13320[13320] {pull}13505[13505]
- Parse query steps in PostgreSQL slowlogs. {issue}13496[13496] {pull}13701[13701]
- Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. {pull}13776[13776]
- Add support to set the document id in the json reader. {pull}5844[5844]
- Add input httpjson. {issue}13545[13545] {pull}13546[13546]
- Filebeat Netflow input: Remove beta label. {pull}13858[13858]
- Remove `event.timezone` from events that don't need it in some modules that support log formats with and without timezones. {pull}13918[13918]
- Add ExpandEventListFromField config option in the kafka input. {pull}13965[13965]
- Add ELB fileset to AWS module. {pull}14020[14020]
- Add module for MISP (Malware Information Sharing Platform). {pull}13805[13805]
- Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. {pull}13776[13776] {pull}14033[14033] {pull}14107[14107]
- Add support for all the ObjectCreated events in S3 input. {pull}14077[14077]
- Add `source.bytes` and `source.packets` for uni-directional netflow events. {pull}14111[14111]
- Add Kibana Dashboard for MISP module. {pull}14147[14147]
- Add support for gzipped files in S3 input {pull}13980[13980]
- Add Filebeat Azure Dashboards {pull}14127[14127]
- Add support for space or time sync character before timestamp in syslog input. {pull}13278[13278] {issue}13269[13269]
- Add support for thread ID in Filebeat Kafka module. {pull}19463[19463]

*Heartbeat*

- Add non-privileged icmp on linux and darwin(mac). {pull}13795[13795] {issue}11498[11498]
- Allow `hosts` to be used to configure http monitors {pull}13703[13703]

*Metricbeat*

- Add refresh list of perf counters at every fetch {issue}13091[13091]
- Add proc/vmstat data to the system/memory metricset on linux {pull}13322[13322]
- Add support for NATS version 2. {pull}13601[13601]
- Add `docker.cpu.*.norm.pct` metrics for `cpu` metricset of Docker Metricbeat module. {pull}13695[13695]
- Add `instance` label by default when using Prometheus collector. {pull}13737[13737]
- Add azure module. {pull}13196[13196] {pull}13859[13859] {pull}13988[13988]
- Add Apache Tomcat module {pull}13491[13491]
- Add ECS `container.id` and `container.runtime` to kubernetes `state_container` metricset. {pull}13884[13884]
- Add `job` label by default when using Prometheus collector. {pull}13878[13878]
- Add `state_resourcequota` metricset for Kubernetes module. {pull}13693[13693]
- Add tags filter in ec2 metricset. {pull}13872[13872] {issue}13145[13145]
- Add cloud.account.id and cloud.account.name into events from aws module. {issue}13551[13551] {pull}13558[13558]
- Add `metrics_path` as known hint for autodiscovery {pull}13996[13996]
- Leverage KUBECONFIG when creating k8s client. {pull}13916[13916]
- Add ability to filter by tags for cloudwatch metricset. {pull}13758[13758] {issue}13145[13145]
- Release cloudwatch, s3_daily_storage, s3_request, sqs and rds metricset as GA. {pull}14114[14114] {issue}14059[14059]
- Add `elasticsearch/enrich` metricset. {pull}14243[14243] {issue}14221[14221]
- Add new dashboards for Azure vms, vm guest metrics, vm scale sets {pull}14000[14000]
- Add vpc metricset for aws module. {pull}16111[16111] {issue}14854[14854]

*Functionbeat*

- Make `bulk_max_size` configurable in outputs. {pull}13493[13493]

*Winlogbeat*

- Fill `event.provider`. {pull}13937[13937]
- Add support for user management events to the Security module. {pull}13530[13530]

==== Deprecated

*Metricbeat*

- `kubernetes.container.id` field for `state_container` is deprecated in favour of ECS `container.id` and `container.runtime`. {pull}13884[13884]

[[release-notes-7.4.2]]
=== Beats version 7.4.2
https://github.com/elastic/beats/compare/v7.4.1\...v7.4.2[View commits]

==== Bugfixes

*Filebeat*

- panw module: Use geo.name instead of geo.country_iso_code for free-form location. {issue}13272[13272]

[[release-notes-7.4.1]]
=== Beats version 7.4.1
https://github.com/elastic/beats/compare/v7.4.0\...v7.4.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Recover from panics in the javascript process and log details about the failure to aid in future debugging. {pull}13690[13690]
- Make the script processor concurrency-safe. {issue}13690[13690] {pull}13857[13857]

*Filebeat*

- Fixed early expiration of templates (Netflow v9 and IPFIX). {pull}13821[13821]
- Fixed bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). {pull}13821[13821]
- cisco asa and ftd filesets: Fix parsing of message 106001. {issue}13891[13891] {pull}13903[13903]
- Fix merging of fields specified in global scope with fields specified under an input's scope. {issue}3628[3628] {pull}13909[13909]
- Fix delay in enforcing close_renamed and close_removed options. {issue}13488[13488] {pull}13907[13907]
- Fix missing netflow fields in index template. {issue}13768[13768] {pull}13914[13914]
- Fix cisco module's asa and ftd filesets parsing of domain names where an IP address is expected. {issue}14034[14034]

*Metricbeat*

- Mark Kibana usage stats as collected only if API call succeeds. {pull}13881[13881]

[[release-notes-7.4.0]]
=== Beats version 7.4.0
https://github.com/elastic/beats/compare/v7.3.1\...v7.4.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Update to Golang 1.12.7. {pull}12931[12931]
- Remove `in_cluster` configuration parameter for Kuberentes, now in-cluster configuration is used only if no other kubeconfig is specified {pull}13051[13051]

*Auditbeat*

- Socket dataset: New implementation using Kprobes for finer-grained monitoring and UDP support. {pull}13058[13058]

*Filebeat*

- Fix a race condition in the TCP input when close the client socket. {pull}13038[13038]
- cisco/asa fileset: Renamed log.original to event.original and cisco.asa.list_id to cisco.asa.rule_name. {pull}13286[13286]
- cisco/asa fileset: Fix parsing of 302021 message code. {pull}13476[13476]

*Metricbeat*

- Add new Dashboard for PostgreSQL database stats {pull}13187[13187]
- Add new dashboard for CouchDB database {pull}13198[13198]
- Add new dashboard for Ceph cluster stats {pull}13216[13216]
- Add new dashboard for Aerospike database stats {pull}13217[13217]
- Add new dashboard for Couchbase cluster stats {pull}13212[13212]
- Add new dashboard for Prometheus server stats {pull}13126[13126]
- Add statistic option into cloudwatch metricset. If there is no statistic method specified, default is to collect Average, Sum, Maximum, Minimum and SampleCount. {issue}12370[12370] {pull}12840[12840]
- Fix rds metricset dashboard. {pull}13721[13721]

*Functionbeat*

- Separate management and functions in Functionbeat. {pull}12939[12939]

==== Bugfixes

*Affecting all Beats*

- ILM: Use GET instead of HEAD when checking for alias to expose detailed error message. {pull}12886[12886]
- Fix unexpected stops on docker autodiscover when a container is restarted before `cleanup_timeout`. {issue}12962[12962] {pull}13127[13127]
- Fix some incorrect types and formats in field.yml files. {pull}13188[13188]
- Load DLLs only from Windows system directory. {pull}13234[13234] {pull}13384[13384]
- Fix mapping for kubernetes.labels and kubernetes.annotations in add_kubernetes_metadata. {issue}12638[12638] {pull}13226[13226]
- Fix case insensitive regular expressions not working correctly. {pull}13250[13250]

*Auditbeat*

- Host dataset: Export Host fields to gob encoder. {pull}12940[12940]

*Filebeat*

- Fix filebeat autodiscover fileset hint for container input. {pull}13296[13296]
- Fix incorrect references to index patterns in AWS and CoreDNS dashboards. {pull}13303[13303]
- Fix timezone parsing of system module ingest pipelines. {pull}13308[13308]
- Fix timezone parsing of elasticsearch module ingest pipelines. {pull}13367[13367]
- Change iis url path grok pattern from URIPATH to NOTSPACE. {issue}12710[12710] {pull}13225[13225] {issue}7951[7951] {pull}13378[13378]
- Add timezone information to apache error fileset. {issue}12772[12772] {pull}13304[13304]
- Fix timezone parsing of nginx module ingest pipelines. {pull}13369[13369]
- Allow path variables to be used in files loaded from modules.d. {issue}13184[13184]
- Fix incorrect field references in envoyproxy dashboard {issue}13420[13420] {pull}13421[13421]

*Heartbeat*

- Fix integer comparison on JSON responses. {pull}13348[13348]

*Metricbeat*

- Ramdisk is not filtered out when collecting disk performance counters in diskio metricset {issue}12814[12814] {pull}12829[12829]
- Fix redis key metricset dashboard references to index pattern. {pull}13303[13303]
- Check if fields in DBInstance is nil in rds metricset. {pull}13294[13294] {issue}13037[13037]
- Fix silent failures in kafka and prometheus module. {pull}13353[13353] {issue}13252[13252]
- Fix module-level fields in Kubernetes metricsets. {pull}13433[13433] {pull}13544[13544]
- Fix panic in Redis Key metricset when collecting information from a removed key. {pull}13426[13426]
- In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn't report load average. {pull}12866[12866]
- Print errors that were being omitted in vSphere metricsets. {pull}12816[12816]
- Fix issue with aws cloudwatch module where dimensions and/or namespaces that contain space are not being parsed correctly {pull}13389[13389]
- Fix reporting empty events in cloudwatch metricset. {pull}13458[13458]
- Fix data race affecting config validation at startup. {issue}13005[13005]

*Packetbeat*

- Fix parsing the extended RCODE in the DNS parser. {pull}12805[12805]

*Functionbeat*

- Fix Cloudwatch logs timestamp to use timestamp of the log record instead of when the record was processed {pull}13291[13291]
- Look for the keystore under the correct path. {pull}13332[13332]

==== Added

*Affecting all Beats*

- Add support for reading the `network.iana_number` field by default to the community_id processor. {pull}12701[12701]
- Add a check so alias creation explicitely fails if there is an index with the same name. {pull}13070[13070]
- Update kubernetes watcher to use official client-go libraries. {pull}13051[13051]
- Add support for unix epoch time values in the `timestamp` processor. {pull}13319[13319]
- add_host_metadata is now GA. {pull}13148[13148]
- Add an `ignore_missing` configuration option the `drop_fields` processor. {pull}13318[13318]
- Add `registered_domain` processor for deriving the registered domain from a given FQDN. {pull}13326[13326]
- Add support for RFC3339 time zone offsets in JSON output. {pull}13227[13227]
- Added `monitoring.cluster_uuid` setting to associate Beat data with specified ES cluster in Stack Monitoring UI. {pull}13182[13182]

*Filebeat*

- Add netflow dashboards based on Logstash netflow. {pull}12857[12857]
- Parse more fields from Elasticsearch slowlogs. {pull}11939[11939]
- Update module pipelines to enrich events with autonomous system fields. {pull}13036[13036]
- Add module for ingesting IBM MQ logs. {pull}8782[8782]
- Add S3 input to retrieve logs from AWS S3 buckets. {pull}12640[12640] {issue}12582[12582]
- Add aws module s3access metricset. {pull}13170[13170] {issue}12880[12880]
- Update Suricata module to populate ECS DNS fields and handle EVE DNS version 2. {issue}13320[13320] {pull}13329[13329]
- Update PAN-OS fileset to use the ECS NAT fields. {issue}13320[13320] {pull}13330[13330]
- Add fields to the Zeek DNS fileset for ECS DNS. {issue}13320[13320] {pull}13324[13324]
- Add container image in Kubernetes metadata {pull}13356[13356] {issue}12688[12688]
- Add module for ingesting Cisco FTD logs over syslog. {pull}13286[13286]

*Heartbeat*

- Record HTTP body metadata and optionally contents in `http.response.body.*` fields. {pull}13022[13022]

*Metricbeat*

- Add Kubernetes proxy dashboard to Kubernetes module {pull}12734[12734]
- Add Kubernetes controller manager dashboard to Kubernetes module {pull}12744[12744]
- Add metrics to kubernetes apiserver metricset. {pull}12922[12922]
- Add Kubernetes scheduler dashboard to Kubernetes module {pull}12749[12749]
- Collect client provided name for rabbitmq connection. {issue}12851[12851] {pull}12852[12852]
- Add support to load default aws config file to get credentials. {pull}12727[12727] {issue}12708[12708]
- Add statistic option into cloudwatch metricset. {issue}12370[12370] {pull}12840[12840]
- Add support for kubernetes cronjobs {pull}13001[13001]
- Add cgroup memory stats to docker/memory metricset {pull}12916[12916]
- Add AWS elb metricset. {pull}12952[12952] {issue}11701[11701]
- Add AWS ebs metricset. {pull}13167[13167] {issue}11699[11699]
- Add `metricset.period` field with the configured fetching period. {pull}13242[13242] {issue}12616[12616]
- Add rate metrics for ec2 metricset. {pull}13203[13203]
- Add Performance metricset to Oracle module {pull}12547[12547]
- Use DefaultMetaGeneratorConfig in MetadataEnrichers to initialize configurations {pull}13414[13414]
- Add module for statsd. {pull}13109[13109]

*Packetbeat*

- Update DNS protocol plugin to produce events with ECS fields for DNS. {issue}13320[13320] {pull}13354[13354]

*Functionbeat*

- Add timeout option to reference configuration. {pull}13351[13351]
- Configurable tags for Lambda functions. {pull}13352[13352]
- Add input for Cloudwatch logs through Kinesis. {pull}13317[13317]
- Enable Logstash output. {pull}13345[13345]

*Winlogbeat*

- Add support for event ID 4634 and 4647 to the Security module. {pull}12906[12906]
- Add `network.community_id` to Sysmon network events (event ID 3). {pull}13034[13034]
- Add `event.module` to Winlogbeat modules. {pull}13047[13047]
- Add `event.category: process` and `event.type: process_start/process_end` to Sysmon process events (event ID 1 and 5). {pull}13047[13047]
- Add support for event ID 4672 to the Security module. {pull}12975[12975]
- Add support for event ID 22 (DNS query) to the Sysmon module. {pull}12960[12960]
- Add support for event ID 4634 and 4647 to the Security module. {pull}12906[12906]
- Add `network.community_id` to Sysmon network events (event ID 3). {pull}13034[13034]
- Add `event.module` to Winlogbeat modules. {pull}13047[13047]
- Add `event.category: process` and `event.type: process_start/process_end` to Sysmon process events (event ID 1 and 5). {pull}13047[13047]
- Add support for event ID 4672 to the Security module. {pull}12975[12975]
- Add support for event ID 22 (DNS query) to the Sysmon module. {pull}12960[12960]
- Add certain winlog.event_data.* fields to the index template. {issue}13700[13700] {pull}13704[13704]

[[release-notes-7.3.2]]
=== Beats version 7.3.2
https://github.com/elastic/beats/compare/v7.3.1\...v7.3.2[View commits]

==== Bugfixes

*Filebeat*

- Fix filebeat autodiscover fileset hint for container input. {pull}13296[13296]
- Fix timezone parsing of system module ingest pipelines. {pull}13308[13308]
- Fix timezone parsing of elasticsearch module ingest pipelines. {pull}13367[13367]
- Fix timezone parsing of nginx module ingest pipelines. {pull}13369[13369]

*Metricbeat*

- Fix module-level fields in Kubernetes metricsets. {pull}13433[13433] {pull}13544[13544]
- Fix panic in Redis Key metricset when collecting information from a removed key. {pull}13426[13426]

[[release-notes-7.3.1]]
=== Beats version 7.3.1
https://github.com/elastic/beats/compare/v7.3.0\...v7.3.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix install-service.ps1's ability to set Windows service's delay start configuration. {pull}13173[13173]
- Fix `decode_base64_field` processor. {pull}13092[13092], {pull}13144[13144]

*Filebeat*

- Fix multiline pattern in Postgres which was too permissive. {issue}12078[12078] {pull}13069[13069]

*Metricbeat*

- Fix `logstash/node_stats` metricset to also collect `logstash_stats.events.duration_in_millis` field when `xpack.enabled: true` is set. {pull}13082[13082]
- Fix `logstash/node` metricset to also collect `logstash_state.pipeline.representation.{type,version,hash}` fields when `xpack.enabled: true` is set. {pull}13133[13133]

==== Added

*Metricbeat*

- Make the `beat` module defensive about determining ES cluster UUID when `xpack.enabled: true` is set. {pull}13020[13020]

[[release-notes-7.3.0]]
=== Beats version 7.3.0
https://github.com/elastic/beats/compare/v7.2.0\...v7.3.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Update to ECS 1.0.1. {pull}12284[12284] {pull}12317[12317]
- Default of output.kafka.metadata.full is set to false by now. This reduced the amount of metadata to be queried from a kafka cluster. {pull}12738[12738]

*Filebeat*

- `convert_timezone` option is removed and locale is always added to the event so timezone is used when parsing the timestamp, this behaviour can be overriden with processors. {pull}12410[12410]

==== Bugfixes

*Affecting all Beats*

- Fix typo in TLS renegotiation configuration and setting the option correctly {issue}10871[10871], {pull}12354[12354]
- Add configurable bulk_flush_frequency in kafka output. {pull}12254[12254]
- Fixed setting bulk max size in kafka output. {pull}12254[12254]
- Add additional nil pointer checks to Docker client code to deal with vSphere Integrated Containers {pull}12628[12628]
- Fix seccomp policy preventing some features to function properly on 32bit Linux systems. {issue}12990[12990] {pull}13008[13008]

*Auditbeat*

- Package dataset: Close librpm handle. {pull}12215[12215]
- Package dataset: Improve dpkg parsing. {pull}12325[12325]
- Host dataset: Fix reboot detection logic. {pull}12591[12591]
- Add syscalls used by librpm for the system/package dataset to the default Auditbeat seccomp policy. {issue}12578[12578] {pull}12617[12617]
- Host dataset: Export Host fields to gob encoder. {pull}12940[12940]

*Filebeat*

- Parse timezone in PostgreSQL logs as part of the timestamp {pull}12338[12338]
- When TLS is configured for the TCP input and a `certificate_authorities` is configured we now default to `required` for the `client_authentication`. {pull}12584[12584]
- Syslog input will now omit the `process` object from events if it is empty. {pull}12700[12700]
- Apply `max_message_size` to incoming message buffer. {pull}11966[11966]

*Heartbeat*


*Journalbeat*

- Iterate over journal correctly, so no duplicate entries are sent. {pull}12716[12716]
- Preserve host name when reading from remote journal. {pull}12714[12714]

*Metricbeat*

- Refactored Windows perfmon metricset: replaced method to retrieve counter paths with PdhExpandWildCardPathW, separated code by responsibility, removed unused functions {pull}12212[12212]
- Validate that kibana/status metricset cannot be used when xpack is enabled. {pull}12264[12264]
- In the kibana/stats metricset, only log error (don't also index it) if xpack is enabled. {pull}12265[12265]
- Fix an issue listing all processes when run under Windows as a non-privileged user. {issue}12301[12301] {pull}12475[12475]
- When TLS is configured for the http metricset and a `certificate_authorities` is configured we now default to `required` for the `client_authentication`. {pull}12584[12584]
- Reuse connections in PostgreSQL metricsets. {issue}12504[12504] {pull}12603[12603]
- PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function.{issue}12590[12590]{pull}12622[12622]
- Print errors that were being omitted in vSphere metricsets {pull}12816[12816]
- In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn't report load average. {pull}12866[12866]
- Fix incoherent behaviour in redis key metricset when keyspace is specified both in host URL and key pattern {pull}12913[12913]
- Fix connections leak in redis module {pull}12914[12914] {pull}12950[12950]

*Packetbeat*


==== Added

*Affecting all Beats*

- Add `proxy_disable` output flag to explicitly ignore proxy environment variables. {issue}11713[11713] {pull}12243[12243]
- Processor `add_cloud_metadata` adds fields `cloud.account.id` and `cloud.image.id` for AWS EC2. {pull}12307[12307]
- Add `decode_base64_field` processor for decoding base64 field. {pull}11914[11914]
- Add aws overview dashboard. {issue}11007[11007] {pull}12175[12175]
- Add `decompress_gzip_field` processor. {pull}12733[12733]
- Add `timestamp` processor for parsing time fields. {pull}12699[12699]
- Add Oracle Tablespaces Dashboard {pull}12736[12736]
- Add `proxy_disable` output flag to explicitly ignore proxy environment variables. {issue}11713[11713] {pull}12243[12243]

*Auditbeat*


*Filebeat*

- Add timeouts on communication with docker daemon. {pull}12310[12310]
- Add specific date processor to convert timezones so same pipeline can be used when convert_timezone is enabled or disabled. {pull}12253[12253]
- Add MSSQL module {pull}12079[12079]
- Add ISO8601 date parsing support for system module. {pull}12568[12568] {pull}12578[12579]
- Update Kubernetes deployment manifest to use `container` input. {pull}12632[12632]
- Add `google-pubsub` input type for consuming messages from a Google Cloud Pub/Sub topic subscription. {pull}12746[12746]
- Add module for ingesting Cisco IOS logs over syslog. {pull}12748[12748]
- Add module for ingesting Google Cloud VPC flow logs. {pull}12747[12747]
- Report host metadata for Filebeat logs in Kubernetes. {pull}12790[12790]

*Metricbeat*

- Add overview dashboard to Consul module {pull}10665[10665]
- New fields were added in the mysql/status metricset. {pull}12227[12227]
- Add Kubernetes metricset `proxy`. {pull}12312[12312]
- Always report Pod UID in the `pod` metricset. {pull}12345[12345]
- Add Vsphere Virtual Machine operating system to `os` field in Vsphere virtualmachine module. {pull}12391[12391]
- Add CockroachDB module. {pull}12467[12467]
- Add support for metricbeat modules based on existing modules (a.k.a. light modules) {issue}12270[12270] {pull}12465[12465]
- Add a system/entropy metricset {pull}12450[12450]
- Add kubernetes metricset `controllermanager` {pull}12409[12409]
- Allow redis URL format in redis hosts config. {pull}12408[12408]
- Add tags into ec2 metricset. {issue}12263[12263] {pull}12372[12372]
- Add kubernetes metricset `scheduler` {pull}12521[12521]
- Add Kubernetes scheduler dashboard to Kubernetes module {pull}12749[12749]
- Add `beat` module. {pull}12181[12181] {pull}12615[12615]
- Collect tags for cloudwatch metricset in aws module. {issue}12263[12263] {pull}12480[12480]
- Add AWS RDS metricset. {pull}11620[11620] {issue}10054[10054]
- Add Oracle Module {pull}11890[11890]
- Add Kubernetes proxy dashboard to Kubernetes module {pull}12734[12734]
- Add Kubernetes controller manager dashboard to Kubernetes module {pull}12744[12744]

*Functionbeat*

- Export automation templates used to create functions. {pull}11923[11923]
- Configurable Amazon endpoint. {pull}12369[12369]

==== Deprecated

*Filebeat*

- `postgresql.log.timestamp` field is deprecated in favour of `@timestamp`. {pull}12338[12338]

[[release-notes-7.2.1]]
=== Beats version 7.2.1
https://github.com/elastic/beats/compare/v7.2.0\...v7.2.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix Central Management enroll under Windows {issue}12797[12797] {pull}12799[12799]
- Fixed a crash under Windows when fetching processes information. {pull}12833[12833]

*Filebeat*

- Add support for client addresses with port in Apache error logs {pull}12695[12695]
- Load correct pipelines when system module is configured in modules.d. {pull}12340[12340]

*Metricbeat*

- Fix wrong uptime reporting by system/uptime metricset under Windows. {pull}12915[12915]

*Packetbeat*

- Limit memory usage of Redis replication sessions. {issue}12657[12657]

[[release-notes-7.2.0]]
=== Beats version 7.2.0
https://github.com/elastic/beats/compare/v7.1.1\...v7.2.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Update to Golang 1.12.4. {pull}11782[11782]

*Auditbeat*

- Auditd module: Normalized value of `event.category` field from `user-login` to `authentication`. {pull}11432[11432]
- Auditd module: Unset `auditd.session` and `user.audit.id` fields are removed from audit events. {issue}11431[11431] {pull}11815[11815]
- Socket dataset: Exclude localhost by default {pull}11993[11993]

*Filebeat*

- Add read_buffer configuration option. {pull}11739[11739]

*Heartbeat*

- Removed the `add_host_metadata` and `add_cloud_metadata` processors from the default config. These don't fit well with ECS for Heartbeat and were rarely used.

*Journalbeat*

*Metricbeat*

- Add new option `OpMultiplyBuckets` to scale histogram buckets to avoid decimal points in final events {pull}10994[10994]
- system/raid metricset now uses /sys/block instead of /proc/mdstat for data. {pull}11613[11613]

*Packetbeat*

- Add support for mongodb opcode 2013 (OP_MSG). {issue}6191[6191] {pull}8594[8594]
- NFSv4: Always use opname `ILLEGAL` when failed to match request to a valid nfs operation. {pull}11503[11503]

*Winlogbeat*

*Functionbeat*

==== Bugfixes

*Affecting all Beats*

- Ensure all beat commands respect configured settings. {pull}10721[10721]
- Add missing fields and test cases for libbeat add_kubernetes_metadata processor. {issue}11133[11133], {pull}11134[11134]
- decode_json_field: process objects and arrays only {pull}11312[11312]
- decode_json_field: do not process arrays when flag not set. {pull}11318[11318]
- Report faulting file when config reload fails. {pull}11304[11304]
- Fix a typo in libbeat/outputs/transport/client.go by updating `c.conn.LocalAddr()` to `c.conn.RemoteAddr()`. {pull}11242[11242]
- Management configuration backup file will now have a timestamps in their name. {pull}11034[11034]
- [CM] Parse enrollment_token response correctly {pull}11648[11648]
- Not hiding error in case of http failure using elastic fetcher {pull}11604[11604]
- Escape BOM on JsonReader before trying to decode line {pull}11661[11661]
- Fix matching of string arrays in contains condition. {pull}11691[11691]
- Replace wmi queries with win32 api calls as they were consuming CPU resources {issue}3249[3249] and {issue}11840[11840]
- Fix queue.spool.write.flush.events config type. {pull}12080[12080]
- Fixed a memory leak when using the add_process_metadata processor under Windows. {pull}12100[12100]
- Fix of docker json parser for missing "log" jsonkey in docker container's log {issue}11464[11464]
- Fixed Beat ID being reported by GET / API. {pull}12180[12180]
- Add host.os.codename to fields.yml. {pull}12261[12261]
- Fix `@timestamp` being duplicated in events if `@timestamp` is set in a
  processor (or by any code utilizing `PutValue()` on a `beat.Event`).
- Fix leak in script processor when using Javascript functions in a processor chain. {pull}12600[12600]

*Auditbeat*

- Process dataset: Fixed a memory leak under Windows. {pull}12100[12100]
- Login dataset: Fix re-read of utmp files. {pull}12028[12028]
- Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. {issue}12147[12147] {pull}12168[12168]
- Fix formatting of config files on macOS and Windows. {pull}12148[12148]
- Fix direction of incoming IPv6 sockets. {pull}12248[12248]
- Package dataset: Auto-detect package directories. {pull}12289[12289]
- System module: Start system module without host ID. {pull}12373[12373]

*Filebeat*

- Add support for Cisco syslog format used by their switch. {pull}10760[10760]
- Cover empty request data, url and version in Apache2 module{pull}10730[10730]
- Fix registry entries not being cleaned due to race conditions. {pull}10747[10747]
- Improve detection of file deletion on Windows. {pull}10747[10747]
- Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. {pull}11591[11591]
- Reduce memory usage if long lines are truncated to fit `max_bytes` limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. {pull}11524[11524]
- Fix memory leak in Filebeat pipeline acker. {pull}12063[12063]
- Fix goroutine leak caused on initialization failures of log input. {pull}12125[12125]
- Fix goroutine leak on non-explicit finalization of log input. {pull}12164[12164]
- Require client_auth by default when ssl is enabled for tcp input {pull}12333[12333]
- Fix timezone offset parsing in system/syslog. {pull}12529[12529]

*Heartbeat*

- Fix NPEs / resource leaks when executing config checks. {pull}11165[11165]
- Fix duplicated IPs on `mode: all` monitors. {pull}12458[12458]

*Journalbeat*

- Use backoff when no new events are found. {pull}11861[11861]

*Metricbeat*

- Change diskio metrics retrieval method (only for Windows) from wmi query to DeviceIOControl function using the IOCTL_DISK_PERFORMANCE control code {pull}11635[11635]
- Call GetMetricData api per region instead of per instance. {issue}11820[11820] {pull}11882[11882]
- Update documentation with cloudwatch:ListMetrics permission. {pull}11987[11987]
- Check permissions in system socket metricset based on capabilities. {pull}12039[12039]
- Get process information from sockets owned by current user when system socket metricset is run without privileges. {pull}12039[12039]
- Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. {issue}8264[8264] {pull}12086[12086]
- Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. {pull}11393[11393]
- Change some field type from scaled_float to long in aws module. {pull}11982[11982]
- Fixed RabbitMQ `queue` metricset gathering when `consumer_utilisation` is set empty at the metrics source {pull}12089[12089]
- Fix direction of incoming IPv6 sockets. {pull}12248[12248]
- Ignore prometheus metrics when their values are NaN or Inf. {pull}12084[12084] {issue}10849[10849]
- Require client_auth by default when ssl is enabled for module http metricset server{pull}12333[12333]
- The `elasticsearch/index_summary` metricset gracefully handles an empty Elasticsearch cluster when `xpack.enabled: true` is set. {pull}12489[12489] {issue}12487[12487]

*Packetbeat*

- Prevent duplicate packet loss error messages in HTTP events. {pull}10709[10709]
- Fixed a memory leak when using process monitoring under Windows. {pull}12100[12100]
- Improved debug logging efficiency in PGQSL module. {issue}12150[12150]

*Winlogbeat*

*Functionbeat*

- Fix function name reference for Kinesis streams in CloudFormation templates {pull}11646[11646]

==== Added

*Affecting all Beats*

- Add an option to append to existing logs rather than always rotate on start. {pull}11953[11953]
- Add `network` condition to processors for matching IP addresses against CIDRs. {pull}10743[10743]
- Add if/then/else support to processors. {pull}10744[10744]
- Add `community_id` processor for computing network flow hashes. {pull}10745[10745]
- Add output test to kafka output {pull}10834[10834]
- Gracefully shut down on SIGHUP {pull}10704[10704]
- New processor: `copy_fields`. {pull}11303[11303]
- Add `error.message` to events when `fail_on_error` is set in `rename` and `copy_fields` processors. {pull}11303[11303]
- New processor: `truncate_fields`. {pull}11297[11297]
- Allow a beat to ship monitoring data directly to an Elasticsearch monitoring clsuter. {pull}9260[9260]
- Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. {pull}NNNN[NNNN]
- Add `add_observer_metadata` processor. {pull}11394[11394]
- Add `decode_csv_fields` processor. {pull}11753[11753]
- Add `convert` processor for converting data types of fields. {issue}8124[8124] {pull}11686[11686]
- New `extract_array` processor. {pull}11761[11761]
- Add number of goroutines to reported metrics. {pull}12135[12135]

*Auditbeat*

- Auditd module: Add `event.outcome` and `event.type` for ECS. {pull}11432[11432]
- Process: Add file hash of process executable. {pull}11722[11722]
- Socket: Add network.transport and network.community_id. {pull}12231[12231]
- Host: Fill top-level host fields. {pull}12259[12259]

*Filebeat*

- Add more info to message logged when a duplicated symlink file is found {pull}10845[10845]
- Add option to configure docker input with paths {pull}10687[10687]
- Add Netflow module to enrich flow events with geoip data. {pull}10877[10877]
- Set `event.category: network_traffic` for Suricata. {pull}10882[10882]
- Allow custom default settings with autodiscover (for example, use of CRI paths for logs). {pull}12193[12193]
- Allow to disable hints based autodiscover default behavior (fetching all logs). {pull}12193[12193]
- Change Suricata module pipeline to handle `destination.domain` being set if a reverse DNS processor is used. {issue}10510[10510]
- Add the `network.community_id` flow identifier to field to the IPTables, Suricata, and Zeek modules. {pull}11005[11005]
- New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. {pull}11200[11200]
- New module for Cisco ASA logs. {issue}9200[9200] {pull}11171[11171]
- Added support for Cisco ASA fields to the netflow input. {pull}11201[11201]
- Configurable line terminator. {pull}11015[11015]
- Add Filebeat envoyproxy module. {pull}11700[11700]
- Add apache2(httpd) log path (`/var/log/httpd`) to make apache2 module work out of the box on Redhat-family OSes. {issue}11887[11887] {pull}11888[11888]
- Add support to new MongoDB additional diagnostic information {pull}11952[11952]
- New module `panw` for Palo Alto Networks PAN-OS logs. {pull}11999[11999]
- Add RabbitMQ module. {pull}12032[12032]
- Add new `container` input. {pull}12162[12162]

*Heartbeat*

- Enable `add_observer_metadata` processor in default config. {pull}11394[11394]

*Metricbeat*

- Add AWS SQS metricset. {pull}10684[10684] {issue}10053[10053]
- Add AWS s3_request metricset. {pull}10949[10949] {issue}10055[10055]
- Add s3_daily_storage metricset. {pull}10940[10940] {issue}10055[10055]
- Add `coredns` metricbeat module. {pull}10585[10585]
- Add SSL support for Metricbeat HTTP server. {pull}11482[11482] {issue}11457[11457]
- The `elasticsearch.index` metricset (with `xpack.enabled: true`) now collects `refresh.external_total_time_in_millis` fields from Elasticsearch. {pull}11616[11616]
- Allow module configurations to have variants {pull}9118[9118]
- Add `timeseries.instance` field calculation. {pull}10293[10293]
- Added new disk states and raid level to the system/raid metricset. {pull}11613[11613]
- Added `path_name` and `start_name` to service metricset on windows module {issue}8364[8364] {pull}11877[11877]
- Add check on object name in the counter path if the instance name is missing {issue}6528[6528] {pull}11878[11878]
- Add AWS cloudwatch metricset. {pull}11798[11798] {issue}11734[11734]
- Add `regions` in aws module config to specify target regions for querying cloudwatch metrics. {issue}11932[11932] {pull}11956[11956]
- Keep `etcd` followers members from reporting `leader` metricset events {pull}12004[12004]
- Add validation for elasticsearch and kibana modules' metricsets when `xpack.enabled` is set to `true`. {pull}12386[12386]

*Functionbeat*

- Add new options to configure roles and VPC. {pull}11779[11779]

*Winlogbeat*

- Add support for reading from `.evtx` files. {issue}4450[4450]

==== Deprecated

*Filebeat*

- Deprecate `docker` input in favor of `container`. {pull}12162[12162]

*Functionbeat*

==== Known Issue

*Journalbeat*

[[release-notes-7.1.1]]
=== Beats version 7.1.1
https://github.com/elastic/beats/compare/v7.1.0\...v7.1.1[View commits]

No changes in this release.

[[release-notes-7.1.0]]
=== Beats version 7.1.0
https://github.com/elastic/beats/compare/v7.0.0\...v7.1.0[View commits]

* Updates to support changes to licensing of security features.
+
Some Elastic Stack security features, such as encrypted communications, file and native authentication, and
role-based access control, are now available in more subscription levels. For details, see https://www.elastic.co/subscriptions.

[[release-notes-7.0.1]]
=== Beats version 7.0.1
https://github.com/elastic/beats/compare/v7.0.0\...v7.0.1[View commits]

==== Breaking changes

*Metricbeat*

- Change cloud.provider from ec2 to aws and from gce to gcp in add_cloud_metadata to align with ECS. {issue}10775[10775] {pull}11687[11687]

==== Bugfixes

*Affecting all Beats*

- Fix formatting for `event.duration`, "human readable" was not working well for this. {pull}11675[11675]
- Fix initialization of the TCP input logger. {pull}11605[11605]

*Auditbeat*

- Package dataset: Log error when Homebrew is not installed. {pull}11667[11667]

*Heartbeat*

- Fix NPE on some monitor configuration errors. {pull}11910[11910]

*Metricbeat*

- Change `add_cloud_metadata` processor to not overwrite `cloud` field when it already exist in the event. {pull}11612[11612] {issue}11305[11305]

==== Added

*Auditbeat*

- Add support to the system package dataset for the SUSE OS family. {pull}11634[11634]

==== Deprecated

*Metricbeat*

- Prevent the docker/memory metricset from processing invalid events before container start {pull}11676[11676]

include::libbeat/docs/release-notes/7.0.0.asciidoc[]

[[release-notes-7.0.0-ga]]
=== Beats version 7.0.0-GA
https://github.com/elastic/beats/compare/v7.0.0-rc2\...v7.0.0[View commits]

The list below covers the changes between 7.0.0-rc2 and 7.0.0 GA only.

==== Bugfixes

*Affecting all Beats*

- Relax validation of the X-Pack license UID value. {issue}11640[11640]
- Fix a parsing error with the X-Pack license check on 32-bit system. {issue}11650[11650]
- Fix ILM policy always being overwritten. {pull}11671[11671]
- Fix template always being overwritten. {pull}11671[11671]

*Auditbeat*

- Package dataset: Nullify Librpm's rpmsqEnable. {pull}11628[11628]

*Filebeat*

- Fix `add_docker_metadata` source matching, using `log.file.path` field now. {pull}11577[11577]

[[release-notes-7.0.0-rc2]]
=== Beats version 7.0.0-rc2
https://github.com/elastic/beats/compare/v7.0.0-rc1\...v7.0.0-rc2[Check the HEAD diff]

==== Breaking changes

*Auditbeat*

- Process dataset: Only report processes with executable. {pull}11232[11232]
- Shorten entity IDs. {pull}11405[11405]

*Metricbeat*

- Add connection and request timeouts for HTTP helper. {pull}11032[11032]

==== Bugfixes

*Affecting all Beats*

- Fixed OS family classification in `add_host_metadata` for Amazon Linux, Raspbian, and RedHat Linux. {issue}9134[9134] {pull}11494[11494]
- Allow 'ilm.rollover_alias' to expand global fields like `agent.version`. {issue}12233[12233]

*Auditbeat*

- Package dataset: dlopen versioned librpm shared objects. {pull}11565[11565]

*Filebeat*

- Don't apply multiline rules in Logstash json logs. {pull}11346[11346]
- Fix panic in add_kubernetes_metadata processor when key `log` does not exist. {issue}11543[11543] {pull}11549[11549]
- Fix goroutine leak happening when harvesters are dynamically stopped. {pull}11263[11263]

*Metricbeat*

- Add _bucket to histogram metrics in Prometheus Collector {pull}11578[11578]

==== Added

*Auditbeat*

- Login dataset: Add event category and type. {pull}11339[11339]

*Filebeat*

- Add support for MySQL 8.0, Percona 8.0 and MariaDB 10.3. {pull}11417[11417]

[[release-notes-7.0.0-rc1]]
=== Beats version 7.0.0-rc1
https://github.com/elastic/beats/compare/v7.0.0-beta1\...v7.0.0-rc1[Check the HEAD diff]

==== Breaking changes

*Affecting all Beats*

- On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project
  info from the cloud.machine.type and cloud.availability_zone. {issue}10968[10968]
- Add `cleanup_timeout` option to docker autodiscover, to wait some time before removing configurations after a container is stopped. {issue}10374[10374] {pull}10905[10905]
- Empty `meta.json` file will be treated as a missing meta file. {issue}8558[8558]
- Rename `migration.enabled` config to `migration.6_to_7.enabled`. {pull}11284[11284]
- Initialize the Paths before the keystore and save the keystore into `data/{beatname}.keystore`. {pull}10706[10706]
- Beats Xpack now checks for Basic license on connect. {pull}11296[11296]

*Auditbeat*

- Process dataset: Only report processes with executable. {pull}11232[11232]

*Filebeat*

- Set `ecs: true` in user_agent processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. {issue}10655[10655] {pull}10875[10875]

*Metricbeat*

- Migrate docker module to ECS. {pull}10927[10927]

*Functionbeat*

- Correctly extract Kinesis Data field from the Kinesis Record. {pull}11141[11141]

==== Bugfixes

*Affecting all Beats*

- Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. {pull}10988[10988]
- Add missing host.* fields to fields.yml. {pull}11016[11016]
- Include ip and boolean type when generating index pattern. {pull}10995[10995]
- Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn't exist. {pull}10936[10936]
- Cancelling enrollment of a beat will not enroll the beat. {issue}10150[10150]
- Allow to configure Kafka fetching strategy for the topic metadata. {pull}10682[10682]

*Auditbeat*

- Package: Disable librpm signal handlers. {pull}10694[10694]
- Login: Handle different bad login UTMP types. {pull}10865[10865]
- System module: Fix and unify bucket closing logic. {pull}10897[10897]
- User dataset: Numerous fixes to error handling. {pull}10942[10942]

*Filebeat*

- Fix errors in filebeat Zeek dashboard and README files. Add notice.log support. {pull}10916[10916]
- Fix a bug when converting NetFlow fields to snake_case. {pull}10950[10950]
- Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. {issue}11004[11004] {pull}11105[11105]
- Fix issue preventing docker container events to be stored if the container has a network interface without ip address. {issue}11225[11225] {pull}11247[11247]
- Change URLPATH grok pattern to support brackets. {issue}11135[11135] {pull}11252[11252]
- Add support for iis log with different address format. {issue}11255[11255] {pull}11256[11256]

*Heartbeat*

- Fix checks for TCP send/receive data {pull}11118[11118]

*Metricbeat*

- Migrate docker autodiscover to ECS. {issue}10757[10757] {pull}10862[10862]
- Fix issue in kubernetes module preventing usage percentages to be properly calculated. {pull}10946[10946]
- Fix for not reusable http client leading to connection leaks in Jolokia module {pull}11014[11014]
- Fix parsing error using GET in Jolokia module. {pull}11075[11075] {issue}11071[11071]
- Collect metrics when EC2 instances are not in running state. {issue}11008[11008] {pull}11023[11023]
- Change ECS field cloud.provider to aws. {pull}11023[11023]
- Add documentation about jolokia autodiscover fields. {issue}10925[10925] {pull}10979[10979]
- Add missing aws.ec2.instance.state.name into fields.yml. {issue}11219[11219] {pull}11221[11221]
- Fix ec2 metricset to collect metrics from Cloudwatch with the same timestamp. {pull}11142[11142]
- Fix potential memory leak in stopped docker metricsets {pull}11294[11294]

*Packetbeat*

- Avoid reporting unknown MongoDB opcodes more than once. {pull}10878[10878]

*Winlogbeat*

- Prevent Winlogbeat from dropping events with invalid XML. {pull}11006[11006]
- Fix Winlogbeat escaping CR, LF and TAB characters. {issue}11328[11328] {pull}11357[11357]

*Functionbeat*

==== Added

*Affecting all Beats*

- Add ip fields to default_field in Elasticsearch template. {pull}11035[11035]

*Auditbeat*

- Move System module to beta. {pull}10800[10800]

*Filebeat*

- Add ISO8601 timestamp support in syslog metricset. {issue}8716[8716] {pull}10736[10736]
- Add support for loading custom NetFlow and IPFIX field definitions to netflow input. {pull}10945[10945] {pull}11223[11223]
- Added categorization fields for SSH login events in the system/auth fileset. {pull}11334[11334]

*Metricbeat*

- Add filters and pie chart for AWS EC2 dashboard. {pull}10596[10596]

*Winlogbeat*

- Add an `index` option to all event logs to specify the output index for events from that source. {pull}15062[15062]


==== Known Issue

*Journalbeat*

- Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

[[release-notes-7.0.0-beta1]]
=== Beats version 7.0.0-beta1
https://github.com/elastic/beats/compare/v7.0.0-alpha2\...v7.0.0-beta1[Check the HEAD diff]

==== Breaking changes

*Affecting all Beats*

- Embedded html is not escaped anymore by default. {pull}9914[9914]
- Remove port settings from Logstash and Redis output. {pull}9934[9934]
- Rename `process.exe` to `process.executable` in add_process_metadata to align with ECS. {pull}9949[9949]
- Import ECS change https://github.com/elastic/ecs/pull/308[ecs#308]:
  leaf field `user.group` is now the `group` field set. {pull}10275[10275]
- Update the code of Central Management to align with the new returned format. {pull}10019[10019]
- Docker and Kubernetes labels/annotations will be "dedoted" by default. {pull}10338[10338]
- Remove --setup command line flag. {pull}10138[10138]
- Remove --version command line flag. {pull}10138[10138]
- Remove --configtest command line flag. {pull}10138[10138]
- Move output.elasticsearch.ilm settings to setup.ilm. {pull}10347[10347]
- ILM will be available by default if Elasticsearch > 7.0 is used. {pull}10347[10347]

*Auditbeat*

- Rename `process.exe` to `process.executable` in auditd module to align with ECS. {pull}9949[9949]
- Rename `process.cwd` to `process.working_directory` in auditd module to align with ECS. {pull}10195[10195]
- Change data type of `process.pid` and `process.ppid` to number in JSON output
  of the auditd module. {pull}10195[10195]
- Change data type of `file.uid` and `file.gid` to string in JSON output of the
  FIM module. {pull}10195[10195]
- Field `file.origin` changed type from `text` to `keyword`. {pull}10544[10544]
- Rename user fields to ECS in auditd module. {pull}10456[10456]
- Rename `event.type` to `auditd.message_type` in auditd module because event.type is reserved for future use by ECS. {pull}10536[10536]
- Rename `auditd.messages` to `event.original` and `auditd.warnings` to `error.message`. {pull}10577[10577]

*Filebeat*

- Rename many `kibana.log.*` fields to map to ECS. {pull}9301[9301]
- Modify apache/error dataset to follow ECS. {pull}8963[8963]
- Rename many `traefik.access.*` fields to map to ECS. {pull}9005[9005]
- Fix parsing of GC entries in elasticsearch server log. {issue}9513[9513] {pull}9810[9810]
- Rename `read_timestamp` to `event.created` for Redis input. {pull}9924[9924]
- Rename a few `elasticsearch.audit.*` fields to map to ECS. {pull}9293[9293]
- Rename `read_timestamp` to `event.created` for all Filebeat modules using it. {pull}10139[10139]
- Rename many `iis.error.*` fields to map to ECS. {pull}9955[9955]
- Adjust fileset `haproxy.log` to map to ECS. {pull}10143[10143]
- Rename a few `logstash.*` fields to map to ECS, remove logstash.slowlog.message. {pull}9935[9935]
- Rename a few `mongodb.*` fields to map to ECS. {pull}10009[10009]
- Rename a few `mysql.*` fields to map to ECS. {pull}10008[10008]
- Rename a few `nginx.error.*` fields to map to ECS. {pull}10007[10007]
- Rename many `auditd.log.*` fields to map to ECS. {pull}10192[10192]
- Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001]
- Remove service.name from Elastcsearch module. Replace by service.type. {pull}10042[10042]
- Remove numeric coercions for `user.id` and `group.id`. IDs should be `keyword`. {pull}10233[10233]
- Add grok pattern to support redis 5.0.3 log timestamp. {issue}9819[9819] {pull}10033[10033]
- Now save the 'first seen' timestamp in `event.created` (previously `read_timestamp`),
  instead of saving the parsed date. Now aligned with `event.created` semantics elsewhere. {pull}10139[10139]
- Rename `mysql.error.thread_id` and `mysql.slowlog.id` to `mysql.thread_id`. {pull}10161[10161]
- Remove `mysql.error.timestamp`  and `mysql.slowlog.timestamp`. {pull}10161[10161]
- Migrate multiple fields to `event.duration`, from modules "apache", "elasticsearch",
  "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik",
  including `http.response.elapsed_time` (ECS). {pull}10188[10188], {pull}10274[10274]
- Rename multiple fields to `http.response.body.bytes`, from modules "apache", "iis",
  "kibana", "nginx" and "traefik", including `http.response.content_length` (ECS). {pull}10188[10188]
- Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, `raw_request_line`, `mode`. {pull}10397[10397]
- Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. {pull}10401[10401]
- Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above {pull}10352[10352]
- Migrate Elasticsearch audit logs fields to ECS {pull}10352[10352]
- Several text fields in the Logstash module are now indexed as `keyword` fields with `text` multi-fields (ECS). {pull}10417[10417]
- Several text fields in the Elasticsearch module are now indexed as `keyword` fields with `text` multi-fields (ECS). {pull}10414[10414]
- Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. {pull}10442[10442]
- The `elasticsearch/deprecation` fileset now indexes the `component` field under `elasticsearch` instead of `elasticsearch.server`. {pull}10445[10445]
- Remove field `kafka.log.trace.full` from kafka.log fielset. {pull}10398[10398]
- Change field `kafka.log.class` for kafka.log fileset from text to keyword. {pull}10398[10398]
- Address add_kubernetes_metadata processor issue where old source field is
  still used for matcher. {issue}10505[10505] {pull}10506[10506]
- Change type of haproxy.source from text to keyword. {pull}10506[10506]
- Rename `event.type` to `suricata.eve.event_type` in Suricata module because event.type is reserved for future use by ECS. {pull}10575[10575]
- Populate more ECS fields in the Suricata module. {pull}10006[10006]
- Rename setting `filebeat.registry_flush` to `filebeat.registry.flush`. {pull}10504[10504]
- Rename setting `filebeat.registry_file_permission` to `filebeat.registry.file_permission`. {pull}10504[10504]
- Remove setting `filebeat.registry_file` in favor of `filebeat.registry.path`. The registry file will be stored in a sub-directory by now. {pull}10504[10504]

*Heartbeat*

- Remove monitor generator script that was rarely used. {pull}9648[9648]
- monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you'll need to set the `id` property explicitly. {pull}9697[9697]
- A number of fields have been aliased to their relevant counterparts in the `url.*` field. Existing visualizations should mostly work. The fields that have been moved are `monitor.scheme -> url.scheme`, `monitor.host -> url.domain`, `resolve.host -> url.domain`, `http.url -> url.full`,  `tcp.port -> url.port`. In addition to these moves the new fields `url.username`, `url.password`, `url.path`, and `url.query` are now present. It should be noted that the `url.password` field does not contain actual password values, but rather the text `<hidden>` {pull}9570[9570].
- The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. {pull}10294[10294]

*Journalbeat*

- Rename read_timestamp to event.created to align with ECS. {pull}10043[10043], {pull}10139[10139]
- Rename host.name to host.hostname to align with ECS. {pull}10043[10043]
- Fix typo in the field name `container.id_truncated`. {pull}10525[10525]
- Rename `container.image.tag` to `container.log.tag`. {pull}10561[10561]
- Change type of `text` fields to `keyword`. {pull}10542[10542]

*Metricbeat*

- Migrate system process metricset fields to ECS. {pull}10332[10332]
- Refactor Prometheus metric mappings {pull}9948[9948]
- Removed Prometheus stats metricset in favor of just using Prometheus collector {pull}9948[9948]
- Migrate system socket metricset fields to ECS. {pull}10339[10339]
- Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. {pull}10339[10339]
- Adjust Redis.info metricset fields to ECS. {pull}10319[10319]
- Change type of field docker.container.ip_addresses to `ip` instead of `keyword`. {pull}10364[10364]
- Rename http.request.body field to http.request.body.content. {pull}10315[10315]
- Adjust php_fpm.process metricset fields to ECS. {pull}10366[10366]
- Adjust mongodb.status metricset to to ECS. {pull}10368[10368]
- Refactor munin module to collect an event per plugin and to have more strict field mappings. `namespace` option has been removed, and will be replaced by `service.name`. {pull}10322[10322]
- Change the following fields from type text to keyword: {pull}10318[10318]
  - ceph.osd_df.name
  - ceph.osd_tree.name
  - ceph.osd_tree.children
  - kafka.consumergroup.meta
  - kibana.stats.name
  - mongodb.metrics.replication.executor.network_interface
  - php_fpm.process.request_uri
  - php_fpm.process.script
- Add `service.name` option to all modules to explicitly set `service.name` if it is unset. {pull}10427[10427]
- Update a few elasticsearch.* fields to map to ECS. {pull}10350[10350]
- Update a few logstash.* fields to map to ECS. {pull}10350[10350]
- Update a few kibana.* fields to map to ECS. {pull}10350[10350]
- Update rabbitmq.* fields to map to ECS. {pull}10563[10563]
- Update haproxy.* fields to map to ECS. {pull}10558[10558] {pull}10568[10568]
- Collect all EC2 meta data from all instances in all states. {pull}10628[10628]
- Fix MongoDB dashboard that had some incorrect field names from `status` Metricset {pull}9795[9795] {issue}9715[9715]

*Packetbeat*

- Adjust Packetbeat `http` fields to ECS Beta 2 {pull}9645[9645]
  - `http.request.body` moves to `http.request.body.content`
  - `http.response.body` moves to `http.response.body.content`
- Changed Packetbeat fields to align with ECS. {issue}7968[7968]
- Removed trailing dot from domain names reported by the DNS protocol. {pull}9941[9941]

*Winlogbeat*

- Adjust Winlogbeat fields to map to ECS. {pull}10333[10333]

*Functionbeat*

- Correctly normalize Cloudformation resource name. {issue}10087[10087]
- Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]
- Allow functionbeat to use the keystore. {issue}9009[9009]

==== Bugfixes

*Affecting all Beats*

- Fix config appender registration. {pull}9873[9873]
- Gracefully handle TLS options when enrolling a Beat. {issue}9129[9129]
- The backing off now implements jitter to better distribute the load. {issue}10172[10172]
- Fix TLS certificate DoS vulnerability. {pull}10302[10302]
- Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. {pull}10289[10289]
- Fix encoding of timestamps when using disk spool. {issue}10099[10099]
- Fix stopping of modules started by kubernetes autodiscover. {pull}10476[10476]
- Fix a issue when remote and local configuration didn't match when fetching configuration from Central Management. {issue}10587[10587]
- Fix unauthorized error when loading dashboards by adding username and password into kibana config. {issue}10513[10513] {pull}10675[10675]
- Fix exclude_labels when there are dotted keys {pull}10154[10154]
- Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). {pull}9920[9920]

*Auditbeat*

- Enable System module config on Windows. {pull}10237[10237]

*Filebeat*

- Support IPv6 addresses with zone id in IIS ingest pipeline.
  {issue}9836[9836] error log: {pull}9869[9869], access log: {pull}9955[9955].
- Support haproxy log lines without captured headers. {issue}9463[9463] {pull}9958[9958]
- Make elasticsearch/audit fileset be more lenient in parsing node name. {issue}10035[10035] {pull}10135[10135]
- Fix bad bytes count in `docker` input when filtering by stream. {pull}10211[10211]
- Fixed data types for roles and indices fields in `elasticsearch/audit` fileset {pull}10307[10307]
- Ensure `source.address` is always populated by the nginx module (ECS). {pull}10418[10418]
- Support mysql 5.7.22 slowlog starting with time information. {issue}7892[7892] {pull}9647[9647]

*Heartbeat*

- Made monitors.d configuration part of the default config. {pull}9004[9004]
- Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace.  {pull}9566[9566]

*Journalbeat*

- Do not stop collecting events when journal entries change. {pull}9994[9994]

*Metricbeat*

- Fix panics in vsphere module when certain values where not returned by the API. {pull}9784[9784]
- Fix pod UID metadata enrichment in Kubernetes module. {pull}10081[10081]
- Fix issue that would prevent collection of processes without command line on Windows. {pull}10196[10196]
- Fixed data type for tags field in `docker/container` metricset {pull}10307[10307]
- Fixed data type for tags field in `docker/image` metricset {pull}10307[10307]
- Fixed data type for isr field in `kafka/partition` metricset {pull}10307[10307]
- Fixed data types for various hosts fields in `mongodb/replstatus` metricset {pull}10307[10307]
- Added function to close sql database connection. {pull}10355[10355]
- Fix issue with `elasticsearch/node_stats` metricset (x-pack) not indexing `source_node` field. {pull}10639[10639]

*Packetbeat*

- Fix DHCPv4 dashboard that wouldn't load in Kibana. {issue}9850[9850]
- Fixed a crash when using af_packet capture {pull}10477[10477]

*Winlogbeat*

- Close handle on signalEvent. {pull}9838[9838]

*Functionbeat*

- Ensure that functionbeat is logging at info level not debug. {issue}10262[10262]
- Add the required permissions to the role when deployment SQS functions. {issue}9152[9152]

==== Added

*Affecting all Beats*

- Update field definitions for `http` to ECS Beta 2 {pull}9645[9645]
- Add `agent.id` and `agent.ephemeral_id` fields to all beats. {pull}9404[9404]
- Add `name` config option to `add_host_metadata` processor. {pull}9943[9943]
- Add `add_labels` and `add_tags` processors. {pull}9973[9973]
- Add missing file encoding to readers. {pull}10080[10080]
- Introduce `migration.enabled` configuration. {pull}9805[9805]
- Add alias field support in Kibana index pattern. {pull}10075[10075]
- Add `add_fields` processor. {pull}10119[10119]
- Add Kibana field formatter to bytes fields. {pull}10184[10184]
- Document a few more `auditd.log.*` fields. {pull}10192[10192]
- Support Kafka 2.1.0. {pull}10440[10440]
- Add ILM mode `auto` to setup.ilm.enabled setting. This new default value detects if ILM is available {pull}10347[10347]
- Add support to read ILM policy from external JSON file. {pull}10347[10347]
- Add `overwrite` and `check_exists` settings to ILM support. {pull}10347[10347]
- Generate Kibana index pattern on demand instead of using a local file. {pull}10478[10478]
- Calls to Elasticsearch X-Pack APIs made by Beats won't cause deprecation logs in Elasticsearch logs. {9656}9656[9656]
- Allow to unenroll a Beat from the UI. {issue}9452[9452]
- Release Jolokia autodiscover as GA. {pull}9706[9706]
- Allow Central Management to send events back to kibana. {issue}9382[9382]

*Auditbeat*

- Add system module. {pull}9546[9546]
- Add `user.id` (UID) and `user.name` for ECS. {pull}10195[10195]
- Add `group.id` (GID) and `group.name` for ECS. {pull}10195[10195]
- System module `process` dataset: Add user information to processes. {pull}9963[9963]
- Add system `package` dataset. {pull}10225[10225]
- Add system module `login` dataset. {pull}9327[9327]
- Add `entity_id` fields. {pull}10500[10500]
- Add seven dashboards for the system module. {pull}10511[10511]

*Filebeat*

- Add `convert_timezone` option to Elasticsearch module to convert dates to UTC. {issue}9756[9756] {pull}9761[9761]
- Added module for parsing Google Santa logs. {pull}9540[9540]
- Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. {issue}9399[9399]
- Add option to modules.yml file to indicate that a module has been moved {pull}9432[9432].
- Add support for ssl_request_log in apache2 module. {issue}8088[8088] {pull}9833[9833]
- Add support for iis 7.5 log format. {issue}9753[9753] {pull}9967[9967]
- Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with `service.type` config. {pull}10042[10042]
- Add support for MariaDB in the `slowlog` fileset of `mysql` module. {pull}9731[9731]
- Apache module's error fileset now performs GeoIP lookup, like the access fileset. {pull}10273[10273]
- Elasticsearch module's slowlog now populates `event.duration` (ECS). {pull}9293[9293]
- HAProxy module now populates `event.duration` and `http.response.bytes` (ECS). {pull}10143[10143]
- Teach elasticsearch/audit fileset to parse out some more fields. {issue}10134[10134] {pull}10137[10137]
- Add convert_timezone to nginx module. {issue}9839[9839] {pull}10148[10148]
- Add support for Percona in the `slowlog` fileset of `mysql` module. {issue}6665[6665] {pull}10227[10227]
- Added support for ingesting structured Elasticsearch audit logs {pull}10352[10352]
- Added support for ingesting structured Elasticsearch slow logs {pull}10445[10445]
- Added support for ingesting structured Elasticsearch deprecation logs {pull}10445[10445]
- New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. {issue}8781[8781] {pull}10176[10176]
- Added support for ingesting structured Elasticsearch server logs {pull}10428[10428]
- Populate more ECS fields in the Suricata module. {pull}10006[10006]
- Add module zeek. {issue}9931[9931] {pull}10034[10034]

*Heartbeat*

- Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you'll see the correct fields under the `docker` key. {pull}10258[10258]

*Journalbeat*

- Migrate registry from previously incorrect path. {pull}10486[10486]

*Metricbeat*

- Add `key` metricset to the Redis module. {issue}9582[9582] {pull}9657[9657] {pull}9746[9746]
- Add `socket_summary` metricset to system defaults, removing experimental tag and supporting Windows {pull}9709[9709]
- Add docker `event` metricset. {pull}9856[9856]
- Add 'performance' metricset to x-pack mssql module {pull}9826[9826]
- Add DeDot for kubernetes labels and annotations. {issue}9860[9860] {pull}9939[9939]
- Add more meaningful metrics to 'performance' Metricset on 'MSSQL' module {pull}10011[10011]
- Rename some fields in `performance` Metricset on MSSQL module to match the updated documentation from Microsoft {pull}10074[10074]
- Add AWS EC2 module. {pull}9257[9257] {issue}9300[9300]
- Release windows Metricbeat module as GA. {pull}10163[10163]
- Release traefik Metricbeat module as GA. {pull}10166[10166]
- Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. {pull}10094[10094]
- List filesystems on Windows that have an access path but not an assigned letter {issue}8916[8916] {pull}10196[10196]
- Add `nats` module. {issue}10071[10071]
- Release uswgi Metricbeat module GA. {pull}10164[10164]
- Release php_fpm module as GA. {pull}10198[10198]
- Release Memcached module as GA. {pull}10199[10199]
- Release etcd module as GA. {pull}10200[10200]
- Release Ceph module as GA. {pull}10202[10202]
- Release aerospike module as GA. {pull}10203[10203]
- Release kubernetes apiserver and event metricsets as GA {pull}10212[10212]
- Release Couchbase module as GA. {pull}10201[10201]
- Release RabbitMQ module GA. {pull}10165[10165]
- Release envoyproxy module GA. {pull}10223[10223]
- Release mongodb.metrics and mongodb.replstatus as GA. {pull}10242[10242]
- Release mysql.galera_status as GA. {pull}10242[10242]
- Release postgresql.statement as GA. {pull}10242[10242]
- Release RabbitMQ Metricbeat module GA. {pull}10165[10165]
- Release Dropwizard module as GA. {pull}10240[10240]
- Release Graphite module as GA. {pull}10240[10240]
- Release kvm module as beta. {pull}10279[10279]
- Release http.server metricset as GA. {pull}10240[10240]
- Release Nats module as GA. {pull}10281[10281]
- Release munin module as GA. {pull}10311[10311]
- Release Golang module as GA. {pull}10312[10312]
- Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. {pull}10222[10222]
- Add support for MySQL 8.0 and tests also for Percona and MariaDB. {pull}10261[10261]
- Rename 'db' Metricset to 'transaction_log' in MSSQL Metricbeat module {pull}10109[10109]
- Add process arguments and the path to its executable file in the system process metricset {pull}10332[10332]
- Added 'server' Metricset to Zookeeper Metricbeat module {issue}8938[8938] {pull}10341[10341]
- Release AWS module as GA. {pull}10345[10345]
- Add overview dashboard to Zookeeper Metricbeat module {pull}10379[10379]

*Packetbeat*

- Add `network.community_id` to Packetbeat flow events. {pull}10061[10061]
- Add aliases for flow fields that were renamed. {issue}7968[7968] {pull}10063[10063]
- Add support to decode mysql prepare statement command. {pull}8084[8084]

*Functionbeat*

- Mark Functionbeat  as GA. {pull}10564[10564]

[[release-notes-7.0.0-alpha2]]
=== Beats version 7.0.0-alpha2
https://github.com/elastic/beats/compare/v7.0.0-alpha1\...v7.0.0-alpha2[Check the HEAD diff]

==== Breaking changes

*Affecting all Beats*

- Update add_cloud_metadata fields to adjust to ECS. {pull}9265[9265]
- Automaticall cap signed integers to 63bits. {pull}8991[8991]
- Rename beat.timezone to event.timezone. {pull}9458[9458]
- Use _doc as document type. {pull}9056[9056]
- Removed dashboards and index patterns generation for Kibana 5. {pull}8927[8927]
- On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. {pull}8942[8942].

*Auditbeat*

- Remove warning for deprecated option: "filters". {pull}9002[9002]

*Filebeat*

- Allow beats to blacklist certain part of the configuration while using Central Management. {pull}9099[9099]
- Remove warnings for deprecated options: "spool_size", "publish_async", "idle_timeout". {pull}9002[9002]
- Rename many `haproxy.*` fields to map to ECS. {pull}9117[9117]
- Rename many `iis.access.*` fields to map to ECS. {pull}9084[9084]
- IIS module's user agent string is no longer encoded (`+` replaced with spaces). {pull}9084[9084]
- Rename many `system.syslog.*` fields to map to ECS. {pull}9135[9135]
- Rename many `nginx.access.*` fields to map to ECS. {pull}9081[9081]
- Rename many `system.auth.*` fields to map to ECS. {pull}9138[9138]
- Rename many `apache2.access.*` fields to map to ECS. {pull}9245[9245]
- Rename `apache2` module to `apache`. {pull}9402[9402]

*Metricbeat*

- Allow beats to blacklist certain part of the configuration while using Central Management. {pull}9099[9099]
- Remove warning for deprecated option: "filters". {pull}9002[9002]

*Packetbeat*

- Renamed the flow event fields to follow Elastic Common Schema. {pull}9121[9121]
- Renamed several client and server fields. IP, port, and process metadata are
  now contained under the client and server namespaces. {issue}9303[9303]

*Functionbeat*

- The CLI will now log CloudFormation Stack events. {issue}8912[8912]
- Function concurrency is now set to 5 instead of unreserved. {pull}8992[8992]

==== Bugfixes

*Affecting all Beats*

- Propagate Sync error when running SafeFileRotate. {pull}9069[9069]
- Fix autodiscover configurations stopping when metadata is missing. {pull}8851[8851]
- Log events at the debug level when dropped by encoding problems. {pull}9251[9251]
- Refresh host metadata in add_host_metadata. {pull}9359[9359]
- When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. {issue}6271[6271] {pull}9383[9383]
- Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. {pull}9016[9016]
- Ignore non index fields in default_field for Elasticsearch. {pull}9549[9549]
- Update Kibana index pattern attributes for objects that are disabled. {pull}9644[9644]
- Enforce validation for the Central Management access token. {issue}9621[9621]
- Update to Golang 1.11.4. {pull}9627[9627]

*Auditbeat*

*Filebeat*

- Correctly parse `December` or `Dec` in the Syslog input. {pull}9349[9349]
- Fix installation of haproxy dashboard. {issue}9307[9307] {pull}9313[9313]
- Don't generate incomplete configurations when logs collection is disabled by hints. {pull}9305[9305]
- Stop runners disabled by hints after previously being started. {pull}9305[9305]
- Fix saved objects in filebeat haproxy dashboard. {pull}9417[9417]
- Use `log.source.address` instead of `log.source.ip` for network input sources. {pull}9487[9487]
- Rename many `redis.log.*` fields to map to ECS. {pull}9315[9315]
- Rename many `icinga.*` fields to map to ECS. {pull}9294[9294]
- Rename many `postgresql.log.*` fields to map to ECS. {pull}9308[9308]
- Rename many `kafka.log.*` fields to map to ECS. {pull}9297[9297]
- Add `convert_timezone` option to Logstash module to convert dates to UTC. {issue}9756[9756] {pull}9797[9797]

*Metricbeat*

- Fix issue preventing diskio metrics collection for idle disks. {issue}9124[9124] {pull}9125[9125]
- Fix panic on docker healthcheck collection on dockers without healthchecks. {pull}9171[9171]
- Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. {pull}9179[9179]
- The `node.name` field in the `elasticsearch/node` metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. {pull}9209[9209]

*Packetbeat*

- Fix issue with process monitor associating traffic to the wrong process. {issue}9151[9151] {pull}9443[9443]

==== Added

*Affecting all Beats*
- Unify dashboard exporter tools. {pull}9097[9097]

- Unify dashboard exporter tools. {pull}9097[9097]
- Add cache.ttl to add_host_metadata. {pull}9359[9359]
- Add support for index lifecycle management (beta). {pull}7963[7963]
- Always include Pod UID as part of Pod metadata. {pull}9517[9517]
- Autodiscovery no longer requires that the `condition` field be set. If left unset all configs will be matched. {pull}9029[9029]
- Add geo fields to `add_host_metadata` processor. {pull}9392[9392]

*Filebeat*
- Added `detect_null_bytes` selector to detect null bytes from a io.reader. {pull}9210[9210]

- Added the `redirect_stderr` option that allows panics to be logged to log files. {pull}8430[8430]
- Added `detect_null_bytes` selector to detect null bytes from a io.reader. {pull}9210[9210]
- Added `syslog_host` variable to HAProxy module to allow syslog listener to bind to configured host. {pull}9366[9366]
- Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format {issue}8015[8015] {issue}6111[6111] {pull}8768[8768].
- Add support for multi-core thread_id in postgresql module {issue}9156[9156] {pull}9482[9482]

*Heartbeat*

- Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. {pull}9022[9022]
- Add central management support. {pull}9254[9254]

*Journalbeat*

- Add cursor_seek_fallback option. {pull}9234[9234]

*Metricbeat*

- Add settings to disable docker and cgroup cpu metrics per core. {issue}9187[9187] {pull}9194[9194] {pull}9589[9589]
- The `elasticsearch/node` metricset now reports the Elasticsearch cluster UUID. {pull}8771[8771]
- Add service.type field to Metricbeat. {pull}8965[8965]
- Support GET requests in Jolokia module. {issue}8566[8566] {pull}9226[9226]
- Add freebsd support for the uptime metricset. {pull}9413[9413]
- Add `host.os.name` field to add_host_metadata processor. {issue}8948[8948] {pull}9405[9405]
- Add more TCP statuses to `socket_summary` metricset. {pull}9430[9430]
- Remove experimental tag from ceph metricsets. {pull}9708[9708]
- Add MS SQL module to X-Pack {pull}9414[9414]

==== Deprecated

*Metricbeat*

- event.duration is now in nano and not microseconds anymore. {pull}8941[8941]

[[release-notes-7.0.0-alpha1]]
=== Beats version 7.0.0-alpha1
https://github.com/elastic/beats/compare/v6.5.0\...v7.0.0-alpha1[View commits]

==== Breaking changes

*Affecting all Beats*

- Dissect syntax change, use * instead of ? when working with field reference. {issue}8054[8054]

*Auditbeat*

- Use `initial_scan` action for new paths. {pull}7954[7954]
- Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.
- Rename `source.hostname` to `source.domain` in the auditd module. {pull}9027[9027]

*Filebeat*

- Rename `fileset.name` to `event.name`. {pull}8879[8879]
- Rename `fileset.module` to `event.module`. {pull}8879[8879]
- Rename source to log.file.path and log.source.ip {pull}8902[8902]
- Remove the deprecated `prospector(s)` option in the configuration use `input(s)` instead. {pull}8909[8909]
- Rename `offset` to `log.offset`. {pull}8923[8923]
- Rename `source_ecs` to `source` in the Filebeat Suricata module. {pull}8983[8983]

==== Bugfixes

*Affecting all Beats*

- Fixed `-d` CLI flag by trimming spaces from selectors. {pull}7864[7864]
- Fixed Support `add_docker_metadata` in Windows by identifying systems' path separator. {issue}7797[7797]
- Do not panic when no tokenizer string is configured for a dissect processor. {issue}8895[8895]
- Start autodiscover consumers before producers. {pull}7926[7926]

*Filebeat*

- Fixed a memory leak when harvesters are closed. {pull}7820[7820]
- Fix improperly set config for CRI Flag in Docker Input {pull}8899[8899]
- Just enabling the `elasticsearch` fileset and starting Filebeat no longer causes an error. {pull}8891[8891]
- Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]

*Heartbeat*

- Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn't used since the connection would be closed soon after the headers were sent, but before the entire body was. {pull}8894[8894]
- `Host` header can now be overridden for HTTP requests sent by Heartbeat monitors. {pull}9148[9516]

*Metricbeat*

- Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. {pull}7789[7789]
- Add missing namespace field in http server metricset {pull}7890[7890]
- Fix race condition when enriching events with kubernetes metadata. {issue}9055[9055] {issue}9067[9067]

*Packetbeat*

- Fixed the mysql missing transactions if monitoring a connection from the start. {pull}8173[8173]


==== Added

*Affecting all Beats*

- Add field `host.os.kernel` to the add_host_metadata processor and to the
  internal monitoring data. {issue}7807[7807]
- Add debug check to logp.Logger {pull}7965[7965]
- Count HTTP 429 responses in the elasticsearch output {pull}8056[8056]
- Allow Bus to buffer events in case listeners are not configured. {pull}8527[8527]
- Dissect will now flag event on parsing error. {pull}8751[8751]
- add_cloud_metadata initialization is performed asynchronously to avoid delays on startup. {pull}8845[8845]
- Add DeDot method in add_docker_metadata processor in libbeat. {issue}9350[9350] {pull}9505[9505]

*Filebeat*

- Make inputsource generic taking bufio.SplitFunc as input {pull}7746[7746]
- Add custom unpack to log hints config to avoid env resolution {pull}7710[7710]
- Make docker input check if container strings are empty {pull}7960[7960]
- Keep unparsed user agent information in user_agent.original. {pull}8537[8537]
- Allow to force CRI format parsing for better performance {pull}8424[8424]

*Heartbeat*

- Add automatic config file reloading. {pull}8023[8023]

*Journalbeat*

- Add the ability to check against JSON HTTP bodies with conditions. {pull}8667[8667]

*Metricbeat*

- Add metrics about cache size to memcached module {pull}7740[7740]
- Add experimental socket summary metricset to system module {pull}6782[6782]
- Collect custom cluster `display_name` in `elasticsearch/cluster_stats` metricset. {pull}8445[8445]
- Test etcd module with etcd 3.3. {pull}9068[9068]
- All `elasticsearch` metricsets now have module-level `cluster.id` and `cluster.name` fields. {pull}8770[8770] {pull}8771[8771] {pull}9164[9164] {pull}9165[9165] {pull}9166[9166] {pull}9168[9168]
- All `elasticsearch` node-level metricsets now have `node.id` and `node.name` fields. {pull}9168[9168] {pull}9209[9209]

*Packetbeat*

- Add support to decode HTTP bodies compressed with `gzip` and `deflate`. {pull}7915[7915]
- Added support to calculate certificates' fingerprints (MD5, SHA-1, SHA-256). {issue}8180[8180]
- Support new TLS version negotiation introduced in TLS 1.3. {issue}8647[8647].

[[release-notes-6.8.13]]
=== Beats version 6.8.13
https://github.com/elastic/beats/compare/v6.8.12\...v6.8.13[View commits]

==== Added

*Filebeat*

- Add container image in Kubernetes metadata. {pull}13356[13356] {issue}12688[12688]

[[release-notes-6.8.12]]
=== Beats version 6.8.12
https://github.com/elastic/beats/compare/v6.8.11\...v6.8.12[View commits]

==== Bugfixes

*Filebeat*

- Fix Filebeat OOMs on very long lines {issue}19500[19500], {pull}19552[19552]

[[release-notes-6.8.11]]
=== Beats version 6.8.11
https://github.com/elastic/beats/compare/v6.8.10\...v6.8.11[View commits]

==== Bugfixes

*Metricbeat*

- Fix bug incorrect parsing of float numbers as integers in Couchbase module {issue}18949[18949] {pull}19055[19055]

[[release-notes-6.8.10]]
=== Beats version 6.8.10
https://github.com/elastic/beats/compare/v6.8.9\...v6.8.10[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix `add_cloud_metadata` to better support modifying sub-fields with other processors. {pull}13808[13808]

[[release-notes-6.8.9]]
=== Beats version 6.8.9
https://github.com/elastic/beats/compare/v6.8.8\...v6.8.9[View commits]

==== Bugfixes

*Heartbeat*

- Fix crashes when multiple TCP ports are specified. {pull}17262[17262]

[[release-notes-6.8.8]]
=== Beats version 6.8.8
https://github.com/elastic/beats/compare/v6.8.7\...v6.8.8[View commits]

==== Bugfixes

*Filebeat*

- Add support for Cisco syslog format used by their switch. {pull}10760[10760]

[[release-notes-6.8.7]]
=== Beats version 6.8.7
https://github.com/elastic/beats/compare/v6.8.6\...v6.8.7[View commits]

==== Bugfixes

*Metricbeat*

- Fix bug with `elasticsearch/cluster_stats` metricset not recording license expiration date correctly. {issue}14541[14541] {pull}14591[14591]
- Make `kibana` module more resilient to Kibana unavailability. {issue}15258[15258] {pull}15270[15270]

[[release-notes-6.8.6]]
=== Beats version 6.8.6
https://github.com/elastic/beats/compare/v6.8.5\...v6.8.6[View commits]

==== Bugfixes

*Heartbeat*

- Fix recording of SSL cert metadata for Expired/Unvalidated x509 certs. {pull}13687[13687]

*Metricbeat*

- Fix marshaling of ms-since-epoch values in `elasticsearch/cluster_stats` metricset. {pull}14378[14378]
- Fix bug with `elasticsearch/cluster_stats` metricset not recording license ID in the correct field. {pull}14592[14592]

[[release-notes-6.8.5]]
=== Beats version 6.8.5
https://github.com/elastic/beats/compare/v6.8.4\...v6.8.5[View commits]

==== Bugfixes

*Metricbeat*

- Convert indexed ms-since-epoch timestamp fields in `elasticsearch/ml_job` metricset to ints from float64s. {issue}14220[14220] {pull}14222[14222]

[[release-notes-6.8.4]]
=== Beats version 6.8.4
https://github.com/elastic/beats/compare/v6.8.3\...v6.8.4[View commits]

==== Breaking changes

*Filebeat*

- Fix delay in enforcing close_renamed and close_removed options. {issue}13488[13488] {pull}13907[13907]

==== Bugfixes

*Filebeat*

- Fix merging of fields specified in global scope with fields specified under an input's scope. {issue}3628[3628] {pull}13909[13909]
- Fix early expiration of templates (Netflow v9 and IPFIX). {pull}13821[13821]
- Fix bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). {pull}13821[13821]
- Fix increased memory usage with large files when multiline pattern does not match. {issue}14068[14068]

*Metricbeat*

- Mark Kibana usage stats as collected only if API call succeeds. {pull}13881[13881]

[[release-notes-6.8.3]]
=== Beats version 6.8.3
https://github.com/elastic/beats/compare/v6.8.2\...v6.8.3[View commits

==== Bugfixes

*Journalbeat*

- Iterate over journal correctly, so no duplicate entries are sent. {pull}12716[12716]

*Metricbeat*

- Fix panic in Redis Key metricset when collecting information from a removed key. {pull}13426[13426]

==== Added

*Metricbeat*

- Remove _nodes field from under cluster_stats as it's not being used. {pull}13010[13010]
- Collect license expiry date fields as well. {pull}11652[11652]

[[release-notes-6.8.2]]
=== Beats version 6.8.2
https://github.com/elastic/beats/compare/v6.8.1\...v6.8.2[View commits]

==== Bugfixes

*Auditbeat*

- Process dataset: Do not show non-root warning on Windows. {pull}12740[12740]

*Filebeat*

- Skipping unparsable log entries from docker json reader {pull}12268[12268]

*Packetbeat*

- Limit memory usage of Redis replication sessions. {issue}12657[12657]

[[release-notes-6.8.1]]
=== Beats version 6.8.1
https://github.com/elastic/beats/compare/v6.8.0\...v6.8.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fixed a memory leak when using the add_process_metadata processor under Windows. {pull}12100[12100]

*Auditbeat*

- Package dataset: Log error when Homebrew is not installed. {pull}11667[11667]
- Process dataset: Fixed a memory leak under Windows. {pull}12100[12100]
- Login dataset: Fix re-read of utmp files. {pull}12028[12028]
- Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. {issue}12147[12147] {pull}12168[12168]
- Fix direction of incoming IPv6 sockets. {pull}12248[12248]
- Package dataset: Auto-detect package directories. {pull}12289[12289]
- System module: Start system module without host ID. {pull}12373[12373]
- Host dataset: Fix reboot detection logic. {pull}12591[12591]

*Filebeat*

- Fix goroutine leak happening when harvesters are dynamically stopped. {pull}11263[11263]
- Fix initialization of the TCP input logger. {pull}11605[11605]
- Fix goroutine leak caused on initialization failures of log input. {pull}12125[12125]
- Fix memory leak in Filebeat pipeline acker. {pull}12063[12063]
- Fix goroutine leak on non-explicit finalization of log input. {pull}12164[12164]
- When TLS is configured for the TCP input and a `certificate_authorities` is configured we now default to `required` for the `client_authentication`. {pull}12584[12584]

*Metricbeat*

- Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. {issue}8264[8264] {pull}12086[12086]
- Fix direction of incoming IPv6 sockets. {pull}12248[12248]
- Validate that kibana/status metricset cannot be used when xpack is enabled. {pull}12264[12264]
- In the kibana/stats metricset, only log error (don't also index it) if xpack is enabled. {pull}12353[12353]
- The `elasticsearch/index_summary` metricset gracefully handles an empty Elasticsearch cluster when `xpack.enabled: true` is set. {pull}12489[12489] {issue}12487[12487]
- When TLS is configured for the http metricset and a `certificate_authorities` is configured we now default to `required` for the `client_authentication`. {pull}12584[12584]

*Packetbeat*

- Fixed a memory leak when using process monitoring under Windows. {pull}12100[12100]
- Improved debug logging efficiency in PGQSL module. {issue}12150[12150]

==== Added

*Auditbeat*

- Add support to the system package dataset for the SUSE OS family. {pull}11634[11634]

*Metricbeat*

- Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. {pull}12386[12386]

[[release-notes-6.8.0]]
=== Beats version 6.8.0

* Updates to support changes to licensing of security features.
+
Some Elastic Stack security features, such as encrypted communications, file and native authentication, and
role-based access control, are now available in more subscription levels. For details, see https://www.elastic.co/subscriptions.

[[release-notes-6.7.2]]
=== Beats version 6.7.2
https://github.com/elastic/beats/compare/v6.7.1\...v6.7.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Relax validation of the X-Pack license UID value. {issue}11640[11640]
- Fix a parsing error with the X-Pack license check on 32-bit system. {issue}11650[11650]
- Fix OS family classification in `add_host_metadata` for Amazon Linux, Raspbian, and RedHat Linux. {issue}9134[9134] {pull}11494[11494]
- Fix false positives reported in the `host.containerized` field added by `add_host_metadata`. {pull}11494[11494]
- Fix the add_host_metadata's `host.id` field on older Linux versions. {pull}11494[11494]

*Auditbeat*

- Package dataset: dlopen versioned librpm shared objects. {pull}11565[11565]
- Package dataset: Nullify Librpm's rpmsqEnable. {pull}11628[11628]

*Filebeat*

- Don't apply multiline rules in Logstash json logs. {pull}11346[11346]
- Fix goroutine leak happening when harvesters are dynamically stopped. {pull}11263[11263]
- Fix initialization of the TCP input logger. {pull}11605[11605]

*Metricbeat*

- Prevent the docker/memory metricset from processing invalid events before container start {pull}11676[11676]

==== Added

*Auditbeat*

- Add support to the system package dataset for the SUSE OS family. {pull}11634[11634]

[[release-notes-6.7.1]]
=== Beats version 6.7.1
https://github.com/elastic/beats/compare/v6.7.0\...v6.7.1[View commits]

==== Breaking changes

*Affecting all Beats*

- Initialize the Paths before the keystore and save the keystore into `data/{beatname}.keystore`. {pull}10706[10706]

==== Bugfixes

*Affecting all Beats*

- Remove IP fields from default_field in Elasticsearch template. {pull}11399[11399]

[[release-notes-6.7.0]]
=== Beats version 6.7.0
https://github.com/elastic/beats/compare/v6.6.2\...v6.7.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Port settings have been deprecated in redis/logstash output and will be removed in 7.0. {pull}9915[9915]
- Update the code of Central Management to align with the new returned format. {pull}10019[10019]
- Allow Central Management to send events back to kibana. {issue}9382[9382]
- Fix panic if fields settting is used to configure `hosts.x` fields. {issue}10824[10824] {pull}10935[10935]
- Introduce query.default_field as part of the template. {pull}11205[11205]
- Beats Xpack now checks for Basic license on connect. {pull}11296[11296]

*Filebeat*

- Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001]
- Add grok pattern to support redis 5.0.3 log timestamp. {issue}9819[9819] {pull}10033[10033]
- Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above {pull}8852[8852]
- Remove `ecs` option from user_agent processors when loading pipelines with Filebeat 6.7.x into Elasticsearch < 6.7.0. {issue}10655[10655] {pull}11362[11362]

*Heartbeat*

- Remove monitor generator script that was rarely used. {pull}9648[9648]

==== Bugfixes

*Affecting all Beats*

- Fix TLS certificate DoS vulnerability. {pull}10303[10303]
- Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. {pull}10289[10289]
- Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. {pull}9016[9016]
- Do not panic when no tokenizer string is configured for a dissect processor. {issue}8895[8895]
- Fix a issue when remote and local configuration didn't match when fetching configuration from Central Management. {issue}10587[10587]
- Add ECS-like selectors and dedotting to docker autodiscover. {issue}10757[10757] {pull}10862[10862]
- Fix encoding of timestamps when using disk spool. {issue}10099[10099]
- Include ip and boolean type when generating index pattern. {pull}10995[10995]
- Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn't exist. {pull}10936[10936]
- Cancelling enrollment of a beat will not enroll the beat. {issue}10150[10150]
- Remove IP fields from default_field in Elasticsearch template. {pull}11399[11399]

*Auditbeat*

- Package: Disable librpm signal handlers. {pull}10694[10694]
- Login: Handle different bad login UTMP types. {pull}10865[10865]
- Fix hostname references in System module dashbords. {pull}11064[11064]
- User dataset: Numerous fixes to error handling. {pull}10942[10942]

*Filebeat*

- Support IPv6 addresses with zone id in IIS ingest pipeline. {issue}9836[9836] error log: {pull}9869[9869] access log: {pull}10029[10029]
- Fix bad bytes count in `docker` input when filtering by stream. {pull}10211[10211]
- Fixed data types for roles and indices fields in `elasticsearch/audit` fileset {pull}10307[10307]
- Cover empty request data, url and version in Apache2 module{pull}10846[10846]
- Fix a bug with the convert_timezone option using the incorrect timezone field. {issue}11055[11055] {pull}11164[11164]
- Change URLPATH grok pattern to support brackets. {issue}11135[11135] {pull}11252[11252]
- Add support for iis log with different address format. {issue}11255[11255] {pull}11256[11256]
- Add fix to parse syslog message with priority value 0. {issue}11010[11010]

*Heartbeat*

- `Host` header can now be overridden for HTTP requests sent by Heartbeat monitors. {pull}9148[9516]
- Fix checks for TCP send/receive data {pull}10777[10777]

*Journalbeat*

- Do not stop collecting events when journal entries change. {pull}9994[9994]

*Metricbeat*

- Fix MongoDB dashboard that had some incorrect field names from `status` Metricset {pull}9795[9795] {issue}9715[9715]
- Fix issue that would prevent collection of processes without command line on Windows. {pull}10196[10196]
- Fixed data type for tags field in `docker/container` metricset {pull}10307[10307]
- Fixed data type for tags field in `docker/image` metricset {pull}10307[10307]
- Fixed data type for isr field in `kafka/partition` metricset {pull}10307[10307]
- Fixed data types for various hosts fields in `mongodb/replstatus` metricset {pull}10307[10307]
- Added function to close sql database connection. {pull}10355[10355]
- Fix parsing error using GET in Jolokia module. {pull}11075[11075] {issue}11071[11071]

*Winlogbeat*

- Fix Winlogbeat escaping CR, LF and TAB characters. {issue}11328[11328] {pull}11357[11357]

*Functionbeat*

- Correctly extract Kinesis Data field from the Kinesis Record. {pull}11141[11141]
- Add the required permissions to the role when deployment SQS functions. {issue}9152[9152]

==== Added

*Affecting all Beats*

- Add ip fields to default_field in Elasticsearch template. {pull}11035[11035]
- Add `cleanup_timeout` option to docker autodiscover, to wait some time before removing configurations after a container is stopped. {issue}10374[10374] {pull}10905[10905]

*Auditbeat*

- System module `process` dataset: Add user information to processes. {pull}9963[9963]
- Add system `package` dataset. {pull}10225[10225]
- Add system module `login` dataset. {pull}9327[9327]
- Add `entity_id` fields. {pull}10500[10500]
- Add seven dashboards for the system module. {pull}10511[10511]

*Filebeat*

- Add field log.source.address and log.file.path to replace source. {pull}9435[9435]
- Support mysql 5.7.22 slowlog starting with time information. {issue}7892[7892] {pull}9647[9647]
- Add support for ssl_request_log in apache2 module. {issue}8088[8088] {pull}9833[9833]
- Add support for iis 7.5 log format. {issue}9753[9753] {pull}9967[9967]
- Add support for MariaDB in the `slowlog` fileset of `mysql` module. {pull}9731[9731]
- Add convert_timezone to nginx module. {issue}9839[9839] {pull}10148[10148]
- Add support for Percona in the `slowlog` fileset of `mysql` module. {issue}6665[6665] {pull}10227[10227]
- Added support for ingesting structured Elasticsearch audit logs {pull}8852[8852]
- New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. {issue}8781[8781] {pull}10176[10176]
- Populate more ECS fields in the Suricata module. {pull}10006[10006]

*Heartbeat*

- Made monitors.d configuration part of the default config. {pull}9004[9004]
- Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you'll see the correct fields under the `docker` key. {pull}10258[10258]

*Metricbeat*

- Add field `event.dataset` which is `{module}.{metricset}`.
- Add more TCP statuses to `socket_summary` metricset. {pull}9430[9430]
- Remove experimental tag from ceph metricsets. {pull}9708[9708]
- Add `key` metricset to the Redis module. {issue}9582[9582] {pull}9657[9657]
- Add DeDot for kubernetes labels and annotations. {issue}9860[9860] {pull}9939[9939]
- Add docker `event` metricset. {pull}9856[9856]
- Release Ceph module as GA. {pull}10202[10202]
- Release windows Metricbeat module as GA. {pull}10163[10163]
- Release traefik Metricbeat module as GA. {pull}10166[10166]
- List filesystems on Windows that have an access path but not an assigned letter {issue}8916[8916] {pull}10196[10196]
- Release uswgi Metricbeat module GA. {pull}10164[10164]
- Release php_fpm module as GA. {pull}10198[10198]
- Release Memcached module as GA. {pull}10199[10199]
- Release etcd module as GA. {pull}10200[10200]
- Release kubernetes apiserver and event metricsets as GA {pull}10212[10212]
- Release Couchbase module as GA. {pull}10201[10201]
- Release aerospike module as GA. {pull}10203[10203]
- Release envoyproxy module GA. {pull}10223[10223]
- Release mongodb.metrics and mongodb.replstatus as GA. {pull}10242[10242]
- Release mysql.galera_status as Beta. {pull}10242[10242]
- Release postgresql.statement as GA. {pull}10242[10242]
- Release RabbitMQ Metricbeat module GA. {pull}10165[10165]
- Release Dropwizard module as GA. {pull}10240[10240]
- Release Graphite module as GA. {pull}10240[10240]
- Release http.server metricset as GA. {pull}10240[10240]
- Add support for MySQL 8.0 and tests also for Percona and MariaDB. {pull}10261[10261]
- Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. {pull}10222[10222]
- Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. {pull}10094[10094]
- Add remaining memory metrics of pods in Kubernetes metricbeat module {pull}10157[10157]
- Added 'server' Metricset to Zookeeper Metricbeat module {issue}8938[8938] {pull}10341[10341]
- Add overview dashboard to Zookeeper Metricbeat module {pull}10379[10379]

*Functionbeat*

- Mark Functionbeat  as GA. {pull}10564[10564]
- Functionbeat can now deploy a function for Kinesis. {pull}10116[10116]
- Allow functionbeat to use the keystore. {issue}9009[9009]

==== Deprecated

*Filebeat*

- Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. {pull}9435[9435]

*Metricbeat*

- Deprecate field `metricset.rtt`. Replaced by `event.duration` which is in nano instead of micro seconds.

*Packetbeat*

- Support new TLS version negotiation introduced in TLS 1.3. {issue}8647[8647].

==== Known Issue

*Journalbeat*

- Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

[[release-notes-6.6.2]]
=== Beats version 6.6.2
https://github.com/elastic/beats/compare/v6.6.1\...6.6.2[View commits]

==== Bugfixes

*Auditbeat*

- System module: Fix and unify bucket closing logic. {pull}10897[10897]

*Filebeat*

- Fix a bug when converting NetFlow fields to snake_case. {pull}10950[10950]

*Metricbeat*

- Fix issue in kubernetes module preventing usage percentages to be properly calculated. {pull}10946[10946]

*Packetbeat*

- Avoid reporting unknown MongoDB opcodes more than once. {pull}10878[10878]

*Winlogbeat*

- Prevent Winlogbeat from dropping events with invalid XML. {pull}11006[11006]

[[release-notes-6.6.1]]
=== Beats version 6.6.1
https://github.com/elastic/beats/compare/v6.6.0\...6.6.1[View commits]

==== Breaking changes

*Affecting all Beats*

- Fix stopping of modules started by kubernetes autodiscover. {pull}10476[10476]

*Auditbeat*

- Enable System module config on Windows. {pull}10237[10237]

*Filebeat*

- Fix bad bytes count in `docker` input when filtering by stream. {pull}10211[10211]
- Add `convert_timezone` option to Logstash module to convert dates to UTC. {issue}9756[9756] {pull}9797[9797]
- Add `convert_timezone` option to Elasticsearch module to convert dates to UTC. {issue}9756[9756] {pull}9761[9761]
- Make elasticsearch/audit fileset be more lenient in parsing node name. {issue}10035[10035] {pull}10135[10135]

*Journalbeat*

- Fix fields.yml indentation of audit group which had the effect of creating an incomplete Elasticsearch index template. {pull}10556[10556]

*Metricbeat*

- Fix issue with `elasticsearch/node_stats` metricset (x-pack) not indexing `source_node` field. {pull}10639[10639]

*Packetbeat*

- Fixed a crash when using af_packet capture {pull}10477[10477]

*Functionbeat*

- Ensure that functionbeat is logging at info level not debug. {issue}10262[10262]

==== Added

*Filebeat*

- Teach elasticsearch/audit fileset to parse out some more fields. {issue}10134[10134] {pull}10137[10137]

*Journalbeat*

- Migrate registry from previously incorrect path. {pull}10486[10486]

[[release-notes-6.6.0]]
=== Beats version 6.6.0
https://github.com/elastic/beats/compare/v6.5.4\...6.6[View commits]

==== Breaking changes

*Affecting all Beats*

- Dissect syntax change, use * instead of ? when working with field reference. {issue}8054[8054]

*Filebeat*

- Allow beats to blacklist certain part of the configuration while using Central Management. {pull}9099[9099]

*Metricbeat*

- Allow beats to blacklist certain part of the configuration while using Central Management. {pull}9099[9099]

*Functionbeat*

- The CLI will now log CloudFormation Stack events. {issue}8912[8912]
- Correctly normalize Cloudformation resource name. {issue}10087[10087]

==== Bugfixes

*Affecting all Beats*

- Fix autodiscover configurations stopping when metadata is missing. {pull}8851[8851]
- Refresh host metadata in add_host_metadata. {pull}9359[9359]
- When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. {issue}6271[6271] {pull}9383[9383]
- Ignore non index fields in default_field for Elasticsearch. {pull}9549[9549]
- Update Golang to 1.10.6. {pull}9563[9563]
- Update Kibana index pattern attributes for objects that are disabled. {pull}9644[9644]
- Enforce validation for the Central Management access token. {issue}9621[9621]
- Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). {pull}9920[9920]
- Gracefully handle TLS options when enrolling a Beat. {issue}9129[9129]
- Allow to unenroll a Beat from the UI. {issue}9452[9452]
- The backing off now implements jitter to better distribute the load. {issue}10172[10172]
- Fix config appender registration. {pull}9873[9873]
- Fix TLS certificate DoS vulnerability. {pull}10304[10304]

*Filebeat*

- Fix improperly set config for CRI Flag in Docker Input {pull}8899[8899]
- Just enabling the `elasticsearch` fileset and starting Filebeat no longer causes an error. {pull}8891[8891]
- Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]
- Support IPv6 addresses with zone id in IIS ingest pipeline. {issue}9836[9836] error log: {pull}9869[9869] access log: {pull}10030[10030]
- Support haproxy log lines without captured headers. {issue}9463[9463] {pull}9958[9958]

*Heartbeat*

- Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn't used since the connection would be closed soon after the headers were sent, but before the entire body was. {pull}8894[8894]

*Metricbeat*

- Add missing namespace field in http server metricset {pull}7890[7890]
- Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. {pull}9179[9179]
- The `node.name` field in the `elasticsearch/node` metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. {pull}9209[9209]
- Fix panics in vsphere module when certain values where not returned by the API. {pull}9784[9784]
- Fix pod UID metadata enrichment in Kubernetes module. {pull}10081[10081]


*Packetbeat*

- Fix issue with process monitor associating traffic to the wrong process. {issue}9151[9151] {pull}9443[9443]
- Fix DHCPv4 dashboard that wouldn't load in Kibana. {issue}9850[9850]


==== Added

*Affecting all Beats*

- Unify dashboard exporter tools. {pull}9097[9097]
- Dissect will now flag event on parsing error. {pull}8751[8751]
- Added the `redirect_stderr` option that allows panics to be logged to log files. {pull}8430[8430]
- Add cache.ttl to add_host_metadata. {pull}9359[9359]
- Add support for index lifecycle management (beta). {pull}7963[7963]
- Always include Pod UID as part of Pod metadata. {pull}9517[9517]
- Release Jolokia autodiscover as GA. {pull}9706[9706]

*Auditbeat*

- Add system module. {pull}9546[9546]

*Filebeat*
- Added `detect_null_bytes` selector to detect null bytes from a io.reader. {pull}9210[9210]
- Added `syslog_host` variable to HAProxy module to allow syslog listener to bind to configured host. {pull}9366[9366]
- Allow to force CRI format parsing for better performance {pull}8424[8424]
- Add event.dataset to module events. {pull}9457[9457]
- Add field log.source.address and log.file.path to replace source. {pull}9435[9435]
- Add support for multi-core thread_id in postgresql module {issue}9156[9156] {pull}9482[9482]
- Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. {issue}9399[9399]

*Journalbeat*

- Add the ability to check against JSON HTTP bodies with conditions. {pull}8667[8667]
- Add cursor_seek_fallback option. {pull}9234[9234]

*Metricbeat*

- Collect custom cluster `display_name` in `elasticsearch/cluster_stats` metricset. {pull}8445[8445]
- Test etcd module with etcd 3.3. {pull}9068[9068]
- All `elasticsearch` metricsets now have module-level `cluster.id` and `cluster.name` fields. {pull}8770[8770] {pull}8771[8771] {pull}9164[9164] {pull}9165[9165] {pull}9166[9166] {pull}9168[9168]
- All `elasticsearch` node-level metricsets now have `node.id` and `node.name` fields. {pull}9168[9168] {pull}9209[9209]
- Add settings to disable docker and cgroup cpu metrics per core. {issue}9187[9187] {pull}9194[9194] {pull}9589[9589]
- The `elasticsearch/node` metricset now reports the Elasticsearch cluster UUID. {pull}8771[8771]
- Support GET requests in Jolokia module. {issue}8566[8566] {pull}9226[9226]
- Add freebsd support for the uptime metricset. {pull}9413[9413]
- Add `host.os.name` field to add_host_metadata processor. {issue}8948[8948] {pull}9405[9405]
- Add field `event.dataset` which is `{module}.{metricset)`. {pull}9393[9393]

==== Deprecated

*Filebeat*
- Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. {pull}9435[9435]

*Metricbeat*

- Deprecate field `metricset.rtt`. Replaced by `event.duration` which is in nano instead of micro seconds. {pull}9393[9393]

*Packetbeat*

- Support new TLS version negotiation introduced in TLS 1.3. {issue}8647[8647].



[[release-notes-6.5.4]]
=== Beats version 6.5.4
https://github.com/elastic/beats/compare/v6.5.3\...v6.5.4[View commits]

==== Bugfixes

*Affecting all Beats*

- Update Golang to 1.10.6. This fixes an issue in remote certificate validation CVE-2018-16875. {pull}9563[9563]

*Filebeat*

- Fix saved objects in filebeat haproxy dashboard. {pull}9417[9417]
- Fixed a memory leak when harvesters are closed. {pull}7820[7820]

==== Added

*Filebeat*

- Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format {issue}8015[8015] {issue}6111[6111] {pull}8768[8768].


[[release-notes-6.5.3]]
=== Beats version 6.5.3
https://github.com/elastic/beats/compare/v6.5.2\...v6.5.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Log events at the debug level when dropped by encoding problems. {pull}9251[9251]

*Filebeat*

- Correctly parse `December` or `Dec` in the Syslog input. {pull}9349[9349]
- Don't generate incomplete configurations when logs collection is disabled by hints. {pull}9305[9305]
- Stop runners disabled by hints after previously being started. {pull}9305[9305]
- Fix installation of haproxy dashboard. {issue}9307[9307] {pull}9313[9313]

[[release-notes-6.5.2]]
=== Beats version 6.5.2
https://github.com/elastic/beats/compare/v6.5.1\...v6.5.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Propagate Sync error when running SafeFileRotate. {pull}9069[9069]

*Metricbeat*

- Fix panic on docker healthcheck collection on dockers without healthchecks. {pull}9171[9171]
- Fix issue preventing diskio metrics collection for idle disks. {issue}9124[9124] {pull}9125[9125]

[[release-notes-6.5.1]]
=== Beats version 6.5.1
https://github.com/elastic/beats/compare/v6.5.0\...v6.5.1[View commits]

==== Bugfixes

*Affecting all Beats*
- Fix windows binaries not having an enroll command. {issue}9096[9096] {pull}8836[8836]

*Journalbeat*
- Fix journalbeat sometimes hanging if output is unavailable. {pull}9106[9106]

*Metricbeat*
- Fix race condition when enriching events with kubernetes metadata. {issue}9055[9055] {issue}9067[9067]

==== Added

*Journalbeat*
- Add minimal kibana dashboard. {pull}9106[9106]


[[release-notes-6.5.0]]
=== Beats version 6.5.0
https://github.com/elastic/beats/compare/v6.4.0\...v6.5.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Fixed `add_host_metadata` not initializing correctly on Windows. {issue}7715[7715]
- Fixed missing file unlock in spool file on Windows, so file can be reopened and locked. {pull}7859[7859]
- Fix spool file opening/creation failing due to file locking on Windows. {pull}7859[7859]
- Fix size of maximum mmaped read area in spool file on Windows. {pull}7859[7859]
- Fix potential data loss on OS X in spool file by using fcntl with F_FULLFSYNC. {pull}7859[7859]
- Improve fsync on linux, by assuming the kernel resets error flags of failed writes. {pull}7859[7859]
- Remove unix-like permission checks on Windows, so files can be opened. {issue}7849[7849]
- Replace index patterns in TSVB visualizations. {pull}7929[7929]
- Deregister pipeline loader callback when inputsRunner is stopped. {pull}[7893][7893]
- Add backoff support to x-pack monitoring outputs. {issue}7966[7966]
- Removed execute permissions systemd unit file. {pull}7873[7873]
- Fix a race condition with the `add_host_metadata` and the event serialization. {pull}8223[8223] {pull}8653[8653]
- Enforce that data used by k8s or docker doesn't use any reference. {pull}8240[8240]
- Switch to different UUID lib due to to non-random generated UUIDs. {pull}8485[8485]
- Fix race condition when publishing monitoring data. {pull}8646[8646]
- Fix bug in loading dashboards from zip file. {issue}8051[8051]
- Fix in-cluster kubernetes configuration on IPv6. {pull}8754[8754]
- The export config subcommand should not display real value for field reference. {pull}8769[8769]
- The setup command will not fail if no dashboard is available to import. {pull}8977[8977]
- Fix central management configurations reload when a configuration is removed in Kibana. {issue}9010[9010]

*Auditbeat*

- Fixed a crash in the file_integrity module under Linux. {issue}7753[7753]
- Fixed the RPM by designating the config file as configuration data in the RPM spec. {issue}8075[8075]
- Fixed a concurrent map write panic in the auditd module. {pull}8158[8158]
- Fixed a data race in the file_integrity module. {issue}8009[8009]
- Fixed a deadlock in the file_integrity module. {pull}8027[8027]

*Filebeat*

- Fix date format in Mongodb Ingest pipeline. {pull}7974[7974]
- Fixed a docker input error due to the offset update bug in partial log join.{pull}8177[8177]
- Update CRI format to support partial/full tags. {pull}8265[8265]
- Fix some errors happening when stopping syslog input. {pull}8347[8347]
- Fix RFC3339 timezone and nanoseconds parsing with the syslog input. {pull}8346[8346]
- Mark the TCP and UDP input as GA. {pull}8125[8125]
- Support multiline logs in logstash/log fileset of Filebeat. {pull}8562[8562]
- Support different timestamp format in postgresql module. {issue}9494[9494] {pull}9650[9650]

*Heartbeat*

- Fixed bug where HTTP responses with larger bodies would incorrectly report connection errors. {pull}8660[8660]

*Metricbeat*

- Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. {pull}7789[7789]
- Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. {issue}8075[8075]
- Fixed the location of the modules.d dir in Deb and RPM packages. {issue}8104[8104]
- Add docker diskio stats on Windows. {issue}6815[6815] {pull}8126[8126]
- Fix incorrect type conversion of average response time in Haproxy dashboards {pull}8404[8404]
- Added io disk read and write times to system module {issue}8473[8473] {pull}8508[8508]
- Avoid mapping issues in kubernetes module. {pull}8487[8487]
- Recover metrics for old apache versions removed by mistake on #6450. {pull}7871[7871]
- Fix dropwizard module parsing of metric names. {issue}8365[8365] {pull}6385[8385]
- Fix issue that would prevent kafka module to find a proper broker when port is not set {pull}8613[8613]
- Fix range colors in multiple visualizations. {issue}8633[8633] {pull}8634[8634]
- Fix incorrect header parsing on http metricbeat module {issue}8564[8564] {pull}8585[8585]
- Fixed a panic when the kvm module cannot establish a connection to libvirtd. {issue}7792[7792].

*Packetbeat*

- Fixed a seccomp related error where the `fcntl64` syscall was not permitted
  on 32-bit Linux and the sniffer failed to start. {issue}7839[7839]
- Added missing `cmdline` and `client_cmdline` fields to index template. {pull}8258[8258]

==== Added

*Affecting all Beats*

- Added time-based log rotation. {pull}8349[8349]
- Add backoff on error support to redis output. {pull}7781[7781]
- Allow for cloud-id to specify a custom port. This makes cloud-id work in ECE contexts. {pull}7887[7887]
- Add support to grow or shrink an existing spool file between restarts. {pull}7859[7859]
- Make kubernetes autodiscover ignore events with empty container IDs {pull}7971[7971]
- Implement CheckConfig in RunnerFactory to make autodiscover check configs {pull}7961[7961]
- Add DNS processor with support for performing reverse lookups on IP addresses. {issue}7770[7770]
- Support for Kafka 2.0.0 in kafka output {pull}8399[8399]
- Add setting `setup.kibana.space.id` to support Kibana Spaces {pull}7942[7942]
- Better tracking of number of open file descriptors. {pull}7986[7986]
- Report number of open file handles on Windows. {pull}8329[8329]
- Added the `add_process_metadata` processor to enrich events with process information. {pull}6789[6789]
- Add Beats Central Management {pull}8559[8559]
- Report configured queue type. {pull}8091[8091]
- Enable `host` and `cloud` metadata processors by default. {pull}8596[8596]

*Filebeat*

- Add tag "truncated" to "log.flags" if incoming line is longer than configured limit. {pull}7991[7991]
- Add haproxy module. {pull}8014[8014]
- Add tag "multiline" to "log.flags" if event consists of multiple lines. {pull}7997[7997]
- Release `docker` input as GA. {pull}8328[8328]
- Keep unparsed user agent information in user_agent.original. {pull}7823[7832]
- Added default and TCP parsing formats to HAproxy module {issue}8311[8311] {pull}8637[8637]
- Add Suricata IDS/IDP/NSM module. {issue}8153[8153] {pull}8693[8693]
- Support for Kafka 2.0.0 {pull}8853[8853]

*Heartbeat*

- Heartbeat is marked as GA.
- Add automatic config file reloading. {pull}8023[8023]
- Added autodiscovery support {pull}8415[8415]
- Added support for extra TLS/x509 metadata. {pull}7944[7944]
- Added stats and state metrics for number of monitors and endpoints started. {pull}8621[8621]
- Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. {pull}9022[9022]

*Journalbeat*

- Add journalbeat. {pull}8703[8703]

*Metricbeat*

- Add `replstatus` metricset to MongoDB module {pull}7604[7604]
- Add experimental socket summary metricset to system module {pull}6782[6782]
- Move common kafka fields (broker, topic and partition.id) to the module level to facilitate events correlation {pull}7767[7767]
- Add fields for memory fragmentation, memory allocator stats, copy on write, master-slave status, and active defragmentation to `info` metricset of Redis module. {pull}7695[7695]
- Increase ignore_above for system.process.cmdline to 2048. {pull}8101[8100]
- Add support to renamed fields planned for redis 5.0. {pull}8167[8167]
- Allow TCP helper to support delimiters and graphite module to accept multiple metrics in a single payload. {pull}8278[8278]
- Added 'died' PID state to process_system metricset on system module {pull}8275[8275]
- Add `metrics` metricset to MongoDB module. {pull}7611[7611]
- Added `ccr` metricset to Elasticsearch module. {pull}8335[8335]
- Support for Kafka 2.0.0 {pull}8399[8399]
- Added support for query params in configuration {issue}8286[8286] {pull}8292[8292]
- Add container image for docker metricsets. {issue}8214[8214] {pull}8438[8438]
- Precalculate composed id fields for kafka dashboards. {pull}8504[8504]
- Add support for `full` status page output for php-fpm module as a separate metricset called `process`. {pull}8394[8394]
- Add Kafka dashboard. {pull}8457[8457]
- Release Kafka module as GA. {pull}8854[8854]

*Packetbeat*

- Added DHCP protocol support. {pull}7647[7647]

*Functionbeat*

- Initial version of Functionbeat. {pull}8678[8678]

==== Deprecated

*Heartbeat*

- watch.poll_file is now deprecated and superceded by automatic config file reloading.

*Metricbeat*

- Redis `info` `replication.master_offset` has been deprecated in favor of `replication.master.offset`.{pull}7695[7695]
- Redis `info` clients fields `longest_output_list` and `biggest_input_buf` have been renamed to `max_output_buffer` and `max_input_buffer` based on the names they will have in Redis 5.0, both fields will coexist during a time with the same value {pull}8167[8167].
- Move common kafka fields (broker, topic and partition.id) to the module level {pull}7767[7767].



[[release-notes-6.4.3]]
=== Beats version 6.4.3
https://github.com/elastic/beats/compare/v6.4.2\...v6.4.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix a race condition with the `add_host_metadata` and the event serialization. {pull}8223[8223] {pull}8653[8653]
- Fix race condition when publishing monitoring data. {pull}8646[8646]
- Fix bug in loading dashboards from zip file. {issue}8051[8051]
- The export config subcommand should not display real value for field reference. {pull}8769[8769]

*Filebeat*

- Fix typo in Filebeat IIS Kibana visualization. {pull}8604[8604]

*Metricbeat*

- Recover metrics for old Apache versions removed by mistake on #6450. {pull}7871[7871]
- Avoid mapping issues in Kubernetes module. {pull}8487[8487]
- Fixed a panic when the KVM module cannot establish a connection to libvirtd. {issue}7792[7792]


[[release-notes-6.4.2]]
=== Beats version 6.4.2
https://github.com/elastic/beats/compare/v6.4.1\...v6.4.2[View commits]

==== Bugfixes

*Filebeat*

- Fix some errors happening when stopping syslog input. {pull}8347[8347]
- Fix RFC3339 timezone and nanoseconds parsing with the syslog input. {pull}8346[8346]

*Metricbeat*

- Fix incorrect type conversion of average response time in Haproxy dashboards {pull}8404[8404]
- Fix dropwizard module parsing of metric names. {issue}8365[8365] {pull}6385[8385]

[[release-notes-6.4.1]]
=== Beats version 6.4.1
https://github.com/elastic/beats/compare/v6.4.0\...v6.4.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Add backoff support to x-pack monitoring outputs. {issue}7966[7966]
- Removed execute permissions systemd unit file. {pull}7873[7873]
- Fix a race condition with the `add_host_metadata` and the event serialization. {pull}8223[8223]
- Enforce that data used by k8s or docker doesn't use any reference. {pull}8240[8240]
- Implement CheckConfig in RunnerFactory to make autodiscover check configs {pull}7961[7961]
- Make kubernetes autodiscover ignore events with empty container IDs {pull}7971[7971]

*Auditbeat*

- Fixed a concurrent map write panic in the auditd module. {pull}8158[8158]
- Fixed the RPM by designating the config file as configuration data in the RPM spec. {issue}8075[8075]

*Filebeat*

- Fixed a docker input error due to the offset update bug in partial log join.{pull}8177[8177]
- Update CRI format to support partial/full tags. {pull}8265[8265]

*Metricbeat*

- Fixed the location of the modules.d dir in Deb and RPM packages. {issue}8104[8104]
- Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. {issue}8075[8075]
- Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. {pull}7789[7789]

*Packetbeat*

- Added missing `cmdline` and `client_cmdline` fields to index template. {pull}8258[8258]

[[release-notes-6.4.0]]
=== Beats version 6.4.0
https://github.com/elastic/beats/compare/v6.3.1\...v6.4.0[View commits]

==== Known issue

Due to a packaging mistake, the `modules.d` configuration directory is
installed in the wrong path in the Metricbeat DEB and RPM packages.  This issue
results in an empty list when you run `metricbeat modules list` and failures
when you try to enable or disable modules. To work around this issue, run the
following command:

[source,sh]
-----------
sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/
-----------

This issue affects all new installations on DEB and RPM. Upgrades will run, but
use old configurations defined in the `modules.d` directory from the previous
installation.

The issue will be fixed in the 6.4.1 release.

==== Breaking changes

*Affecting all Beats*

- Set default kafka version to 1.0.0 in kafka output. Older versions are still supported by configuring the `version` setting. Minimally supported version is 0.11 (older versions might work, but are untested). {pull}7025[7025]

*Heartbeat*

- Rename http.response.status to http.response.status_code to align with ECS. {pull}7274[7274]
- Remove `type` field as not needed. {pull}7307[7307]

*Metricbeat*

- Fixed typo in values for `state_container` `status.phase`, from `terminate` to `terminated`. {pull}6916[6916]
- RabbitMQ management plugin path is now configured at the module level instead of having to do it in each of the metricsets. New `management_path_prefix` option should be used now {pull}7074[7074]
- RabbitMQ node metricset only collects metrics of the instance it connects to, `node.collect: cluster` can be used to collect all nodes as before. {issue}6556[6556] {pull}6971[6971]
- Change http/server metricset to put events by default under http.server and prefix config options with server.. {pull}7100[7100]
- Disable dedotting in docker module configuration. This will change the out-of-the-box behaviour, but not the one of already configured instances. {pull}7485[7485]
- Fix typo in etcd/self metricset fields from *.bandwithrate to *.bandwidthrate. {pull}7456[7456]
- Changed the definition of the `system.cpu.total.pct` and `system.cpu.total.norm.cou` fields to exclude the IOWait time. {pull}7691[7691]

==== Bugfixes

*Affecting all Beats*

- Error out on invalid Autodiscover template conditions settings. {pull}7200[7200]
- Allow to override the `ignore_above` option when defining new field with the type keyword. {pull}7238[7238]
- Fix a panic on the Dissect processor when we have data remaining after the last delimiter. {pull}7449[7449]
- When we fail to build a Kubernetes' indexer or matcher we produce a warning but we don't add them to the execution. {pull}7466[7466]
- Fix default value for logging.files.keepfiles. It was being set to 0 and now
  it's set to the documented value of 7. {issue}7494[7494]
- Retain compatibility with older Docker server versions. {issue}7542[7542]
- Fix errors unpacking configs modified via CLI by ignoring `-E key=value` pairs with missing value. {pull}7599[7599]

*Auditbeat*

- Allow `auditbeat setup` to run without requiring elevated privileges for the audit client. {issue}7111[7111]
- Fix goroutine leak that occurred when the auditd module was stopped. {pull}7163[7163]

*Filebeat*

- Fix a data race between stopping and starting of the harvesters. {issue}6879[6879]
- Fix an issue when parsing ISO8601 dates with timezone definition {issue}7367[7367]
- Fix Grok pattern of MongoDB module. {pull}7568[7568]
- Fix registry duplicates and log resending on upgrade. {issue}7634[7634]

*Metricbeat*

- Fix Windows service metricset when using a 32-bit binary on a 64-bit OS. {pull}7294[7294]
- Do not report Metricbeat container host as hostname in Kubernetes deployment. {issue}7199[7199]
- Ensure metadata updates don't replace existing pod metrics. {pull}7573[7573]
- Fix kubernetes pct fields reporting. {pull}7677[7677]
- Add support for new `kube_node_status_condition` in Kubernetes `state_node`. {pull}7699[7699]

==== Added

*Affecting all Beats*

- Add dissect processor. {pull}6925[6925]
- Add IP-addresses and MAC-addresses to add_host_metadata. {pull}6878[6878]
- Added a seccomp (secure computing) filter on Linux that whitelists the
  necessary system calls used by each Beat. {issue}5213[5213]
- Ship fields.yml as part of the binary {pull}4834[4834]
- Added options to dev-tools/cmd/dashboards/export_dashboard.go: -indexPattern to include index-pattern in output, -quiet to be quiet. {pull}7101[7101]
- Add Indexer indexing by pod uid. Enable pod uid metadata gathering in add_kubernetes_metadata. Extended Matcher log_path matching to support volume mounts {pull}7072[7072]
- Add default_fields to Elasticsearch template when connecting to Elasticsearch >= 7.0. {pull}7015[7015]
- Add support for loading a template.json file directly instead of using fields.yml. {pull}7039[7039]
- Add support for keyword multifields in field.yml. {pull}7131[7131]
- Add experimental Jolokia Discovery autodiscover provider. {pull}7141[7141]
- Add owner object info to Kubernetes metadata. {pull}7231[7231]
- Add Beat export dashboard command. {pull}7239[7239]
- Add support for docker autodiscover to monitor containers on host network {pull}6708[6708]
- Add ability to define input configuration as stringified JSON for autodiscover. {pull}7372[7372]
- Add processor definition support for hints builder {pull}7386[7386]
- Add support to disable html escaping in outputs. {pull}7445[7445]
- Refactor error handing in schema.Apply(). {pull}7335[7335]
- Add additional types to Kubernetes metadata {pull}7457[7457]
- Add module state reporting for Beats Monitoring. {pull}7075[7075]
- Release the `rename` processor as GA. {pull}7656[7656]
- Add support for Openstack Nova in `add_cloud_metadata` processor. {pull}7663[7663]
- Add support to set Beats services to automatic-delayed start on Windows. {pull}8720[8711]

*Auditbeat*

- Added XXH64 hash option for file integrity checks. {pull}7311[7311]
- Added the `show auditd-rules` and `show auditd-status` commands to show kernel rules and status. {pull}7114[7114]
- Add Kubernetes specs for auditbeat file integrity monitoring {pull}7642[7642]

*Filebeat*

- Add Kibana module with log fileset. {pull}7052[7052]
- Support MySQL 5.7.19 by mysql/slowlog {pull}6969[6969]
- Correctly join partial log lines when using `docker` input. {pull}6967[6967]
- Add support for TLS with client authentication to the TCP input {pull}7056[7056]
- Converted part of pipeline from treafik/access metricSet to dissect to improve efficiency. {pull}7209[7209]
- Add GC fileset to the Elasticsearch module. {pull}7305[7305]
- Add Audit log fileset to the Elasticsearch module. {pull}7365[7365]
- Add Slow log fileset to the Elasticsearch module. {pull}7473[7473]
- Add deprecation fileset to the Elasticsearch module. {pull}7474[7474]
- Add `convert_timezone` option to Kafka module to convert dates to UTC. {issue}7546[7546] {pull}7578[7578]
- Add patterns for kafka 1.1 logs. {pull}7608[7608]
- Move debug messages in tcp input source {pull}7712[7712]

*Metricbeat*

- Add experimental Elasticsearch index metricset. {pull}6881[6881]
- Add dashboards and visualizations for haproxy metrics. {pull}6934[6934]
- Add Jolokia agent in proxy mode. {pull}6475[6475]
- Add message rates to the RabbitMQ queue metricset {issue}6442[6442] {pull}6606[6606]
- Add exchanges metricset to the RabbitMQ module {issue}6442[6442] {pull}6607[6607]
- Add Elasticsearch index_summary metricset. {pull}6918[6918]
- Add shard metricset to Elasticsearch module. {pull}7006[7006]
- Add apiserver metricset to Kubernetes module. {pull}7059[7059]
- Add maxmemory to redis info metricset. {pull}7127[7127]
- Set guest as default user in RabbitMQ module. {pull}7107[7107]
- Add postgresql statement metricset. {issue}7048[7048] {pull}7060[7060]
- Update `state_container` metricset to support latest `kube-state-metrics` version. {pull}7216[7216]
- Add TLS support to MongoDB module. {pull}7401[7401]
- Added Traefik module with health metricset. {pull}7413[7413]
- Add Elasticsearch ml_job metricsets. {pull}7196[7196]
- Add support for bearer token files to HTTP helper. {pull}7527[7527]
- Add Elasticsearch index recovery metricset. {pull}7225[7225]
- Add `locks`, `global_locks`, `oplatencies` and `process` fields to `status` metricset of MongoDB module. {pull}7613[7613]
- Run Kafka integration tests on version 1.1.0 {pull}7616[7616]
- Release raid and socket metricset from system module as GA. {pull}7658[7658]
- Release elasticsearch module and all its metricsets as beta. {pull}7662[7662]
- Release munin and traefik module as beta. {pull}7660[7660]
- Add envoyproxy module. {pull}7569[7569]
- Release prometheus collector metricset as GA. {pull}7660[7660]
- Add Elasticsearch `cluster_stats` metricset. {pull}7638[7638]
- Added `basepath` setting for HTTP-based metricsets {pull}7700[7700]
- Add couchdb module. {pull}9406[9406]

*Packetbeat*

- The process monitor now reports the command-line for all processes, under Linux and Windows. {pull}7135[7135]
- Updated the TLS protocol parser with new cipher suites added to TLS 1.3. {issue}7455[7455]
- Flows are enriched with process information using the process monitor. {pull}7507[7507]
- Added UDP support to process monitor. {pull}7571[7571]

==== Deprecated

*Metricbeat*

- Kubernetes `state_container` `cpu.limit.nanocores` and `cpu.request.nanocores` have been
deprecated in favor of `cpu.*.cores`. {pull}6916[6916]

[[release-notes-6.3.1]]
=== Beats version 6.3.1
https://github.com/elastic/beats/compare/v6.3.0\...v6.3.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Allow index-pattern only setup when setup.dashboards.only_index=true. {pull}7285[7285]
- Preserve the event when source matching fails in `add_docker_metadata`. {pull}7133[7133]
- Negotiate Docker API version from our client instead of using a hardcoded one. {pull}7165[7165]
- Fix duplicating dynamic_fields in template when overwriting the template. {pull}7352[7352]

*Auditbeat*

- Fixed parsing of AppArmor audit messages. {pull}6978[6978]

*Filebeat*

- Comply with PostgreSQL database name format {pull}7198[7198]
- Optimize PostgreSQL ingest pipeline to use anchored regexp and merge multiple regexp into a single expression. {pull}7269[7269]
- Keep different registry entry per container stream to avoid wrong offsets. {issue}7281[7281]
- Fix offset field pointing at end of a line. {issue}6514[6514]
- Commit registry writes to stable storage to avoid corrupt registry files. {issue}6792[6792]

*Metricbeat*

- Fix field mapping for the system process CPU ticks fields. {pull}7230[7230]
- Ensure canonical naming for JMX beans is disabled in Jolokia module. {pull}7047[7047]
- Fix Jolokia attribute mapping when using wildcards and MBean names with multiple properties. {pull}7321[7321]

*Packetbeat*

- Fix an out of bounds access in HTTP parser caused by malformed request. {pull}6997[6997]
- Fix missing type for `http.response.body` field. {pull}7169[7169]

==== Added

*Auditbeat*

- Added caching of UID and GID values to auditd module. {pull}6978[6978]
- Updated syscall tables for Linux 4.16. {pull}6978[6978]
- Added better error messages for when the auditd module fails due to the
  Linux kernel not supporting auditing (CONFIG_AUDIT=n). {pull}7012[7012]

*Metricbeat*

- Collect accumulated docker network metrics and mark old ones as deprecated. {pull}7253[7253]



[[release-notes-6.3.0]]
=== Beats version 6.3.0
https://github.com/elastic/beats/compare/v6.2.3\...v6.3.0[View commits]

==== Breaking changes

*Affecting all Beats*

- De dot keys of labels and annotations in kubernetes meta processors to prevent collisions. {pull}6203[6203]
- Rename `beat.cpu.*.time metrics` to `beat.cpu.*.time.ms`. {pull}6449[6449]
- Add `host.name` field to all events, to avoid mapping conflicts. This could be breaking Logstash configs if you rely on the `host` field being a string. {pull}7051[7051]

*Filebeat*

- Add validation for Stdin, when Filebeat is configured with Stdin and any other inputs, Filebeat
  will now refuse to start. {pull}6463[6463]
- Mark `system.syslog.message` and `system.auth.message` as `text` instead of `keyword`. {pull}6589[6589]

*Metricbeat*

- De dot keys in kubernetes/event metricset to prevent collisions. {pull}6203[6203]
- Add config option for windows/perfmon metricset to ignore non existent counters. {pull}6432[6432]
- Refactor docker CPU calculations to be more consistent with `docker stats`. {pull}6608[6608]
- Update logstash.node_stats metricset to write data under `logstash.node.stats.*`. {pull}6714[6714]

==== Bugfixes

*Affecting all Beats*

- Fix panic when Events containing a float32 value are normalized. {pull}6129[6129]
- Fix `setup.dashboards.always_kibana` when using Kibana 5.6. {issue}6090[6090]
- Fix for Kafka logger. {pull}6430[6430]
- Remove double slashes in Windows service script. {pull}6491[6491]
- Ensure Kubernetes labels/annotations don't break mapping {pull}6490[6490]
- Ensure that the dashboard zip files can't contain files outside of the kibana directory. {pull}6921[6921]
- Fix map overwrite panics by cloning shared structs before doing the update. {pull}6947[6947]
- Fix delays on autodiscovery events handling caused by blocking runner stops. {pull}7170[7170]
- Do not emit Kubernetes autodiscover events for Pods without IP address. {pull}7235[7235]
- Fix self metrics when containerized {pull}6641[6641]

*Auditbeat*

- Add hex decoding for the name field in audit path records. {pull}6687[6687]
- Fixed a deadlock in the file_integrity module under Windows. {issue}6864[6864]
- Fixed parsing of AppArmor audit messages. {pull}6978[6978]
- Allow `auditbeat setup` to run without requiring elevated privileges for the audit client. {issue}7111[7111]
- Fix goroutine leak that occurred when the auditd module was stopped. {pull}7163[7163]

*Filebeat*

- Fix panic when log prospector configuration fails to load. {issue}6800[6800]
- Fix memory leak in log prospector when files cannot be read. {issue}6797[6797]
- Add raw JSON to message field when JSON parsing fails. {issue}6516[6516]
- Commit registry writes to stable storage to avoid corrupt registry files. {pull}6877[6877]
- Fix a parsing issue in the syslog input for RFC3339 timestamp and time with nanoseconds. {pull}7046[7046]
- Fix an issue with an overflowing wait group when using the TCP input. {issue}7202[7202]

*Heartbeat*

- Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. {issue}6616[6616]

*Metricbeat*

- Fix the default configuration for Logstash to include the default port. {pull}6279[6279]
- Fix dealing with new process status codes in Linux kernel 4.14+. {pull}6306[6306]
- Add filtering option by exact device names in system.diskio. `diskio.include_devices`. {pull}6085[6085]
- Add connections metricset to RabbitMQ module {pull}6548[6548]
- Fix panic in http dependent modules when invalid config was used. {pull}6205[6205]
- Fix system.filesystem.used.pct value to match what df reports. {issue}5494[5494]
- Fix namespace disambiguation in Kubernetes state_* metricsets. {issue}6281[6281]
- Fix Windows perfmon metricset so that it sends metrics when an error occurs. {pull}6542[6542]
- Fix Kubernetes calculated fields store. {pull}6564[6564]
- Exclude bind mounts in fsstat and filesystem metricsets. {pull}6819[6819]
- Don't stop Metricbeat if aerospike server is down. {pull}6874[6874]
- disk reads and write count metrics in RabbitMQ queue metricset made optional. {issue}6876[6876]
- Add mapping for docker metrics per cpu. {pull}6843[6843]

*Winlogbeat*

- Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. {pull}6247[6247]
- Fix config validation to allow `event_logs.processors`. {pull}6217[6217]

==== Added

*Affecting all Beats*

- Update Golang 1.9.4 {pull}6326[6326]
- Add the ability to log to the Windows Event Log. {pull}5913[5913]
- The node name can be discovered automatically by machine-id matching when beat deployed outside Kubernetes cluster. {pull}6146[6146]
- Panics will be written to the logger before exiting. {pull}6199[6199]
- Add builder support for autodiscover and annotations builder {pull}6408[6408]
- Add plugin support for autodiscover builders, providers {pull}6457[6457]
- Preserve runtime from container statuses in Kubernetes autodiscover {pull}6456[6456]
- Experimental feature setup.template.append_fields added. {pull}6024[6024]
- Add appender support to autodiscover {pull}6469[6469]
- Add add_host_metadata processor {pull}5968[5968]
- Retry configuration to load dashboards if Kibana is not reachable when the beat starts. {pull}6560[6560]
- Add `has_fields` conditional to filter events based on the existence of all the given fields. {issue}6285[6285] {pull}6653[6653]
- Add support for spooling to disk to the beats event publishing pipeline. {pull}6581[6581]
- Added logging of system info at Beat startup. {issue}5946[5946]
- Do not log errors if X-Pack Monitoring is enabled but Elastisearch X-Pack is not. {pull}6627[6627]
- Add rename processor. {pull}6292[6292]
- Allow override of dynamic template `match_mapping_type` for fields with object_type. {pull}6691[6691]

*Filebeat*

- Add IIS module to parse access log and error log. {pull}6127[6127]
- Renaming of the prospector type to the input type and all prospectors are now moved to the input
  folder, to maintain backward compatibility type aliasing was used to map the old type to the new
  one. This change also affect YAML configuration. {pull}6078[6078]
- Addition of the TCP input {pull}6700[6700]
- Add option to convert the timestamps to UTC in the system module. {pull}5647[5647]
- Add Logstash module support for main log and the slow log, support the plain text or structured JSON format {pull}5481[5481]
- Add stream filtering when using `docker` prospector. {pull}6057[6057]
- Add support for CRI logs format. {issue}5630[5630]
- Add json.ignore_decoding_error config to not log json decoding erors. {issue}6547[6547]
- Make registry file permission configurable. {pull}6455[6455]
- Add MongoDB module. {pull}6283[6238]
- Add Ingest pipeline loading to setup. {pull}6814[6814]
- Add support of log_format combined to NGINX access logs. {pull}6858[6858]
- Release config reloading feature as GA.
- Add support human friendly size for the UDP input. {pull}6886[6886]
- Add Syslog input to ingest RFC3164 Events via TCP and UDP {pull}6842[6842]
- Remove the undefined `username` option from the Redis input and clarify the documentation. {pull}6662[6662]
- Add CometD input {pull}30089[30089]

*Heartbeat*

- Made the URL field of Heartbeat aggregateable. {pull}6263[6263]
- Use `match.Matcher` for checking Heartbeat response bodies with regular expressions. {pull}6539[6539]

*Metricbeat*

- Support apache status pages for versions older than 2.4.16. {pull}6450[6450]
- Add support for huge pages on Linux. {pull}6436[6436]
- Support to optionally 'de dot' keys in http/json metricset to prevent collisions. {pull}5970[5970]
- Add graphite protocol metricbeat module. {pull}4734[4734]
- Add http server metricset to support push metrics via http. {pull}4770[4770]
- Make config object public for graphite and http server {pull}4820[4820]
- Add system uptime metricset. {issue}4848[4848]
- Add experimental `queue` metricset to RabbitMQ module. {pull}4788[4788]
- Add additional php-fpm pool status kpis for Metricbeat module {pull}5287[5287]
- Add etcd module. {issue}4970[4970]
- Add ip address of docker containers to event. {pull}5379[5379]
- Add ceph osd tree information to metricbeat {pull}5498[5498]
- Add ceph osd_df to metricbeat {pull}5606[5606]
- Add basic Logstash module. {pull}5540[5540]
- Add dashboard for Windows service metricset. {pull}5603[5603]
- Add pct calculated fields for Pod and container CPU and memory usages. {pull}6158[6158]
- Add statefulset support to Kubernetes module. {pull}6236[6236]
- Refactor prometheus endpoint parsing to look similar to upstream prometheus {pull}6332[6332]
- Making the http/json metricset GA. {pull}6471[6471]
- Add support for array in http/json metricset. {pull}6480[6480]
- Making the jolokia/jmx module GA. {pull}6143[6143]
- Making the MongoDB module GA. {pull}6554[6554]
- Allow to disable labels `dedot` in Docker module, in favor of a safe way to keep dots. {pull}6490[6490]
- Add experimental module to collect metrics from munin nodes. {pull}6517[6517]
- Add support for wildcards and explicit metrics grouping in jolokia/jmx. {pull}6462[6462]
- Set `collector` as default metricset in Prometheus module. {pull}6636[6636] {pull}6747[6747]
- Set `mntr` as default metricset in Zookeeper module. {pull}6674[6674]
- Set default metricsets in vSphere module. {pull}6676[6676]
- Set `status` as default metricset in Apache module. {pull}6673[6673]
- Set `namespace` as default metricset in Aerospike module. {pull}6669[6669]
- Set `service` as default metricset in Windows module. {pull}6675[6675]
- Set all metricsets as default metricsets in uwsgi module. {pull}6688[6688]
- Allow autodiscover to monitor unexposed ports {pull}6727[6727]
- Mark kubernetes.event metricset as beta. {pull}6715[6715]
- Set all metricsets as default metricsets in couchbase module. {pull}6683[6683]
- Mark uwsgi module and metricset as beta. {pull}6717[6717]
- Mark Golang module and metricsets as beta. {pull}6711[6711]
- Mark system.raid metricset as beta. {pull}6710[6710]
- Mark http.server metricset as beta. {pull}6712[6712]
- Mark metricbeat logstash module and metricsets as beta. {pull}6713[6713]
- Set all metricsets as default metricsets in Ceph module. {pull}6676[6676]
- Set `container`, `cpu`, `diskio`, `healthcheck`, `info`, `memory` and `network` in docker module as default. {pull}6718[6718]
- Set `cpu`, `load`, `memory`, `network`, `process` and `process_summary` as default metricsets in system module. {pull}6689[6689]
- Set `collector` as default metricset in Dropwizard module. {pull}6669[6669]
- Set `info` and `keyspace` as default metricsets in redis module. {pull}6742[6742]
- Set `connection` as default metricset in rabbitmq module. {pull}6743[6743]
- Set all metricsets as default metricsets in Elasticsearch module. {pull}6755[6755]
- Set all metricsets as default metricsets in Etcd module. {pull}6756[6756]
- Set server metricsets as default in Graphite module. {pull}6757[6757]
- Set all metricsets as default metricsets in HAProxy module. {pull}6758[6758]
- Set all metricsets as default metricsets in Kafka module. {pull}6759[6759]
- Set all metricsets as default metricsets in postgresql module. {pull}6761[6761]
- Set status metricsets as default in Kibana module. {pull}6762[6762]
- Set all metricsets as default metricsets in Logstash module. {pull}6763[6763]
- Set `container`, `node`, `pod`, `system`, `volume` as default in Kubernetes module. {pull} 6764[6764]
- Set `stats` as default in memcached module. {pull}6765[6765]
- Set all metricsets as default metricsets in Mongodb module. {pull}6766[6766]
- Set `pool` as default metricset for php_fpm module. {pull}6768[6768]
- Set `status` as default metricset for mysql module. {pull} 6769[6769]
- Set `stubstatus` as default metricset for nginx module. {pull}6770[6770]
- Added support for haproxy 1.7 and 1.8. {pull}6793[6793]
- Add accumulated I/O stats to diskio in the line of `docker stats`. {pull}6701[6701]
- Ignore virtual filesystem types by default in system module. {pull}6819[6819]
- Release config reloading feature as GA. {pull}6891[6891]
- Kubernetes deployment: Add ServiceAccount config to system metricbeat. {pull}6824[6824]
- Kubernetes deployment: Add DNS Policy to system metricbeat. {pull}6656[6656]

*Packetbeat*

- Add support for condition on bool type {issue}5659[5659] {pull}5954[5954]
- Fix high memory usage on HTTP body if body is not published. {pull}6680[6680]
- Allow to capture the HTTP request or response bodies independently. {pull}6784[6784]
- HTTP publishes an Error event for unmatched requests or responses. {pull}6794[6794]

*Winlogbeat*

- Use bookmarks to persist the last published event. {pull}6150[6150]

[[release-notes-6.2.3]]
=== Beats version 6.2.3
https://github.com/elastic/beats/compare/v6.2.2\...v6.2.3[View commits]

==== Breaking changes

*Affecting all Beats*

- Fix conditions checking on autodiscover Docker labels. {pull}6412[6412]

==== Bugfixes

*Affecting all Beats*

- Avoid panic errors when processing nil Pod events in add_kubernetes_metadata. {issue}6372[6372]
- Fix infinite failure on Kubernetes watch {pull}6504[6504]

*Metricbeat*

- Fix Kubernetes overview dashboard views for non default time ranges. {issue}6395[6395]


[[release-notes-6.2.2]]
=== Beats version 6.2.2
https://github.com/elastic/beats/compare/v6.2.1\...v6.2.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Add logging when monitoring cannot connect to Elasticsearch. {pull}6365[6365]
- Fix infinite loop when event unmarshal fails in Kubernetes pod watcher. {pull}6353[6353]

*Filebeat*

- Fix a conversion issue for time related fields in the Logstash module for the slowlog
  fileset. {issue}6317[6317]

[[release-notes-6.2.1]]
=== Beats version 6.2.1
https://github.com/elastic/beats/compare/v6.2.0\...v6.2.1[View commits]

No changes in this release.

[[release-notes-6.2.0]]
=== Beats version 6.2.0
https://github.com/elastic/beats/compare/v6.1.3\...v6.2.0[View commits]

==== Breaking changes

*Affecting all Beats*

- The log format may differ due to logging library changes. {pull}5901[5901]
- The default value for pipelining is reduced to 2 to avoid high memory in the Logstash beats input. {pull}6250[6250]

*Auditbeat*

- Split the audit.kernel and audit.file metricsets into their own modules
  named auditd and file_integrity, respectively. This change requires
  existing users to update their config. {issue}5422[5422]
- Renamed file_integrity module fields. {issue}5423[5423] {pull}5995[5995]
- Renamed auditd module fields. {issue}5423[5423] {pull}6080[6080]

*Metricbeat*

- Rename `golang.heap.system.optained` field to `golang.heap.system.obtained`. {issue}5703[5703]
- De dot keys in jolokia/jmx metricset to prevent collisions. {pull}5957[5957]

==== Bugfixes

*Auditbeat*

- Fixed an issue where the proctitle value was being truncated. {pull}6080[6080]
- Fixed an issue where values were incorrectly interpreted as hex data. {pull}6080[6080]
- Fixed parsing of the `key` value when multiple keys are present. {pull}6080[6080]
- Fix possible resource leak if file_integrity module is used with config
  reloading on Windows or Linux. {pull}6198[6198]

*Filebeat*

- Fix variable name for `convert_timezone` in the system module. {pull}5936[5936]

*Metricbeat*

- Fix error `datastore '*' not found` in Vsphere module. {issue}4879[4879]
- Fix error `NotAuthenticated` in Vsphere module. {issue}4673[4673]
- Fix mongodb session consistency mode to allow command execution on secondary nodes. {issue}4689[4689]
- Fix kubernetes `state_pod` `status.phase` so that the active phase is returned instead of `unknown`. {pull}5980[5980]
- Fix error collecting network_names in Vsphere module. {pull}5962[5962]
- Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp. {issue}6033[6033]
- Fix kafka OffsetFetch request missing topic and partition parameters. {pull}5880[5880]

*Packetbeat*

- Fix mysql SQL parser to trim `\r` from Windows Server `SELECT\r\n\t1`. {pull}5572[5572]


==== Added

*Affecting all Beats*

- Adding a local keystore to allow user to obfuscate password {pull}5687[5687]
- Add autodiscover for kubernetes. {pull}6055[6055]
- Add Beats metrics reporting to Xpack. {issue}3422[3422]
- Update the command line library cobra and add support for zsh completion {pull}5761[5761]
- Update to Golang 1.9.2
- Moved `ip_port` indexer for `add_kubernetes_metadata` to all beats. {pull}5707[5707]
- `ip_port` indexer now index both IP and IP:port pairs. {pull}5721[5721]
- Add the ability to write structured logs. {pull}5901[5901]
- Use structured logging for the metrics that are periodically logged via the
  `logging.metrics` feature. {pull}5915[5915]
- Improve Elasticsearch output metrics to count number of dropped and duplicate (if event ID is given) events. {pull}5811[5811]
- Add the ability for the add_docker_metadata process to enrich based on process ID. {pull}6100[6100]
- The `add_docker_metadata` and `add_kubernetes_metadata` processors are now GA, instead of Beta. {pull}6105[6105]
- Update go-ucfg library to support top level key reference and cyclic key reference for the
  keystore {pull}6098[6098]

*Auditbeat*

- Auditbeat is marked as GA, no longer Beta. {issue}5432[5432]
- Add support for BLAKE2b hash algorithms to the file integrity module. {pull}5926[5926]
- Add support for recursive file watches. {pull}5575[5575] {pull}5833[5833]

*Filebeat*

- Add Osquery module. {pull}5971[5971]
- Add stream filtering when using `docker` prospector. {pull}6057[6057]

*Metricbeat*

- Add ceph osd_df to metricbeat {pull}5606[5606]
- Add field network_names of hosts and virtual machines. {issue}5646[5646]
- Add experimental system/raid metricset. {pull}5642[5642]
- Add a dashboard for the Nginx module. {pull}5991[5991]
- Add experimental mongodb/collstats metricset. {pull}5852[5852]
- Update the MySQL dashboard to use the Time Series Visual Builder. {pull}5996[5996]
- Add experimental uwsgi module. {pull}6006[6006]
- Docker and Kubernetes modules are now GA, instead of Beta. {pull}6105[6105]
- Support haproxy stats gathering using http (additionally to tcp socket). {pull}5819[5819]
- Support to optionally 'de dot' keys in http/json metricset to prevent collisions. {pull}5957[5957]

*Packetbeat*

- Configure good defaults for `add_kubernetes_metadata`. {pull}5707[5707]

[[release-notes-6.1.3]]
=== Beats version 6.1.3
https://github.com/elastic/beats/compare/v6.1.2\...v6.1.3[View commits]

No changes in this release.

[[release-notes-6.1.2]]
=== Beats version 6.1.2
https://github.com/elastic/beats/compare/v6.1.1\...v6.1.2[View commits]

==== Bugfixes

*Auditbeat*

- Add an error check to the file integrity scanner to prevent a panic when
  there is an error reading file info via lstat. {issue}6005[6005]

==== Added

*Filebeat*

- Switch to docker prospector in sample manifests for Kubernetes deployment {pull}5963[5963]

[[release-notes-6.1.1]]
=== Beats version 6.1.1
https://github.com/elastic/beats/compare/v6.1.0\...v6.1.1[View commits]

No changes in this release.

[[release-notes-6.1.0]]
=== Beats version 6.1.0
https://github.com/elastic/beats/compare/v6.0.1\...v6.1.0[View commits]

==== Breaking changes

*Auditbeat*

- Changed `audit.file.path` to be a multi-field so that path is searchable. {pull}5625[5625]

*Metricbeat*

- Rename `heap_init` field to `heap.init` in the Elasticsearch module. {pull}5320[5320]
- Rename `http.response.status_code` field to `http.response.code` in the HTTP module. {pull}5521[5521]

==== Bugfixes

*Affecting all Beats*

- Remove ID() from Runner interface {issue}5153[5153]
- Correctly send configured `Host` header to the remote server. {issue}4842[4842]
- Change add_kubernetes_metadata to attempt detection of namespace. {pull}5482[5482]
- Avoid double slash when join url and path {pull}5517[5517]
- Fix console color output for Windows. {issue}5611[5611]
- Fix logstash output debug message. {pull}5799{5799]
- Fix isolation of modules when merging local and global field settings. {issue}5795[5795]
- Report ephemeral ID and uptime in monitoring events on all platforms {pull}6501[6501]

*Filebeat*

- Add support for adding string tags {pull}5395[5395]
- Fix race condition when limiting the number of harvesters running in parallel {issue}5458[5458]
- Fix relative paths in the prospector definitions. {pull}5443[5443]
- Fix `recursive_globe.enabled` option. {pull}5443[5443]

*Metricbeat*

- Change field type of http header from nested to object {pull}5258[5258]
- Fix the fetching of process information when some data is missing under MacOS X. {issue}5337[5337]
- Change `MySQL active connections` visualization title to `MySQL total connections`. {issue}4812[4812]
- Fix `ProcState` on Linux and FreeBSD when process names contain parentheses. {pull}5775[5775]
- Fix incorrect `Mem.Used` calculation under linux. {pull}5775[5775]
- Fix `open_file_descriptor_count` and `max_file_descriptor_count` lost in zookeeper module {pull}5902[5902]
- Fix system process metricset for kernel processes. {issue}5700[5700]
- Change kubernetes.node.cpu.allocatable.cores to float. {pull}6130[6130]

*Packetbeat*

- Fix http status phrase parsing not allow spaces. {pull}5312[5312]
- Fix http parse to allow to parse get request with space in the URI. {pull}5495[5495]
- Fix mysql SQL parser to trim `\r` from Windows Server `SELECT\r\n\t1`. {pull}5572[5572]
- Fix corruption when parsing repeated headers in an HTTP request or response. {pull}6325[6325]
- Fix panic when parsing partial AMQP messages. {pull}6384[6384]
- Fix out of bounds access to slice in MongoDB parser. {pull}6256[6256]
- Fix sniffer hanging on exit under Linux. {pull}6535[6535]
- Fix bounds check error in http parser causing a panic. {pull}6750[6750]

*Winlogbeat*

- Fix the registry file. It was not correctly storing event log names, and
  upon restart it would begin reading at the start of each event log. {issue}5813[5813]

==== Added

*Affecting all Beats*

- Support dashboard loading without Elasticsearch {pull}5653[5653]
- Changed the hashbang used in the beat helper script from `/bin/bash` to `/usr/bin/env bash`. {pull}5051[5051]
- Changed beat helper script to use `exec` when running the beat. {pull}5051[5051]
- Fix reloader error message to only print on actual error {pull}5066[5066]
- Add support for enabling TLS renegotiation. {issue}4386[4386]
- Add Azure VM support for add_cloud_metadata processor {pull}5355[5355]
- Add `output.file.permission` config option. {pull}4638[4638]
- Refactor add_kubernetes_metadata to support autodiscovery {pull}5434[5434]
- Improve custom flag handling and CLI flags usage message. {pull}5543[5543]
- Add number_of_routing_shards config set to 30 {pull}5570[5570]
- Set log level for kafka output. {pull}5397[5397]
- Move TCP UDP start up into `server.Start()` {pull}4903[4903]
- Update to Golang 1.9.2

*Auditbeat*

- Add support for SHA3 hash algorithms to the file integrity module. {issue}5345[5345]
- Add dashboards for Linux audit framework events (overview, executions, sockets). {pull}5516[5516]

*Filebeat*

- Add PostgreSQL module with slowlog support. {pull}4763[4763]
- Add Kafka log module. {pull}4885[4885]
- Add support for `/var/log/containers/` log path in `add_kubernetes_metadata` processor. {pull}4981[4981]
- Remove error log from runnerfactory as error is returned by API. {pull}5085[5085]
- Add experimental Docker `json-file` prospector . {pull}5402[5402]
- Add experimental Docker autodiscover functionality. {pull}5245[5245]
- Add option to convert the timestamps to UTC in the system module. {pull}5647[5647]
- Add Logstash module support for main log and the slow log, support the plain text or structured JSON format {pull}5481[5481]

*Metricbeat*

- Add graphite protocol metricbeat module. {pull}4734[4734]
- Add http server metricset to support push metrics via http. {pull}4770[4770]
- Make config object public for graphite and http server {pull}4820[4820]
- Add system uptime metricset. {issue}4848[4848]
- Add experimental `queue` metricset to RabbitMQ module. {pull}4788[4788]
- Add additional php-fpm pool status kpis for Metricbeat module {pull}5287[5287]
- Add etcd module. {issue}4970[4970]
- Add ip address of docker containers to event. {pull}5379[5379]
- Add ceph osd tree information to Metricbeat {pull}5498[5498]
- Add basic Logstash module. {pull}5540[5540]
- Add dashboard for Windows service metricset. {pull}5603[5603]
- Add experimental Docker autodiscover functionality. {pull}5245[5245]
- Add Windows service metricset in the windows module. {pull}5332[5332]
- Update gosigar to v0.6.0. {pull}5775[5775]

*Packetbeat*

- Add support for decoding the TLS envelopes. {pull}5476[5476]
- HTTP parses successfully on empty status phrase. {issue}6176[6176]
- HTTP parser supports broken status line. {pull}6631[6631]

[[release-notes-6.0.1]]
=== Beats version 6.0.1
https://github.com/elastic/beats/compare/v6.0.0\...v6.0.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix documentation links in README.md files. {pull}5710[5710]
- Fix `add_docker_metadata` dropping some containers. {pull}5788[5788]

*Heartbeat*

- Fix the "HTTP up status" visualization. {pull}5564[5564]

*Metricbeat*

- Fix map overwrite in docker diskio module. {issue}5582[5582]
- Fix connection leak in mongodb module. {issue}5688[5688]
- Fix the include top N processes feature for cases where there are fewer
  processes than N. {pull}5729[5729]


include::libbeat/docs/release-notes/6.0.0.asciidoc[]

[[release-notes-6.0.0-ga]]
=== Beats version 6.0.0-GA
https://github.com/elastic/beats/compare/v6.0.0-rc2\...v6.0.0[View commits]

The list below covers the changes between 6.0.0-rc2 and 6.0.0 GA only.

==== Bugfixes

*Filebeat*

- Fix machine learning jobs setup for dynamic modules. {pull}5509[5509]

*Packetbeat*

- Fix missing length check in the PostgreSQL module. {pull}5457[5457]
- Fix panic in ACK handler if event is dropped on blocked queue {issue}5524[5524]

==== Added

*Filebeat*

- Add Kubernetes manifests to deploy Filebeat. {pull}5349[5349]
- Add container short ID matching to add_docker_metadata. {pull}6172[6172]

*Metricbeat*

- Add Kubernetes manifests to deploy Metricbeat. {pull}5349[5349]


[[release-notes-6.0.0-rc2]]
=== Beats version 6.0.0-rc2
https://github.com/elastic/beats/compare/v6.0.0-rc1\...v6.0.0-rc2[View commits]

==== Breaking changes

*Packetbeat*

- Remove not-working `runoptions.uid` and `runoptions.gid` options in Packetbeat. {pull}5261[5261]

==== Bugfixes

*Affecting all Beats*

- Fix data race accessing watched containers. {issue}5147[5147]
- Do not require template if index change and template disabled {pull}5319[5319]
- Fix missing ACK in redis output. {issue}5404[5404]

*Filebeat*

- Fix default paths for redis 4.0.1 logs on macOS {pull}5173[5173]
- Fix Filebeat not starting if command line and modules configs are used together. {issue}5376[5376]
- Fix double `@timestamp` field when JSON decoding was used. {pull}5436[5436]

*Metricbeat*

- Use `beat.name` instead of `beat.hostname` in the Host Overview dashboard. {pull}5340[5340]
- Fix the loading of 5.x dashboards. {issue}5277[5277]

==== Added

*Metricbeat*

- Auto-select a hostname (based on the host on which the Beat is running) in the Host Overview dashboard. {pull}5340[5340]

==== Deprecated

*Filebeat*

- The `filebeat.config_dir` option is deprecated. Use `filebeat.config.prospector` options instead. {pull}5321[5321]

[[release-notes-6.0.0-rc1]]
=== Beats version 6.0.0-rc1
https://github.com/elastic/beats/compare/v6.0.0-beta2\...v6.0.0-rc1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix the `/usr/bin/beatname` script to accept `-d "*"` as a parameter. {issue}5040[5040]
- Combine `fields.yml` properties when they are defined in different sources. {issue}5075[5075]
- Keep Docker & Kubernetes pod metadata after container dies while they are needed by processors. {pull}5084[5084]
- Fix `fields.yml` lookup when using `export template` with a custom `path.config` param. {issue}5089[5089]
- Remove runner creation from every reload check {pull}5141[5141]
- Fix add_kubernetes_metadata matcher registry lookup. {pull}5159[5159]

*Metricbeat*

- Fix a memory allocation issue where more memory was allocated than needed in the windows-perfmon metricset. {issue}5035[5035]
- Don't start metricbeat if external modules config is wrong and reload is disabled {pull}5053[5053]
- The MongoDB module now connects on each fetch, to avoid stopping the whole Metricbeat instance if MongoDB is not up when starting. {pull}5120[5120]
- Fix kubernetes events module to be able to index time fields properly. {issue}5093[5093]
- Fixed `cmd_set` and `cmd_get` being mixed in the Memcache module. {pull}5189[5189]


==== Added

*Affecting all Beats*

- Enable flush timeout by default. {pull}5150[5150]
- Add @metadata.version to events send to Logstash. {pull}5166[5166]

*Auditbeat*

- Changed the number of shards in the default configuration to 3. {issue}5095[5095]
- Add support for receiving audit events using a multicast socket. {issue}4850[4850]

*Filebeat*

- Changed the number of shards in the default configuration to 3. {issue}5095[5095]
- Don't start filebeat if external modules/prospectors config is wrong and reload is disabled {pull}5053[5053]
- Add `filebeat.registry_flush` setting, to delay the registry updates. {pull}5146[5146]

*Heartbeat*

- Changed the number of shards in the default configuration to 1. {issue}5095[5095]

*Packetbeat*

- Changed the number of shards in the default configuration to 3. {issue}5095[5095]

*Winlogbeat*

- Changed the number of shards in the default configuration to 3. {issue}5095[5095]

[[release-notes-6.0.0-beta2]]
=== Beats version 6.0.0-beta2
https://github.com/elastic/beats/compare/v6.0.0-beta1\...v6.0.0-beta2[View commits]

==== Breaking changes

*Affecting all Beats*

- The log directory (`path.log`) for Windows services is now set to `C:\ProgramData\[beatname]\logs`. {issue}4764[4764]
- The _all field is disabled in Elasticsearch 6.0. This means that searching by individual
  words only work on text fields. {issue}4901[4901]
- Fail if removed setting output.X.flush_interval is explicitly configured.
- Rename the `/usr/bin/beatname.sh` script (e.g. `metricbeat.sh`) to `/usr/bin/beatname`. {pull}4933[4933]
- Beat does not start if elasticsearch index pattern was modified but not the template name and pattern. {issue}4769[4769]
- Fail if removed setting output.X.flush_interval is explicitly configured. {pull}4880[4880]

==== Bugfixes

*Affecting all Beats*

- Register kubernetes `field_format` matcher and remove logger in `Encode` API {pull}4888[4888]
- Fix go plugins not loaded when beat starts {pull}4799[4799]
- Add support for `initContainers` in `add_kubernetes_metadata` processor. {issue}4825[4825]
- Eliminate deprecated _default_ mapping in 6.x {pull}4864[4864]
- Fix pod name indexer to use both namespace, pod name to frame index key {pull}4775[4775]

*Filebeat*

- Fix issue where the `fileset.module` could have the wrong value. {issue}4761[4761]

*Heartbeat*

- Fix monitor.name being empty by default. {issue}4852[4852]
- Fix wrong event timestamps. {issue}4851[4851]

*Metricbeat*

- Added missing mongodb configuration file to the `modules.d` folder. {pull}4870[4870]
- Fix wrong MySQL CRUD queries timelion visualization {pull}4857[4857]
- Add new metrics to CPU metricset {pull}4969[4969]

*Packetbeat*

- Update flow timestamp on each packet being received. {issue}4895[4895]

==== Added

*Affecting all Beats*

- Add setting to enable/disable the slow start in logstash output. {pull}4972[4972]
- Update init scripts to use the `test config` subcommand instead of the deprecated `-configtest` flag. {issue}4600[4600]
- Get by default the credentials for connecting to Kibana from the Elasticsearch output configuration. {pull}4867[4867]
- Added `cloud.id` and `cloud.auth` settings, for simplifying using Beats with the Elastic Cloud. {issue}4959[4959]
- Add lz4 compression support to kafka output. {pull}4977[4977]
- Add newer kafka versions to kafka output. {pull}4977[4977]
- Configure the index name when loading the dashboards and the index pattern. {pull}4949[4949]

*Metricbeat*

- Add `filesystem.ignore_types` to system module for ignoring filesystem types. {issue}4685[4685]
- Add support to exclude labels from kubernetes pod metadata. {pull}4757[4757]

[[release-notes-6.0.0-beta1]]
=== Beats version 6.0.0-beta1
https://github.com/elastic/beats/compare/v6.0.0-alpha2\...v6.0.0-beta1[View commits]

==== Breaking changes

*Affecting all Beats*

- Rename `kubernetes` processor to `add_kubernetes_metadata`. {pull}4473[4473]
- Rename `*.full.yml` config files to `*.reference.yml`. {pull}4563[4563]
- The `scripts/import_dashboards` is removed from packages. Use the `setup` command instead. {pull}4586[4586]
- Change format of the saved kibana dashboards to have a single JSON file for each dashboard {pull}4413[4413]
- Rename `configtest` command to `test config`. {pull}4590[4590]
- Remove setting `queue_size` and `bulk_queue_size`. {pull}4650[4650]
- Remove setting `dashboard.snapshot` and `dashboard.snapshot_url`. They are no longer needed because the
  dashboards are included in the packages by default. {pull}4675[4675]
- Beats can no longer be launched from Windows Explorer (GUI), command line is required. {pull}4420[4420]

*Auditbeat*

- Changed file metricset config to make `file.paths` a list instead of a dictionary. {pull}4796[4796]

*Heartbeat*

- Renamed the heartbeat RPM/DEB name to `heartbeat-elastic`. {pull}4601[4601]

*Metricbeat*

- Change all `system.cpu.*.pct` metrics to be scaled by the number of CPU cores.
  This will make the CPU usage percentages from the system cpu metricset consistent
  with the system process metricset. The documentation for these metrics already
  stated that on multi-core systems the percentages could be greater than 100%. {pull}4544[4544]
- Remove filters setting from metricbeat modules. {pull}4699[4699]
- Added `type` field to filesystem metrics. {pull}4717[4717]

*Packetbeat*

- Remove the already unsupported `pf_ring` sniffer option. {pull}4608[4608]

==== Bugfixes

*Affecting all Beats*

- Don't stop with error loading the ES template if the ES output is not enabled. {pull}4436[4436]
- Fix race condition in internal logging rotator. {pull}4519[4519]
- Normalize all times to UTC to ensure proper index naming. {issue}4569[4569]
- Fix issue with loading dashboards to ES 6.0 when .kibana index did not already exist. {issue}4659[4659]

*Auditbeat*

- Fix `file.max_file_size` config option for the audit file metricset. {pull}4796[4796]

*Filebeat*

- Fix issue where the `fileset.module` could have the wrong value. {issue}4761[4761]

*Metricbeat*

- Fix issue affecting Windows services timing out at startup. {pull}4491[4491]
- Fix incorrect docker.diskio.total metric calculation. {pull}4507[4507]
- Vsphere module: used memory field corrected. {issue}4461[4461]

*Packetbeat*

- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. {pull}4442[4442]

*Winlogbeat*

- Removed validation of top-level config keys. This behavior was inconsistent with other Beats
  and caused maintainability issues. {pull}4657[4657]

==== Added

*Affecting all Beats*

- New cli subcommands interface. {pull}4420[4420]
- Allow source path matching in `add_docker_metadata` processor. {pull}4495[4495]
- Add support for analyzers and multifields in fields.yml. {pull}4574[4574]
- Add support for JSON logging. {pull}4523[4523]
- Add `test output` command, to test Elasticsearch and Logstash output settings. {pull}4590[4590]
- Introduce configurable event queue settings: queue.mem.events, queue.mem.flush.min_events and queue.mem.flush.timeout. {pull}4650[4650]
- Enable pipelining in Logstash output by default. {pull}4650[4650]
- Added 'result' field to Elasticsearch QueryResult struct for compatibility with 6.x Index and Delete API responses. {issue}4661[4661]
- The sample dashboards are now included in the Beats packages. {pull}4675[4675]
- Add `pattern` option to be used in the fields.yml to specify the pattern for a number field. {pull}4731[4731]

*Auditbeat*

- Added `file.hash_types` config option for controlling the hash types. {pull}4796[4796]
- Added the ability to specify byte unit suffixes to `file.max_file_size`. {pull}4796[4796]

*Filebeat*

- Add experimental Redis module. {pull}4441[4441]
- Nginx module: use the first not-private IP address as the remote_ip. {pull}4417[4417]
- Load Ingest Node pipelines when the Elasticsearch connection is established, instead of only once at startup. {pull}4479[4479]
- Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. {pull}4506[4506] {pull}4609[4609]

- Add udp prospector type. {pull}4452[4452]
- Enabled Cgo which means libc is dynamically compiled. {pull}4546[4546]
- Add Beta module config reloading mechanism {pull}4566[4566]
- Remove spooler and publisher components and settings. {pull}4644[4644]

*Heartbeat*

- Enabled Cgo which means libc is dynamically compiled. {pull}4546[4546]

*Metricbeat*

- Add random startup delay to each metricset to avoid the thundering herd problem. {issue}4010[4010]
- Add the ability to configure audit rules to the kernel module. {pull}4482[4482]
- Add the ability to configure kernel's audit failure mode. {pull}4516[4516]
- Add experimental Aerospike module. {pull}4560[4560]
- Vsphere module: collect custom fields from virtual machines. {issue}4464[4464]
- Add `test modules` command, to test modules expected output. {pull}4656[4656]
- Add `processors` setting to metricbeat modules. {pull}4699[4699]
- Support `npipe` protocol (Windows) in Docker module. {pull}4751[4751]

*Winlogbeat*

- Add the ability to use LevelRaw if Level isn't populated in the event XML. {pull}4257[4257]

*Auditbeat*

- Add file integrity metricset to the audit module. {pull}4486[4486]

[[release-notes-6.0.0-alpha2]]
=== Beats version 6.0.0-alpha2
https://github.com/elastic/beats/compare/v6.0.0-alpha1\...v6.0.0-alpha2[View commits]

==== Breaking changes

*Filebeat*

- Rename `input_type` field to `prospector.type` {pull}4294[4294]
- The `@metadata.type` field, added by the Logstash output, is now hardcoded to `doc` and will be removed in future versions. {pull}4331[4331].

==== Bugfixes

*Affecting all Beats*

- Fix importing the dashboards when the limit for max open files is too low. {issue}4244[4244]
- Fix configuration documentation for kubernetes processor {pull}4313[4313]
- Fix misspelling in `add_locale` configuration option for abbreviation.

*Filebeat*

- Fix race condition on harvester stopping with reloading enabled. {issue}3779[3779]
- Fix recursive glob config parsing and resolution across restarts. {pull}4269[4269]
- Allow string characters in user agent patch version (NGINX and Apache) {pull}4415[4415]
- Fix grok pattern in filebeat module system/auth without hostname. {pull}4224[4224]

*Metricbeat*

- Set correct format for percent fields in memory module. {pull}4619[4619]
- Fix a debug statement that said a module wrapper had stopped when it hadn't. {pull}4264[4264]
- Use MemAvailable value from /proc/meminfo on Linux 3.14. {pull}4316[4316]
- Fix panic when events were dropped by filters. {issue}4327[4327]
- Add filtering to system filesystem metricset to remove relative mountpoints like those
  from Linux network namespaces. {pull}4370[4370]
- Remove unnecessary print statement in schema apis. {pull}4355[4355]
- Fix type of field `haproxy.stat.check.health.last`. {issue}4407[4407]

*Packetbeat*
- Enable memcache filtering only if a port is specified in the config file. {issue}4335[4335]
- Enable memcache filtering only if a port is specified in the config file. {issue}4335[4335]

==== Added

*Affecting all Beats*

- Upgraded to Golang 1.8.3. {pull}4401[4401]
- Added the possibility to set Elasticsearch mapping template settings from the Beat configuration file. {pull}4284[4284] {pull}4317[4317]
- Add a variable to the SysV init scripts to make it easier to change the user. {pull}4340[4340]
- Add the option to write the generated Elasticsearch mapping template into a file. {pull}4323[4323]
- Add `instance_name` in GCE add_cloud_metadata processor. {pull}4414[4414]
- Add `add_docker_metadata` processor. {pull}4352[4352]
- Add `logging.files` `permissions` option. {pull}4295[4295]

*Filebeat*
- Added ability to sort harvested files. {pull}4374[4374]
- Add experimental Redis slow log prospector type. {pull}4180[4180]

*Metricbeat*

- Add macOS implementation of the system diskio metricset. {issue}4144[4144]
- Add process_summary metricset that records high level metrics about processes. {pull}4231[4231]
- Add `kube-state-metrics` based metrics to `kubernetes` module {pull}4253[4253]
- Add debug logging to Jolokia JMX metricset. {pull}4341[4341]
- Add events metricset for kubernetes metricbeat module {pull}4315[4315]
- Change Metricbeat default configuration file to be better optimized for most users. {pull}4329[4329]
- Add experimental RabbitMQ module. {pull}4394[4394]
- Add Kibana dashboard for the Kubernetes modules. {pull}4138[4138]

*Packetbeat*

*Winlogbeat*

==== Deprecated

*Affecting all Beats*

- The `@metadata.type` field, added by the Logstash output, is deprecated, hardcoded to `doc` and will be removed in future versions. {pull}4331[4331].

*Filebeat*

- Deprecate `input_type` prospector config. Use `type` config option instead. {pull}4294[4294]

==== Known Issue

- If the Elasticsearch output is not enabled, but `setup.template` options are
  present (like it's the case in the default Metricbeat configuration), the
  Beat stops with an error: "Template loading requested but the Elasticsearch
  output is not configured/enabled". To avoid this error, disable the template
  loading explicitly `setup.template.enabled: false`.

[[release-notes-6.0.0-alpha1]]
=== Beats version 6.0.0-alpha1
https://github.com/elastic/beats/compare/v5.4.0\...v6.0.0-alpha1[View commits]

==== Breaking changes

*Affecting all Beats*

- Introduce beat version in the Elasticsearch index and mapping template {pull}3527[3527]
- Usage of field `_type` is now ignored and hardcoded to `doc`. {pull}3757[3757]
- Change vendor manager from glide to govendor. {pull}3851[3851]
- Rename `error` field to `error.message`. {pull}3987[3987]
- Change `dashboards.*` config options to `setup.dashboards.*`. {pull}3921[3921]
- Change `outputs.elasticsearch.template.* to `setup.template.*` {pull}4080[4080]

*Filebeat*

- Remove code to convert states from 1.x. {pull}3767[3767]
- Remove deprecated config options `force_close_files` and `close_older`. {pull}3768[3768]
- Change `clean_removed` behaviour to also remove states for files which cannot be found anymore under the same name. {pull}3827[3827]
- Remove `document_type` config option. Use `fields` instead. {pull}4204[4204]
- Move `json_error` under `error.message` and `error.key`. {pull}4167[4167]

*Packetbeat*

- Remove deprecated `geoip`. {pull}3766[3766]
- Replace `waitstop` command line argument by `shutdown_timeout` in configuration file. {pull}3588[3588]

*Winlogbeat*

- Remove metrics endpoint. Replaced by http endpoint in libbeat (see #3717). {pull}3901[3901]

==== Bugfixes

*Affecting all Beats*

- Add `_id`, `_type`, `_index` and `_score` fields in the generated index pattern. {pull}3282[3282]

*Filebeat*

- Fix the Mysql slowlog parsing of IP addresses. {pull}4183[4183]
- Fix issue that new prospector was not reloaded on conflict {pull}4128[4128]

*Heartbeat*

- Use IP type of elasticsearch for ip field. {pull}3926[3926]

*Metricbeat*

- Support `common.Time` in `mapstriface.toTime()` {pull}3812[3812]
- Fix MongoDB `dbstats` fields mapping. {pull}4025[4025]
- Fixing prometheus collector to aggregate metrics based on metric family. {pull}4075[4075]
- Fixing multiEventFetch error reporting when no events are returned {pull}4153[4153]

==== Added

*Affecting all Beats*

- Initialize a beats UUID from file on startup. {pull}3615[3615]
- Add new `add_locale` processor to export the local timezone with an event. {pull}3902[3902]
- Add http endpoint. {pull}3717[3717]
- Updated to Go 1.8.1. {pull}4033[4033]
- Add kubernetes processor {pull}3888[3888]
- Add support for `include_labels` and `include_annotations` in kubernetes processor {pull}4043[4043]
- Support new `index_patterns` field when loading templates for Elasticsearch >= 6.0 {pull}4056[4056]
- Adding goimports support to make check and fmt {pull}4114[4114]
- Make kubernetes indexers/matchers pluggable {pull}4151[4151]
- Abstracting pod interface in kubernetes plugin to enable easier vendoring {pull}4152[4152]

*Filebeat*

- Restructure `input.Event` to be inline with `outputs.Data` {pull}3823[3823]
- Add base for supporting prospector level processors {pull}3853[3853]
- Add `filebeat.config.path` as replacement for `config_dir`. {pull}4051[4051]
- Add a `recursive_glob.enabled` setting to expand `**` in patterns. {pull}3980[3980]
- Add Icinga module. {pull}3904[3904]
- Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address.

*Heartbeat*

- Event format and field naming changes in Heartbeat and sample Dashboard. {pull}4091[4091]

*Metricbeat*

- Add experimental metricset `perfmon` to Windows module. {pull}3758[3758]
- Add memcached module with stats metricset. {pull}3693[3693]
- Add the `process.cmdline.cache.enabled` config option to the System Process Metricset. {pull}3891[3891]
- Add new MetricSet interfaces for developers (`Closer`, `ReportingFetcher`, and `PushMetricSet`). {pull}3908[3908]
- Add kubelet module {pull}3916[3916]
- Add dropwizard module {pull}4022[4022]
- Adding query APIs for metricsets and modules from metricbeat registry {pull}4102[4102]
- Fixing nil pointer on prometheus collector when http response is nil {pull}4119[4119]
- Add http module with json metricset. {pull}4092[4092]
- Add the option to the system module to include only the first top N processes by CPU and memory. {pull}4127[4127].
- Add experimental Vsphere module. {pull}4028[4028]
- Add experimental Elasticsearch module. {pull}3903[3903]
- Add experimental Kibana module. {pull}3895[3895]
- Move elasticsearch metricset node_stats under node.stats namespace. {pull}4142[4142]
- Make IP port indexer constructor public {pull}4434[4434]

*Packetbeat*

- Add `fields` and `fields_under_root` to Packetbeat protocols configurations. {pull}3518[3518]
- Add list style Packetbeat protocols configurations. This change supports specifying multiple configurations of the same protocol analyzer. {pull}3518[3518]

*Winlogbeat*

==== Deprecated

*Affecting all Beats*

- Usage of field `_type` is deprecated. It should not be used in queries or dashboards. {pull}3409[3409]

*Packetbeat*

- Deprecate dictionary style protocols configuration. {pull}3518[3518]

*Winlogbeat*

==== Known Issue

*Filebeat*

- Prospector reloading only works properly with new files. {pull}3546[3546]


[[release-notes-5.6.14]]
=== Beats version 5.6.14
https://github.com/elastic/beats/compare/v5.6.13\...v5.6.14[View commits]

No changes in this version.

[[release-notes-5.6.13]]
=== Beats version 5.6.13
https://github.com/elastic/beats/compare/v5.6.12\...v5.6.13[View commits]

No changes in this version.

[[release-notes-5.6.12]]
=== Beats version 5.6.12
https://github.com/elastic/beats/compare/v5.6.11\...v5.6.12[View commits]

No changes in this version.

[[release-notes-5.6.11]]
=== Beats version 5.6.11
https://github.com/elastic/beats/compare/v5.6.10\...v5.6.11[View commits]

No changes in this version.

[[release-notes-5.6.10]]
=== Beats version 5.6.10
https://github.com/elastic/beats/compare/v5.6.9\...v5.6.10[View commits]

==== Bugfixes

*Packetbeat*

- Fix an out of bounds access in HTTP parser caused by malformed request. {pull}6997[6997]

[[release-notes-5.6.9]]
=== Beats version 5.6.9
https://github.com/elastic/beats/compare/v5.6.8\...v5.6.9[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix a type issue when specifying certicate authority when using the `import_dashboards` command. {pull}6678[6678]

*Packetbeat*

- Fix http status phrase parsing not allow spaces. {pull}5312[5312]
- Fix http parse to allow to parse get request with space in the URI. {pull}5495[5495]
- Fix mysql SQL parser to trim `\r` from Windows Server `SELECT\r\n\t1`. {pull}5572[5572]
- Fix corruption when parsing repeated headers in an HTTP request or response. {pull}6325[6325]
- Fix panic when parsing partial AMQP messages. {pull}6384[6384]
- Fix out of bounds access to slice in MongoDB parser. {pull}6256[6256]
- Fix sniffer hanging on exit under Linux. {pull}6535[6535]
- Fix bounds check error in http parser causing a panic. {pull}6750[6750]
- HTTP parses successfully on empty status phrase. {issue}6176[6176]
- HTTP parser supports broken status line. {pull}6631[6631]


[[release-notes-5.6.8]]
=== Beats version 5.6.8
https://github.com/elastic/beats/compare/v5.6.7\...v5.6.8[View commits]

==== Bugfixes

*Winlogbeat*

- Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. {pull}6247[6247]


[[release-notes-5.6.7]]
=== Beats version 5.6.7
https://github.com/elastic/beats/compare/v5.6.6\...v5.6.7[View commits]

No changes in this release.


[[release-notes-5.6.6]]
=== Beats version 5.6.6
https://github.com/elastic/beats/compare/v5.6.5\...v5.6.6[View commits]

No changes in this release.


[[release-notes-5.6.5]]
=== Beats version 5.6.5
https://github.com/elastic/beats/compare/v5.6.4\...v5.6.5[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix duplicate batches of events in retry queue. {pull}5520[5520]

*Metricbeat*

- Clarify meaning of percentages reported by system core metricset. {pull}5565[5565]
- Fix map overwrite in docker diskio module. {issue}5582[5582]

[[release-notes-5.6.4]]
=== Beats version 5.6.4
https://github.com/elastic/beats/compare/v5.6.3\...v5.6.4[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix race condition in internal logging rotator. {pull}4519[4519]

*Packetbeat*

- Fix missing length check in the PostgreSQL module. {pull}5457[5457]

==== Added

*Affecting all Beats*

- Add support for enabling TLS renegotiation. {issue}4386[4386]
- Add setting to enable/disable the slow start in logstash output. {pull}5400[5400]

[[release-notes-5.6.3]]
=== Beats version 5.6.3
https://github.com/elastic/beats/compare/v5.6.2\...v5.6.3[View commits]

No changes in this release.

[[release-notes-5.6.2]]
=== Beats version 5.6.2
https://github.com/elastic/beats/compare/v5.6.1\...v5.6.2[View commits]

No changes in this release.

[[release-notes-5.6.1]]
=== Beats version 5.6.1
https://github.com/elastic/beats/compare/v5.6.0\...v5.6.1[View commits]

No changes in this release.

[[release-notes-5.6.0]]
=== Beats version 5.6.0
https://github.com/elastic/beats/compare/v5.5.3\...v5.6.0[View commits]

==== Breaking changes

*Affecting all Beats*

- The _all.norms setting in the Elasticsearch template is no longer disabled.
  This increases the storage size with one byte per document, but allows for a
  better upgrade experience to 6.0. {issue}4901[4901]


==== Bugfixes

*Filebeat*

- Fix issue where the `fileset.module` could have the wrong value. {issue}4761[4761]

*Packetbeat*

- Update flow timestamp on each packet being received. {issue}4895[4895]

*Metricbeat*

- Fix a debug statement that said a module wrapper had stopped when it hadn't. {pull}4264[4264]
- Use MemAvailable value from /proc/meminfo on Linux 3.14. {pull}4316[4316]
- Fix panic when events were dropped by filters. {issue}4327[4327]

==== Added

*Affecting all Beats*

- Add option to the import_dashboards script to load the dashboards via Kibana API. {pull}4682[4682]

*Filebeat*

- Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. {pull}4506[4506] {pull}4609[4609]
-  Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address. {pull}4351[4351]

*Metricbeat*

- Add `filesystem.ignore_types` to system module for ignoring filesystem types. {issue}4685[4685]

==== Deprecated

*Affecting all Beats*

- Loading more than one output is deprecated and will be removed in 6.0. {pull}4907[4907]

[[release-notes-5.5.3]]
=== Beats version 5.5.3
https://github.com/elastic/beats/compare/v5.5.2\...v5.5.3[View commits]

No changes in this release.

[[release-notes-5.5.2]]
=== Beats version 5.5.2
https://github.com/elastic/beats/compare/v5.5.1\...v5.5.2[View commits]

No changes in this release.
[[release-notes-5.5.1]]
=== Beats version 5.5.1
https://github.com/elastic/beats/compare/v5.5.0\...v5.5.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Normalize all times to UTC to ensure proper index naming. {issue}4569[4569]

[[release-notes-5.5.0]]
=== Beats version 5.5.0
https://github.com/elastic/beats/compare/v5.4.2\...v5.5.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Usage of field `_type` is now ignored and hardcoded to `doc`. {pull}3757[3757]

*Metricbeat*
- Change all `system.cpu.*.pct` metrics to be scaled by the number of CPU cores.
  This will make the CPU usage percentages from the system cpu metricset consistent
  with the system process metricset. The documentation for these metrics already
  stated that on multi-core systems the percentages could be greater than 100%. {pull}4544[4544]

==== Bugfixes

*Affecting all Beats*

- Fix console output. {pull}4045[4045]

*Filebeat*

- Allow string characters in user agent patch version (NGINX and Apache) {pull}4415[4415]

*Metricbeat*

- Fix type of field `haproxy.stat.check.health.last`. {issue}4407[4407]

*Packetbeat*

- Fix `packetbeat.interface` options that contain underscores (e.g. `with_vlans` or `bpf_filter`). {pull}4378[4378]
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. {pull}4442[4442]

==== Deprecated

*Filebeat*

- Deprecate `document_type` prospector config option as _type is removed in elasticsearch 6.0. Use fields instead. {pull}4225[4225]

*Winlogbeat*

- Deprecated metrics endpoint. It is superseded by a libbeat feature that can serve metrics on an HTTP endpoint. {pull}4145[4145]

[[release-notes-5.4.2]]
=== Beats version 5.4.2
https://github.com/elastic/beats/compare/v5.4.1\...v5.4.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Removed empty sections from the template files, causing indexing errors for array objects. {pull}4488[4488]

*Metricbeat*

- Fix issue affecting Windows services timing out at startup. {pull}4491[4491]
- Add filtering to system filesystem metricset to remove relative mountpoints like those
  from Linux network namespaces. {pull}4370[4370]

*Packetbeat*

- Clean configured geoip.paths before attempting to open the database. {pull}4306[4306]

[[release-notes-5.4.1]]
=== Beats version 5.4.1
https://github.com/elastic/beats/compare/v5.4.0\...v5.4.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix importing the dashboards when the limit for max open files is too low. {issue}4244[4244]
- Fix console output. {pull}4045[4045]

*Filebeat*

- Fix issue that new prospector was not reloaded on conflict. {pull}4128[4128]
- Fix grok pattern in filebeat module system/auth without hostname. {pull}4224[4224]
- Fix the Mysql slowlog parsing of IP addresses. {pull}4183[4183]

==== Added

*Affecting all Beats*

- Binaries upgraded to Go 1.7.6 which contains security fixes. {pull}4400[4400]

*Winlogbeat*

- Add the ability to use LevelRaw if Level isn't populated in the event XML. {pull}4257[4257]

[[release-notes-5.4.0]]
=== Beats version 5.4.0
https://github.com/elastic/beats/compare/v5.3.2\...v5.4.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Improve error message when downloading the dashboards fails. {pull}3805[3805]
- Fix potential Elasticsearch output URL parsing error if protocol scheme is missing. {pull}3671[3671]
- Downgrade Elasticsearch per batch item failure log to debug level. {issue}3953[3953]
- Make `@timestamp` accessible from format strings. {pull}3721[3721]

*Filebeat*

- Allow log lines without a program name in the Syslog fileset. {pull}3944[3944]
- Don't stop Filebeat when modules are used with the Logstash output. {pull}3929[3929]

*Metricbeat*

- Fixing panic on the Prometheus collector when label has a comma. {pull}3947[3947]
- Make system process metricset honor the `cpu_ticks` config option. {issue}3590[3590]

*Winlogbeat*

- Fix null terminators include in raw XML string when include_xml is enabled. {pull}3943[3943]

==== Added

*Affecting all Beats*

- Update index mappings to support future Elasticsearch 6.X. {pull}3778[3778]

*Filebeat*

- Add auditd module for reading audit logs on Linux. {pull}3750[3750] {pull}3941[3941]
- Add fileset for the Linux authorization logs. {pull}3669[3669]

*Heartbeat*

- Add default ports in HTTP monitor. {pull}3924[3924]

*Metricbeat*

- Add beta Jolokia module. {pull}3844[3844]
- Add dashboard for the MySQL module. {pull}3716[3716]
- Module configuration reloading is now beta instead of experimental. {pull}3841[3841]
- Marked http fields from the HAProxy module optional to improve compatibility with 1.5. {pull}3788[3788]
- Add support for custom HTTP headers and TLS for the Metricbeat modules. {pull}3945[3945]

*Packetbeat*

- Add DNS dashboard for an overview the DNS traffic. {pull}3883[3883]
- Add DNS Tunneling dashboard to highlight domains with large numbers of subdomains or high data volume. {pull}3884[3884]

[[release-notes-5.3.2]]
=== Beats version 5.3.2
https://github.com/elastic/beats/compare/v5.3.1\...v5.3.2[View commits]

==== Bugfixes

*Filebeat*

- Properly shut down crawler in case one prospector is misconfigured. {pull}4037[4037]
- Fix panic in JSON decoding code if the input line is "null". {pull}4042[4042]


[[release-notes-5.3.1]]
=== Beats version 5.3.1
https://github.com/elastic/beats/compare/v5.3.0\...v5.3.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix panic when testing regex-AST to match against date patterns. {issue}3889[3889]
- Fix panic due to race condition in kafka output. {pull}4098[4098]

*Filebeat*

- Fix modules default file permissions. {pull}3879[3879]
- Allow `-` in Apache access log byte count. {pull}3863[3863]

*Metricbeat*

- Avoid errors when some Apache status fields are missing. {issue}3074[3074]


[[release-notes-5.3.0]]
=== Beats version 5.3.0
https://github.com/elastic/beats/compare/v5.2.2\...v5.3.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Configuration files must be owned by the user running the Beat or by root, and they must not be writable by others. {pull}3544[3544] {pull}3689[3689]
- Change Beat generator. Use `$GOPATH/src/github.com/elastic/beats/script/generate.py` to generate a beat. {pull}3452[3452]

*Filebeat*

- Always use absolute path for event and registry. This can lead to issues when relative paths were used before. {pull}3328[3328]

*Metricbeat*

- Linux cgroup metrics are now enabled by default for the system process metricset. The configuration option for the feature was renamed from `cgroups` to `process.cgroups.enabled`. {pull}3519[3519]
- Change field names `couchbase.node.couch.*.actual_disk_size.*` to `couchbase.node.couch.*.disk_size.*` {pull}3545[3545]

==== Bugfixes

*Affecting all Beats*

- Add `_id`, `_type`, `_index` and `_score` fields in the generated index pattern. {pull}3282[3282]

*Filebeat*
- Always use absolute path for event and registry. {pull}3328[3328]
- Raise an exception in case there is a syntax error in one of the configuration files available under
  filebeat.config_dir. {pull}3573[3573]
- Fix empty registry file on machine crash. {issue}3537[3537]

*Metricbeat*

- Add error handling to system process metricset for when Linux cgroups are missing from the kernel. {pull}3692[3692]
- Add labels to the Docker healthcheck metricset output. {pull}3707[3707]

*Winlogbeat*

- Fix handling of empty strings in event_data. {pull}3705[3705]

==== Added

*Affecting all Beats*

- Files created by Beats (logs, registry, file output) will have 0600 permissions. {pull}3387[3387].
- RPM/deb packages will now install the config file with 0600 permissions. {pull}3382[3382]
- Add the option to pass custom HTTP headers to the Elasticsearch output. {pull}3400[3400]
- Unify `regexp` and `contains` conditionals, for both to support array of strings and convert numbers to strings if required. {pull}3469[3469]
- Add the option to load the sample dashboards during the Beat startup phase. {pull}3506[3506]
- Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. {pull}3528[3528]
- Using environment variables in the configuration file is now GA, instead of experimental. {pull}3525[3525]

*Filebeat*

- Add Filebeat modules for system, apache2, mysql, and nginx. {issue}3159[3159]
- Add the `pipeline` config option at the prospector level, for configuring the Ingest Node pipeline ID. {pull}3433[3433]
- Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. {pull}3469[3469]
- The `symlinks` and `harvester_limit` settings are now GA, instead of experimental. {pull}3525[3525]
- close_timeout is also applied when the output is blocking. {pull}3511[3511]
- Improve handling of different path variants on Windows. {pull}3781[3781]
- Add multiline.flush_pattern option, for specifying the 'end' of a multiline pattern {pull}4019[4019]

*Heartbeat*

- Add `tags`, `fields` and `fields_under_root` in monitors configuration. {pull}3623[3623]

*Metricbeat*

- Add experimental dbstats metricset to MongoDB module. {pull}3228[3228]
- Use persistent, direct connections to the configured nodes for MongoDB module. {pull}3228[3228]
- Add dynamic configuration reloading for modules. {pull}3281[3281]
- Add docker health metricset {pull}3357[3357]
- Add docker image metricset {pull}3467[3467]
- System module uses new matchers for white-listing processes. {pull}3469[3469]
- Add Beta CEPH module with health metricset. {pull}3311[3311]
- Add Beta php_fpm module with pool metricset. {pull}3415[3415]
- The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. {pull}3525[3525]
- The HAProxy module is now GA, instead of experimental. {pull}3525[3525]
- Add the ability to collect the environment variables from system processes. {pull}3337[3337]

==== Deprecated

*Affecting all Beats*

- Usage of field `_type` is deprecated. It should not be used in queries or dashboards. {pull}3409[3409]

*Filebeat*

- The experimental `publish_async` option is now deprecated and is planned to be removed in 6.0. {pull}3525[3525]


[[release-notes-5.2.2]]
=== Beats version 5.2.2
https://github.com/elastic/beats/compare/v5.2.1\...v5.2.2[View commits]

*Metricbeat*

- Fix bug docker module hanging when docker container killed. {issue}3610[3610]
- Set timeout to period instead of 1s by default as documented. {pull}3612[3612]

[[release-notes-5.2.1]]
=== Beats version 5.2.1
https://github.com/elastic/beats/compare/v5.2.0\...v5.2.1[View commits]

==== Bugfixes

*Metricbeat*

- Fix go routine leak in docker module. {pull}3492[3492]

*Packetbeat*

- Fix error in the NFS sample dashboard. {pull}3548[3548]

*Winlogbeat*

- Fix error in the Winlogbeat sample dashboard. {pull}3548[3548]

[[release-notes-5.2.0]]
=== Beats version 5.2.0
https://github.com/elastic/beats/compare/v5.1.2\...v5.2.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix overwriting explicit empty config sections. {issue}2918[2918]

*Filebeat*

- Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. {issue}3273[3273]

*Metricbeat*

- Fix service times-out at startup. {pull}3056[3056]
- Kafka module case sensitive host name matching. {pull}3193[3193]
- Fix interface conversion panic in couchbase module {pull}3272[3272]

*Packetbeat*

- Fix issue where some Cassandra visualizations were showing data from all protocols. {issue}3314[3314]

==== Added

*Affecting all Beats*

- Add support for passing list and dictionary settings via -E flag.
- Support for parsing list and dictionary setting from environment variables.
- Added new flags to import_dashboards (-cacert, -cert, -key, -insecure). {pull}3139[3139] {pull}3163[3163]
- The limit for the number of fields is increased via the mapping template. {pull}3275[3275]
- Updated to Go 1.7.4. {pull}3277[3277]
- Added a NOTICE file containing the notices and licenses of the dependencies. {pull}3334[3334].

*Heartbeat*

- First release, containing monitors for ICMP, TCP, and HTTP.

*Filebeat*

- Add enabled config option to prospectors. {pull}3157[3157]
- Add target option for decoded_json_field. {pull}3169[3169]

*Metricbeat*

- Kafka module broker matching enhancements. {pull}3129[3129]
- Add a couchbase module with metricsets for node, cluster and bucket. {pull}3081[3081]
- Export number of cores for CPU module. {pull}3192[3192]
- Experimental Prometheus module. {pull}3202[3202]
- Add system socket module that reports all TCP sockets. {pull}3246[3246]
- Kafka consumer groups metricset. {pull}3240[3240]
- Add jolokia module with dynamic jmx metricset. {pull}3570[3570]

*Winlogbeat*

- Reduced amount of memory allocated while reading event log records. {pull}3113[3113] {pull}3118[3118]

[[release-notes-5.1.2]]
=== Beats version 5.1.2
https://github.com/elastic/beats/compare/v5.1.1\...v5.1.2[View commits]

==== Bugfixes

*Filebeat*

- Fix registry migration issue from old states where files were only harvested after second restart. {pull}3322[3322]

*Packetbeat*

- Fix error on importing dashboards due to colons in the Cassandra dashboard. {issue}3140[3140]
- Fix error on importing dashboards due to the wrong type for the geo_point fields. {pull}3147[3147]

*Winlogbeat*

- Fix for "The array bounds are invalid" error when reading large events. {issue}3076[3076]

[[release-notes-5.1.1]]
=== Beats version 5.1.1
https://github.com/elastic/beats/compare/v5.0.2\...v5.1.1[View commits]

==== Breaking changes

*Metricbeat*

- Change data structure of experimental haproxy module. {pull}3003[3003]

*Filebeat*

- If a file is falling under `ignore_older` during startup, offset is now set to end of file instead of 0.
  With the previous logic the whole file was sent in case a line was added and it was inconsistent with
  files which were harvested previously. {pull}2907[2907]
- `tail_files` is now only applied on the first scan and not for all new files. {pull}2932[2932]

==== Bugfixes

*Affecting all Beats*

- Fix empty benign errors logged by processor actions. {pull}3046[3046]

*Metricbeat*

- Calculate the fsstat values per mounting point, and not filesystem. {pull}2777[2777]

==== Added

*Affecting all Beats*

- Add add_cloud_metadata processor for collecting cloud provider metadata. {pull}2728[2728]
- Added decode_json_fields processor for decoding fields containing JSON strings. {pull}2605[2605]
- Add Tencent Cloud provider for add_cloud_metadata processor. {pull}4023[4023]
- Add Alibaba Cloud provider for add_cloud_metadata processor. {pull}4111[4111]

*Metricbeat*

- Add experimental Docker module. Provided by Ingensi and @douaejeouit based on dockbeat.
- Add a sample Redis Kibana dashboard. {pull}2916[2916]
- Add support for MongoDB 3.4 and WiredTiger metrics. {pull}2999[2999]
- Add experimental kafka module with partition metricset. {pull}2969[2969]
- Add raw config option for mysql/status metricset. {pull}3001[3001]
- Add command fields for mysql/status metricset. {pull}3251[3251]

*Filebeat*

- Add command line option `-once` to run Filebeat only once and then close. {pull}2456[2456]
- Only load matching states into prospector to improve state handling {pull}2840[2840]
- Reset all states ttl on startup to make sure it is overwritten by new config {pull}2840[2840]
- Persist all states for files which fall under `ignore_older` to have consistent behaviour {pull}2859[2859]
- Improve shutdown behaviour with large number of files. {pull}3035[3035]

*Winlogbeat*

- Add `event_logs.batch_read_size` configuration option. {pull}2641[2641]

[[release-notes-5.1.0]]
=== Beats version 5.1.0 (skipped)

Version 5.1.0 doesn't exist because, for a short period of time, the Elastic
Yum and Apt repositories included unreleased binaries labeled 5.1.0. To avoid
confusion and upgrade issues for the people that have installed these without
realizing, we decided to skip the 5.1.0 version and release 5.1.1 instead.

[[release-notes-5.0.2]]
=== Beats version 5.0.2
https://github.com/elastic/beats/compare/v5.0.1\...v5.0.2[View commits]

==== Bugfixes

*Metricbeat*

- Fix the `password` option in the MongoDB module. {pull}2995[2995]


[[release-notes-5.0.1]]
=== Beats version 5.0.1
https://github.com/elastic/beats/compare/v5.0.0\...v5.0.1[View commits]

==== Bugfixes

*Metricbeat*

- Fix `system.process.start_time` on Windows. {pull}2848[2848]
- Fix `system.process.ppid` on Windows. {issue}2860[2860]
- Fix system process metricset for Windows XP and 2003. `cmdline` will be unavailable. {issue}1704[1704]
- Fix access denied issues in system process metricset by enabling SeDebugPrivilege on Windows. {issue}1897[1897]
- Fix system diskio metricset for Windows XP and 2003. {issue}2885[2885]

*Packetbeat*

- Fix 'index out of bounds' bug in Packetbeat DNS protocol plugin. {issue}2872[2872]

*Filebeat*

- Fix registry cleanup issue when files falling under ignore_older after restart. {issue}2818[2818]


==== Added

*Metricbeat*

- Add username and password config options to the PostgreSQL module. {pull}2889[2890]
- Add username and password config options to the MongoDB module. {pull}2889[2889]
- Add system core metricset for Windows. {pull}2883[2883]

*Packetbeat*

- Define `client_geoip.location` as geo_point in the mappings to be used by the GeoIP processor in the Ingest Node pipeline.
  {pull}2795[2795]

*Filebeat*

- Stop Filebeat on registrar loading error. {pull}2868[2868]


include::libbeat/docs/release-notes/5.0.0.asciidoc[]

[[release-notes-5.0.0-ga]]
=== Beats version 5.0.0-GA
https://github.com/elastic/beats/compare/v5.0.0-rc1\...v5.0.0[View commits]

The list below covers the changes between 5.0.0-rc1 and 5.0.0 GA only.

==== Bugfixes

*Affecting all Beats*

- Fix kafka output re-trying batches with too large events. {issue}2735[2735]
- Fix kafka output protocol error if `version: 0.10` is configured. {issue}2651[2651]
- Fix kafka output connection closed by broker on SASL/PLAIN. {issue}2717[2717]

*Metricbeat*

- Fix high CPU usage on macOS when encountering processes with long command lines. {issue}2747[2747]
- Fix high value of `system.memory.actual.free` and `system.memory.actual.used`. {issue}2653[2653]
- Change several `OpenProcess` calls on Windows to request the lowest possible access privilege.  {issue}1897[1897]
- Fix system.memory.actual.free high value on Windows. {issue}2653[2653]

*Filebeat*

- Fix issue when clean_removed and clean_inactive were used together that states were not directly removed from the registry.
- Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. {pull}2792[2792]

==== Added

*Affecting all Beats*

- Add beat.version fields to all events.

[[release-notes-5.0.0-rc1]]
=== Beats version 5.0.0-rc1
https://github.com/elastic/beats/compare/v5.0.0-beta1\...v5.0.0-rc1[View commits]

==== Breaking changes

*Affecting all Beats*

- A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. {pull}2688[2688]

==== Bugfixes

*Affecting all Beats*

- Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. {pull}2627[2627]
- Fix ignoring all fields from drop_fields in case the first field is unknown. {pull}2685[2685]
- Fix dynamic configuration int/uint to float type conversion. {pull}2698[2698]
- Fix primitive types conversion if values are read from environment variables. {pull}2698[2698]

*Metricbeat*

- Fix default configuration file on Windows to not enabled the `load` metricset. {pull}2632[2632]

*Packetbeat*

- Fix the `bpf_filter` setting. {issue}2660[2660]

*Filebeat*

- Fix input buffer on encoding problem. {pull}2416[2416]

==== Deprecated

*Affecting all Beats*

- Setting `port` has been deprecated in Redis and Logstash outputs. {pull}2620[2620]


[[release-notes-5.0.0-beta1]]
=== Beats version 5.0.0-beta1
https://github.com/elastic/beats/compare/v5.0.0-alpha5\...v5.0.0-beta1[View commits]

==== Breaking changes

*Affecting all Beats*

- Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. {pull}2119[2119]
- Replace `output.kafka.use_type` by `output.kafka.topic` accepting a format string. {pull}2188[2188]
- If the path specified by the `-c` flag is not absolute and `-path.config` is not specified, it
  is considered relative to the current working directory. {pull}2245[2245]
- rename `tls` configurations section to `ssl`. {pull}2330[2330]
- rename `certificate_key` configuration to `key`. {pull}2330[2330]
- replace `tls.insecure` with `ssl.verification_mode` setting. {pull}2330[2330]
- replace `tls.min/max_version` with `ssl.supported_protocols` setting requiring full protocol name. {pull}2330[2330]

*Metricbeat*

- Change field type system.process.cpu.start_time from keyword to date. {issue}1565[1565]
- redis/info metricset fields were renamed up according to the naming conventions.

*Packetbeat*

- Group HTTP fields under `http.request` and `http.response` {pull}2167[2167]
- Export `http.request.body` and `http.response.body` when configured under `include_body_for` {pull}2167[2167]
- Move `ignore_outgoing` config to `packetbeat.ignore_outgoing` {pull}2393[2393]

*Filebeat*

- Set close_inactive default to 5 minutes (was 1 hour before)
- Set clean_removed and close_removed to true by default

==== Bugfixes

*Affecting all Beats*

- Fix logstash output handles error twice when asynchronous sending fails. {pull}2441[2441]
- Fix Elasticsearch structured error response parsing error. {issue}2229[2229]
- Fixed the run script to allow the overriding of the configuration file. {issue}2171[2171]
- Fix logstash output crash if no hosts are configured. {issue}2325[2325]
- Fix array value support in -E CLI flag. {pull}2521[2521]
- Fix merging array values if -c CLI flag is used multiple times. {pull}2521[2521]
- Fix beats failing to start due to invalid duplicate key error in configuration file. {pull}2521[2521]
- Fix panic on non writable logging directory. {pull}2571[2571]

*Metricbeat*

- Fix module filters to work properly with drop_event filter. {issue}2249[2249]

*Packetbeat*

- Fix mapping for some Packetbeat flow metrics that were not marked as being longs. {issue}2177[2177]
- Fix handling of messages larger than the maximum message size (10MB). {pull}2470[2470]

*Filebeat*

- Fix processor failure in Filebeat when using regex, contain, or equals with the message field. {issue}2178[2178]
- Fix async publisher sending empty events {pull}2455[2455]
- Fix potential issue with multiple harvester per file on large file numbers or slow output {pull}2541[2541]

*Winlogbeat*

- Fix corrupt registry file that occurs on power loss by disabling file write caching. {issue}2313[2313]

==== Added

*Affecting all Beats*

- Add script to generate the Kibana index-pattern from fields.yml. {pull}2122[2122]
- Enhance Redis output key selection based on format string. {pull}2169[2169]
- Configurable Redis `keys` using filters and format strings. {pull}2169[2169]
- Add format string support to `output.kafka.topic`. {pull}2188[2188]
- Add `output.kafka.topics` for more advanced kafka topic selection per event. {pull}2188[2188]
- Add support for Kafka 0.10. {pull}2190[2190]
- Add SASL/PLAIN authentication support to kafka output. {pull}2190[2190]
- Make Kafka metadata update configurable. {pull}2190[2190]
- Add Kafka version setting (optional) enabling kafka broker version support. {pull}2190[2190]
- Add Kafka message timestamp if at least version 0.10 is configured. {pull}2190[2190]
- Add configurable Kafka event key setting. {pull}2284[2284]
- Add settings for configuring the kafka partitioning strategy. {pull}2284[2284]
- Add partitioner settings `reachable_only` to ignore partitions not reachable by network. {pull}2284[2284]
- Enhance contains condition to work on fields that are arrays of strings. {issue}2237[2237]
- Lookup the configuration file relative to the `-path.config` CLI flag. {pull}2245[2245]
- Re-write import_dashboards.sh in Golang. {pull}2155[2155]
- Update to Go 1.7. {pull}2306[2306]
- Log total non-zero internal metrics on shutdown. {pull}2349[2349]
- Add support for encrypted private key files by introducing `ssl.key_passphrase` setting. {pull}2330[2330]
- Add experimental symlink support with `symlinks` config {pull}2478[2478]
- Improve validation of registry file on startup.

*Metricbeat*

- Use the new scaled_float Elasticsearch type for the percentage values. {pull}2156[2156]
- Add experimental cgroup metrics to the system/process MetricSet. {pull}2184[2184]
- Added a PostgreSQL module. {pull}2253[2253]
- Improve mapping by converting half_float to scaled_float and integers to long. {pull}2430[2430]
- Add experimental haproxy module. {pull}2384[2384]
- Add Kibana dashboard for cgroups data {pull}2555[2555]

*Packetbeat*

- Add Cassandra protocol analyzer to Packetbeat. {pull}1959[1959]
- Match connections with IPv6 addresses to processes {pull}2254[2254]
- Add IP address to -devices command output {pull}2327[2327]
- Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. {pull}2470[2470]

*Filebeat*

- Introduce close_timeout harvester options {issue}1926[1926]
- Strip BOM from first message in case of BOM files {issue}2351[2351]
- Add harvester_limit option {pull}2417[2417]

==== Deprecated

*Affecting all Beats*

- Topology map is deprecated. This applies to the settings: refresh_topology_freq, topology_expire, save_topology, host_topology, password_topology, db_topology.


[[release-notes-5.0.0-alpha5]]
=== Beats version 5.0.0-alpha5
https://github.com/elastic/beats/compare/v5.0.0-alpha4\...v5.0.0-alpha5[View commits]

==== Breaking changes

*Affecting all Beats*

- Rename the `filters` section to `processors`. {pull}1944[1944]
- Introduce the condition with `when` in the processor configuration. {pull}1949[1949]
- The Elasticsearch template is now loaded by default. {pull}1993[1993]
- The Redis output `index` setting is renamed to `key`. `index` still works but it's deprecated. {pull}2077[2077]
- The undocumented file output `index` setting was removed. Use `filename` instead. {pull}2077[2077]

*Metricbeat*

- Create a separate metricSet for load under the system module and remove load information from CPU stats. {pull}2101[2101]
- Add `system.load.norm.1`, `system.load.norm.5` and `system.load.norm.15`. {pull}2101[2101]
- Add threads fields to mysql module. {pull}2484[2484]

*Packetbeat*

- Set `enabled` ` in `packetbeat.protocols.icmp` configuration to `true` by default. {pull}1988[1988]

==== Bugfixes

*Affecting all Beats*

- Fix sync publisher `PublishEvents` return value if client is closed concurrently. {pull}2046[2046]

*Metricbeat*

- Do not send zero values when no value was present in the source. {issue}1972[1972]

*Filebeat*

- Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. {issue}2041[2041]
- Fix open file handler issue. {issue}2028[2028] {pull}2020[2020]
- Fix filtering of JSON events when using integers in conditions. {issue}2038[2038]

*Winlogbeat*

- Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. {issue}2041[2041]

==== Added

*Affecting all Beats*

- Periodically log internal metrics. {pull}1955[1955]
- Add enabled setting to all output modules. {pull}1987[1987]
- Command line flag `-c` can be used multiple times. {pull}1985[1985]
- Add OR/AND/NOT to the condition associated with the processors. {pull}1983[1983]
- Add `-E` CLI flag for overwriting single config options via command line. {pull}1986[1986]
- Choose the mapping template file based on the Elasticsearch version. {pull}1993[1993]
- Check stdout being available when console output is configured. {issue}2035[2035]

*Metricbeat*

- Add pgid field to process information. {pull} 2021[2021]

*Packetbeat*

- Add enabled setting to Packetbeat protocols. {pull}1988[1988]
- Add enabled setting to Packetbeat network flows configuration. {pull}1988[1988]

*Filebeat*

- Introduce `close_removed` and `close_renamed` harvester options. {issue}1600[1600]
- Introduce `close_eof` harvester option. {issue}1600[1600]
- Add `clean_removed` and `clean_inactive` config option. {issue}1600[1600]

==== Deprecated

*Filebeat*

- Deprecate `close_older` option and replace it with `close_inactive`. {issue}2051[2051]
- Deprecate `force_close_files` option and replace it with `close_removed` and `close_renamed`. {issue}1600[1600]

[[release-notes-5.0.0-alpha4]]
=== Beats version 5.0.0-alpha4
https://github.com/elastic/beats/compare/v5.0.0-alpha3\...v5.0.0-alpha4[View commits]

==== Breaking changes

*Affecting all Beats*

- The topology_expire option of the Elasticsearch output was removed. {pull}1907[1907]

*Filebeat*

- Stop following symlink. Symlinks are now ignored: {pull}1686[1686]

==== Bugfixes

*Affecting all Beats*

- Reset backoff factor on partial ACK. {issue}1803[1803]
- Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. {issue}1829[1829]
- Fix logstash output with pipelining mode enabled not reconnecting. {issue}1876[1876]
- Empty configuration sections become merge-able with variables containing full path. {pull}1900[1900]
- Fix error message about required fields missing not printing the missing field name. {pull}1900[1900]

*Metricbeat*

- Fix the CPU values returned for each core. {issue}1863[1863]

*Packetbeat*

- Add missing nil-check to memcached GapInStream handler. {issue}1162[1162]
- Fix NFSv4 Operation returning the first found first-class operation available in compound requests. {pull}1821[1821]
- Fix TCP overlapping segments not being handled correctly. {pull}1898[1898]

*Winlogbeat*

- Fix issue with rendering forwarded event log records. {pull}1891[1891]

==== Added

*Affecting all Beats*

- Improve error message if compiling regular expression from config files fails. {pull}1900[1900]
- Compression support in the Elasticsearch output. {pull}1835[1835]

*Metricbeat*

- Add MongoDB module. {pull}1837[1837]


[[release-notes-5.0.0-alpha3]]
=== Beats version 5.0.0-alpha3
https://github.com/elastic/beats/compare/v5.0.0-alpha2\...v5.0.0-alpha3[View commits]

==== Breaking changes

*Affecting all Beats*

- All configuration settings under `shipper:` are moved to be top level configuration settings. I.e.
  `shipper.name:` becomes `name:` in the configuration file. {pull}1570[1570]

*Topbeat*

- Topbeat is replaced by Metricbeat.

*Filebeat*

- The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled.

==== Bugfixes

*Winlogbeat*

- Adding missing argument to the "Stop processing" log message. {pull}1590[1590]

==== Added

*Affecting all Beats*

- Add conditions to generic filtering. {pull}1623[1623]

*Metricbeat*

- First public release, containing the following modules: apache, mysql, nginx, redis, system, and zookeeper.

*Filebeat*

- The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. {pull}1703[1703]

==== Deprecated

*Affecting all Beats*

- The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. {pull}1601[1601]


[[release-notes-5.0.0-alpha2]]
=== Beats version 5.0.0-alpha2
https://github.com/elastic/beats/compare/v5.0.0-alpha1\...v5.0.0-alpha2[View commits]

==== Breaking changes

*Affecting all Beats*

- On DEB/RPM installations, the binary files are now found under `/usr/share/{{beat_name}}/bin`, not in `/usr/bin`. {pull}1385[1385]
- The logs are written by default to self rotating files, instead of syslog. {pull}1371[1371]
- Remove deprecated `host` option from elasticsearch, logstash and redis outputs. {pull}1474[1474]

*Packetbeat*

- Configuration of redis topology support changed. {pull}1353[1353]
- Move all Packetbeat configuration options under the packetbeat namespace {issue}1417[1417]

*Filebeat*

- Default location for the registry file was changed to be `data/registry` from the binary directory,
  rather than `.filebeat` in the current working directory. This affects installations for zip/tar.gz/source,
  the location for DEB and RPM packages stays the same. {pull}1373[1373]

==== Bugfixes

*Affecting all Beats*

- Drain response buffers when pipelining is used by Redis output. {pull}1353[1353]
- Unterminated environment variable expressions in config files will now cause an error {pull}1389[1389]
- Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. {issue}1321[1321]
- Fix bug affecting -cpuprofile, -memprofile, and -httpprof CLI flags {pull}1415[1415]
- Fix race when multiple outputs access the same event with logstash output manipulating event {issue}1410[1410] {pull}1428[1428]
- Seed random number generator using crypto.rand package. {pull}1503{1503]
- Fix beats hanging in -configtest {issue}1213[1213]
- Fix kafka log message output {pull}1516[1516]

*Filebeat*

- Improvements in registrar dealing with file rotation. {pull}1281[1281]
- Fix issue with JSON decoding where `@timestamp` or `type` keys with the wrong type could cause Filebeat
  to crash. {issue}1378[1378]
- Fix issue with JSON decoding where values having `null` as values could crash Filebeat. {issue}1466[1466]
- Multiline reader normalizing newline to use `\n`. {pull}1552[1552]

*Winlogbeat*

- Fix panic when reading messages larger than 32K characters on Windows XP and 2003. {pull}1498[1498]
- Fix panic that occurs when reading a large events on Windows Vista and newer. {pull}1499[1499]

==== Added

*Affecting all Beats*

- Add support for TLS to Redis output. {pull}1353[1353]
- Add SOCKS5 proxy support to Redis output. {pull}1353[1353]
- Failover and load balancing support in redis output. {pull}1353[1353]
- Multiple-worker per host support for redis output. {pull}1353[1353]
- Added ability to escape `${x}` in config files to avoid environment variable expansion {pull}1389[1389]
- Configuration options and CLI flags for setting the home, data and config paths. {pull}1373[1373]
- Configuration options and CLI flags for setting the default logs path. {pull}1437[1437]
- Update to Go 1.6.2 {pull}1447[1447]
- Add Elasticsearch template files compatible with Elasticsearch 2.x. {pull}1501[1501]
- Add scripts for managing the dashboards of a single Beat {pull}1359[1359]

*Packetbeat*

- Fix compile issues for OpenBSD. {pull}1347[1347]

*Topbeat*

- Updated elastic/gosigar version so Topbeat can compile on OpenBSD. {pull}1403[1403]


[[release-notes-5.0.0-alpha1]]
=== Beats version 5.0.0-alpha1
https://github.com/elastic/beats/compare/v1.2.0\...v5.0.0-alpha1[View commits]

==== Breaking changes

*libbeat*

- Run function to start a Beat now returns an error instead of directly exiting. {pull}771[771]
- The method signature of HandleFlags() was changed to allow returning an error {pull}1249[1249]
- Require braces for environment variable expansion in config files {pull}1304[1304]

*Packetbeat*

- Rename output fields in the dns package. Former flag `recursion_allowed` becomes `recursion_available`. {pull}803[803]
  Former SOA field `ttl` becomes `minimum`. {pull}803[803]
- The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. {pull}803[803]
- Remove the count field from the exported event {pull}1210[1210]

*Topbeat*

- Rename `proc.cpu.user_p` with `proc.cpu.total_p` as it includes CPU time spent in kernel space {pull}631[631]
- Remove `count` field from the exported fields {pull}1207[1207]
- Rename `input` top level config option to `topbeat`

*Filebeat*

- Scalar values in used in the `fields` configuration setting are no longer automatically converted to strings. {pull}1092[1092]
- Count field was removed from event as not used in filebeat {issue}778[778]

*Winlogbeat*

- The `message_inserts` field was replaced with the `event_data` field {issue}1053[1053]
- The `category` field was renamed to `task` to better align with the Windows Event Log API naming {issue}1053[1053]
- Remove the count field from the exported event {pull}1218[1218]


==== Bugfixes

*Affecting all Beats*

- Logstash output will not retry events that are not JSON-encodable {pull}927[927]

*Packetbeat*

- Create a proper BPF filter when ICMP is the only enabled protocol {issue}757[757]
- Check column length in pgsql parser. {issue}565[565]
- Harden pgsql parser. {issue}565[565]

*Topbeat*

- Fix issue with `cpu.system_p` being greater than 1 on Windows {pull}1128[1128]

*Filebeat*

- Stop filebeat if started without any prospectors defined or empty prospectors {pull}644[644] {pull}647[647]
- Improve shutdown of crawler and prospector to wait for clean completion {pull}720[720]
- Omit `fields` from Filebeat events when null {issue}899[899]

*Winlogbeat*

==== Added

*Affecting all Beats*

- Update builds to Golang version 1.6
- Add option to Elasticsearch output to pass http parameters in index operations {issue}805[805]
- Improve Logstash and Elasticsearch backoff behavior. {pull}927[927]
- Add experimental Kafka output. {pull}942[942]
- Add config file option to configure GOMAXPROCS. {pull}969[969]
- Improve shutdown handling in libbeat. {pull}1075[1075]
- Add `fields` and `fields_under_root` options under the `shipper` configuration {pull}1092[1092]
- Add the ability to use a SOCKS5 proxy with the Logstash output {issue}823[823]
- The `-configtest` flag will now print "Config OK" to stdout on success {pull}1249[1249]

*Packetbeat*

- Change the DNS library used throughout the dns package to github.com/miekg/dns. {pull}803[803]
- Add support for NFS v3 and v4. {pull}1231[1231]
- Add support for EDNS and DNSSEC. {pull}1292[1292]

*Topbeat*

- Add `username` to processes {pull}845[845]

*Filebeat*

- Add the ability to set a list of tags for each prospector {pull}1092[1092]
- Add JSON decoding support {pull}1143[1143]


*Winlogbeat*

- Add caching of event metadata handles and the system render context for the wineventlog API {pull}888[888]
- Improve config validation by checking for unknown top-level YAML keys. {pull}1100[1100]
- Add the ability to set tags, fields, and fields_under_root as options for each event log {pull}1092[1092]
- Add additional data to the events published by Winlogbeat. The new fields are `activity_id`,
`event_data`, `keywords`, `opcode`, `process_id`, `provider_guid`, `related_activity_id`,
`task`, `thread_id`, `user_data`, and `version`. {issue}1053[1053]
- Add `event_id`, `level`, and `provider` configuration options for filtering events {pull}1218[1218]
- Add `include_xml` configuration option for including the raw XML with the event {pull}1218[1218]

==== Known issues
* All Beats can hang or panic on shutdown if the next server in the pipeline (e.g. Elasticsearch or Logstash) is
  not reachable. {issue}1319[1319]
* When running the Beats as a service on Windows, you need to manually load the Elasticsearch mapping
  template. {issue}1315[1315]
* The ES template automatic load doesn't work if Elasticsearch is not available when the Beat is starting. {issue}1321[1321]

[[release-notes-1.3.1]]
=== Beats version 1.3.1
https://github.com/elastic/beats/compare/v1.3.0\...v1.3.1[View commits]

==== Bugfixes

*Filebeat*

- Fix a concurrent bug on filebeat startup with a large number of prospectors defined. {pull}2509[2509]

*Packetbeat*

- Fix description for the -I CLI flag. {pull}2480[2480]

*Winlogbeat*

- Fix corrupt registry file that occurs on power loss by disabling file write caching. {issue}2313[2313]

[[release-notes-1.3.0]]
=== Beats version 1.3.0
https://github.com/elastic/beats/compare/v1.2.3\...v1.3.0[View commits]

==== Deprecated

*Filebeat*

- Undocumented support for following symlinks is deprecated. Filebeat will not follow symlinks in version 5.0. {pull}1767[1767]

==== Bugfixes

*Affecting all Beats*

- Fix beats load balancer deadlock if `max_retries: -1` or `publish_async` is enabled in filebeat. {issue}1829[1829]
- Fix output modes backoff counter reset. {issue}1803[1803] {pull}1814[1814] {pull}1818[1818]
- Set logstash output default bulk_max_size to 2048. {issue}1662[1662]
- Seed random number generator using crypto.rand package. {pull}1503[1503]
- Check stdout being available when console output is configured. {issue}2063[2063]

*Packetbeat*

- Add missing nil-check to memcached GapInStream handler. {issue}1162[1162]
- Fix NFSv4 Operation returning the first found first-class operation available in compound requests. {pull}1821[1821]
- Fix TCP overlapping segments not being handled correctly. {pull}1917[1917]

==== Added

*Affecting all Beats*

- Updated to Go 1.7


[[release-notes-1.2.3]]
=== Beats version 1.2.3
https://github.com/elastic/beats/compare/v1.2.2\...v1.2.3[View commits]

==== Bugfixes

*Topbeat*

- Fix high CPU usage when using filtering under Windows. {pull}1598[1598]

*Filebeat*

- Fix rotation issue with ignore_older. {issue}1528[1528]

*Winlogbeat*

- Fix panic when reading messages larger than 32K characters on Windows XP and 2003. {pull}1498[1498]

==== Added

*Filebeat*

- Prevent file opening for files which reached ignore_older. {pull}1649[1649]


[[release-notes-1.2.2]]
=== Beats version 1.2.2
https://github.com/elastic/beats/compare/v1.2.0\...v1.2.2[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix race when multiple outputs access the same event with Logstash output manipulating event. {issue}1410[1410]
- Fix go-daemon (supervisor used in init scripts) hanging when executed over SSH. {issue}1394[1394]

*Filebeat*

- Improvements in registrar dealing with file rotation. {issue}1281[1281]


[[release-notes-1.2.1]]
=== Beats version 1.2.1
https://github.com/elastic/beats/compare/v1.2.0\...v1.2.1[View commits]

==== Breaking changes

*Affecting all Beats*

- Require braces for environment variable expansion in config files {pull}1304[1304]
- Removed deprecation warning for the Redis output. {pull}1282[1282]

*Topbeat*

- Fixed name of the setting `stats.proc` to `stats.process` in the default configuration file. {pull}1343[1343]
- Fix issue with cpu.system_p being greater than 1 on Windows {pull}1128[1128]

==== Added

*Topbeat*

- Add username to processes {pull}845[845]


[[release-notes-1.2.0]]
=== Beats version 1.2.0
https://github.com/elastic/beats/compare/v1.1.2\...v1.2.0[View commits]

==== Breaking changes

*Filebeat*

- Default config for ignore_older is now infinite instead of 24h, means ignore_older is disabled by default. Use close_older to only close file handlers.

==== Bugfixes

*Packetbeat*

- Split real_ip_header value when it contains multiple IPs {pull}1241[1241]

*Winlogbeat*

- Fix invalid `event_id` on Windows XP and Windows 2003 {pull}1227[1227]

==== Added

*Affecting all Beats*

- Add ability to override configuration settings using environment variables {issue}114[114]
- Libbeat now always exits through a single exit method for proper cleanup and control {pull}736[736]
- Add ability to create Elasticsearch mapping on startup {pull}639[639]

*Topbeat*

- Add the command line used to start processes {issue}533[533]

*Filebeat*

- Add close_older configuration option to complete ignore_older https://github.com/elastic/filebeat/issues/181[181]

[[release-notes-1.1.2]]
=== Beats version 1.1.2
https://github.com/elastic/beats/compare/v1.1.1\...v1.1.2[View commits]

==== Bugfixes

*Filebeat*

- Fix registrar bug for rotated files {pull}1010[1010]


[[release-notes-1.1.1]]
=== Beats version 1.1.1
https://github.com/elastic/beats/compare/v1.1.0\...v1.1.1[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix logstash output loop hanging in infinite loop on too many output errors. {pull}944[944]
- Fix critical bug in filebeat and winlogbeat potentially dropping events. {pull}953[953]

[[release-notes-1.1.0]]
=== Beats version 1.1.0
https://github.com/elastic/beats/compare/v1.0.1\...v1.1.0[View commits]

==== Bugfixes

*Affecting all Beats*

- Fix logging issue with file based output where newlines could be misplaced
  during concurrent logging {pull}650[650]
- Reduce memory usage by separate queue sizes for single events and bulk events. {pull}649[649] {issue}516[516]
- Set default default bulk_max_size value to 2048 {pull}628[628]

*Packetbeat*

- Fix setting direction to out and use its value to decide when dropping events if ignore_outgoing is enabled {pull}557[557]
- Fix logging issue with file-based output where newlines could be misplaced
  during concurrent logging {pull}650[650]
- Reduce memory usage by having separate queue sizes for single events and bulk events. {pull}649[649] {issue}516[516]
- Set default bulk_max_size value to 2048 {pull}628[628]
- Fix logstash window size of 1 not increasing. {pull}598[598]

*Packetbeat*

- Fix the condition that determines whether the direction of the transaction is set to "outgoing". Packetbeat uses the
  direction field to determine which transactions to drop when dropping outgoing transactions. {pull}557[557]
- Allow PF_RING sniffer type to be configured using pf_ring or pfring {pull}671[671]

*Filebeat*

- Set spool_size default value to 2048 {pull}628[628]

==== Added

*Affecting all Beats*

- Add include_fields and drop_fields as part of generic filtering {pull}1120[1120]
- Make logstash output compression level configurable. {pull}630[630]
- Some publisher options refactoring in libbeat {pull}684[684]
- Move event preprocessor applying GeoIP to packetbeat {pull}772[772]

*Packetbeat*

- Add support for capturing DNS over TCP network traffic. {pull}486[486] {pull}554[554]

*Topbeat*

- Group all CPU usage per core statistics and export them optionally if cpu_per_core is configured {pull}496[496]

*Filebeat*

- Add multiline support for combining multiple related lines into one event. {issue}461[461]
- Add `exclude_lines` and `include_lines` options for regexp based line filtering. {pull}430[430]
- Add `exclude_files` configuration option. {pull}563[563]
- Add experimental option to enable filebeat publisher pipeline to operate asynchronously {pull}782[782]

*Winlogbeat*

- First public release of Winlogbeat

[[release-notes-1.0.1]]
=== Beats version 1.0.1
https://github.com/elastic/beats/compare/v1.0.0\...v1.0.1[Check 1.0.1 diff]

==== Bugfixes

*Filebeat*

- Fix force_close_files in case renamed file appeared very fast. https://github.com/elastic/filebeat/pull/302[302]

*Packetbeat*

- Improve MongoDB message correlation. {issue}377[377]
- Improve redis parser performance. {issue}442[422]
- Fix panic on nil in redis protocol parser. {issue}384[384]
- Fix errors redis parser when messages are split in multiple TCP segments. {issue}402[402]
- Fix errors in redis parser when length prefixed strings contain sequences of CRLF. {issue}402[402]
- Fix errors in redis parser when dealing with nested arrays. {issue}402[402]

[[release-notes-1.0.0]]
=== Beats version 1.0.0
https://github.com/elastic/beats/compare/1.0.0-rc2\...1.0.0[Check 1.0.0 diff]

==== Breaking changes

*Topbeat*

- Change proc type to process #138


==== Bugfixes

*Affecting all Beats*

- Fix random panic on shutdown by calling shutdown handler only once. elastic/filebeat#204
- Fix credentials are not send when pinging an elasticsearch host. elastic/filebeat#287

*Filebeat*

- Fix problem that harvesters stopped reading after some time and filebeat stopped processing events #257
- Fix line truncating by internal buffers being reused by accident #258
- Set default ignore_older to 24 hours #282




[[release-notes-1.0.0-rc2]]
=== Beats version 1.0.0-rc2
https://github.com/elastic/beats/compare/1.0.0-rc1\...1.0.0-rc2[Check 1.0.0-rc2
diff]

==== Breaking changes

*Affecting all Beats*

- The `shipper` output field is renamed to `beat.name`. #285
- Use of `enabled` as a configuration option for outputs (elasticsearch,
  logstash, etc.) has been removed. #264
- Use of `disabled` as a configuration option for tls has been removed. #264
- The `-test` command line flag was renamed to `-configtest`. #264
- Disable geoip by default. To enable it uncomment in config file. #305


*Filebeat*

- Removed utf-16be-bom encoding support. Support will be added with fix for #205
- Rename force_close_windows_files to force_close_files and make it available for all platforms.


==== Bugfixes

*Affecting all Beats*

- Disable logging to stderr after configuration phase. #276
- Set the default file logging path when not set in config. #275
- Fix bug silently dropping records based on current window size. elastic/filebeat#226
- Fix direction field in published events. #300
- Fix elasticsearch structured errors breaking error handling. #309

*Packetbeat*

- Packetbeat will now exit if a configuration error is detected. #357
- Fixed an issue handling DNS requests containing no questions. #369

*Topbeat*

- Fix leak of Windows handles. #98
- Fix memory leak of process information. #104

*Filebeat*

- Filebeat will now exit if a configuration error is detected. #198
- Fix to enable prospector to harvest existing files that are modified. #199
- Improve line reading and encoding to better keep track of file offsets based
  on encoding. #224
- Set input_type by default to "log"


==== Added

*Affecting all Beats*

- Added `beat.hostname` to contain the hostname where the Beat is running on as
  returned by the operating system. #285
- Added timestamp for file logging. #291

*Filebeat*

- Handling end of line under windows was improved #233



[[release-notes-1.0.0-rc1]]
=== Beats version 1.0.0-rc1
https://github.com/elastic/beats/compare/1.0.0-beta4\...1.0.0-rc1[Check
1.0.0-rc1 diff]

==== Breaking changes

*Affecting all Beats*

- Rename timestamp field with @timestamp. #237

*Packetbeat*

- Rename timestamp field with @timestamp. #343

*Topbeat*

- Rename timestamp field with @timestamp for a better integration with
Logstash. #80

*Filebeat*

- Rename the timestamp field with @timestamp #168
- Rename tail_on_rotate prospector config to tail_files
- Removal of line field in event. Line number was not correct and does not add value. #217


==== Bugfixes

*Affecting all Beats*

- Use stderr for console log output. #219
- Handle empty event array in publisher. #207
- Respect '*' debug selector in IsDebug. #226 (elastic/packetbeat#339)
- Limit number of workers for Elasticsearch output. elastic/packetbeat#226
- On Windows, remove service related error message when running in the console. #242
- Fix waitRetry no configured in single output mode configuration. elastic/filebeat#144
- Use http as the default scheme in the elasticsearch hosts #253
- Respect max bulk size if bulk publisher (collector) is disabled or sync flag is set.
- Always evaluate status code from Elasticsearch responses when indexing events. #192
- Use bulk_max_size configuration option instead of bulk_size. #256
- Fix max_retries=0 (no retries) configuration option. #266
- Filename used for file based logging now defaults to beat name. #267

*Packetbeat*

- Close file descriptors used to monitor processes. #337
- Remove old RPM spec file. It moved to elastic/beats-packer. #334

*Topbeat*

- Don't wait for one period until shutdown #75

*Filebeat*

- Omit 'fields' from event JSON when null. #126
- Make offset and line value of type long in elasticsearch template to prevent overflow. #140
- Fix locking files for writing behaviour. #156
- Introduce 'document_type' config option per prospector to define document type
  for event stored in elasticsearch. #133
- Add 'input_type' field to published events reporting the prospector type being used. #133
- Fix high CPU usage when not connected to Elasticsearch or Logstash. #144
- Fix issue that files were not crawled anymore when encoding was set to something other then plain. #182


==== Added

*Affecting all Beats*

- Add Console output plugin. #218
- Add timestamp to log messages #245
- Send @metadata.beat to Logstash instead of @metadata.index to prevent
  possible name clashes and give user full control over index name used for
  Elasticsearch
- Add logging messages for bulk publishing in case of error #229
- Add option to configure number of parallel workers publishing to Elasticsearch
  or Logstash.
- Set default bulk size for Elasticsearch output to 50.
- Set default http timeout for Elasticsearch to 90s.
- Improve publish retry if sync flag is set by retrying only up to max bulk size
  events instead of all events to be published.

*Filebeat*

- Introduction of backoff, backoff_factor, max_backoff, partial_line_waiting, force_close_windows_files
  config variables to make crawling more configurable.
- All Godeps dependencies were updated to master on 2015-10-21 [#122]
- Set default value for ignore_older config to 10 minutes. #164
- Added the fields_under_root setting to optionally store the custom fields top
level in the output dictionary. #188
- Add more encodings by using x/text/encodings/htmlindex package to select
  encoding by name.




[[release-notes-1.0.0-beta4]]
=== Beats version 1.0.0-beta4
https://github.com/elastic/beats/compare/1.0.0-beta3\...1.0.0-beta4[Check
1.0.0-beta4 diff]


==== Breaking changes

*Affecting all Beats*

- Update tls config options naming from dash to underline #162
- Feature/output modes: Introduction of PublishEvent(s) to be used by beats #118 #115

*Packetbeat*

- Renamed http module config file option 'strip_authorization' to 'redact_authorization'
- Save_topology is set to false by default
- Rename elasticsearch index to [packetbeat-]YYYY.MM.DD

*Topbeat*

- Percentage fields (e.g user_p) are exported as a float between 0 and 1 #34


==== Bugfixes

*Affecting all Beats*

- Determine Elasticsearch index for an event based on UTC time #81
- Fixing ES output's defaultDeadTimeout so that it is 60 seconds #103
- ES outputer: fix timestamp conversion #91
- Fix TLS insecure config option #239
- ES outputer: check bulk API per item status code for retransmit on failure.

*Packetbeat*

- Support for lower-case header names when redacting http authorization headers
- Redact proxy-authorization if redact-authorization is set
- Fix some multithreading issues #203
- Fix negative response time #216
- Fix memcache TCP connection being nil after dropping stream data. #299
- Add missing DNS protocol configuration to documentation #269

*Topbeat*

- Don't divide the reported memory by an extra 1024 #60


==== Added

*Affecting all Beats*

- Add logstash output plugin #151
- Integration tests for Beat -> Logstash -> Elasticsearch added #195 #188 #168 #137 #128 #112
- Large updates and improvements to the documentation
- Add direction field to publisher output to indicate inbound/outbound transactions #150
- Add tls configuration support to elasticsearch and logstash outputers #139
- All external dependencies were updated to the latest version. Update to Golang 1.5.1 #162
- Guarantee ES index is based in UTC time zone #164
- Cache: optional per element timeout #144
- Make it possible to set hosts in different ways. #135
- Expose more TLS config options #124
- Use the Beat name in the default configuration file path #99

*Packetbeat*

- add [.editorconfig file](http://editorconfig.org/)
- add (experimental/unsupported?) saltstack files
- Sample config file cleanup
- Moved common documentation to [libbeat repository](https://github.com/elastic/libbeat)
- Update build to go 1.5.1
- Adding device descriptions to the -device output.
- Generate coverage for system tests
- Move go-daemon dependency to beats-packer
- Rename integration tests to system tests
- Made the `-devices` option more user friendly in case `sudo` is not used.
  Issue #296.
- Publish expired DNS transactions #301
- Update protocol guide to libbeat changes
- Add protocol registration to new protocol guide
- Make transaction timeouts configurable #300
- Add direction field to the exported fields #317

*Topbeat*

- Document fields in a standardized format (etc/fields.yml) #34
- Updated to use new libbeat Publisher #37 #41
- Update to go 1.5.1 #43
- Updated configuration files with comments for all options #65
- Documentation improvements


==== Deprecated

*Affecting all Beats*

- Redis output was deprecated #169 #145
- Host and port configuration options are deprecated. They are replaced by the hosts
 configuration option. #141