Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x](backport #41667) filebeat/inputs/filestream: add metric for messages truncated #42185

Merged
merged 2 commits into from
Dec 31, 2024

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Dec 31, 2024

Proposed commit message

While investigating an SDH, I noticed that although we add the truncated label to log fields, there is no feedback on the amount of messages that are truncated. It often happens that almost all messages are truncated because of a misconfigured input.

This PR adds a input metric that counts the total number of truncated messages.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Create a log file with multiline logs:

cat <<EOF > /tmp/in.log
[beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index]
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver\$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77)
    at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75)
EOF

Configure filebeat to parse the multiline logs with max_lines:

filebeat.inputs:
- type: filestream
  id: foo-bar
  enabled: true
  paths:
    - "/tmp/in.log"
  parsers:
    - multiline:
        type: pattern
        pattern: '^\['
        negate: true
        match: after
        max_lines: 3

output.discard:
  enabled: true
http.enabled: true

Check the metrics:

$ curl -XGET 'localhost:5066/inputs/' | jq
[
  {
    "bytes_processed_total": 596,
    "events_processed_total": 1,
    "files_active": 1,
    "files_closed_total": 0,
    "files_opened_total": 1,
    "id": "foo-bar",
    "input": "filestream",
    "messages_read_total": 1,
    "messages_truncated_total": 1,
    "processing_errors_total": 0,
    "processing_time": {
      "histogram": {
        "count": 1,
        "max": 5000516503,
        "mean": 5000516503,
        "median": 5000516503,
        "min": 5000516503,
        "p75": 5000516503,
        "p95": 5000516503,
        "p99": 5000516503,
        "p999": 5000516503,
        "stddev": 0
      }
    }
  }
]

Related issues


This is an automatic backport of pull request #41667 done by [Mergify](https://mergify.com).

* filebeat: log when multiline reader truncates messages

While investigating an SDH, I noticed that although we add the truncated label
to log fields, there is no feedback explaining why the messages were
truncated.

This PR adds a log statement to provide insight whenever multiline messages
are truncated.

(cherry picked from commit 7806f1a)

# Conflicts:
#	filebeat/input/filestream/metrics_integration_test.go
@mergify mergify bot added the backport label Dec 31, 2024
@mergify mergify bot requested a review from a team as a code owner December 31, 2024 14:08
@mergify mergify bot added the conflicts There is a conflict in the backported pull request label Dec 31, 2024
@mergify mergify bot requested review from khushijain21 and VihasMakwana and removed request for a team December 31, 2024 14:08
Copy link
Contributor Author

mergify bot commented Dec 31, 2024

Cherry-pick of 7806f1a has failed:

On branch mergify/bp/8.x/pr-41667
Your branch is up to date with 'origin/8.x'.

You are currently cherry-picking commit 7806f1a2c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/docs/inputs/input-filestream.asciidoc
	modified:   filebeat/input/filestream/environment_test.go
	modified:   filebeat/input/filestream/input.go
	modified:   filebeat/input/filestream/internal/input-logfile/metrics.go
	modified:   libbeat/reader/multiline/message_buffer.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   filebeat/input/filestream/metrics_integration_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 31, 2024
@mauri870 mauri870 requested review from mauri870 and removed request for khushijain21 and VihasMakwana December 31, 2024 15:06
@mauri870 mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team and removed conflicts There is a conflict in the backported pull request labels Dec 31, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 31, 2024
@mauri870 mauri870 enabled auto-merge (squash) December 31, 2024 15:30
@mauri870 mauri870 merged commit 129374e into 8.x Dec 31, 2024
142 checks passed
@mauri870 mauri870 deleted the mergify/bp/8.x/pr-41667 branch December 31, 2024 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants