Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query finds something that does not exist in logs #108

Open
oalimerko opened this issue Dec 24, 2022 · 0 comments
Open

Query finds something that does not exist in logs #108

oalimerko opened this issue Dec 24, 2022 · 0 comments

Comments

@oalimerko
Copy link

oalimerko commented Dec 24, 2022

i am using this query for querying the logs in elastic

    [query_database_pool_exceeded]
    QueryIndices = logs.*
    QueryIntervalSecs = 300
    QueryJson = {
      "size": 0,
      "query": {
        "query_string": {
          "query": "(message:\"Maximum allowed connections to DB\" OR \"Cannot get a connection, pool error\") AND @timestamp:(>=now-6m AND <now)"
        }
      },
      "aggs": {
        "application": {
          "terms": {
            "field": "kubernetes.labels.app.keyword"
            }
          }
        }
      }

and also i use this prometheus alert rule

    - alert: ElasticExporterDatabaseConnectionPoolExceeded
      expr: database_pool_exceeded_application_doc_count > 0
      for: 6m
      labels:
        severity: critical
        target: project
        topic: database
      annotations:
        targetTemplate: kibana
        summary: "summary"
        description: "description"

The problem is that i get alerted time to time,when i check the metric in prometheus i see that my metric returns some date,but the strange thing is that i do not find those strings in logs.I do not understand how the metric can return data if those strings are not visible in logs.It looks like that my query finds something that does not exist in logs at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant