SearchError ~ percentage_match | container restart failed to sync with existing indices

[aryaa192]

Hi Team,
I'm using this since last 3 months.
Elastic stack version: 7.17.9
ElastAlert2 version: 2.18.0
Environment: Docker

recently I've tried percentage_match. After deployment what I've observed in elastAlert logs,

ERROR:elastalert:Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/elastalert/elastalert.py", line 1324, in alert
    return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/elastalert.py", line 1409, in send_alert
    alert.alert(matches)
  File "/usr/local/lib/python3.12/site-packages/elastalert/alerters/teams.py", line 50, in alert
    title = self.create_title(matches)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/alerts.py", line 196, in create_title
    return self.create_custom_title(matches)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/elastalert/alerts.py", line 219, in create_custom_title
    alert_subject = alert_subject.format(*alert_subject_values)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: Replacement index 1 out of range for positional args tuple

ERROR:elastalert:Uncaught exception running rule [critical] ALERT - Backend Connectivity Error [15m]: Replacement index 1 out of range for positional args tuple

WARNING:elasticsearch:POST http://elasticsearch:9200/<indices*>/_search?ignore_unavailable=true&size=0 [status:400 request:0.003s]
ERROR:elastalert:Error running query: RequestError(400, 'x_content_parse_exception', '[query] query malformed, no start_object after query name')

However, the rule which produced this error has been resolved. It was alert_subject parameter related issue. But, I could see that the error is still persisting into the elastAlert logs.

next attempt I tried,

• change the query_key to different field. (No change)
• removed that rule file. (No change)
• restarted the container. (Now, that rule yaml is no longer exist. still it's persisting.)
• Stopped the container and deleted all the existing indices related to elastalert from elasticsearch. Then restart the container (Still it's persisting.)

My query,

1. Is percentage_match along with match_bucket_filter ES DSL able to query elasticsearch? (This produces SearchError, RequestError - 400)
2. After restarting docker container, A new container with new changes in elastAlert.yaml config and rules/ (new added rules) would be able to sync with existing indices related to elastalert?
3. In my application's logs there's key/value --> "applicationResourceTemplate.keyword":"/*". When I tried to query with this key elastAlert was unable to query with this key and I was

Your feedback is precious and would be appreciated!

Thanks!

[aryaa192]

Here's sample rule configured,

type: percentage_match
index: sample*

query_key:
- field.keyword
- field_2.keyword
   
run_every:
  minutes: 1
timeframe:
  minutes: 5
buffer_time:
  minutes: 1

realert:
  minutes: 30

max_percentage: 1
match_bucket_filter:
  - term:
    ResponseCode: 500

Alert sent to ms_teams.

[jertel]

Enable debug logs and post the debug log output showing the actual query that was sent to Elasticsearch.

[aryaa192]

Thanks. The issue has been resolved for this SearchError. The issue was resolved after deletion of elastalert indices from elasticsearch & container restart.