Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.

[un0of]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

two vectors start to upload a log to one elasticsearch index and a similar error appears in the logs, one vector is on the first server, the second on the second and they upload to 1 elasticsearch index

Configuration

No response

Version

vector 0.33.0

Debug Output

No response

Example Data

{"host":"dc2","message":"Internal log [Service call failed. No retries or retries exhausted.] has been suppressed 9 times.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::service","target":"vector_common::internal_event::service"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:41.576300789Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"error":"None","error_type":"request_failed","host":"dc2","internal_log_rate_limit":true,"message":"Service call failed. No retries or retries exhausted.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::service","target":"vector_common::internal_event::service"},"pid":40900,"request_id":21310,"source_type":"internal_logs","stage":"sending","timestamp":"2024-08-24T23:59:41.576309042Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Events dropped] has been suppressed 9 times.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::component_events_dropped","target":"vector_common::internal_event::component_events_dropped"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:41.576339728Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"count":11,"host":"dc2","intentional":false,"internal_log_rate_limit":true,"message":"Events dropped","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::component_events_dropped","target":"vector_common::internal_event::component_events_dropped"},"pid":40900,"reason":"Service call failed. No retries or retries exhausted.","source_type":"internal_logs","timestamp":"2024-08-24T23:59:41.576344899Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Not retriable; dropping the request.] is being suppressed to avoid flooding.","metadata":{"kind":"event","level":"ERROR","module_path":"vector::sinks::util::retries","target":"vector::sinks::util::retries"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:42.561304282Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::service","target":"vector_common::internal_event::service"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:42.561388700Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Events dropped] is being suppressed to avoid flooding.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::component_events_dropped","target":"vector_common::internal_event::component_events_dropped"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:42.561412918Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Not retriable; dropping the request.] has been suppressed 6 times.","metadata":{"kind":"event","level":"ERROR","module_path":"vector::sinks::util::retries","target":"vector::sinks::util::retries"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:53.581473811Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","internal_log_rate_limit":true,"message":"Not retriable; dropping the request.","metadata":{"kind":"event","level":"ERROR","module_path":"vector::sinks::util::retries","target":"vector::sinks::util::retries"},"pid":40900,"reason":""error type: document_parsing_exception, reason: [1:319] failed to parse field [context.context] of type [text] in document with id 'KoTVhpEBLQAI7rP3BpLa'. Preview of field's value: '{tum=, reqId=9fedf1b2dd8c5199f005de8ba6600b90}'"","source_type":"internal_logs","timestamp":"2024-08-24T23:59:53.581502772Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Service call failed. No retries or retries exhausted.] has been suppressed 6 times.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::service","target":"vector_common::internal_event::service"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:53.581575549Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"error":"None","error_type":"request_failed","host":"dc2","internal_log_rate_limit":true,"message":"Service call failed. No retries or retries exhausted.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::service","target":"vector_common::internal_event::service"},"pid":40900,"request_id":21317,"source_type":"internal_logs","stage":"sending","timestamp":"2024-08-24T23:59:53.581582655Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"host":"dc2","message":"Internal log [Events dropped] has been suppressed 6 times.","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::component_events_dropped","target":"vector_common::internal_event::component_events_dropped"},"pid":40900,"source_type":"internal_logs","timestamp":"2024-08-24T23:59:53.581613612Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}
{"count":14,"host":"dc2","intentional":false,"internal_log_rate_limit":true,"message":"Events dropped","metadata":{"kind":"event","level":"ERROR","module_path":"vector_common::internal_event::component_events_dropped","target":"vector_common::internal_event::component_events_dropped"},"pid":40900,"reason":"Service call failed. No retries or retries exhausted.","source_type":"internal_logs","timestamp":"2024-08-24T23:59:53.581619025Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}

Additional Context

No response

References

No response

[jszwedko]

You can see the actual error in there:

{"host":"dc2","internal_log_rate_limit":true,"message":"Not retriable; dropping the request.","metadata":{"kind":"event","level":"ERROR","module_path":"vector::sinks::util::retries","target":"vector::sinks::util::retries"},"pid":40900,"reason":""error type: document_parsing_exception, reason: [1:319] failed to parse field [context.context] of type [text] in document with id 'KoTVhpEBLQAI7rP3BpLa'. Preview of field's value: '{tum=, reqId=9fedf1b2dd8c5199f005de8ba6600b90}'"","source_type":"internal_logs","timestamp":"2024-08-24T23:59:53.581502772Z","vector":{"component_id":"es_cluster","component_kind":"sink","component_name":"es_cluster","component_type":"elasticsearch"}}

It looks like you might be sending events that don't match the index configuration.

[bigbrobro]

how to get the raw events

[Klar]

I have a similar issue and would like to find the "raw event" which generated this error, I don't know how to find it.

{
  "@timestamp": "2024-10-04T09:20:46.566457015Z",
  "event": {
    "dataset": "vector.internal_logs",
    "module": "vector"
  },
  "host": {
    "name": "servername"
  },
  "internal_log_rate_limit": true,
  "message": "Not retriable; dropping the request.",
  "metadata": {
    "kind": "event",
    "level": "ERROR",
    "module_path": "vector::sinks::util::retries",
    "target": "vector::sinks::util::retries"
  },
  "process": {
    "pid": 13449
  },
  "reason": "\"error type: document_parsing_exception, reason: [1:169] object mapping for [error] tried to parse field [error] as object, but found a concrete value\"",
  "source_type": "internal_logs",
  "timestamp": "2024-10-04T09:20:46.566333563Z",
  "vector": {
    "component_id": "sink_vector",
    "component_kind": "sink",
    "component_type": "elasticsearch"
  }
}

This is from the /var/log/vector.json which I have also configured to write into elasticsearch (sink_vector).