Otel Collector Keep Restarting

[moronmon]

Hi All,
I just start to migrate our APM to SigNoz and use the Open Telemetry to send and process the metrics.
It turns out the Otel Collector and Otel Metrics keep restarting when I check both consume a lot of memory.
If I checked the logs of Otel Collector, the log is :
2022-06-30T07:32:51.855Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "clickhousemetricswrite", "error": "context deadline exceeded; context deadline exceeded; context deadline exceeded", "errorCauses": [{"error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.inTransaction\n\t/src/exporter/clickhousemetricsexporter/clickhouse.go:225\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*clickHouse).Write\n\t/src/exporter/clickhousemetricsexporter/clickhouse.go:305\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1\n\t/src/exporter/clickhousemetricsexporter/exporter.go:279\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}, {"error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.inTransaction\n\t/src/exporter/clickhousemetricsexporter/clickhouse.go:225\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*clickHouse).Write\n\t/src/exporter/clickhousemetricsexporter/clickhouse.go:305\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1\n\t/src/exporter/clickhousemetricsexporter/exporter.go:279\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}, {"error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.inTransaction\n\t/src/exporter/clickhousemetricsexporter/clickhouse.go:225\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*clickHouse).Write\n\t/src/exporter/clickhousemetricsexporter/clickhouse.go:305\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1\n\t/src/exporter/clickhousemetricsexporter/exporter.go:279\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}], "interval": "75.972662ms"}

I use the version:

• SigNoz : 0.8.2
• Otel Collector: 0.45.1-0.3
• Otel Collector Metrics: 0.45.1-0.3
• ClickHouse : 21.12.3.32

Any best practice to use the Otel Collector? I assume our apps that send their metrics are exceeded the Otel Collector's capacity, but when I give more resources to the Otel Collector, it keeps needing more memory to handle the metrics.

[jpkrohling]

There are two issues here: the first is that the collector shouldn't be simply crashing. You should be able to use the memory limiter to prevent the collector from being killed after consuming too much memory. But the underlying issue seems to be that you can't send data to SigNoz. I would check that project's documentation on how to configure the collector to send data to it.

[moronmon]

Hi @jpkrohling , Thank you for the reply.
I do use the memory limiter as below:
otel-collector-config.yaml

processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
    send_batch_max_size: 1500
  signozspanmetrics/prometheus:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
    dimensions_cache_size: 10000
    dimensions:
      - name: service.namespace
        default: default
      - name: deployment.environment
        default: default
  memory_limiter:
  #   # 80% of maximum memory up to 2G
    limit_mib: 3500
  #   # 25% of limit up to 2G
    spike_limit_mib: 500
    check_interval: 5s

otel-collector-metrics-config.yaml

processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
    send_batch_max_size: 1500
  memory_limiter:
  #   # 80% of maximum memory up to 2G
    limit_mib: 3500
  #   # 25% of limit up to 2G
    spike_limit_mib: 500
    check_interval: 5s

But it seems no effect, cause both of the otel still crashing caused by memory usage.
Is there any best practice or the right implementation for the otel-collector to be referred?

[jpkrohling]

Can you please post the logs for the process that is restarting the collector? The collector itself isn't restarting itself, you'd have to check the watchdog's configuration and see if the memory limits set there are in line with the ones from the config you shared here.