Failed to get cluster name error on newrelic_rpm v9.15.0

[bkwaku]

Description

We are encountering multiple signature errors on our monitoring systems. Upon further inspection, the following errors were identified in our logs:

[NewRelic][2024-06-17 20:12:22 +0000 web.1 (60)] ERROR: Failed to get cluster name for elastic search

Elasticsearch::Transport::Transport::Errors::Forbidden [403] {"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. [The remaining error log has been hidden for security reasons].

These requests are being done by new relic ruby agent apparently as part of a health check routine (they trigger multiple times over time generating a tonne of logs for us)

Expected Behavior

No signature errors should appear during the health check routine.

Your Environment

Ruby -> 3.0.6
Rails -> 6.1.0
Elasticsearch -> 6.2.4
newrelic_rpm -> 9.15.0

Additional context

If you need any more information, please don't hesitate to ask. Thank you.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-363256

[hannahramadan]

Hi @bkwaku! Others have seen this issue before (#2730) but we weren't able to reproduce it with supported versions of Elasticsearch (the agent supports Elasticsearch versions 7.0+).

Originally, the agent was attempting to get the cluster name multiple times per client instance. We reduced that in Ruby agent 9.12.0 (#2743) to only attempt to get the cluster name once per client instance, so the error messages were seen much less. It seems you may be creating multiple client instances, which is leading to the influx in error log messages each time the agent attempts to get the cluster name.

Unfortunately without a reproduction on a supported Elasticsearch version, there's not much we can do here to resolve this error. We have however discussed adding a configuration option to simply not collect the cluster name, which should prevent the errors. We will begin working on this.

In the meantime, we haven't seen this issue in Elasticsearch 7.0+, so upgrading may help if that is possible. You can also disable Elasticsearch instrumentation to get rid of those warnings for now.

[hannahramadan]

Hi @bkwaku! We've created an new configuration option elasticsearch.capture_cluster_name that can control the capturing Elasticsearch cluster names (#3038). If disabled, while cluster names will no longer be reported, the related errors will go away.

This new configuration will be available in our next release. I'm going to close this issue for now, but feel free to reopen should anything else come up.