Data race detected in internal/utilization/utilization.go

[cyrilc-pro]

After upgrading to v3.34.0, a data race condition is detected in my tests.

Description

After upgrading from Go 1.22 to Go 1.23 and from NR GoAgent 3.32.0 to 3.34.0, my tests are failing because of the following race condition:

WARNING: DATA RACE
Write at 0x00c002452b10 by goroutine 12799:
  github.com/newrelic/go-agent/v3/internal/utilization.gatherAWS()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/aws.go:39 +0xa7
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2.1()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:159 +0xdc

Previous read at 0x00c002452b10 by goroutine 12800:
  github.com/newrelic/go-agent/v3/internal/utilization.(*vendors).AnySet()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:87 +0x9d
  github.com/newrelic/go-agent/v3/internal/utilization.gatherAzure()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/azure.go:33 +0x43
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2.1()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:159 +0xdc

Goroutine 12799 (running) created at:
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:152 +0x207
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:168 +0x3fa
  github.com/newrelic/go-agent/v3/internal/utilization.Gather()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:119 +0x264
  github.com/newrelic/go-agent/v3/newrelic.config.createConnectJSON()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/config.go:1065 +0x1ad
  github.com/newrelic/go-agent/v3/newrelic.connectAttempt()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/collector.go:349 +0x64e
  github.com/newrelic/go-agent/v3/newrelic.(*app).connectRoutine()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/internal_app.go:141 +0x164
  github.com/newrelic/go-agent/v3/newrelic.newApp.gowrap2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/internal_app.go:465 +0x33

Goroutine 12800 (finished) created at:
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient.func2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:152 +0x207
  github.com/newrelic/go-agent/v3/internal/utilization.gatherWithClient()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:172 +0x431
  github.com/newrelic/go-agent/v3/internal/utilization.Gather()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/internal/utilization/utilization.go:119 +0x264
  github.com/newrelic/go-agent/v3/newrelic.config.createConnectJSON()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/config.go:1065 +0x1ad
  github.com/newrelic/go-agent/v3/newrelic.connectAttempt()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/collector.go:349 +0x64e
  github.com/newrelic/go-agent/v3/newrelic.(*app).connectRoutine()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/internal_app.go:141 +0x164
  github.com/newrelic/go-agent/v3/newrelic.newApp.gowrap2()
      /root/.gvm/pkgsets/go1.23.0/global/pkg/mod/github.com/newrelic/go-agent/[email protected]/newrelic/internal_app.go:465 +0x33

Steps to Reproduce

Tests are run from inside a Docker container.

Expected Behavior

No race condition.

NR Diag results

N/A

Your Environment

N/A

Reproduction case

N/A

Additional context

N/A

[nr-swilloughby]

To ensure that we can verify that the fix we release for this when it's been diagnosed covers your specific case can you provide a testcase which reproduces the problem you are reporting here, in a way that doesn't expose anything proprietary or relies on external components not needed to verify this condition? With a race condition it can be important to know we're trying the same set of call flows that you are when the race condition is occurring.
Thanks.

[nr-swilloughby]

Although I haven't been able to reproduce your error, analysis of the code base has identified a potential source for a race condition that could be triggered in some circumstances. I'm putting in a correction for this issue and we'll see if this resolves the problem you're having.

[cyrilc-pro]

Thanks. It's hard to provide a simple test case, as the issue was detected on an integration test for a full application.

[nr-swilloughby]

I have merged a proposed fix to our develop branch which is where our code gets staged shortly before release. Could you try pulling a copy of the agent from that branch and try compiling with that to see if that solves your issue?

[nr-swilloughby]

Fixed in 3.35.0