Skip to content

feat: do not instrument tracer final batch processor #16870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat: do not instrument tracer final batch processor #16870

wants to merge 8 commits into from

Conversation

kruskall
Copy link
Member

@kruskall kruskall commented May 8, 2025

Motivation/summary

prevent the tracer bulk indexer from generating more metrics and traces

Checklist

For functional changes, consider:

  • Is it observable through the addition of either logging or metrics?
  • Is its use being published in telemetry to enable product improvement?
  • Have system tests been added to avoid regression?

How to test these changes

Related issues

Related to #16838

@kruskall
feat: do not instrument tracer final batch processor
de0bbd8 
prevent the tracer bulk indexer from generating more metrics and traces
@kruskall kruskall requested a review from a team as a code owner May 8, 2025 20:41
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 8, 2025

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@kruskall kruskall added the backport-8.19 Automated backport to the 8.19 branch label May 8, 2025
@kruskall kruskall added the backport-9.0 Automated backport to the 9.0 branch label May 8, 2025
kruskall added 6 commits May 9, 2025 00:37
@kruskall
test: move TestWrapServerAPMInstrumentationTimeout doc assertions out…
dcc746a 
…side processor

the trace server no longer uses the server batch processor to avoid
generating traces

move assertions to validate the documents received from ES
@kruskall
lint: fix linter issues
d99342c
@kruskall
Update server_test.go
b317c41
@kruskall
Update server_test.go
ccc6e74
@kruskall
Update server_test.go
31db4c5
@kruskall
Merge branch 'main' into fix/tracer-processor
edfd525
@rubvs
Copy link
Contributor

rubvs commented May 21, 2025 •
edited
Loading

@kruskall, I still get racing issue when I test it locally on my Mac with your branch.

In my #16838 (comment), I explain why noop works for logger, implying a race with Close. Have you checked the patch provided by @1pkg, see #16838 (comment)? This patch works perfectly for me and seems to address the core issue. Maybe I'm missing something with your solution?

WARNING: DATA RACE
Read at 0x00c001140583 by goroutine 296:
  testing.(*common).logDepth()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1053 +0x8c
  testing.(*common).log()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1046 +0x80
  testing.(*common).Logf()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1097 +0x58
  go.uber.org/zap/zaptest.TestingWriter.Write()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/zaptest/logger.go:146 +0xf0
  go.uber.org/zap/zaptest.(*TestingWriter).Write()
      <autogenerated>:1 +0x64
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/zapcore/core.go:99 +0x11c
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:253 +0x158
  go.uber.org/zap.(*SugaredLogger).log()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/sugar.go:355 +0x104
  go.uber.org/zap.(*SugaredLogger).Debug()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/sugar.go:149 +0x1fc
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Debug()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/[email protected]/logp/logger.go:158 +0x1c8
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/queue_reader.go:68 +0x1a4
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.gowrap1()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/consumer.go:102 +0x40

Previous write at 0x00c001140583 by goroutine 283:
  testing.tRunner.func1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1779 +0x5dc
  runtime.deferreturn()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:610 +0x5c
  testing.(*T).Run.gowrap1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1851 +0x40

Goroutine 296 (running) created at:
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/consumer.go:102 +0x388
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.newOutputController()                                                                                                                                          /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/controller.go:97 +0x70
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.New()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/pipeline.go:149 +0x3f8                                                    github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/module.go:100 +0x294
  github.com/elastic/beats/v7/libbeat/publisher/pipeline.Load()                                                                                                                                                         /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/beats/[email protected]/libbeat/publisher/pipeline/module.go:73 +0x170
  github.com/elastic/apm-server/internal/beater.(*Runner).newLibbeatFinalBatchProcessor()
      /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/beater.go:854 +0x47c                                                                                              github.com/elastic/apm-server/internal/beater.(*Runner).newFinalBatchProcessor()
      /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/beater.go:689 +0x4d8
  github.com/elastic/apm-server/internal/beater.(*Runner).Run()                                                                                                                                                         /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/beater.go:376 +0x197c
  github.com/elastic/apm-server/internal/beater/beatertest.(*Server).Start.func1()
      /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/beatertest/server.go:147 +0xb8                                                                                    golang.org/x/sync/errgroup.(*Group).add.func1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:130 +0xd4
                                                                                                                                                                                                                  Goroutine 283 (running) created at:
  testing.(*T).Run()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1851 +0x684                                                                                         testing.runTests.func1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2279 +0x7c
  testing.tRunner()                                                                                                                                                                                                     /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1792 +0x180
  testing.runTests()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2277 +0x77c                                                                                         testing.(*M).Run()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2142 +0xb68
  main.main()                                                                                                                                                                                                           _testmain.go:99 +0x110
==================
2025/05/21 10:13:29 http: TLS handshake error from 127.0.0.1:51587: read tcp 127.0.0.1:51584->127.0.0.1:51587: use of closed network connection
==================
WARNING: DATA RACE
Read at 0x00c002ea0ac3 by goroutine 5160:
  testing.(*common).logDepth()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1059 +0x380
  testing.(*common).log()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1046 +0x80
  testing.(*common).Logf()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1097 +0x58
  go.uber.org/zap/zaptest.TestingWriter.Write()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/zaptest/logger.go:146 +0xf0
  go.uber.org/zap/zaptest.(*TestingWriter).Write()
      <autogenerated>:1 +0x64
  go.uber.org/zap/zapcore.(*ioCore).Write()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/zapcore/core.go:99 +0x11c
  go.uber.org/zap/zapcore.(*CheckedEntry).Write()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:253 +0x158
  go.uber.org/zap.(*SugaredLogger).log()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/sugar.go:355 +0x104
  go.uber.org/zap.(*SugaredLogger).Error()
      /Users/rubenvanstaden/go/pkg/mod/go.uber.org/[email protected]/sugar.go:167 +0x6c
  github.com/elastic/elastic-agent-libs/logp.(*Logger).Error()
      /Users/rubenvanstaden/go/pkg/mod/github.com/elastic/[email protected]/logp/logger.go:173 +0x30
  github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1()
      /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/middleware/log_middleware.go:60 +0x3c0
  github.com/elastic/apm-server/internal/beater/api.NewMux.(*ContextPool).HTTPHandler.func12()
      /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/request/context_pool.go:47 +0x120
  net/http.HandlerFunc.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2294 +0x48
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x288
  net/http.serverHandler.ServeHTTP()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3301 +0x298
  net/http.(*conn).serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:2102 +0xeb4
  net/http.(*Server).Serve.gowrap3()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3454 +0x4c

Previous write at 0x00c002ea0ac3 by goroutine 5076:
  testing.tRunner.func1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1779 +0x5dc
  runtime.deferreturn()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:610 +0x5c
  testing.(*T).Run.gowrap1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1851 +0x40

Goroutine 5160 (running) created at:
  net/http.(*Server).Serve()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/net/http/server.go:3454 +0x674
  github.com/elastic/apm-server/internal/beater.(*Runner).Run.func11()
      /Users/rubenvanstaden/issue/apm_server/16838/apm-server/kruskall/apm-server/internal/beater/beater.go:503 +0x44
  golang.org/x/sync/errgroup.(*Group).add.func1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:130 +0xd4

Goroutine 5076 (finished) created at:
  testing.(*T).Run()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1851 +0x684
  testing.runTests.func1()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2279 +0x7c
  testing.tRunner()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:1792 +0x180
  testing.runTests()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2277 +0x77c
  testing.(*M).Run()
      /Users/rubenvanstaden/go/pkg/mod/golang.org/[email protected]/src/testing/testing.go:2142 +0xb68
  main.main()
      _testmain.go:99 +0x110

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
backport-8.19 Automated backport to the 8.19 branch backport-9.0 Automated backport to the 9.0 branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kruskall @rubvs