Default sampling rule with 0 rate keeps matching traces

[epomatti]

I don't want the Default sampling rule to apply to unmatched rules. I'm setting it to 0.

I want it to be zero because I want to catch only specific hosts and paths with a custom sampling rule that I will created (this is primarily to avoid health checks). The rest I want to discard.

However, even if I leave only the default rule with with 0, without any other rule, it keeps matching all of the health checks.

Am I doing something wrong? Application is running on App Runner.

This is the runtime stage of my Dockerfile:

### Runtime ###
FROM eclipse-temurin:21-jre-jammy
WORKDIR /opt/app
COPY --from=builder /opt/app/target/*.jar /opt/app/*.jar

# AWS X-Ray
ADD https://github.com/aws-observability/aws-otel-java-instrumentation/releases/latest/download/aws-opentelemetry-agent.jar /opt/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS=-javaagent:/opt/aws-opentelemetry-agent.jar
ENV OTEL_PROPAGATORS=xray
ENV OTEL_TRACES_SAMPLER=xray
ENV OTEL_METRICS_EXPORTER=none
ENV OTEL_SERVICE_NAME=MyApp

EXPOSE 8080

ENTRYPOINT ["java","-jar","*.jar"]

This is the application boot log:

01-14-2024 08:53:09 AM Picked up JAVA_TOOL_OPTIONS: -javaagent:/opt/aws-opentelemetry-agent.jar
01-14-2024 08:53:09 AM OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
01-14-2024 08:53:10 AM [otel.javaagent 2024-01-14 11:53:10:920 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.0-aws
01-14-2024 08:53:25 AM   .   ____          _            __ _ _
01-14-2024 08:53:25 AM  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
01-14-2024 08:53:25 AM ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
01-14-2024 08:53:25 AM  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
01-14-2024 08:53:25 AM   '  |____| .__|_| |_|_| |_\__, | / / / /
01-14-2024 08:53:25 AM  =========|_|==============|___/=/_/_/_/
01-14-2024 08:53:25 AM  :: Spring Boot ::                (v3.2.1)
01-14-2024 08:53:26 AM 2024-01-14T11:53:26.654Z  INFO 1 --- [           main] io.pomatti.app.AppApplication            : Starting AppApplication using Java 21.0.1 with PID 1 (/opt/app/*.jar started by root in /opt/app)
01-14-2024 08:53:26 AM 2024-01-14T11:53:26.682Z  INFO 1 --- [           main] io.pomatti.app.AppApplication            : No active profile set, falling back to 1 default profile: "default"
01-14-2024 08:53:33 AM 2024-01-14T11:53:33.584Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
01-14-2024 08:53:33 AM 2024-01-14T11:53:33.767Z  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
01-14-2024 08:53:33 AM 2024-01-14T11:53:33.768Z  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.17]
01-14-2024 08:53:33 AM 2024-01-14T11:53:33.957Z  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
01-14-2024 08:53:33 AM 2024-01-14T11:53:33.958Z  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 6801 ms
01-14-2024 08:53:36 AM 2024-01-14T11:53:36.456Z  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
01-14-2024 08:53:36 AM 2024-01-14T11:53:36.977Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path ''
01-14-2024 08:53:37 AM 2024-01-14T11:53:37.154Z  INFO 1 --- [           main] io.pomatti.app.AppApplication            : Started AppApplication in 12.979 seconds (process running for 27.986)
01-14-2024 08:53:37 AM 2024-01-14T11:53:37.973Z  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
01-14-2024 08:53:37 AM 2024-01-14T11:53:37.975Z  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
01-14-2024 08:53:37 AM 2024-01-14T11:53:37.977Z  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms

[epomatti]

As suggested by the the other thread aws/aws-xray-daemon#163 comments, adding a custom sampling with reservoir and rate 0 did not solve the issue.

I keep getting samples in X-Ray that are triggered by the health check:

Sampling rule that would ignore other traces:

This is the high priority sampling rule that should capture valid traces:

[epomatti]

With DEBUG enabled I've found this:

Caused by: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:2000

I'll search again the App Runner documentation and try to see what's wrong.

Full stack trace:

[otel.javaagent 2024-01-24 01:53:52:403 +0000] [xray-rules-poller] DEBUG io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler - Failed to update sampler
--
java.io.UncheckedIOException: Failed to execute sampling request.
at io.opentelemetry.contrib.awsxray.XraySamplerClient.executeJsonRequest(XraySamplerClient.java:104)
at io.opentelemetry.contrib.awsxray.XraySamplerClient.getSamplingRules(XraySamplerClient.java:77)
at io.opentelemetry.contrib.awsxray.AwsXrayRemoteSampler.getAndUpdateSampler(AwsXrayRemoteSampler.java:126)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:2000
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
at io.opentelemetry.contrib.awsxray.XraySamplerClient.executeJsonRequest(XraySamplerClient.java:101)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
... 24 more

[vastin]

Hi @epomatti, it seems you use aws-opentelemetry-agent.jar which is ADOT Java agent and this repo is for X-Ray Java SDK. You could open ADOT Java agent issue in https://github.com/aws-observability/aws-otel-java-instrumentation/issues. For X-Ray sampling rule support of ADOT Java agent, please check https://aws-otel.github.io/docs/getting-started/java-sdk/trace-auto-instr#using-x-ray-remote-sampling.