Not able to receive zipkin traces/metrics

[jpicara]

Hello,
I am trying to get the traces and metrics from our zipkin component using Grafana Agent and forward them to Grafana tempo. However for some reasons it was not able to do it.
Grafana agent configuration:

  grafana-agent:
    enabled: true
    controller:
      replicas: 1
    agent:
      mode: flow
      extraPorts:
      - name: http-traces
        port: 4318
        targetPort: 4318
        protocol: "TCP"
      - name: http-zipkin
        port: 9411
        targetPort: 9411
        protocol: "TCP"
      extraEnv:
      - name: PROMETHEUS_ENDPOINT
        value: "http://monitoring-platform-kube-p-prometheus.platform.svc:9090"
      - name: TEMPO_ENDPOINT
        value: "http://tracing-platform-grafana-tempo.platform.svc:4318"
      configMap:
        create: true
        content: |
          otelcol.receiver.zipkin "default" {       
            output {
              traces = [otelcol.processor.batch.default.input]
              metrics = [otelcol.processor.batch.default.input]
            }
          }

          otelcol.processor.batch "default" {
            output {
              metrics = [otelcol.exporter.prometheus.default.input]
              traces  = [otelcol.exporter.otlphttp.tempo.input]
            }
          }
            
          otelcol.exporter.prometheus "default" {
            forward_to = [prometheus.remote_write.prometheus.receiver]
          }
            
          prometheus.remote_write "prometheus" {
            endpoint {
              url = env("PROMETHEUS_ENDPOINT") + "/api/v1/write"
              }
            }
            
          otelcol.exporter.otlphttp "tempo" {
            // Send traces to a locally running Tempo without TLS enabled.
            client {
              endpoint = env("TEMPO_ENDPOINT")
            }
          }

From tempo side, this is my configuration:

  grafana-tempo:
    enabled: true
    tolerations:
      - key: "dedicated-nodes"
        value: "platform-masters"
        operator: "Equal"
        effect: "NoSchedule"
    tempo:
      storage:
        trace:
          backend: local #change to s3 
          local:
            path: /var/tempo/traces
          wal:
            path: /var/tempo/wal
      receivers:
        zipkin:
            endpoint: 0.0.0.0:9411
        jaeger:
          protocols:
            grpc:
              endpoint: 0.0.0.0:14250
            thrift_binary:
              endpoint: 0.0.0.0:6832
            thrift_compact:
              endpoint: 0.0.0.0:6831
            thrift_http:
              endpoint: 0.0.0.0:14268
        opencensus: null
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318

However traces or metrics are not flowing to tempo. Is there anything I missed on this piece of code? No errors seen in agent or tempo.
Thanks in advance!

[ptodev]

Hello! It'd be good to check the localhost:12345/metrics endpoint on the Agent. It contains metrics from various components such as the batch one.

[jpicara]

Sure thing. This is the metrics coming from the grafana agent regarding zipkin receiver.

http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="0"} 0
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="5"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="10"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="25"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="50"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="75"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="100"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="250"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="500"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="750"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="1000"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="2500"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="5000"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="7500"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="10000"} 3
http_server_duration_bucket{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0",le="+Inf"} 3
http_server_duration_sum{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0"} 0.258883
http_server_duration_count{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0"} 3
# HELP http_server_request_content_length_total 
# TYPE http_server_request_content_length_total counter
http_server_request_content_length_total{component_id="otelcol.receiver.otlp.default",http_flavor="1.1",http_method="POST",http_scheme="http",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0"} 99909
http_server_request_content_length_total{component_id="otelcol.receiver.zipkin.default",http_flavor="1.1",http_method="GET",http_scheme="http",http_status_code="400",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.45.0"} 0

And when I try to forward also the zipkin port in the grafana agent I got the following message:
unexpected end of JSON input

Any ideas on what's going on?
Thanks again.

[ptodev]

Thank you! What do you mean by "forward also the zipkin port in the grafana agent"?

The metrics indicate that there is an HTTP error 400 (Bad Request) in the zipkin receiver. I wonder if there are any errors in the applications which send the traces to the collector? Or if there is a way to inspect the HTTP payload somehow?

You could also try to enable debug logging and send us the logs, but I'm not sure if they will be useful in this case. There are also some Golang env vars for HTTP debugging, but I'm also not sure if they will be useful.

By the way, it would be good to remove the metrics = [otelcol.processor.batch.default.input] from otelcol.receiver.zipkin. This component can only output traces. I'll update our docs to reflect this as well - we should only have traces in the output block for it.

[jpicara]

Thanks a lot for your response.
Thank you! What do you mean by "forward also the zipkin port in the grafana agent"? I mean, inside the grafana tempo I added the port 9411 for zipkin in case it is needed.

      - name: http-zipkin
        port: 9411
        targetPort: 9411
        protocol: "TCP"

so when I tried to forward it as localhost:PORT I got the unexpected end of JSON input.
Let me explain a bit why I am trying to get the traces from zipkin. When I set up Beyla, Agent plus Tempo they are able to get the traces from the application I am interesting on. However, traceID does not match with the real traceID I can find in Zipkin and so that, I cannot see the end to end traces as well as unable to convert traces to logs with Loki as it will try to search for a traceID that is not stored in Loki. On the line I have opened this ticket: grafana/beyla#845. However I thought a possible workaround to collect the traces from zipkin and so that, I would have the real traceID in tempo through grafana agent.
Does it make sense to you?