Since v0.27 no traces are exported to http endpoint while tracing_ppender works

[ronnybremer]

I have been using opentelemetry-otlp 0.16 for some time and could export traces and metrics. Traces are logged to a local file and also sent to otel-collector port 4317.

Since I migrated to opentelemetry 0.27 the latter doesn't work anymore. I verified the changelog and the new examples and this is how I set up tracing and metrics:

// Construct MeterProvider and set it as global
fn init_meter_provider(resource: Resource) -> SdkMeterProvider {
    let exporter = MetricExporter::builder()
        .with_tonic()
        .with_endpoint("http://otel-collector:4317")
        .build()
        .unwrap();

    let reader = PeriodicReader::builder(exporter, runtime::Tokio)
        .with_interval(std::time::Duration::from_secs(30))
        .build();

    let meter_provider = SdkMeterProvider::builder()
        .with_resource(resource)
        .with_reader(reader)
        .build();

    opentelemetry::global::set_meter_provider(meter_provider.clone());

    meter_provider
}

// Construct TracerProvider and set it as global
fn init_tracer_provider(resource: Resource) {
    let exporter = SpanExporter::builder()
        .with_tonic()
        .with_endpoint("http://otel-collector:4317")
        .build()
        .unwrap();
    let tracer_provider = TracerProvider::builder()
        .with_config(Config::default().with_resource(resource))
        .with_batch_exporter(exporter, runtime::Tokio)
        .build();

    opentelemetry::global::set_tracer_provider(tracer_provider.clone());
}

#[tokio::main]
async fn main() {
    // Configure the default `tracing` subscriber.
    opentelemetry::global::set_text_map_propagator(TraceContextPropagator::new());
    // create a file appender that rotates hourly
    let file_appender = tracing_appender::rolling::daily("/log/", log_file_name);
    // make the file appender non-blocking
    // the guard exists to make sure buffered logs get flushed to output
    let (non_blocking, _guard) = tracing_appender::non_blocking(file_appender);
    let file_layer = tracing_subscriber::fmt::layer()
        .with_span_events(FmtSpan::CLOSE)
        .with_file(true)
        .with_line_number(true)
        .with_writer(non_blocking);
    let filter_layer = EnvFilter::try_from_default_env()
        .or_else(|_| EnvFilter::try_new("info,sqlx=debug".to_owned()).into())
        .unwrap();
    let opentelemetry_resource = resource(service_name, deployment);
    let meter_provider = init_meter_provider(opentelemetry_resource.clone());
    let _meter_provider_guard = OtelGuard {
        meter_provider: meter_provider.clone(),
    };
    init_tracer_provider(opentelemetry_resource.clone());
    tracing_subscriber::registry()
        .with(filter_layer)
        .with(file_layer)
        .with(MetricsLayer::new(meter_provider.clone()))
        .with(ErrorLayer::default())
        .init();
    color_eyre::install().expect("unable to initialize color_eyre");
    let app_metrics = AppMetrics::new();
}

In the previous code (v0.16) I had to add a layer to the tracing_subscriber, like this:

        .with(OpenTelemetryLayer::new(init_tracer(
            opentelemetry_resource.clone(),
        )))

but that seems to no longer work (or I am unable to get it to work anyway).

The traces are exported fine in the local log file, but are no longer sent to the otel-collector endpoint. Metrics works fine and is exported every 30s.

What am I missing?

Thank you for your help.

[ronnybremer]

I assume it might be related to #1779.

[cijothomas]

Are you using tracing-opentelemetry? I am not seeing that layer added...

[ronnybremer]

I would like to, but I can't figure out how to add the layer with the new version of the crate. Could you give me an example please?

[ronnybremer]

Guess I found it. I needed to add:

    let tracer_layer =
        tracing_opentelemetry::layer().with_tracer(tracer_provider.tracer("app"));
    tracing_subscriber::registry()
        .with(filter_layer)
        .with(file_layer)
        .with(tracer_layer)
        .with(MetricsLayer::new(meter_provider.clone()))
        .with(ErrorLayer::default())
        .init();

However, important seem to be these imports:

use opentelemetry::trace::TracerProvider as _;
use tracing_subscriber::layer::SubscriberExt;

It's a bit tough to figure out while piecing together the various examples from different crates. The original examples inside of opentelemetry_otlp don't show this part.
Thank you @cijothomas for pointing me in the right direction!

[cijothomas]

Using tracing with OTel is somewhat tricky and won't be fully resolved until #1571