Unable to export Micrometer metrics in Prometheus format

[codewithabhi17]

we enable metrics in our application and expose these metrics directly in Prometheus format.

We use these maven dependencies in our spring boot application to expose metrics in prometheus format:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <scope>runtime</scope>
</dependency>

The spring-boot-starter-actuator provides the metrics API and some out-of-the-box metrics. Under the hood, it uses the Micrometer metrics library. The micrometer-registry-prometheus is for exposing Micrometer metrics in Prometheus format.

Sample metrics exposed by micrometer in prometheus format

# HELP executor_pool_core_threads The core number of threads for the pool
# TYPE executor_pool_core_threads gauge
executor_pool_core_threads{name="applicationTaskExecutor",} 8.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 19845.0
# HELP jvm_gc_overhead_percent An approximation of the percent of CPU time used by GC activities over the last lookback period or since monitoring began, whichever is shorter, in the range [0..1]
# TYPE jvm_gc_overhead_percent gauge
jvm_gc_overhead_percent 4.886425550490458E-5
# HELP jdbc_connections_idle Number of established but idle connections.
# TYPE jdbc_connections_idle gauge
jdbc_connections_idle{name="dataSource",} 10.0
# HELP jdbc_connections_min Minimum number of idle connections in the pool.
# TYPE jdbc_connections_min gauge
jdbc_connections_min{name="dataSource",} 10.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="mapped - 'non-volatile memory'",} 0.0
jvm_buffer_count_buffers{id="mapped",} 0.0
jvm_buffer_count_buffers{id="direct",} 2.0
# HELP disk_total_bytes Total space for path
# TYPE disk_total_bytes gauge
disk_total_bytes{path="/dos/dos-api/.",} 4.99963174912E11
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 30.0
# HELP hikaricp_connections_idle Idle connections
# TYPE hikaricp_connections_idle gauge
hikaricp_connections_idle{pool="HikariPool-1",} 10.0
# HELP tomcat_sessions_created_sessions_total  
# TYPE tomcat_sessions_created_sessions_total counter
tomcat_sessions_created_sessions_total 0.0
# HELP tomcat_sessions_expired_sessions_total  
# TYPE tomcat_sessions_expired_sessions_total counter
tomcat_sessions_expired_sessions_total 0.0
# HELP spring_security_filterchains_AnonymousAuthenticationFilter_before_total  
# TYPE spring_security_filterchains_AnonymousAuthenticationFilter_before_total counter
spring_security_filterchains_AnonymousAuthenticationFilter_before_total{security_security_reached_filter_section="before",spring_security_filterchain_position="0",spring_security_filterchain_size="0",} 1.0
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",id="G1 Survivor Space",} -1.0
jvm_memory_max_bytes{area="heap",id="G1 Old Gen",} 4.294967296E9
jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="nonheap",id="CodeCache",} 5.0331648E7
jvm_memory_max_bytes{area="heap",id="G1 Eden Space",} -1.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9

config.yml :

extensions:
  health_check:
  sigv4auth:
    region: us-west-2
    service: aps
    assume_role:
      arn: XXX
      sts_region: eu-west-2

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  awsecscontainermetrics:
    collection_interval: 10s
  prometheus:
    config:
      global:
        scrape_interval: 20s
        #scrape_timeout: 10s
      scrape_configs:
        - job_name: "otel-collector"
          metrics_path: "dos/api/metrics"
          static_configs:
            - targets: [localhost:8080]

processors:
  batch/traces:
    timeout: 1s
    send_batch_size: 50
  batch/metrics:
    timeout: 60s
  resourcedetection:
    detectors:
      - env
      - system
      - ecs
  filter:
    metrics:
      include:
        match_type: strict
        metric_names:
          - ecs.task.memory.reserved
          - ecs.task.memory.utilized
          - ecs.task.cpu.reserved
          - ecs.task.cpu.utilized
          - ecs.task.network.rate.rx
          - ecs.task.network.rate.tx
          - ecs.task.storage.read_bytes
          - ecs.task.storage.write_bytes
          - container.duration

exporters:
  awsxray:
  prometheusremotewrite:
    endpoint: XXX
    auth:
      authenticator: sigv4auth
  logging:
    loglevel: info
#    resource_to_telemetry_conversion:
#      enabled: true

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ resourcedetection, batch/traces ]
      exporters: [ awsxray ]
    metrics/application:
      receivers: [prometheus]
      processors: [resourcedetection, batch/metrics]
      exporters: [prometheusremotewrite]
    metrics:
      receivers: [awsecscontainermetrics]
      processors: [filter]
      exporters: [prometheusremotewrite]

  extensions: [health_check, sigv4auth]

Error

2023-06-01T02:49:57.192Z error exporterhelper/queued_retry.go:401 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 400 Bad Request; err = %!w(<nil>): user=70501407_ws-6a5b6-79c8-4-8d8f-ed2974f: err: out of order sample. timestamp=2023-06-01T02:48:57.719Z, series={__name__=\"executor_queue_remaining_tasks\", instance=\"localhost:8080\", job=\"otel-collector\", name=\"applicationTaskExecutor\"}\n", "dropped_items": 471}

Is the metrics format exposed by prometheus not correct? Please let me know if you need any more info from my end to debug the issue