Gaps/delays in the reporting Cortex metrics

[zulh-civo]

This is my follow-up question for #5101 since this question is for different context.

Cortex is now working and I can see metrics for my external clusters/tenants (which do remote write to Cortex). But, I saw some gaps/delays from some clusters which quite big and busy compared to the happy-and-no-gaps-in-metrics cluster which is smaller and less busy.

Note: abc and def are the big and busy clusters while xyz is the small and less busy cluster

abc cluster

def cluster

xyz cluster

All three clusters are showing up in the <cortex-distributor>/distributor/all_user_stats page:

Questions:

1. Why these gaps/delays appear?
2. What factors that could contribute to this problem?
3. Is this fixable?
4. What keyword/docs on Cortex website that I can read/refer to understand deeper about this problem?
5. What should I look for (e.g. Cortex configuration) to fix this issue so I don't see gaps/delays anymore and able to see latest metrics when I need to without having to wait?

FWIW, this is the Helm values file I use for the Cortex deployment:

alertmanager:
    enabled: false
  clusterDomain: cortex.example.com
  config:
    auth_enabled: true
    blocks_storage:
      backend: s3
      s3:
        access_key_id: <redacted>
      bucket_name: <redacted>
      endpoint: <redacted>
      region: <redacted>
      secret_access_key: <redacted>
    ingester:
      lifecycler:
        ring:
          replication_factor: 3
  nginx:
    enabled: false
  ruler:
    enabled: false
  tags:
    blocks-storage-memcached: true
  compactor:
    resources:
      requests:
        memory: "1Gi"
        cpu: "1000m"
      limits:
        memory: "1Gi"
        cpu: "1000m"
  distributor:
    replicas: 3
    resources:
      requests:
        memory: "4Gi"
        cpu: "2000m"
      limits:
        memory: "4Gi"
        cpu: "2000m"
    extraArgs:
      "-log.level": "debug"
      "-distributor.ingestion-rate-limit": "500000000"
      # To support `-ingester.max-global-series-per-user` flag
      # "-distributor.shard-by-all-labels": "true"
  ingester:
    replicas: 3
    resources:
      requests:
        memory: "35Gi"
        cpu: "4000m"
      limits:
        memory: "35Gi"
        cpu: "4000m"
    extraArgs:
      "-log.level": "debug"
      # "-ingester.max-series-per-user": "500000000" 
      # "-ingester.max-global-series-per-user": "500000000"
      "-ingester.ignore-series-limit-for-metric-names": "node_network_transmit_errs_total,apiserver_request_duration_seconds_bucket,container_blkio_device_usage_total,container_tasks_state,node_network_address_assign_type,node_network_carrier,node_network_carrier_changes_total,node_network_carrier_down_changes_total,node_network_carrier_up_changes_total,node_network_device_id,node_network_dormant,node_network_flags,node_network_iface_id,node_network_iface_link,node_network_iface_link_mode,node_network_info,node_network_mtu_bytes,node_network_name_assign_type,node_network_net_dev_group,node_network_protocol_type,node_network_receive_bytes_total,node_network_receive_compressed_total,node_network_receive_drop_total,node_network_receive_errs_total,node_network_receive_fifo_total,node_network_receive_frame_total,node_network_receive_multicast_total,node_network_receive_packets_total,node_network_transmit_bytes_total,node_network_transmit_carrier_total,node_network_transmit_colls_total,node_network_transmit_compressed_total,node_network_transmit_drop_total,node_network_transmit_fifo_total,node_network_transmit_packets_total,node_network_transmit_queue_length,node_network_up,root_ca_cert_publisher_sync_duration_seconds_bucket"
    statefulSet:
      enabled: true
    persistentVolume:
      enabled: true
      size: 100Gi
      storageClass: ms-xfs-2-replicas
  memcached-blocks:
    resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  memcached-blocks-index:
    resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  memcached-blocks-metadata:
    resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  querier:
    resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  query_frontend:
    resources:
      requests:
        memory: "128Mi"
        cpu: "500m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  store_gateway:
    replicas: 1
    persistentVolume:
      size: 100Gi
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "500m"
  runtimeconfigmap:
    runtime_config:
      overrides:
        abc-cluster:
          max_series_per_metric: 500000000
          max_global_series_per_metric: 500000000
          max_series_per_user: 500000000 
          max_global_series_per_user: 500000000 
          max_label_names_per_series: 5000 
        def-cluster:
          max_series_per_metric: 500000000
          max_global_series_per_metric: 500000000
          max_series_per_user: 500000000 
          max_global_series_per_user: 500000000 
          max_label_names_per_series: 5000 
        xyz-cluster:
          max_series_per_metric: 500000000
          max_global_series_per_metric: 500000000
          max_series_per_user: 500000000 
          max_global_series_per_user: 500000000 
          max_label_names_per_series: 5000 

[friedrichg]

Typically cortex is monitored using prometheus, scrape /metrics endpoint for every cortex pod using prometheus. If you do that you are going to get prometheus metrics for cortex. One of the metrics you can get from cortex is cortex_discarded_samples_total.
This metric will tell you if metrics are getting discarded by cortex.
This is the first thing to check if you see gaps. Let me know if this metric is bigger than zero and which reason label do you see. A dashboard would be fine.

[zulh-civo]

@friedrichg - these are my cortex_discarded_samples_total metrics - any idea what can I do?

[zulh-civo]

@jeromeinsf - what do you mean by 'collector' here? What's pod name?

[jeromeinsf]

@zulh-civo I mean what ever collector/agent you use to remote_write.
The error message you just shared is pretty self-explanatory, you have out of order samples: probably a collision between your label sets.

[friedrichg]

if you have any discards bigger than zero, that is definitely causing some gaps.
The fact that you say it happens in the big cluster probably means you need more replicas in the ingestion (distributors and ingesters).
you should measure latency and errors for your cortex cluster. Latency and ratio errors to requests should be the same for both clusters to get similar results.

The cortex-mixin provides specific guidelines to what to do in each case

Consider installing and configuring the cortex-mixin with alerts and dashboards so you can make use of its playbook.

[zulh-civo]

@friedrichg - thanks for pointing out the cortex-mixin project. I managed to deploy it and I'm now looking to fix these issues.