prometheus endpoint in cortex randomly returns data

[justinrush]

I deployed using the cortex-helm chart (values below) using minio as a backend. I tried using Azure blob as well. I have a single prometheus instance using remote_write to send data; its scraping a few dozen pods in a test cluster.

The issue I'm seeing is that the prometheus endpoint usually responds with no data, but sometimes does. This highlights the issue:

for i in `seq 1 100`
do
  len=`curl -s --request POST http://cortex-query-frontend:8080/prometheus/api/v1/series --data-urlencode "match[]=up" |jq ".data |length"`
  if [ "$len" == "0" ]
  then
    echo "${i}: no data"
  else
    echo "${i}: data: ${len}"
  fi

  sleep 1
done

Outputs something like this:

1: data: 116
2: no data
3: no data
4: data: 116
5: no data
6: no data
7: no data
8: no data
9: no data
10: data: 116
11: no data
12: no data
13: no data
14: no data
15: no data
16: no data
17: no data
18: no data
19: data: 116
...

I don't see any errors getting logged in any of the cortex pods and prometheus isn't complaining about remote_writes. It doesn't seem to be writing any data to minio (e.g. the /data/cortex directory is empty).

I'm not sure where to go from here - the test configuration is all below - hoping someone can point out what I'm missing.

Install Etcd

cat << EOF > etcd.yaml
auth:
  rbac:
    enabled: false
    allowNoneAuthentication: true
replicaCount: 3
EOF

helm install cortex-etcd bitnami/etcd -n cortex --create-namespace -f etcd.yaml

install minio

cat << EOF > minio.yaml
fullnameOverride: cortex-minio
accessKey:
  password: my-voice-is-my-password
secretKey:
  password: verify-me
defaultBuckets: cortex
EOF

helm install cortex-minio bitnami/minio -n cortex -f minio.yaml

Install Cortex

cat << EOF > cortex.yaml
nginx:
  enabled: true
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: management
    nginx.ingress.kubernetes.io/rewrite-target: /\$1
  hosts:
    - host: my-host.io
      paths:
        - /cortex
  tls:
    - hosts:
      - my-host.io
config:
  api:
    prometheus_http_prefix: /prometheus
  auth_enabled: false
  ingester:
    lifecycler:
      ring:
        kvstore:
          store: etcd
          # https://cortexmetrics.io/docs/configuration/configuration-file/#etcd_config
          etcd:
            endpoints:
              - cortex-etcd-0.cortex-etcd-headless.cortex.svc.cluster.local:2379
              - cortex-etcd-1.cortex-etcd-headless.cortex.svc.cluster.local:2379
              - cortex-etcd-2.cortex-etcd-headless.cortex.svc.cluster.local:2379
  storage:
    engine: blocks
  # https://cortexmetrics.io/docs/configuration/configuration-file/#blocks_storage_config
  blocks_storage:
    backend: s3
    s3:
      endpoint: cortex-minio.cortex.svc.cluster.local:9000
      bucket_name: cortex
      secret_access_key: verify-me
      access_key_id: my-voice-is-my-password
      insecure: true
    tsdb:
      dir: "/data"
    bucket_store:
      sync_dir: "/data"
  querier:
    # Used when the blocks sharding is disabled.
    store_gateway_addresses: "cortex-store-gateway-headless:9095"
  # https://cortexmetrics.io/docs/configuration/configuration-file/#query_frontend_config
  frontend:

memcached:
  enabled: true
memcached-index-read:
  enabled: true
memcached-index-write:
  enabled: true
memcached-frontend:
  enabled: true
EOF

helm install cortex cortex-helm/cortex -n cortex -f cortex.yaml

Update Prom

Add this to the top level of the prometheus config file:

  remote_write:
    - url: http://cortex-ingester.cortex.svc.cluster.local:8080/push

[justinrush]

I'm not sure what was causing the issue, but using the test CI values from the helm project as a base fixed the issue i was having:
https://github.com/cortexproject/cortex-helm-chart/blob/master/ci/test-values.yaml