Query front end not working with querier

[vicmania2007]

Hi, i am close to get cortex block storage cluster together. I am currently facing issue with respect to connecting query front end with queriers. based on the documentation mentioned here - https://cortexmetrics.io/docs/architecture/#query-frontend i have added the CLI flag to querier regarding front end address. Not sure whats wrong here.

• I turned on debug.
• Working on cortex version 1.5

What is working:
I am observing, querier is working. It only works if I provide the query-frontend-headless service address via the CLI flag. Somehow it doesnt work with query-frontend service.

What is not working?

• the front end is crashing as it in not ready stage i.e /ready => 503 and then it throws error that "number of queriers is 0" See error below.

{"caller":"main.go:187","level":"info","msg":"Starting Cortex","ts":"2021-02-23T14:12:01.784412371Z","version":"(version=1.5.0, branch=HEAD, revision=86d95e7)"}
{"caller":"server.go:225","grpc":"[::]:9095","http":"[::]:8080","level":"info","msg":"server listening on addresses","ts":"2021-02-23T14:12:01.784743035Z"}
{"auth":false,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/config","ts":"2021-02-23T14:12:01.784892493Z"}
{"auth":false,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/","ts":"2021-02-23T14:12:01.785001519Z"}
{"auth":false,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/debug/fgprof","ts":"2021-02-23T14:12:01.785028381Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"POST","msg":"api: registering route","path":"/prometheus/api/v1/read","ts":"2021-02-23T14:12:01.785135505Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST","msg":"api: registering route","path":"/prometheus/api/v1/query","ts":"2021-02-23T14:12:01.785171608Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST","msg":"api: registering route","path":"/prometheus/api/v1/query_range","ts":"2021-02-23T14:12:01.785206433Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST","msg":"api: registering route","path":"/prometheus/api/v1/labels","ts":"2021-02-23T14:12:01.785242284Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/prometheus/api/v1/label/{name}/values","ts":"2021-02-23T14:12:01.785275323Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST,DELETE","msg":"api: registering route","path":"/prometheus/api/v1/series","ts":"2021-02-23T14:12:01.78533109Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/prometheus/api/v1/metadata","ts":"2021-02-23T14:12:01.785366064Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"POST","msg":"api: registering route","path":"/api/prom/api/v1/read","ts":"2021-02-23T14:12:01.785400258Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST","msg":"api: registering route","path":"/api/prom/api/v1/query","ts":"2021-02-23T14:12:01.785432Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST","msg":"api: registering route","path":"/api/prom/api/v1/query_range","ts":"2021-02-23T14:12:01.785463638Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST","msg":"api: registering route","path":"/api/prom/api/v1/labels","ts":"2021-02-23T14:12:01.785502149Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/api/prom/api/v1/label/{name}/values","ts":"2021-02-23T14:12:01.785532356Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET,POST,DELETE","msg":"api: registering route","path":"/api/prom/api/v1/series","ts":"2021-02-23T14:12:01.785580006Z"}
{"auth":true,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/api/prom/api/v1/metadata","ts":"2021-02-23T14:12:01.785611143Z"}
{"auth":false,"caller":"api.go:99","level":"debug","methods":"GET","msg":"api: registering route","path":"/services","ts":"2021-02-23T14:12:01.785657168Z"}
{"caller":"module_service.go:48","level":"debug","module":"query-frontend","msg":"module waiting for initialization","ts":"2021-02-23T14:12:01.785737099Z","waiting_for":"query-frontend-tripperware"}
{"caller":"module_service.go:58","level":"info","module":"server","msg":"initialising","ts":"2021-02-23T14:12:01.785773822Z"}
{"caller":"module_service.go:58","level":"info","module":"query-frontend-tripperware","msg":"initialising","ts":"2021-02-23T14:12:01.785796426Z"}
{"caller":"module_service.go:58","level":"info","module":"query-frontend","msg":"initialising","ts":"2021-02-23T14:12:01.785837278Z"}
{"caller":"cortex.go:355","level":"info","msg":"Cortex started","ts":"2021-02-23T14:12:01.785860161Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.489562ms","traceID":"403cf84d4a0e1297","ts":"2021-02-23T14:12:02.690988328Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.454956ms","traceID":"3c6c49af8c06247d","ts":"2021-02-23T14:12:07.689435501Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.243893ms","traceID":"32e65f0533c50471","ts":"2021-02-23T14:12:12.690317812Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 6.026251ms","traceID":"8fe6ecb4fd95fa","ts":"2021-02-23T14:12:17.692156972Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.586538ms","traceID":"692e526cef8cdd64","ts":"2021-02-23T14:12:22.689383287Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.675916ms","traceID":"a118ee33558fcef","ts":"2021-02-23T14:12:27.689590791Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.241062ms","traceID":"724b8d54cb809961","ts":"2021-02-23T14:12:32.690335659Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.874954ms","traceID":"37395532616fb54a","ts":"2021-02-23T14:12:37.689844683Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.455755ms","traceID":"28a88ca94997993f","ts":"2021-02-23T14:12:42.689390862Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.05361ms","traceID":"ae5c4afef0986c1","ts":"2021-02-23T14:12:47.689921653Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.850721ms","traceID":"4c1c3710cdcef576","ts":"2021-02-23T14:12:52.689696991Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.487919ms","traceID":"1fad02ee9fe46359","ts":"2021-02-23T14:12:57.689747443Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.305545ms","traceID":"5e3a2e8434991ddd","ts":"2021-02-23T14:13:02.68930999Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.094205ms","traceID":"2eaeb03ee88bb9ad","ts":"2021-02-23T14:13:07.68998483Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.523525ms","traceID":"35633841dcbd7ef9","ts":"2021-02-23T14:13:12.689449636Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.234168ms","traceID":"214d8bd78d0fece6","ts":"2021-02-23T14:13:17.690189931Z"}
{"caller":"frontend.go:538","level":"info","msg":"not ready: number of queriers connected to query-frontend is 0","ts":"2021-02-23T14:13:21.149919787Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /ready (503) 122.747µs","traceID":"6e3ea3d46fe7e9b3","ts":"2021-02-23T14:13:21.150011842Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.453233ms","traceID":"3bf34982eb1a0359","ts":"2021-02-23T14:13:22.691088023Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.147714ms","traceID":"4fc124173e688fce","ts":"2021-02-23T14:13:27.690141232Z"}
{"caller":"frontend.go:538","level":"info","msg":"not ready: number of queriers connected to query-frontend is 0","ts":"2021-02-23T14:13:29.393217673Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /ready (503) 99.653µs","traceID":"6582da23c61a74da","ts":"2021-02-23T14:13:29.393294131Z"}
{"caller":"frontend.go:538","level":"info","msg":"not ready: number of queriers connected to query-frontend is 0","ts":"2021-02-23T14:13:31.149618509Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /ready (503) 83.941µs","traceID":"2cad8adffa091a09","ts":"2021-02-23T14:13:31.149690513Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 4.146647ms","traceID":"72d1801e9a088a10","ts":"2021-02-23T14:13:32.689972785Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.630865ms","traceID":"5478f61137f4c5f5","ts":"2021-02-23T14:13:37.689691193Z"}
{"caller":"frontend.go:538","level":"info","msg":"not ready: number of queriers connected to query-frontend is 0","ts":"2021-02-23T14:13:39.393288011Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /ready (503) 85.071µs","traceID":"2e714e5613bf0fa4","ts":"2021-02-23T14:13:39.393348247Z"}
{"caller":"frontend.go:538","level":"info","msg":"not ready: number of queriers connected to query-frontend is 0","ts":"2021-02-23T14:13:41.150045562Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /ready (503) 83.744µs","traceID":"62b46ab2e08f3db2","ts":"2021-02-23T14:13:41.15010562Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.491225ms","traceID":"2af7bbee5700864b","ts":"2021-02-23T14:13:42.689580508Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /metrics (200) 3.947689ms","traceID":"1d07c7c171ad1293","ts":"2021-02-23T14:13:47.689933528Z"}
{"caller":"frontend.go:538","level":"info","msg":"not ready: number of queriers connected to query-frontend is 0","ts":"2021-02-23T14:13:49.393342455Z"}
{"caller":"logging.go:66","level":"debug","msg":"GET /ready (503) 113.06µs","traceID":"2eed3a06f81131df","ts":"2021-02-23T14:13:49.393418563Z"}
{"caller":"signals.go:55","level":"info","msg":"=== received SIGINT/SIGTERM ===\n*** exiting","ts":"2021-02-23T14:13:49.422329321Z"}
{"caller":"module_service.go:80","level":"debug","module":"query-frontend","msg":"stopping","ts":"2021-02-23T14:13:49.422503719Z"}
{"caller":"module_service.go:80","level":"debug","module":"server","msg":"stopping","ts":"2021-02-23T14:13:49.422516018Z"}
{"caller":"module_service.go:103","level":"debug","module":"query-frontend-tripperware","msg":"module waiting for","ts":"2021-02-23T14:13:49.422520554Z","waiting_for":"query-frontend"}
{"caller":"module_service.go:90","level":"info","module":"query-frontend","msg":"module stopped","ts":"2021-02-23T14:13:49.422572318Z"}
{"caller":"module_service.go:80","level":"debug","module":"query-frontend-tripperware","msg":"stopping","ts":"2021-02-23T14:13:49.422596714Z"}
{"caller":"module_service.go:90","level":"info","module":"query-frontend-tripperware","msg":"module stopped","ts":"2021-02-23T14:13:49.422619535Z"}
{"caller":"server_service.go:50","level":"info","msg":"server stopped","ts":"2021-02-23T14:13:49.422763352Z"}
{"caller":"module_service.go:90","level":"info","module":"server","msg":"module stopped","ts":"2021-02-23T14:13:49.422781988Z"} 

Here is my Cortex config.

auth_enabled: false
blocks_storage:
  azure:
    account_key: masked     
    account_name: masked
    container_name: masked
  backend: azure
  bucket_store:
    sync_dir: /data/cortex/tsdb-sync
  tsdb:
    block_ranges_period:
    - 0h1m0s
    dir: /data/cortex/tsdb
distributor:
  pool:
    health_check_ingesters: true
  ring:
    kvstore:
      store: etcd
  shard_by_all_labels: true
ingester:
  lifecycler:
    final_sleep: 0s
    join_after: 0s
    num_tokens: 256
    ring:
      kvstore:
        store: etcd
      replication_factor: 2
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    use_gzip_compression: false
limits:
  enforce_metric_name: false
  ingestion_burst_size: 1000000
  ingestion_rate: 750000
  ingestion_rate_strategy: local
  reject_old_samples: true
  reject_old_samples_max_age: 168h
querier:
  default_evaluation_interval: 15s
  store_gateway_addresses: cortex-store-gateway-headless.default.svc.cluster.local:9095
server:
  grpc_listen_port: 9095
  grpc_server_max_concurrent_streams: 1000
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  http_listen_port: 8080
  log_format: "json"
  log_level: "debug"
storage:
  engine: blocks
store_gateway:
  sharding_enabled: false

Here is my querier yaml

---
# Source: cortex/templates/querier-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: cortex-querier
  namespace: default
  labels:
    app: cortex-querier
    chart: cortex-0.2.0
    release: cortex
    heritage: Tiller
  annotations:
    {}
spec:
  type: ClusterIP
  ports:
    - port: 8080
      protocol: TCP
      name: http-metrics
      targetPort: http-metrics
  # type: NodePort
  # ports:
  #   - nodePort: 31500
  #     port: 9090
  #     targetPort: 8080
  selector:
    app: cortex-querier
    release: cortex
---
# Source: cortex/templates/querier-dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cortex-querier
  namespace: default
  labels:
    app: cortex-querier
    chart: cortex-0.2.0
    release: cortex
    heritage: Tiller
  annotations:
    {}
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cortex-querier
      release: cortex
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: cortex-querier
        name: cortex-querier
        target: querier
        release: cortex
      annotations:
        checksum/config: 6dd406f10b037c8f1727c9703989ae342b09846df41aea9dacbd8d76a7c6fde4
        prometheus.io/port: http-metrics
        prometheus.io/scrape: "true"
    spec:
      securityContext:
        {}
      initContainers:
        []
      containers:
        - name: cortex
          image: "quay.io/cortexproject/cortex:v1.5.0" 
          imagePullPolicy: IfNotPresent
          args:
            - "-target=querier"
            - "-config.file=/etc/cortex/cortex.yaml"
            - "-etcd.endpoints=http://etcd-client.default.svc.cluster.local:2379"
            - "-querier.frontend-address=cortex-query-frontend-headless.default.svc.cluster.local"
          volumeMounts:
            - name: config
              mountPath: /etc/cortex
            - name: storage
              mountPath: "/data"
              subPath: 
          ports:
            - name: http-metrics
              containerPort: 8080
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 120
          readinessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 120
          resources:
            limits:
              cpu: 600m
              memory: 4Gi
            requests:
              cpu: 600m
              memory: 2Gi
          env:
      nodeSelector:
        {}
      affinity:
        {}
      tolerations:
        []
      terminationGracePeriodSeconds: 180
      volumes:
        - name: config
          secret:
            secretName: cortex
        - name: storage
          # emptyDir: {}
          hostPath: 
            path: "/querier/data"
            type: DirectoryOrCreate
---

here is my query front end yaml

# Source: cortex/templates/query-frontend-svc-headless.yaml
apiVersion: v1
kind: Service
metadata:
  name: cortex-query-frontend-headless
  namespace: default
  labels:
    app: cortex-query-frontend
    chart: cortex-0.2.0
    release: cortex
    heritage: Tiller
  annotations:
    {}
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 8080
      protocol: TCP
      name: http-metrics
      targetPort: http-metrics
    - port: 9095
      protocol: TCP
      name: grpc
      targetPort: grpc
  selector:
    app: cortex-query-frontend
    release: cortex
---
# Source: cortex/templates/query-frontend-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: cortex-query-frontend
  namespace: default
  labels:
    app: cortex-query-frontend
    chart: cortex-0.2.0
    release: cortex
    heritage: Tiller
  annotations:
    {}
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      name: http-metrics
      targetPort: http-metrics
    - port: 9095
      protocol: TCP
      name: grpc
      targetPort: grpc
  selector:
    app: cortex-query-frontend
    release: cortex
---
# Source: cortex/templates/query-frontend-dep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cortex-query-frontend
  namespace: default
  labels:
    app: cortex-query-frontend
    chart: cortex-0.2.0
    release: cortex
    heritage: Tiller
  annotations:
    {}
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cortex-query-frontend
      release: cortex
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: cortex-query-frontend
        name: cortex-query-frontend
        target: query-frontend
        release: cortex
      annotations:
        checksum/config: 8bdd71502c179b0c6469127667e1879c28e172ceee09aa6de11b8a5a83f6be92
        prometheus.io/port: http-metrics
        prometheus.io/scrape: "true"
    spec:
      securityContext:
        {}
      initContainers:
        []
      containers:
        - name: cortex
          image: "quay.io/cortexproject/cortex:v1.5.0"
          imagePullPolicy: IfNotPresent
          args:
            - "-target=query-frontend"
            - "-config.file=/etc/cortex/cortex.yaml"
          volumeMounts:
            - name: config
              mountPath: /etc/cortex
          ports:
            - name: http-metrics
              containerPort: 8080
              protocol: TCP
            - name: grpc
              containerPort: 9095
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 80
          readinessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 80
          resources:
            limits:
              cpu: 1
              memory: 512Mi
            requests:
              cpu: 200m
              memory: 128Mi
          securityContext:
            {}
          env:
      nodeSelector:
        {}
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: target
                  operator: In
                  values:
                  - query-frontend
              topologyKey: kubernetes.io/hostname
            weight: 100
      tolerations:
        []
      terminationGracePeriodSeconds: 180
      volumes:
        - name: config
          secret:
            secretName: cortex
---

[pracucci]

I am observing, querier is working. It only works if I provide the query-frontend-headless service address via the CLI flag. Somehow it doesnt work with query-frontend service.

The query-frontend is not ready until it has at least 1 querier connected. A Kubernetes service doesn't expose IPs of not-ready ops, so you need to create another query-frontend service with publishNotReadyAddresses enabled (maybe it's what you mean by "headless" service). The querier needs to be configured with the query-frontend publishNotReadyAddresses-enabled service so that it can connect even to not-ready query-frontends. Once it's connected to the query-frontend, the query-frontend becomes ready.