Using thanos query component to query from sidecar

[rishabhkumar92]

Hi team,

I am trying to setup thanos sidecar which point to Grafana Mimir's QFE and want to setup querier to read from it without setting up object storage. I am not seeing option to point thanos querier to sidecar's grpc endpoints.
Is it possible to do that?

[dosubot[bot]]

Hey there, @rishabhkumar92! I'm here to assist you with any bugs, questions, or contributions you might have. Let's get to the bottom of this together.

Yes, it is possible to set up Thanos querier to read from Thanos sidecar's gRPC endpoints without setting up object storage. The Thanos sidecar can expose the StoreAPI to allow Queriers to query Prometheus data without the need for object storage.

Here is an example configuration for Thanos sidecar without object storage:

prometheus \
  --storage.tsdb.max-block-duration=2h \
  --storage.tsdb.min-block-duration=2h \
  --web.enable-lifecycle

thanos sidecar \
    --tsdb.path        "/path/to/prometheus/data/dir" \
    --prometheus.url   "http://localhost:9090"

Example configuration for Thanos querier:

thanos query \
    --http-address 0.0.0.0:19192 \                                # HTTP Endpoint for Thanos Querier UI
    --endpoint     1.2.3.4:19090 \                                # Static gRPC Store API Address for the query node to query
    --endpoint     1.2.3.5:19090 \                                # Also repeatable
    --endpoint     dnssrv+_grpc._tcp.thanos-store.monitoring.svc  # Supports DNS A & SRV records

For more details, you can refer to the Thanos documentation ^[1][2].

To continue talking to Dosu, mention @dosu.

[rishabhkumar92]

is tsdb.path mandatory while running sidecar?
While configuring querier to interact with sidecar, I am seeing query stuck in loop displaying this statement. Not sure why MinTime/MaxTime are being set to such high range.

ts=2024-09-12T20:42:18.233619Z caller=proxy.go:310 level=debug name=query-0 component=proxy request="min_time:1726173434268 max_time:1726173734268 matchers:<name:\"__name__\" value:\"cortex_bucket_blocks_count\" > aggregates:COUNT aggregates:SUM " msg="Series: started fanout streams" status="Store Addr: 0.0.0.0:10901 LabelSets: MinTime: -9223372036854775808 MaxTime: 9223372036854775807 queried"
Here is my setup:

PROMETHEUS_URL points to Grafana Mimir QFE http endpoint.

Sidecar:

 thanos sidecar \
    --debug.name sidecar-0 \
    --log.level debug \
    --grpc-address 0.0.0.0:10901 \
    --grpc-grace-period 1s \
    --prometheus.url "${PROMETHEUS_URL}"

Query:

thanos query \
    --debug.name query-0 \
    --log.level debug \
    --endpoint 0.0.0.0:10901 \
    --http-address 0.0.0.0:9091 \
    --http-grace-period 1s \

and I am querying 9091 with a sample query and other argument.

[dosubot[bot]]

Yes, tsdb.path is mandatory while running Thanos sidecar. The --tsdb.path flag specifies the data directory of TSDB, which is essential for the sidecar to function correctly.

thanos sidecar \
    --tsdb.path        "/path/to/prometheus/data/dir" \
    --prometheus.url   "${PROMETHEUS_URL}" \
    --grpc-address     0.0.0.0:10901 \
    --grpc-grace-period 1s \
    --debug.name       sidecar-0 \
    --log.level        debug

If your query is stuck in a loop, it could be due to improper configuration or missing mandatory flags like --tsdb.path. Ensure that all required flags are correctly set. Additionally, verify that your Prometheus servers and Thanos components are properly configured and running. For example, the scripts/quickstart.sh script provided by Thanos demonstrates a complete setup with multiple Prometheus servers and sidecars, which can serve as a reference for your configuration ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[rishabhkumar92]

upon more debugging seems like minTime and maxTime is being set from here when stores are not set. I am planning to use querier with sidecar without setting up store gateway/specifying tsdb path, is it possible to do that?

[dosubot[bot]]

Yes, it is possible to use Thanos querier with Thanos sidecar without setting up a store gateway or specifying tsdb.path. The Thanos sidecar can expose the StoreAPI directly from the Prometheus instance, allowing the Thanos querier to query Prometheus data without needing a store gateway ^[1][2].

However, the tsdb.path flag is mandatory for the Thanos sidecar to function correctly. It specifies the data directory of the TSDB, which is essential for the sidecar to read the blocks of data that Prometheus produces ^[1].

If you are seeing issues with minTime and maxTime being set to extreme values, it might be due to the absence of the tsdb.path configuration. Ensure that you include the --tsdb.path flag in your sidecar configuration to avoid such issues.

To continue talking to Dosu, mention @dosu.

[MichaHoffmann]

Those times are set properly when a block is uploaded usually. For sidecars that do not upload it's always the full range ( except if you configure
--min/max-time ). If I remember correctly

[rishabhkumar92]

cc: @MichaHoffmann

[rishabhkumar92]

It stuck in infinite loop with min and max time which i linked here - #7740 (comment)

[MichaHoffmann]

It looks like you point the querier to itself, 0.0.0.0:10901 is the default grpc address and you are not overwriting it. Are sidecar and querier on the same machine?

[rishabhkumar92]

ahh, good catch.

current setup:

http://localhost:9090 -> points to Mimir QFE API

Sidecar:

thanos  sidecar \
    --debug.name sidecar-"${i}" \
    --log.level debug \
    --grpc-address 0.0.0.0:10905 \
    --http-address 0.0.0.0:10906 \
    --grpc-grace-period 1s \
    --prometheus.url http://localhost:9090

Query:

thanos query \
    --debug.name query-"${i}" \
    --log.level debug \
    --endpoint 0.0.0.0:10905 \
    --grpc-address 0.0.0.0:10907 \
    --http-address 0.0.0.0:9095 \
    --http-grace-period 1s \

Request I am making to thanos querier:

curl --location 'localhost:9095/api/v1/query?query=cortex_bucket_blocks_count\{\}' --header 'Accept: application/json' --header 'X-Scope-OrgID: mimir_tenant'

this solves loop issues but gives me error related to StoreAPIs not matching, do i need to pass this label explicitly?

ts=2024-09-13T07:02:51.871908Z caller=proxy.go:268 level=debug name=query-0 component=proxy request="min_time:1726210671871 max_time:1726210971871 matchers:<name:\"__name__\" value:\"cortex_bucket_blocks_count\" > aggregates:COUNT aggregates:SUM " err="No StoreAPIs matched for this query" stores=

[MichaHoffmann]

Can you check in the UI if there are any stores attached to this querier?

[rishabhkumar92]

yeah I was able to associate stores with query but couldn't invoke Mimir's query APIs which is probably due to the fact that only remote read is supported.