Running into spikes in metrics after deduplicaiton because vertical Compaction removing label makes it a new timeseries, is this expected?

[jmichalek132]

Hi, I have a question regarding usage of Vertical Compaction for case of one-to-one duplication (because of using thanos receive with replication factor higher than 1).

When querying the data ingested via thanos receive I noticed weird drops in metrics like this:

After removing the rate function we can see that the counter did truly stopped increasing for a few minutes:

After disabling deduplication and focusing on data from only one prometheus instance (from the HA pair), we can this is caused by fourth timeseriers ending after 5 minute because Staleness.

The other 3 timeseries come from thanos receive instance, the fourth one is coming from thanos store.

However it no longer has the receive_replica label since the Vertical Compaction compated the three blocks into one without the label.
The issue for us are the spikes caused by this in metrics.
I know we could potentially avoid this by disabling Vertical Compaction but that would lead to significant increase in data stored in S3 for us.

Is this expected? Is it a known issue (couldn't find one describing this)? Is there a workaround for this?

[jmichalek132]

Also could this be potentially addressed by keeping the label receive_replica when doing the vertical compaction?

[bwplotka]

It looks like vertical compacted (dedup of 1:1 data) has some artefact (or generate artefact due to some staleness maybe?) so it's bug.

Can you give the raw data from Series API in a JSON from:

• Short time of compacted series
• Short time of one, non compacted series (the same time range)

Using: https://github.com/thanos-io/thanos/blob/main/scripts/insecure_grpcurl_series.sh

First thing we need to answer: Is this artefact of query time or dedup time.

[jmichalek132]

Just out of interested to see what happens I build thanos query locally with a bit of code to insert the receive_replica into the metrics when it's not there with value being taken from one of the other metrics with the label. With this change the issue is not present.

Query without the change:

Query with the change:

I added this code snippet:

	receiveReplicaSeen := make([]bool, len(resp.seriesSet))
	receiveReplicaLabelValue := ""
	for i, s := range resp.seriesSet {
		for _, l := range s.Labels {
			if l.Name == "receive_replica" {
				receiveReplicaSeen[i] = true
				receiveReplicaLabelValue = l.Value
			}
		}

	}

	for j, v := range receiveReplicaSeen {
		if !v {
			resp.seriesSet[j].Labels = append(resp.seriesSet[j].Labels , labelpb.ZLabel{
				Name:  "receive_replica",
				Value: receiveReplicaLabelValue,
			})
		}
	}

At this line.

[jmichalek132]

Result of the query against non compacted block:

❯ scripts/insecure_grpcurl_series.sh localhost:10901 '[{"type": 0, "name": "__name__", "value":"prometheus_remote_storage_samples_total"}, {"type": 0, "name": "k8s_pod", "value":"prometheus-s0-0"}]' 1623311400 1623319200000
{
  "series": {
    "labels": [
      {
        "name": "__name__",
        "value": "prometheus_remote_storage_samples_total"
      },
      {
        "name": "dc",
        "value": "dc"
      },
      {
        "name": "hashring",
        "value": "production"
      },
      {
        "name": "instance",
        "value": "ip"
      },
      {
        "name": "job",
        "value": "prometheus-operated"
      },
      {
        "name": "k8s_container",
        "value": "prometheus"
      },
      {
        "name": "k8s_namespace",
        "value": "namespace"
      },
      {
        "name": "k8s_pod",
        "value": "prometheus-s0-0"
      },
      {
        "name": "k8s_service",
        "value": "prometheus-operated"
      },
      {
        "name": "prom_shard",
        "value": "s0"
      },
      {
        "name": "prometheus_replica",
        "value": "prometheus-s0-0"
      },
      {
        "name": "receive_replica",
        "value": "thanos-receive-production-1"
      },
      {
        "name": "remote_name",
        "value": "thanos-receive"
      },
      {
        "name": "thanos_receive_tenant_id",
        "value": "tenant"
      },
      {
        "name": "url",
        "value": "url"
      }
    ],
    "chunks": [
      {
        "minTime": "1623312055474",
        "maxTime": "1623319195474",
        "raw": {
          "data": "AHjk8oHPvl5CAjvgZogAAODUA/aEk/uq38cUoCyG0DIuiA9ilByQ+gL70QMkUod2G0BJtiBjjlAWQug5HsQHscoGS60BfDKPk7RAuzegJIZQcdaICx60DJDKA9qxDyTugL4ZQMnWIN2+0BJFKBjhdAWP4n5IpQHsNoGTdEBfFKDkh9Au3ogJIpQ8cNoCxbEDJHKA9hdByTSgL+8QMl1o92GUBI2iBsm9ARwyg5K0QHv1oGSGUBfF6HkvRAuwygJIrQdi+ICOKUDJC6A9/xxa/+SKUAF8NoAZF0QBuRSgAlh9ADZNKACOtEA8kNoAPfpQAyV4gAvhdAHJNKAFxvEAEt1oD7IZQAR9ogBkm9AB7DKAOSK0AFi6IAZ4ZQDuRegAlvRADZDKACOK0Acm+IAPYpQAyQugAsfxA/PFKAFyG0AEt0oA7O0QARw+gBkmlAB7WiAeSG0AFj0oAZ68QBuQugAlmlADZHaACPbEB8kMoAPY3QAyPYgAshlAHPFaAFzdEAEsMoB7IvQAR3ogBkhlAB7FaAOR+UAFlWIAZ4XQfuTygAl6xADZDYA="
        }
      }
    ]
  }
}

Result of the query against compacted block:

❯ scripts/insecure_grpcurl_series.sh localhost:10901 '[{"type": 0, "name": "__name__", "value":"prometheus_remote_storage_samples_total"}, {"type": 0, "name": "k8s_pod", "value":"prometheus-s0-0"}]' 1623311400 1623319200000
{
  "series": {
    "labels": [
      {
        "name": "__name__",
        "value": "prometheus_remote_storage_samples_total"
      },
      {
        "name": "dc",
        "value": "dc"
      },
      {
        "name": "hashring",
        "value": "production"
      },
      {
        "name": "instance",
        "value": "ip"
      },
      {
        "name": "job",
        "value": "prometheus-operated"
      },
      {
        "name": "k8s_container",
        "value": "prometheus"
      },
      {
        "name": "k8s_namespace",
        "value": "namespace"
      },
      {
        "name": "k8s_pod",
        "value": "prometheus-s0-0"
      },
      {
        "name": "k8s_service",
        "value": "prometheus-operated"
      },
      {
        "name": "prom_shard",
        "value": "s0"
      },
      {
        "name": "prometheus_replica",
        "value": "prometheus-s0-0"
      },
      {
        "name": "remote_name",
        "value": "thanos-receive"
      },
      {
        "name": "thanos_receive_tenant_id",
        "value": "tenant"
      },
      {
        "name": "url",
        "value": "url"
      }
    ],
    "chunks": [
      {
        "minTime": "1623312055474",
        "maxTime": "1623319195474",
        "raw": {
          "data": "AHjk8oHPvl5CAjvgZogAAODUA/aEk/uq38cUoCyG0DIuiA9ilByQ+gL70QMkUod2G0BJtiBjjlAWQug5HsQHscoGS60BfDKPk7RAuzegJIZQcdaICx60DJDKA9qxDyTugL4ZQMnWIN2+0BJFKBjhdAWP4n5IpQHsNoGTdEBfFKDkh9Au3ogJIpQ8cNoCxbEDJHKA9hdByTSgL+8QMl1o92GUBI2iBsm9ARwyg5K0QHv1oGSGUBfF6HkvRAuwygJIrQdi+ICOKUDJC6A9/xxa/+SKUAF8NoAZF0QBuRSgAlh9ADZNKACOtEA8kNoAPfpQAyV4gAvhdAHJNKAFxvEAEt1oD7IZQAR9ogBkm9AB7DKAOSK0AFi6IAZ4ZQDuRegAlvRADZDKACOK0Acm+IAPYpQAyQugAsfxA/PFKAFyG0AEt0oA7O0QARw+gBkmlAB7WiAeSG0AFj0oAZ68QBuQugAlmlADZHaACPbEB8kMoAPY3QAyPYgAshlAHPFaAFzdEAEsMoB7IvQAR3ogBkhlAB7FaAOR+UAFlWIAZ4XQfuTygAl6xADZDYA="
        }
      }
    ]
  }
}
{
  "series": {
    "labels": [
      {
        "name": "__name__",
        "value": "prometheus_remote_storage_samples_total"
      },
      {
        "name": "dc",
        "value": "dc"
      },
      {
        "name": "hashring",
        "value": "production"
      },
      {
        "name": "instance",
        "value": "ip"
      },
      {
        "name": "job",
        "value": "prometheus-operated"
      },
      {
        "name": "k8s_container",
        "value": "prometheus"
      },
      {
        "name": "k8s_namespace",
        "value": "namespace"
      },
      {
        "name": "k8s_pod",
        "value": "prometheus-s0-0"
      },
      {
        "name": "k8s_service",
        "value": "prometheus-operated"
      },
      {
        "name": "prom_shard",
        "value": "s0"
      },
      {
        "name": "prometheus_replica",
        "value": "prometheus-s0-1"
      },
      {
        "name": "remote_name",
        "value": "thanos-receive"
      },
      {
        "name": "thanos_receive_tenant_id",
        "value": "tenant"
      },
      {
        "name": "url",
        "value": "utl"
      }
    ],
    "chunks": [
      {
        "minTime": "1623312017803",
        "maxTime": "1623319157803",
        "raw": {
          "data": "AHiWpv3Ovl5CAjvW8KgAAODUA/SPyGUkax1W/tndASQygY6sQbn9oCWKUDZC6AnvxD+RSgJYbQNi6ICSKUHHD6Bc/RASxSj7IbQEi2IH+OUBJC6DtmlASV4gY/rQ7kMoCWtEDZN6AnhlB+NogJLrQNsMoCTrE/HO6BchlASxWg7F8QEkUoH+F0BJfiHtilASQ2gY90QbkUoCWH0DY+iAkilH/w2gJLpQNveICSF0HHNKBcrxASPWh7YZQElaIH+b0BJDKDtitASbogY4ZcWv/ci9ABI+iAG2GUAEkVoA/98QASRSgBthdABJfiAeOKUALkNoAJHpQB21ogAkh9AD/NKACTtEB9sNoAJLpQAuN4gDjhdABJNKAG2vEAEn1oB/4ZQASRugBtuxABJDKAOOK0ALi6IAJIZQP2xegAk/RAD/DKACSK0Adi+IAJYpQAuQugHjnlABJViAG2G0AEn0oA/60QASQ+gBtmlABI2iA9yG0AMd0oAJI/QB28ogAkmlAD/HaACRbEA9kMoAJY3QAuXYgDjhlABJFaAG3dEAEkMoP/4vQASTyA=="
        }
      }
    ]
  }
}

[bwplotka]

Amazing work @jmichalek132 , you essentially solved the problem, let me try to implement this fix quickly 💪🏽

[bwplotka]

Those test cases shows the issue:

[bwplotka]

WIP fix: #4375

I take this carefully and adding benchmarks to make sure we can improve dedu latency too