compactor never starts

[dharapvj]

Hi,

We use cortex as multi-tenant prometheus storage and the cortex-compactor-0 pod for the statefulset is failing to start. it kills after 15 minutes. I have tried to increase the readiness probe to even 1h.. but it still cannot finish the "block cleanup and maintenance"

What can we be way out?

Relevant portion of logs for cortex-compactor..

level=info ts=2021-12-09T07:47:35.593894286Z caller=main.go:188 msg="Starting Cortex" version="(version=1.9.0, branch=HEAD, revision=ed4f339)"
level=info ts=2021-12-09T07:47:35.594434941Z caller=server.go:239 http=[::]:8080 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2021-12-09T07:47:35.609109981Z caller=module_service.go:59 msg=initialising module=server
level=info ts=2021-12-09T07:47:35.609339837Z caller=module_service.go:59 msg=initialising module=runtime-config
level=info ts=2021-12-09T07:47:35.609663886Z caller=module_service.go:59 msg=initialising module=memberlist-kv
level=info ts=2021-12-09T07:47:35.610771314Z caller=module_service.go:59 msg=initialising module=compactor
level=info ts=2021-12-09T07:47:35.647857736Z caller=lifecycler.go:529 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2021-12-09T07:47:35.646691471Z caller=memberlist_client.go:380 msg="Using memberlist cluster node name" name=cortex-compactor-0-69c8834b
level=info ts=2021-12-09T07:47:35.662088968Z caller=lifecycler.go:558 msg="instance not found in ring, adding with no tokens" ring=compactor
level=info ts=2021-12-09T07:47:35.662756469Z caller=lifecycler.go:405 msg="auto-joining cluster after timeout" ring=compactor
level=info ts=2021-12-09T07:47:35.661938217Z caller=ring.go:271 msg="ring doesn't exist in KV store yet"
level=info ts=2021-12-09T07:47:35.663072363Z caller=compactor.go:395 component=compactor msg="waiting until compactor is ACTIVE in the ring"
level=info ts=2021-12-09T07:47:35.696464994Z caller=memberlist_client.go:492 msg="joined memberlist cluster" reached_nodes=3
level=info ts=2021-12-09T07:47:36.88865993Z caller=compactor.go:399 component=compactor msg="compactor is ACTIVE in the ring"
level=info ts=2021-12-09T07:47:36.888742298Z caller=compactor.go:409 component=compactor msg="waiting until compactor ring topology is stable" min_waiting=1m0s max_waiting=5m0s
level=info ts=2021-12-09T07:48:36.889141066Z caller=compactor.go:413 component=compactor msg="compactor is ring topology is stable"
level=info ts=2021-12-09T07:48:36.889232102Z caller=blocks_cleaner.go:139 component=cleaner msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:34.508642556Z caller=blocks_cleaner.go:294 component=cleaner org_id=9vnttgqwb4 msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:34.508656305Z caller=blocks_cleaner.go:294 component=cleaner org_id=b6drbdbn2t msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:34.508700414Z caller=blocks_cleaner.go:294 component=cleaner org_id=kwjkvtqvlg msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:34.508713601Z caller=blocks_cleaner.go:294 component=cleaner org_id=nn267rkbrx msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:34.50872993Z caller=blocks_cleaner.go:294 component=cleaner org_id=p4b2pm4fwm msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:34.509492673Z caller=blocks_cleaner.go:294 component=cleaner org_id=582cm447qx msg="started blocks cleanup and maintenance"
level=info ts=2021-12-09T07:50:37.582499605Z caller=blocks_cleaner.go:308 component=cleaner org_id=9vnttgqwb4 msg="migrated block deletion marks to the global markers location"
level=info ts=2021-12-09T07:57:16.363953204Z caller=blocks_cleaner.go:299 component=cleaner org_id=9vnttgqwb4 msg="completed blocks cleanup and maintenance" duration=6m41.855311606s
level=info ts=2021-12-09T08:00:28.734627462Z caller=blocks_cleaner.go:308 component=cleaner org_id=582cm447qx msg="migrated block deletion marks to the global markers location"
level=info ts=2021-12-09T08:05:51.333085248Z caller=blocks_cleaner.go:299 component=cleaner org_id=582cm447qx msg="completed blocks cleanup and maintenance" duration=15m16.823591791s
level=info ts=2021-12-09T08:20:22.674446594Z caller=signals.go:55 msg="=== received SIGINT/SIGTERM ===\n*** exiting"

Kindly advice.