Mimir Support #1221

msvechla · 2024-07-18T15:15:36Z

Adds support for optionally provisioning Rules to Grafana Mimir, see: #1192

Configuration

The following new CLI options have been added:

      --mimir-url=                      The URL to the Mimir API. If specified provisions rules via Mimir instead of Prometheus
      --mimir-basic-auth-username=""    The HTTP basic authentication username
      --mimir-basic-auth-password=""    The HTTP basic authentication password
      --mimir-write-alerting-rules      If alerting rules should be provisioned to the Mimir Ruler.

This will provision recording rules and optionally alerting rules to Grafana Mimir, when a mimir-url is specified.

Implementation

I used the mimirtool client to provision the rules. Unfortunately this required me to use a fork of the prometheus package (see comments in this PR). We might have to use an alternative here.

A finalizer has been added to handle deletion of rules, when a servicelevelobjective is deleted. The rbac permissions in the examples have been adapted accordingly.

Testing

I added new targets to the Makefile to spin up a test cluster with a minimal Mimir setup.
I'm also already using this version for testing at work and its working great so far.

Open Questions

Pyrra UI

The Pyrra UI works just fine when configuring it against the Mimir API, when the /prometheus http prefix is specified. However of course the direct links to the prometheus UI are not working, as Mimir does not come with its own UI.

In a first release we could probably just live with that, but we might want to consider to link to Grafana instead?

Please let me know your thoughts, we probably also need to update the README a bit when this is finalized.

msvechla · 2024-07-18T15:18:14Z

go.mod

 	k8s.io/utils v0.0.0-20240310230437-4693a0247e57 // indirect
 	sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
 	sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
 )
+
+replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20240704133652-fb0cb30e280c


I used the Mimir API client from mimirtool. Unfortunately Mimir uses a fork of the prometheus module, this is why this replace has been added. I could not get it to work without it.

This is probably not ideal. If there is another solution or we need to switch to our own client, please let me know

Given we use the rulefmt wrapper by Mimir and only two methods from the mimir client itself, I wonder if we can get away with not having these extra dependencies and simply write the two API calls with the stdlib HTTP library. Shouldn't be too cumbersome.

This would avoid bringing in this big dependency.

Makes sense, I added our own simple Mimir client with basic auth support in mimir/. Let me know what you think!

msvechla · 2024-08-08T14:51:08Z

@metalmatze could you take a look at this and let me know if anything needs adjustment?
Thanks a lot!

metalmatze · 2024-08-09T14:34:22Z

Hey @msvechla,
Thank you so much for sending this PR.
I think there are some things around Makefiles, etc, which I would like to change.

Shortly after you sent the PR I looked at it but dropped the ball since I'm super busy organizing PromCon after I'm done with my day job.

Hopefully I can find some more time soon. Thank you for your patience!

msvechla · 2024-08-10T17:15:26Z

No worries, I totally understand. Enjoy PromCon and let me know when you find some time to continue here!

metalmatze

Super cool.
I haven't tried running against the mimir cluster yet. I'm not exactly sure about the command's flags to run with. But the mimir cluster is up and running. 😎

Left some comments around the directory structure and whether we really need the mimir Go dependency.

This should be enough to make some next changes.

By the way, if you want to separate the finalizer changes, we could get them in even quicker. Depends on how much you want to rework this PR.

Thank you for your patience and for sending this PR!

metalmatze · 2024-08-23T16:14:51Z

Makefile

+ensure-kind-cluster: kind
+	@if ! kind get clusters | grep -q "^mimir$$"; then \
+		kind create cluster --config $(KIND_CONFIG_FILE); \
+	else \
+		echo "Cluster is already configured"; \
+	fi


While this is nice if people use kind there are many different ways of running Kubernetes these days. minikube, k3s, or simply via Docker on Mac. Let's just assume there's going to be a Kubernetes cluster available.
It also makes sense to avoid including it in the Makefile and instead include these scripts in a Mimir helper script.

I'm thinking similar to this: https://github.com/pyrra-dev/pyrra/tree/main/examples/docker-compose

Makes sense. I used kind because I could assign a fixed NodePort to easily route the traffic to the service, but I can also just add a port-forward command.

I was also torn between adding it to the Makefile vs a script, will do that!

I added examples/mimir/ which includes a README with the instructions. Let me know what you think.

metalmatze · 2024-08-23T16:31:34Z

kubernetes/controllers/servicelevelobjective.go


 	kitlog "github.com/go-kit/log"
 	"github.com/go-kit/log/level"
+	mimircli "github.com/grafana/mimir/pkg/mimirtool/client"
+	"github.com/grafana/mimir/pkg/mimirtool/rules/rwrulefmt"


Given this simply wraps Prometheus' rulefmt. Do we really need that dependency?

// RuleGroup is a list of sequentially evaluated recording and alerting rules. type RuleGroup struct { rulefmt.RuleGroup `yaml:",inline"` // RWConfigs is used by the remote write forwarding ruler RWConfigs []RemoteWriteConfig `yaml:"remote_write,omitempty"` }

metalmatze · 2024-08-23T16:33:17Z

testdata/README.md

+This will use [kind](https://kind.sigs.k8s.io/) to spin up a local kubernetes cluster and install the [Mimir Helm Chart](https://github.com/grafana/mimir/tree/main/operations/helm/charts/mimir-distributed) in a minimal configuration.
+The mimir endpoint will be available at: <http://localhost:30950>


Can you add instructions on how to run the api and kubernetes commands with which flags?
That would be super helpful!

Makes sense, I outlined it in ./examples/mimir/README.md

testdata/kind-cluster.yaml

metalmatze · 2024-08-23T16:34:25Z

testdata/mimir-values.yaml

Maybe move to the example/mimir folder that's to be created. See my comment above.

metalmatze · 2024-08-23T16:35:52Z

go.mod

 	k8s.io/utils v0.0.0-20240310230437-4693a0247e57 // indirect
 	sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
 	sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
 )
+
+replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20240704133652-fb0cb30e280c


Given we use the rulefmt wrapper by Mimir and only two methods from the mimir client itself, I wonder if we can get away with not having these extra dependencies and simply write the two API calls with the stdlib HTTP library. Shouldn't be too cumbersome.

This would avoid bringing in this big dependency.

msvechla · 2024-09-06T14:44:58Z

@metalmatze I think I incorporated most of your feedback, let me know what you think. I can also rebase the commits into logical blocks when this MR is ready

metalmatze · 2024-10-14T10:51:15Z

Amazing work! 👏
I followed the README and got it up and running for the most part. 👍
Mimir seems to have a problem with its ring. I can start another cluster and see again.

{"status":"error","errorType":"internal","error":"{\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: too many unhealthy instances in the ring\"}"}

Regardless, I'm happy to merge this now once we rebase and and trigger CI again.

Thank you for your work and your patience in the last weeks!

msvechla · 2024-10-14T11:34:10Z

Awesome, thanks! I might be able to take a quick look at whats causing this when I rebase.

Thanks for your feedback and help!

…specified

This moves the mimir test setup to the examples directory, removing the Makefile targets and describing the required steps in the examples README.

msvechla · 2024-10-14T12:04:11Z

@metalmatze I rebased the PR.

Regarding the mimir issues, I can not reproduce them. When I run the commands as specified in the README, the cluster becomes healthy just fine:

❯ helm upgrade -i mimir grafana/mimir-distributed --version=5.4.0 -n mimir --create-namespace -f ./examples/mimir/mimir-values.yaml --wait
Release "mimir" does not exist. Installing it now.
W1014 14:00:17.622366   53846 warnings.go:70] metadata.name: this is used in Pod names and hostnames, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]
NAME: mimir
LAST DEPLOYED: Mon Oct 14 14:00:14 2024
NAMESPACE: mimir
STATUS: deployed
REVISION: 1

❯ kubectl get pods -n mimir
NAME                                      READY   STATUS      RESTARTS   AGE
mimir-compactor-0                         1/1     Running     0          3m35s
mimir-distributor-7bf8b8c4cd-qwsx9        1/1     Running     0          3m35s
mimir-gateway-6668576657-s6ltv            1/1     Running     0          3m35s
mimir-ingester-0                          1/1     Running     0          3m35s
mimir-make-minio-buckets-5.0.14-drd7x     0/1     Completed   0          3m35s
mimir-minio-5d8b7d8f79-pkxrc              1/1     Running     0          3m35s
mimir-querier-c66568668-69z9x             1/1     Running     0          3m35s
mimir-query-frontend-98464565-phhf4       1/1     Running     0          3m35s
mimir-query-scheduler-85cd695988-psjfj    1/1     Running     0          3m35s
mimir-rollout-operator-7f4847889f-bxssr   1/1     Running     0          3m35s
mimir-ruler-5d587b578c-jwstb              1/1     Running     0          3m35s
mimir-store-gateway-0                     1/1     Running     0          3m35s

metalmatze · 2024-10-14T12:25:40Z

Ok. I'll try again. The cluster is healthy for me too. It just doesn't serve actual queries due to the above error.

PabloPie · 2024-10-14T15:08:59Z

@msvechla I understand making Pyrra compatible with the multi-tenancy of Mimir is outside of the scope of this PR, but should the tenant ID be at least configurable similar to what mimirtool supports? This way you can at least use different Pyrra instances for different tenants.

msvechla · 2024-10-15T08:58:13Z

@msvechla I understand making Pyrra compatible with the multi-tenancy of Mimir is outside of the scope of this PR, but should the tenant ID be at least configurable similar to what mimirtool supports? This way you can at least use different Pyrra instances for different tenants.

I can upate the PR and add another flag to specify an optional mimir tenant ID if we want to include this already here, no problem.

metalmatze · 2024-10-15T11:09:11Z

Another day, another minikube cluster.
Mimir has started just fine it seems.

NAME                                      READY   STATUS      RESTARTS   AGE
mimir-compactor-0                         1/1     Running     0          31m
mimir-distributor-7bf8b8c4cd-xzkn6        1/1     Running     0          31m
mimir-gateway-6668576657-gtkpw            1/1     Running     0          31m
mimir-ingester-0                          1/1     Running     0          31m
mimir-make-minio-buckets-5.0.14-mqzr2     0/1     Completed   0          31m
mimir-minio-5d8b7d8f79-5w8bd              1/1     Running     0          31m
mimir-querier-c66568668-5mwjn             1/1     Running     0          31m
mimir-query-frontend-98464565-flzrm       1/1     Running     0          31m
mimir-query-scheduler-85cd695988-hqzbn    1/1     Running     0          31m
mimir-rollout-operator-7f4847889f-b2nvw   1/1     Running     0          31m
mimir-ruler-5d587b578c-n47bw              1/1     Running     0          31m
mimir-store-gateway-0                     1/1     Running     0          31m

Still, I get the same error talking to http://localhost:8080/prometheus/api/v1/query_range from the Pyrra API.

metalmatze · 2024-10-15T11:12:22Z

It would be great to figure this out so that I can help maintain this part of the code in the future.
I'll try another type of Kubernetes cluster, like kind.

In any case, once we've fixed the golangci-lint errors, let's finally get this in, and then we can figure out the rest.

metalmatze · 2024-10-15T21:39:55Z

hm, I tried running the mimir integration on my MacBook now with Docker for Mac on their Kubernetes. Same error.
Is the README missing a single step to get the ring up and running?

msvechla · 2024-10-16T11:02:56Z

@metalmatze can you try:

kubectl port-forward -n mimir svc/mimir-ingester 8080:8080

and visit http://localhost:8080/ingester/ring in your browser?

There should only be a single instance in your ring.

metalmatze · 2024-10-16T20:12:46Z

Thank you for sending the instructions for the ingester ring debugging.
It seems like everything is fine on that side. The query part still doesn't work.

Let's merge this PR and I'll try to figure out what's happening next. Thankfully, being part of Prometheus team, I know many Mimir maintainers, I'm going to ask.

Thank you so much for this incredible contribution!!!

msvechla · 2024-10-17T08:15:00Z

Thanks a lot for your help, this will allow us to go back to the upstream version!

…

On Wed, 16 Oct 2024, 22:13 Matthias Loibl, ***@***.***> wrote: Merged #1221 <#1221> into main. — Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACEIRJ5ZPLAL4MW6SMMQ4LLZ33CH3AVCNFSM6AAAAABLC4UWR6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGY4DQNBXGMYDSMA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

metalmatze · 2024-10-17T20:24:44Z

Makes sense.
I'll take a look if there are some other smaller things we can get in and then cut v0.8! 🎉

msvechla commented Jul 18, 2024

View reviewed changes

msvechla mentioned this pull request Jul 18, 2024

Grafana Alerting & Mimir support #1192

Open

msvechla changed the title ~~DRAFT: Mimir Support~~ Mimir Support Aug 2, 2024

metalmatze reviewed Aug 23, 2024

View reviewed changes

msvechla added 9 commits October 14, 2024 13:55

generate mimir rulegroups from prometheus rulegroups

f34c224

draft first test writing to mimir api

243d5b0

add configuration options for mimir client

5e21998

add generic rules to mimir rule generation

68b6154

only provision rules to Mimir instead of Prometheus when MimirURL is …

976d3b3

…specified

add finalizer to remove mimir rules on deletion

21e2174

add permissions to update finalizers for SLOs

81cdd71

re-generate examples for updated rbac permissions

f460b50

add test setup and instructions

ec18294

msvechla force-pushed the mimir_support branch from a3de4a3 to 526bf28 Compare October 14, 2024 11:58

msvechla added 2 commits October 14, 2024 13:58

move mimir test setup to examples

883e238

This moves the mimir test setup to the examples directory, removing the Makefile targets and describing the required steps in the examples README.

switch to our own basic mimir client based on stdlib

7e3fb6a

msvechla force-pushed the mimir_support branch from 526bf28 to 7e3fb6a Compare October 14, 2024 11:59

fix linting issues

c7d953a

metalmatze merged commit 03758b0 into pyrra-dev:main Oct 16, 2024
9 of 10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mimir Support #1221

Mimir Support #1221

msvechla commented Jul 18, 2024 •

edited

Loading

msvechla Jul 18, 2024

metalmatze Aug 23, 2024

msvechla Sep 6, 2024

msvechla commented Aug 8, 2024

metalmatze commented Aug 9, 2024

msvechla commented Aug 10, 2024 •

edited

Loading

metalmatze left a comment

metalmatze Aug 23, 2024

msvechla Sep 6, 2024

msvechla Sep 6, 2024

metalmatze Aug 23, 2024

msvechla Sep 6, 2024

metalmatze Aug 23, 2024

msvechla Sep 6, 2024

metalmatze Aug 23, 2024

metalmatze Aug 23, 2024

msvechla commented Sep 6, 2024

metalmatze commented Oct 14, 2024

msvechla commented Oct 14, 2024

msvechla commented Oct 14, 2024

metalmatze commented Oct 14, 2024

PabloPie commented Oct 14, 2024

msvechla commented Oct 15, 2024

metalmatze commented Oct 15, 2024

metalmatze commented Oct 15, 2024

metalmatze commented Oct 15, 2024

msvechla commented Oct 16, 2024

metalmatze commented Oct 16, 2024

msvechla commented Oct 17, 2024 via email

metalmatze commented Oct 17, 2024

		This will use [kind](https://kind.sigs.k8s.io/) to spin up a local kubernetes cluster and install the [Mimir Helm Chart](https://github.com/grafana/mimir/tree/main/operations/helm/charts/mimir-distributed) in a minimal configuration.
		The mimir endpoint will be available at: <http://localhost:30950>

Mimir Support #1221

Mimir Support #1221

Conversation

msvechla commented Jul 18, 2024 • edited Loading

Configuration

Implementation

Testing

Open Questions

Pyrra UI

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msvechla commented Aug 8, 2024

metalmatze commented Aug 9, 2024

msvechla commented Aug 10, 2024 • edited Loading

metalmatze left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msvechla commented Sep 6, 2024

metalmatze commented Oct 14, 2024

msvechla commented Oct 14, 2024

msvechla commented Oct 14, 2024

metalmatze commented Oct 14, 2024

PabloPie commented Oct 14, 2024

msvechla commented Oct 15, 2024

metalmatze commented Oct 15, 2024

metalmatze commented Oct 15, 2024

metalmatze commented Oct 15, 2024

msvechla commented Oct 16, 2024

metalmatze commented Oct 16, 2024

msvechla commented Oct 17, 2024 via email

metalmatze commented Oct 17, 2024

msvechla commented Jul 18, 2024 •

edited

Loading

msvechla commented Aug 10, 2024 •

edited

Loading