Empty regex causes metrics inside pods to be not scraped

[flokli]

I deployed the agent according to https://github.com/grafana/agent/#getting-started, which essentially just uses https://github.com/grafana/agent/blob/v0.12.0/production/kubernetes/agent.yaml under the hood.

For some reason, the "empty regex" relabel_config at

agent/production/kubernetes/agent.yaml

Lines 51 to 54 in e9bd098

	- action: drop
	regex: ""
	source_labels:
	- __meta_kubernetes_pod_label_name

seems to be problematic - with it, I couldn't scrape pod workloads exporting their own metrics on a /metrics endpoint.

What's that line supposed to be doing in first place?

[rfratto]

Hi! It's looking for a label named name. Do your pods have that?

[flokli]

Is there a document describing some of the design ideas of that operator?

Helm charts to install the Operator, so we can move away from the (frankly) super hacky install scripts

Please still provide manifests people can trivially kubectl apply, refer from kustomize, or import into terraform via things like k2tf.

Regarding #439, let's move that discussion bit over there.

[rfratto]

Please still provide manifests people can trivially kubectl apply, refer from kustomize, or import into terraform via things like k2tf.

We're planning on keeping the Jsonnet configs around, at least, but I'm inclined to remove the bash scripts. Running bash scripts tend to make people uncomfortable, and it's hard to come up with sane plain YAML that would work for most users. The defaults we've been running with tend to be really expensive when you run it in a cloud environment.

As-as now, the plan is still for Operator Helm charts for the very-easy-get-started with customization, and Jsonnet for power users.

[flokli]

I assume with more work going into the grafana agent operator, this stance has shifted a bit more to "let's use the ServiceMonitor from the prometheus CRD to describe what to monitor", over more complicated custom grafana-agent configs?

[flokli]

(from #734 (comment))

[rfratto]

I assume with more work going into the grafana agent operator, this stance has shifted a bit more to "let's use the ServiceMonitor from the prometheus CRD to describe what to monitor", over more complicated custom grafana-agent configs?

Yeah, generally I feel like that has the highest chance of ensuring a uniform experience and generally just making it easier to configure.