fix: add generated target for all node IPs

[catsby]

Description

Adds a new generator / target called KubeNodes that contains the internal IP addresses of nodes in the cluster.

NOTE: I have no idea (yet) wher the docs/reference/ file changes came from. They appear to be missing on main.

Related Issue

Relates to #970 . Steps to Validate include steps to verify 970 gets fixed.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Steps to Validate

Setup and verify behavior of the target

Create a k3d cluster named uds (we use names later for adding nodes):

k3d cluster create uds

Deploy slim-dev:

uds run slim-dev

Create and deploy monitoring layer:

uds run -f ./tasks/create.yaml single-layer-callable --set LAYER=monitoring

uds run -f ./tasks/deploy.yaml single-layer-callable --set LAYER=monitoring

Create and deploy metrics-server layer:

uds run -f ./tasks/create.yaml single-layer-callable --set LAYER=metrics-server

uds run -f ./tasks/deploy.yaml single-layer-callable --set LAYER=metrics-server

Inspect the network policy for scraping of kube nodes:

kubectl describe networkpolicy allow-prometheus-stack-egress-metrics-scraping-of-kube-nodes -n monitoring

The spec: part is the relevant part, and should contain the IPs of the nodes:

Spec:
  PodSelector:     app.kubernetes.io/name=prometheus
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: <any> (traffic allowed to all ports)
    To:
      IPBlock:
        CIDR: 172.28.0.2/32
        Except:
  Policy Types: Egress

Add a node:

k3d node create extra1 --cluster uds --wait --memory 500M

Verify the internal IP of the new node:

kubectl get nodes -o custom-columns="NAME:.metadata.name,INTERNAL-IP:.status.addresses[?(@.type=='InternalIP')].address"

Re-get the netpol to verify the new ip is in the spec: block:

kubectl describe networkpolicy allow-prometheus-stack-egress-metrics-scraping-of-kube-nodes -n monitorin

Should now be something like this:

Spec:
  PodSelector:     app.kubernetes.io/name=prometheus
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: <any> (traffic allowed to all ports)
    To:
      IPBlock:
        CIDR: 172.28.0.2/32
        Except:
    To:
      IPBlock:
        CIDR: 172.28.0.4/32
        Except:
  Policy Types: Egress

Verify Prometheus can read things

Connect directly to prometheus:

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090

Visit http://localhost:9090/

Execute this expression to see all node/cpu data:

node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate

To see just info from the extra1 node:

node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{node=~"^k3d-extra.*"}

Add a new node:

k3d node create extra2 --cluster uds --wait --memory 500M

Verify the netpol updates:

kubectl describe networkpolicy allow-prometheus-stack-egress-metrics-scraping-of-kube-nodes -n monitorin

Re-execute the Prometheus query from above. It make take a few minutes for extra2 to show up though. Not sure why.

Delete a node and verify the spec updates again:

kubectl delete node k3d-extra1-0 && k3d node delete k3d-extra1-0

Re-reading the netpol should should the removal of that IP

Checklist before merging

• Test, docs, adr added or updated as needed
• Contributor Guide followed

[catsby]

Rebased with main, and I think I've addressed all the feedback

[mjnagel]

A few follow-on comments here, the code logic looks good at this point though (comments are on comments/docs). I'll deploy and validate this resolves the prometheus error, wanted to post these comments first though.