This document describes the Kubernetes network policies deployed by Gardener into the Seed cluster.
For network policies deployed into the Shoot kube-system namespace, please see the usage section.
Network policies deployed by Gardener have names and annotations describing their purpose, so this document does only highlight a subset of the policies in detail.
The network policies in the Shoot namespace in the Seed can roughly be grouped into policies required for the control plane components and for logging & monitoring.
The network policy deny-all plays a special role. This policy denies all ingress and egress traffic from each pod in the Shoot namespace.
So per default, a pod running in the control plane cannot talk to any other pod in the whole Seed cluster.
This means the pod needs to have labels matching to appropriate network policies allowing it to talk to exactly the components required to execute its desired functionality.
This has also implications for Gardener extensions that need to deploy additional components into the Shoot's control plane.
This section highlights a selection of network policies that exist in the Shoot namespace in the Seed cluster. In general, the control plane components serve different purposes and thus need access to different pods and network ranges.
In contrast to other network policies, the policy allow-to-shoot-networks is tailored to the individual Shoot cluster,
because it is based on the network configuration in the Shoot manifest.
It allows pods with the label networking.gardener.cloud/to-shoot-networks=allowed to access pods in the Shoot pod,
service and node CIDR range. This is used by the Shoot API Server and the prometheus pods to communicate over VPN/proxy with pods in the Shoot cluster.
The policy allow-to-blocked-cidrs allows pods with the label networking.gardener.cloud/to-blocked-cidrs=allowed to access IPs that are explicitly blocked for all control planes in a Seed cluster (configurable via spec.networks.blockCIDRS).
This is used for instance to block the cloud provider's metadata service.
Another network policy to be highlighted is allow-to-seed-apiserver.
Some components need access to the Seed API Server. This can be allowed by labeling the pod with networking.gardener.cloud/to-seed-apiserver=allowed.
This policy allows exactly the IPs of the kube-apiserver of the Seed.
While all other policies have a static set of permissions (do not change during the lifecycle of the Shoot), the policy allow-to-seed-apiserver is reconciled to reflect the endpoints in the default namespace.
This is required because endpoint IPs are not necessarily stable (think of scaling the Seed API Server pods or hibernating the Seed cluster (acting as a shooted Seed) in a local development environment).
Furthermore, the following network policies exist in the Shoot namespace. These policies are the same for every Shoot control plane.
NAME POD-SELECTOR
# Pods that need to access the Shoot API server. Used by all Kubernetes control plane components.
allow-to-shoot-apiserver networking.gardener.cloud/to-shoot-apiserver=allowed
# allows access to kube-dns/core-dns pods for DNS queries
allow-to-dns networking.gardener.cloud/to-dns=allowed
# allows access to private IP address ranges
allow-to-private-networks networking.gardener.cloud/to-private-networks=allowed
# allows access to all but private IP address ranges
allow-to-public-networks networking.gardener.cloud/to-public-networks=allowed
# allows Ingress to etcd pods from the Shoot's Kubernetes API Server
allow-etcd app=etcd-statefulset,garden.sapcloud.io/role=controlplane
# used by the Shoot API server to allows ingress from pods labeled
# with'networking.gardener.cloud/to-shoot-apiserver=allowed', from Prometheus, and allows Egress to etcd pods
allow-kube-apiserver app=kubernetes,garden.sapcloud.io/role=controlplane,role=apiserver
Gardener currently introduces a logging stack based on Loki. So this section is subject to change. Please checkout the Community Meeting for more information.
These are the logging and monitoring related network policies:
NAME POD-SELECTOR
allow-from-prometheus networking.gardener.cloud/from-prometheus=allowed
allow-grafana component=grafana,garden.sapcloud.io/role=monitoring
allow-prometheus app=prometheus,garden.sapcloud.io/role=monitoring,role=monitoring
allow-to-aggregate-prometheus networking.gardener.cloud/to-aggregate-prometheus=allowed
allow-to-loki networking.gardener.cloud/to-loki=allowed
Let's take for instance a look at the network policy from-prometheus.
As part of the shoot reconciliation flow, Gardener deploys a shoot-specific Prometheus into the shoot namespace.
Each pod that should be scraped for metrics must be labeled with networking.gardener.cloud/from-prometheus=allowed to allow incoming network requests by the prometheus pod.
Most components of the Shoot cluster's control plane expose metrics and are therefore labeled appropriately.
Gardener extensions sometimes need to deploy additional components into the Shoot namespace in the Seed hosting the control plane.
For example the Gardener extension provider-aws deploys the MachineControllerManager into the Shoot namespace, that is ultimately responsible to create the VMs with the cloud provider AWS.
Every Shoot namespace in the Seed contains the network policy deny-all.
This requires a pod deployed by a Gardener extension to have labels from network policies, that exist in the Shoot namespace, that allow the required network ranges.
Additionally, extensions could also deploy their own network policies. This is used e.g by the Gardener extension provider-aws to serve Admission Webhooks for the Shoot API server that need to be reachable from within the Shoot namespace.
The pod can use an arbitrary combination of network policies.
The network policies in the garden namespace are, with a few exceptions (e.g Kubernetes control plane specific policies), the same as in the Shoot namespaces.
For your reference, these are all the deployed network policies.
NAME POD-SELECTOR
allow-fluentbit app=fluent-bit,gardener.cloud/role=logging,role=logging
allow-from-aggregate-prometheus networking.gardener.cloud/from-aggregate-prometheus=allowed
allow-to-aggregate-prometheus networking.gardener.cloud/to-aggregate-prometheus=allowed
allow-to-all-shoot-apiservers networking.gardener.cloud/to-all-shoot-apiservers=allowed
allow-to-blocked-cidrs networking.gardener.cloud/to-blocked-cidrs=allowed
allow-to-dns networking.gardener.cloud/to-dns=allowed
allow-to-loki networking.gardener.cloud/to-loki=allowed
allow-to-private-networks networking.gardener.cloud/to-private-networks=allowed
allow-to-public-networks networking.gardener.cloud/to-public-networks=allowed
allow-to-seed-apiserver networking.gardener.cloud/to-seed-apiserver=allowed
deny-all networking.gardener.cloud/to-all=disallowed
This section describes the network policies that are unique to the garden namespace.
The network policy allow-to-all-shoot-apiservers allows pods to access every Shoot API server in the Seed.
This is for instance used by the dependency watchdog to regularly check
the health of all the Shoot API servers.
Gardener deploys a central Prometheus instance in the garden namespace that fetches metrics and data from all seed cluster nodes and all seed cluster pods.
The network policies allow-to-aggregate-prometheus and allow-from-aggregate-prometheus allow traffic from and to this prometheus instance.
Worth mentioning is, that the network policy allow-to-shoot-networks does not exist in the garden namespace. This is to forbid Gardener system components to talk to workload deployed in the Shoot VPC.