Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add generated target for all node IPs #1119

Merged
merged 12 commits into from
Dec 20, 2024
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ tableOfContents:
</tr>
</thead>
<tbody>
<tr><td style="white-space: nowrap;">description</td><td style="white-space: nowrap;">string</td><td>A description of the policy, this will become part of the policy name</td></tr><tr><td style="white-space: nowrap;">direction</td><td style="white-space: nowrap;">string (enum):<ul><li><code>Ingress</code></li><li><code>Egress</code></li></ul></td><td>The direction of the traffic</td></tr><tr><td style="white-space: nowrap;">labels</td><td style="white-space: nowrap;"></td><td>The labels to apply to the policy</td></tr><tr><td style="white-space: nowrap;">podLabels</td><td style="white-space: nowrap;"></td><td>Deprecated: use selector</td></tr><tr><td style="white-space: nowrap;">port</td><td style="white-space: nowrap;">number</td><td>The port to allow (protocol is always TCP)</td></tr><tr><td style="white-space: nowrap;">ports</td><td style="white-space: nowrap;">number[]</td><td>A list of ports to allow (protocol is always TCP)</td></tr><tr><td style="white-space: nowrap;">remoteCidr</td><td style="white-space: nowrap;">string</td><td>Custom generated policy CIDR</td></tr><tr><td style="white-space: nowrap;">remoteGenerated</td><td style="white-space: nowrap;">string (enum):<ul><li><code>KubeAPI</code></li><li><code>IntraNamespace</code></li><li><code>CloudMetadata</code></li><li><code>Anywhere</code></li></ul></td><td>Custom generated remote selector for the policy</td></tr><tr><td style="white-space: nowrap;">remoteNamespace</td><td style="white-space: nowrap;">string</td><td>The remote namespace to allow traffic to/from. Use * or empty string to allow all namespaces</td></tr><tr><td style="white-space: nowrap;">remotePodLabels</td><td style="white-space: nowrap;"></td><td>Deprecated: use remoteSelector</td></tr><tr><td style="white-space: nowrap;">remoteSelector</td><td style="white-space: nowrap;"></td><td>The remote pod selector labels to allow traffic to/from</td></tr><tr><td style="white-space: nowrap;">selector</td><td style="white-space: nowrap;"></td><td>Labels to match pods in the namespace to apply the policy to. Leave empty to apply to all pods in the namespace</td></tr>
<tr><td style="white-space: nowrap;">description</td><td style="white-space: nowrap;">string</td><td>A description of the policy, this will become part of the policy name</td></tr><tr><td style="white-space: nowrap;">direction</td><td style="white-space: nowrap;">string (enum):<ul><li><code>Ingress</code></li><li><code>Egress</code></li></ul></td><td>The direction of the traffic</td></tr><tr><td style="white-space: nowrap;">labels</td><td style="white-space: nowrap;"></td><td>The labels to apply to the policy</td></tr><tr><td style="white-space: nowrap;">podLabels</td><td style="white-space: nowrap;"></td><td>Deprecated: use selector</td></tr><tr><td style="white-space: nowrap;">port</td><td style="white-space: nowrap;">number</td><td>The port to allow (protocol is always TCP)</td></tr><tr><td style="white-space: nowrap;">ports</td><td style="white-space: nowrap;">number[]</td><td>A list of ports to allow (protocol is always TCP)</td></tr><tr><td style="white-space: nowrap;">remoteCidr</td><td style="white-space: nowrap;">string</td><td>Custom generated policy CIDR</td></tr><tr><td style="white-space: nowrap;">remoteGenerated</td><td style="white-space: nowrap;">string (enum):<ul><li><code>KubeAPI</code></li><li><code>KubeNodes</code></li><li><code>IntraNamespace</code></li><li><code>CloudMetadata</code></li><li><code>Anywhere</code></li></ul></td><td>Custom generated remote selector for the policy</td></tr><tr><td style="white-space: nowrap;">remoteNamespace</td><td style="white-space: nowrap;">string</td><td>The remote namespace to allow traffic to/from. Use * or empty string to allow all namespaces</td></tr><tr><td style="white-space: nowrap;">remotePodLabels</td><td style="white-space: nowrap;"></td><td>Deprecated: use remoteSelector</td></tr><tr><td style="white-space: nowrap;">remoteSelector</td><td style="white-space: nowrap;"></td><td>The remote pod selector labels to allow traffic to/from</td></tr><tr><td style="white-space: nowrap;">selector</td><td style="white-space: nowrap;"></td><td>Labels to match pods in the namespace to apply the policy to. Leave empty to apply to all pods in the namespace</td></tr>
</tbody>
</table>
</div>
Expand Down
19 changes: 19 additions & 0 deletions docs/reference/configuration/uds-networking-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,25 @@ This configuration directs the operator to use the specified CIDR range (`172.0.

When configuring a static CIDR range, it is important to make the range as restrictive as possible to limit the potential for unexpected networking access. An overly broad range could inadvertently allow egress traffic to destinations beyond the intended scope. Additionally, careful alignment with the actual IP addresses used by the Kubernetes API server is essential. A mismatch between the specified CIDR range and the cluster's configuration can result in network policy enforcement issues or disrupted connectivity.

## KubeNodes CIDRs

The UDS operator is responsible for dynamically updating network policies that use the `remoteGenerated: KubeNodes` custom selector, in response to changes to nodes in the Kubernetes cluster. As nodes are added, updated, or removed from a cluster, the operator will ensure that policies remain accurate and include all the nodes in the cluster.

UDS operator provides an option to configure a set of static CIDR ranges in place of offering a dynamically updated list by setting an override to `operator.KUBENODE_CIDRS` in your bundle as a value or variable. The value should be a single string of comma (`,`) separated values for the individual IP addresses, using `/32` notation. For example:

```yaml
packages:
- name: uds-core
repository: ghcr.io/defenseunicorns/packages/uds/core
ref: x.x.x
overrides:
uds-operator-config:
uds-operator-config:
values:
- path: operator.KUBENODE_CIDRS
value: "172.28.0.2/32,172.28.0.3/32,172.28.0.4/32"
```

## Additional Network Allowances

Applications deployed in UDS Core utilize [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) with a "Deny by Default" configuration to ensure network traffic is restricted to only what is necessary. Some applications in UDS Core allow for overrides to accommodate environment-specific requirements.
Expand Down
3 changes: 3 additions & 0 deletions src/pepr/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ export const UDSConfig = {
// Static CIDR range to use for KubeAPI instead of k8s watch
kubeApiCidr: process.env.KUBEAPI_CIDR,

// Static CIDRs to use for KubeNodes instead of k8s watch. Comma separated list of CIDRs.
kubeNodeCidrs: process.env.KUBENODE_CIDRS,

// Track if UDS Core identity-authorization layer is deployed
isIdentityDeployed: false,
};
Expand Down
7 changes: 6 additions & 1 deletion src/pepr/operator/controllers/network/generate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { anywhere, anywhereInCluster } from "./generators/anywhere";
import { cloudMetadata } from "./generators/cloudMetadata";
import { intraNamespace } from "./generators/intraNamespace";
import { kubeAPI } from "./generators/kubeAPI";
import { kubeNodes } from "./generators/kubeNodes";
import { remoteCidr } from "./generators/remoteCidr";

function isWildcardNamespace(namespace: string) {
Expand All @@ -26,6 +27,10 @@ function getPeers(policy: Allow): V1NetworkPolicyPeer[] {
peers = kubeAPI();
break;

case RemoteGenerated.KubeNodes:
peers = kubeNodes();
break;

case RemoteGenerated.CloudMetadata:
peers = cloudMetadata;
break;
Expand Down Expand Up @@ -93,7 +98,7 @@ export function generate(namespace: string, policy: Allow): kind.NetworkPolicy {
};
}

// Add the generated policy label (used to track KubeAPI policies)
// Add the generated policy label (used to track KubeAPI and KubeNodes policies)
if (policy.remoteGenerated) {
generated.metadata!.labels!["uds/generated"] = policy.remoteGenerated;
}
Expand Down
218 changes: 218 additions & 0 deletions src/pepr/operator/controllers/network/generators/kubeNodes.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
/**
* Copyright 2024 Defense Unicorns
* SPDX-License-Identifier: AGPL-3.0-or-later OR LicenseRef-Defense-Unicorns-Commercial
*/

import { beforeEach, beforeAll, describe, expect, it, jest } from "@jest/globals";

import {
initAllNodesTarget,
kubeNodes,
updateKubeNodesFromCreateUpdate,
updateKubeNodesFromDelete,
} from "./kubeNodes";
import { K8s, kind } from "pepr";
import { V1NetworkPolicyList } from "@kubernetes/client-node";
import { anywhere } from "./anywhere";

type KubernetesList<T> = {
items: T[];
};

jest.mock("pepr", () => {
const originalModule = jest.requireActual("pepr") as object;
return {
...originalModule,
K8s: jest.fn(),
kind: {
Node: "Node",
NetworkPolicy: "NetworkPolicy",
},
};
});

describe("kubeNodes module", () => {
const mockNodeList = {
items: [
{
metadata: { name: "node1" },
status: {
addresses: [{ type: "InternalIP", address: "10.0.0.1" }],
conditions: [{ type: "Ready", status: "True" }],
},
},
{
metadata: { name: "node2" },
status: {
addresses: [{ type: "InternalIP", address: "10.0.0.2" }],
conditions: [{ type: "Ready", status: "True" }],
},
},
],
};

const mockNetworkPolicyList: V1NetworkPolicyList = {
apiVersion: "networking.k8s.io/v1",
kind: "NetworkPolicyList",
items: [
{
apiVersion: "networking.k8s.io/v1",
kind: "NetworkPolicy",
metadata: {
name: "example-policy",
namespace: "default",
},
spec: {
podSelector: {}, // required field
policyTypes: ["Egress"], // or ["Ingress"], or both
egress: [
{
to: [{ ipBlock: { cidr: "0.0.0.0/0" } }], // an IP we don't want
},
],
},
},
],
};

const mockK8sGetNodes = jest.fn<() => Promise<KubernetesList<kind.Node>>>();
const mockGetNetworkPolicies = jest.fn<() => Promise<KubernetesList<kind.NetworkPolicy>>>();
const mockApply = jest.fn();

beforeAll(() => {
(K8s as jest.Mock).mockImplementation(() => ({
Get: mockK8sGetNodes,
WithLabel: jest.fn(() => ({
Get: mockGetNetworkPolicies,
})),
Apply: mockApply,
}));
});

beforeEach(() => {
jest.clearAllMocks();
});

describe("initAllNodesTarget", () => {
it("should initialize nodeSet with internal IPs from nodes", async () => {
mockK8sGetNodes.mockResolvedValue(mockNodeList);
await initAllNodesTarget();
const cidrs = kubeNodes();
// Should have two IPs from mockNodeList
expect(cidrs).toHaveLength(2);
expect(cidrs).toEqual(
expect.arrayContaining([
{ ipBlock: { cidr: "10.0.0.1/32" } },
{ ipBlock: { cidr: "10.0.0.2/32" } },
]),
);
});
});

describe("nodeCIDRs", () => {
it("should return anywhere if no nodes known", async () => {
mockK8sGetNodes.mockResolvedValue({ items: [] });
await initAllNodesTarget();
const cidrs = kubeNodes();
// expect it to match "anywhere"
expect(cidrs).toEqual([anywhere]);
});
});

describe("updateKubeNodesFromCreateUpdate", () => {
it("should add a node IP if node is ready", async () => {
mockK8sGetNodes.mockResolvedValueOnce({ items: [] });
mockGetNetworkPolicies.mockResolvedValue(mockNetworkPolicyList);
await initAllNodesTarget(); // start empty
await updateKubeNodesFromCreateUpdate(mockNodeList.items[0]);
let cidrs = kubeNodes();
expect(cidrs).toHaveLength(1);
expect(cidrs[0].ipBlock?.cidr).toBe("10.0.0.1/32");
expect(mockApply).toHaveBeenCalled();

await updateKubeNodesFromCreateUpdate(mockNodeList.items[1]);
cidrs = kubeNodes();
expect(cidrs).toHaveLength(2);
expect(cidrs[1].ipBlock?.cidr).toBe("10.0.0.2/32");
expect(mockApply).toHaveBeenCalled();
});

it("should not remove a node that's no longer ready", async () => {
mockK8sGetNodes.mockResolvedValue(mockNodeList);
await initAllNodesTarget();
let cidrs = kubeNodes();
// Should have two IPs from mockNodeList
expect(cidrs).toHaveLength(2);
expect(cidrs).toEqual(
expect.arrayContaining([
{ ipBlock: { cidr: "10.0.0.1/32" } },
{ ipBlock: { cidr: "10.0.0.2/32" } },
]),
);

const notReadyNode = {
metadata: { name: "node2" },
status: {
addresses: [{ type: "InternalIP", address: "10.0.0.1" }],
conditions: [{ type: "Ready", status: "False" }],
},
};
await updateKubeNodesFromCreateUpdate(notReadyNode);
cidrs = kubeNodes();
expect(cidrs).toHaveLength(2);
expect(cidrs).toEqual(
expect.arrayContaining([
{ ipBlock: { cidr: "10.0.0.1/32" } },
{ ipBlock: { cidr: "10.0.0.2/32" } },
]),
);
});

it("should not apply netpol policy changes if a node is already included", async () => {
// setup 1 node in the set and expect 1 application to a policy
mockK8sGetNodes.mockResolvedValueOnce({ items: [] });
mockGetNetworkPolicies.mockResolvedValue(mockNetworkPolicyList);
await initAllNodesTarget(); // start empty
// add a node even if it's not ready
const initialNode = {
metadata: { name: "node1" },
status: {
addresses: [{ type: "InternalIP", address: "10.0.0.9" }],
conditions: [{ type: "Ready", status: "False" }],
},
};
await updateKubeNodesFromCreateUpdate(initialNode);
let cidrs = kubeNodes();
expect(cidrs).toHaveLength(1);
expect(cidrs[0].ipBlock?.cidr).toBe("10.0.0.9/32");
expect(mockApply).toHaveBeenCalled();

// clear out the apply from the setup
mockApply.mockClear();
// change initialNode to set the status to ready
initialNode.status.conditions[0].status = "True";
await updateKubeNodesFromCreateUpdate(initialNode);
cidrs = kubeNodes();
expect(cidrs).toHaveLength(1);
expect(cidrs[0].ipBlock?.cidr).toBe("10.0.0.9/32");

// the apply should not have been called
expect(mockApply).not.toHaveBeenCalled();
});
});

describe("updateKubeNodesFromDelete", () => {
it("should remove the node IP from nodeSet", async () => {
mockK8sGetNodes.mockResolvedValueOnce(mockNodeList);
await initAllNodesTarget();
const cidrsBeforeDelete = kubeNodes();
expect(cidrsBeforeDelete).toHaveLength(2);

await updateKubeNodesFromDelete(mockNodeList.items[0]);
const cidrsAfterDelete = kubeNodes();
expect(cidrsAfterDelete).toHaveLength(1);
expect(cidrsAfterDelete[0].ipBlock?.cidr).toBe("10.0.0.2/32");
expect(mockApply).toHaveBeenCalled();
});
});
});
Loading
Loading