Kubernetes Provider Design

[ghost]

Goal

To create a provider for Kubernetes which can discover and scan assets on a cluster.

Types of Assets

In Kubernetes there are a number of different assets which could be discovered and scanned for issues. To start with Containers and Container Images will be the supported assets, something like a "Pod" could be considered an asset in the future but it is a logical/virtual asset so we'd need asset relationships as a prerequisite for this. For now the "Pod" a container is part of can be considered part of its location.

Discovery

Containers

To discover running containers the provider will list all pods and then inspect their status. Within the status field there is a subfield "containerStatuses" this contains all the information about each container that is part of the pod. If there are multiple containers in a pod then all the statuses are here.

Each container status has:

• containerID this will be used as the "ID" field for the container asset.
• imageID this will be used to fill out the image part of the container asset using the digest not the tag.
• name this will be used to fill out the name part of the container asset

Each container status also has a state field which indicates if the container is running or terminated. If a container is terminated then the asset should be considered terminated too.

containerStatuses:
- containerID: containerd://8a8d582a03508f88c714dd0557f02288a3faaada58102a77282b081738dac9a6
  image: docker.io/bitnami/postgresql:14.6.0-debian-11-r31
  imageID: docker.io/bitnami/postgresql@sha256:5f00817d0f92465eeb1a823380ad43c8e49eecd261fc0f25dd44ab316e8c7b1c
  lastState: {}
  name: postgresql
  ready: true
  restartCount: 0
  started: true
  state:
    running:
      startedAt: "2023-07-31T14:56:10Z"

Container Images

The container images on the cluster can be discovered in a number of ways:

• First is the container images that the running containers (collected above) were started from, this information is available in the containerStatuses. The image IDs are available in a tagged format or sha format. For example:

  image: docker.io/bitnami/postgresql:14.6.0-debian-11-r31
  imageID: docker.io/bitnami/postgresql@sha256:5f00817d0f92465eeb1a823380ad43c8e49eecd261fc0f25dd44ab316e8c7b1c
  

• Second is from the kubernetes node API. Each node has a list of all the container images which are currently stored on that node's container runtime. This will give a more complete picture of all the possible running container images and include historic images if kubelet hasn't garbage collected them yet.

  images:
    - names:
       - ghcr.io/openclarity/freshclam-mirror@sha256:eb50c2f9ca84819656acd9a95ff1fcc7b2e225bd4a45ec633aa3124d7afc6996
       - ghcr.io/openclarity/freshclam-mirror:v0.1.0
       sizeBytes: 323711744
    - names:
       - gcr.io/eticloud/k8sec/k8s_agent@sha256:07b3d2ce6278927d906ce0b7ab51053e0eb047afa47828699404777dc7d3e70e
       - gcr.io/eticloud/k8sec/k8s_agent:811fc54070a0e826e61f67bd03dcbcaad8365aa3
      sizeBytes: 136297504
    - names:
       - docker.io/bitnami/postgresql@sha256:5f00817d0f92465eeb1a823380ad43c8e49eecd261fc0f25dd44ab316e8c7b1c
       - docker.io/bitnami/postgresql:14.6.0-debian-11-r31
  sizeBytes: 91558501
  

• names:
  • docker.io/tehsmash/vmclarity-ui@sha256:f0622bda6b047efa8a05af5f461bdd76c4be574da9814653fdcd86ca2eb43eea
  • docker.io/tehsmash/vmclarity-ui:microservices
    sizeBytes: 73699188

• A third possible source of images is to examine the configurations on the kubernetes cluster in particular Deployments, Jobs, ReplicasSet and Pods. This will potentially provide a different set of images to both the first and second image sources above as these are images which are yet to be deployed and so aren't necessarily running anywhere yet. The the images found here will have a risk of being invalid, and because the container runtime hasn't pulled them yet we will not know the hash.

I think the most complete way to find the images from the cluster is to use the nodes API. This will give us the best data to populate the assets. It gives us size in bytes, it gives us the arch from the node's arch, and os info.

Scanning

Container Images

Container image scanning should take a similar form to what we do in KubeClarity today. The VMClarity CLI accepts a container image ID as an input for some of the families (SBOM and Vulnerability) and some of the others need to be extended to support it (Malware, Misconfiguration).

The VMClarity CLI will then pull the image and if necessary unpack the image into a temporary file system mount. Once this occurs the scan will be the same as any other file system.

This can be run as a Kubernetes Job on the cluster the same as we do for KubeClarity. Similar to KubeClarity we should handle images that require an ImagePullSecret by detecting then using the image pull secret and namespace of another pod using this image in the system. The downside of this is that unless there is a pod running the image we won't be able to discover the credentials required to pull private images for scanning

Containers

Similar to the docker provider containers will be scanned by taking snapshots of their filesystem at the time the scan is run, and then that will be analysed by the VMClarity CLI the same as we do for VMs. In Kubernetes its a little more complicated to perform that it would be on a single docker daemon because the data can be distributed across multiple nodes.

This can be solved by utilising the Kubernetes persistent storage, or co-locating the scan job with the location of the original container.

Using persistent storage would allow a flow to mirror the what we do for virtual machines:

• A new persistent volume is created
• A job is created with node affinity to where the container was running with the PVC attached
• The job mounts the container runtime socket and exports the filesystem of the running container into the PVC
  • For docker this looks something like:
    • docker export
    • tar -zxvf -C /path/to/pvc
  • For containerd this looks something like: (this is currently a WIP https://github.com/containerd/nerdctl/pull/2161/files)
    • nerdctl export
• At that point the volume is successfully populated/created.
• A second job which doesn't require node affinity can then be started which mounts the PVC and runs VMClarity CLI over it.

Without persistent storage we could handle things in a single operation with no need to store the snapshot volume:

• A job is created with node affinity to where the container was running.
• The job mounts the container runtime socket, and then exports the filesystem to a tarball. Same as above.
• The job then extracts the tarball to a /tmp directory and runs the VMClarity CLI over the extracted directory.

Another alternative is to use a container registry as an intermediary:

• A job is created with node affinity to where the container was running
• The job mounts the container runtime socket, and then commits the filesystem of the running container to a new image then saves that image and pushes it to docker registry that VMClarity is hosting in the cluster (this doesn't need persistent storage as things will be short lived).
  • docker:
    • docker commit
    • docker save > image.tar
    • crane or skopeo copy image.tar vmclarity-registry/snapshot:
  • containerd:
    • nerdctl commit
    • nerdctl save > image.tar
    • crane or skopeo copy image.tar vmclarity-registry/snapshot:
• A second job is then started and scans the container image like any other container image except it'll pull from our local registry.

If we want to support the most Kubernetes clusters that we can support, developing a solution which doesn't require persistent storage is the best option IMO. The providers don't have to follow the, take a snapshot, create a volume, mount the volume logic if its possible to do it another way because the Provider API is generic.

We can safely handle different container runtimes by detecting them from the Kubernetes nodes API, and generating a different script for docker/containerd etc to handle the exporting logic. We can make this extensible.

[SukeshRondla]

In your proposed Kubernetes provider setup for asset discovery and scanning, when dealing with container images, you mentioned that images can be discovered from multiple sources such as running containers, the Kubernetes node API, and configurations within Deployments, Jobs, ReplicaSets, and Pods. Among these sources, which one would you prioritize for obtaining the most accurate and comprehensive image data, considering factors like image size, architecture, and potential variations due to deployment stages?

[ghost]

I think the most complete way to find the images from the cluster is to use the nodes API. This will give us the best data to populate the assets. It gives us size in bytes, it gives us the arch from the node's arch, and os info.

This gives us all the images from all the nodes which have actually been pulled and used. Even images which have been used previously and are no longer in use by pods.

[FrimIdan]

If the image is not used right now, what we gain from scanning it?
In cloud accounts we would like to scan all AMIs that exist?

[ghost]

If the image is not used right now, what we gain from scanning it?

The image is pulled and is stored on a node in the cluster, which means its was used, is currently in use by the the cluster, and could be started again. It also means any malicious code in the image is sat on the node's filesystem even if it not currently running. So these are the images we want to scan to provide a base line for the cluster.

In cloud accounts we would like to scan all AMIs that exist?

I expect for cloud accounts we'll scan all AMI's uploaded to the account (even if not in-use) plus any global ones which have a VM created from them (even if the VM is turned off). Cloud Accounts don't really have the same concept of "pulling an image".

[FrimIdan]

image this will be used to fill out the image part of the container asset.

Maybe better to use the imageID that holds the digest of the image?

I think the most complete way to find the images from the cluster is to use the nodes API. This will give us the best data to populate the assets. It gives us size in bytes, it gives us the arch from the node's arch, and os info.

But in that case we are missing the "pod" info, right?

The VMClarity CLI will then pull the image and if necessary unpack the image into a temporary file system mount.

I would say that for all scanners it's better to unpack the image, that way there will not be any changes to the scanners, they all still gets folder path as an input, right? (even sbom analyzer should get the unpack image path)

The job mounts the container runtime socket and exports the filesystem of the running container into the PVC

Can you elaborate more on how it is done?

A second job which doesn't require node affinity can then be started which mounts the PVC and runs VMClarity CLI over it.

Are we sure it is possible for multiple readers from the same PVC in all k8s providers?

If we want to support the most Kubernetes clusters that we can support, developing a solution which doesn't require persistent storage is the best option IMO.

+1 for that.

[FrimIdan]

In the case of images I don't think we want any pod info. The pod relationship is tied to the container that is using the image. When we add a asset called pod, then we build a relationship chain like Pod -> Container -> ContainerImage.

So we are just scanning container images that are known to the cluster, we can do the same by scanning an image registry, it is just another place to get list of images to scan, right?

We can take a shortcut like that to start with, but for the standalone CLI CI/CD use case I think that it would still be good to support container image as an input and have the VMClarity CLI scanners understand how to unpack the image if it needs too.

So the idea is for the VMClarity CLI to receive an image name as an input, pull it once and unpack it to a temp folder and set the folder as input for all families, right?
No need for each family to pull & unpack internally. Is that that idea?

There wouldn't be multiple readers, the first job which populates the volume would be finished and shut down, then the second job which does the scanning would start and mount the PVC.

ok

[ghost]

So we are just scanning container images that are known to the cluster, we can do the same by scanning an image registry, it is just another place to get list of images to scan, right?

100% we could write a Harbour or docker registry provider for VMClarity.

So the idea is for the VMClarity CLI to receive an image name as an input, pull it once and unpack it to a temp folder and set the folder as input for all families, right? No need for each family to pull & unpack internally. Is that that idea?

I think we'd optimise this with some wrapper code so that we only unpack it once, but each scanner not each family needs to understand if it can handle container image or not. For example within the SBOM family, syft and trivy know how to scan a container directly, but cyclonedx-go can only handle a file system. The input therefore should be passed all the way down and then the scanner decides:
a) I can scan that natively
b) I need to unpack that to scan it so call some common code to unpack the image to a tmp FS.
c) I can't scan that input type.

[FrimIdan]

I think we'd optimise this with some wrapper code so that we only unpack it once, but each scanner not each family needs to understand if it can handle container image or not.

Not sure I understand why, I think that all the the scanners can get a folder as an input, so why we need that the scanners that knows how to scan images will do the pull & scan themself and others will use the common code?
I think it will be more consistent if all scanners will get folder as input - we don't need to think if the scanner support image scanning or not. Also, we ideally should pull the image only once and not on each scanner that can scan it natively.

[ghost]

Scanners which understand containers natively perform different operations compared to just being given a file system for example processing the layers. Do we want to lose that capability by always pulling and unpacking the container image to a filesystem?

[FrimIdan]

That is a good point, not sure. Need to think what we are doing with the image layers information.

[FrimIdan]

The job mounts the container runtime socket, and then commits the filesystem of the running container to a new image then saves that image and pushes it to docker registry that VMClarity is hosting in the cluster

Why we need to push to a docker registry and scan it with another scanner? why don't we just commit -> save -> unpack -> scan?
Do we want to mimic the snapshot logic of the VMs scanning?

[ghost]

That is a great point, we could probably avoid a registry if we can unpack the image locally. One disadvantage here is the same the other non-PVC version which is the job doing the scan is forced to run on the same node as the container being scanned.

Do we want to mimic the snapshot logic of the VMs scanning?

I wasn't trying to mimic the VM logic in the case. I was trying to make the scan phase standard between a container image scan and a contain scan. (both are images, its just that we commit/push the image for running containers as a "prep" step)