Skip to content
This repository has been archived by the owner on Nov 18, 2024. It is now read-only.

Latest commit

 

History

History
257 lines (186 loc) · 18.7 KB

BUILD_DESIGN.md

File metadata and controls

257 lines (186 loc) · 18.7 KB
Raw

Falco eBPF Probe Building Design

Falco is an open-source cloud-native runtime security tool that parses Linux system calls from the kernel and alerts when they match a user-defined set of rules.

Overview

Falco's architecture is composed of:

  • A Falco Agent: userspace daemon which processes these syscall events by:
    1. Matching on user-defined rules.
    2. Forwarding matches to user-defined outputs.
  • A Falco Driver: kernel module/eBPF probe which collects Linux kernel syscall events.
  • Falco Configuration which includes rules to match on and where to forward rule-match events.

This design document is only concerned about the building of the Falco Driver which is unique per Linux kernel.

Goals

  • Interoperable: We must be able to build Falco drivers for a wide-range of operating systems and their kernel versions (e.g. Amazon Linux 2, Google Container OS, Ubuntu, etc.).
  • Scalable: Adding support for more operating systems or kernels must not increase the maintenance and build complexity exponentially. This project must not expect regular human maintenance.

Background

Why eBPF?

The Falco Agent supports reading Linux syscall events from 3 types of Falco Drivers.

  • Kernel Module
  • eBPF Probe
    • Advantages:
    • Disadvantages:
      • Only supports Linux kernels >= 4.14.
      • Requires the --privileged container flag. In Linux kernels >= 5.8 CAP_BPF and CAP_PERFMON can be used instead.
  • Userspace instrumentation
    • Advantages:
      • Does not require access to the kernel thus can run entirely unprivileged.
    • Disadvantages:
      • Currently (as of 06/2021) no officially supported implementation.
      • Cannot see root/kernelspace events.

The listing above favours the eBPF Probe driver as all of the limitations are primarily around support for legacy kernel versions. The 4.14 Kernel was released in 11/2017 where the majority of actively supported cloud vendor provided operating systems that can be used with Kubernetes are using a Linux Kernel >= 4.14. With this in mind, eBPF's disadvantages can be considered moot for modern Kubernetes clusters (with a path for the --privileged container flag being mitigated in the future too).

Building a Falco eBPF Probe

Falco provide the following methods to build Falco Drivers:

  • A bash script falco-driver-loader which is used to build the Falco kernel module/eBPF probe at runtime.
  • driverkit which is a command-line tool in active development which FalcoSecurity use to produce their repository of Falco kernel probes.

driverkit is the de-facto way to build probes before runtime, however upon further investigation we found that it does not quite meet our needs:

  • It currently does not support Google Container-Optimized OS (cos) which is non-trivial to add because there is no package repository and Google only publish kernel sources per build of cos.
  • This project is also used to pre-build Linux Kernel Modules in their https://download.falco.org/driver repository which is pushed via this job on their CI system, prow.
    • This job currently does not build eBPF probes, which we desire.
    • This job currently rebuilds every probe and re-uploads them which results in hash changes, which does not suit our static hash verification when fetching external assets.
    • Currently, supporting newer kernel versions requires a pull-request to the repository on GitHub, e.g. falcosecurity/test-infra#419 which makes us dependent on FalcoSecurity's review processes.

With this in mind, we have favoured the falco-driver-loader method to give us broader Operating System support and attempt to resolve some of the current shortcomings of Falco's probe building infrastructure.

Reverse engineering the falco-driver-loader bash script yields the following inputs for building an eBPF probe:

Thus to build a Falco eBPF probe in Docker, we can:

  1. Build a modified falco-driver-loader image (called falco-driver-builder) that allows us to mock these values by patching the falco-driver-loader script.
  2. Obtain Kernel sources and configuration for your chosen kernel.
  3. Mock the resolved Target ID by mocking the /etc/os-release file.
  4. Mock the Kernel Release (output of uname -r) value.
  5. Mock the Kernel Version (output of uname -v) value.
  6. Mock the Kernel Machine (output of uname -m) value.
  7. Build Probe using patched falco-driver-loader script in falco-driver-builder with mocked values, Kernel sources, Kernel configuration and mocked Target ID.

This process is proved by the accompanying scripts which

  • builds a Falco eBPF probe for Amazon Linux 2 with the 4.14.232-176.381.amzn2 kernel which can be executed via:
# A list of Kernel packages for Amazon Linux 2 can be obtained by running:
# $ docker run --rm amazonlinux:2 yum --showduplicates list kernel-devel | tail -n+3 | awk '{ print $2 }'
$ bash ./docs/BUILD_DESIGN_assets/build-amazonlinux2-ebpf-probe.sh "4.14.232-176.381.amzn2"
  • builds a Falco eBPF probe for Google COS with the cos-101-17162-40-20 image which can be executed via:
$ bash ./docs/BUILD_DESIGN_assets/build-cos-ebpf-probe.sh "cos-101-17162-40-20"

Design

In Building a Falco eBPF Probe, we identified 6 inputs to the falco-driver-loader script. These are the requirements for building Falco eBPF probes for any Linux kernel. In order to support additional Operating Systems and their kernels in this project, we can define these 6 required inputs as functions within an Interface to provide a layer of abstraction between different kernels.

// KernelPackage abstracts the implementation of resolving the required inputs
// for building a Falco eBPF probe per Kernel Package.
// The outputs of are not guaranteed to be unique, see "Operating Systems without Package Managers"
// below for an explanation.
// Note: This interface is provided as an example.
// It will be different in the implementation of this design
// to include scope such as error handling.
type KernelPackage interface {
    // GetKernelRelease returns the value to mock as the output of `uname -r`.
    GetKernelRelease() string
    // GetKernelVersion returns the value to mock as the output of `uname -v`.
    GetKernelVersion() string
    // GetKernelMachine returns the value to mock as the output of `uname -m`.
    GetKernelMachine() string
    // GetOSRelease returns the file contents to use as the mock of `/etc/os-release`.
    GetOSRelease() FileContents
    // GetKernelConfiguration returns the volume to mount as `/host/lib/modules/`.
    GetKernelConfiguration() Volume
    // GetKernelSources returns the volume to mount as `/usr/src/`.
    GetKernelSources() Volume
}

// e.g. for the `4.14.232-176.381.amzn2 kernel` on `Amazon Linux 2`:
// GetKernelRelease() returns "4.14.232-176.381.amzn2.x86_64".
// GetKernelVersion() returns "#1 SMP Wed May 19 00:31:54 UTC 2021".
// GetKernelMachine() returns "x86_64".
// GetOSRelease() returns the contents /etc/os-release file from the Amazon Linux 2 docker image.
// GetKernelConfiguration() returns the volume with `/lib/modules/` for the kernel after running `yum install -y kernel-...`.
// GetKernelSources() returns the volume with `/usr/src/` for the kernel after running `yum install -y kernel-...`.

The above interface does not cover step 2 of the Building a Falco eBPF Probe process i.e. 2. Obtain Kernel sources and configuration for your chosen kernel. For the 4.14.232-176.381.amzn2 kernel on Amazon Linux 2, we obtained these by running the yum -y install "kernel-devel-$KERNEL_PACKAGE" "kernel-$KERNEL_PACKAGE" command. This command utilises the yum package manager which is specific to RHEL and its children of which Amazon Linux 2 is one of. However, this command doesn't work in other Operating Systems such as Ubuntu Linux or Google Container OS and thus needs abstraction.

Additionally, this project aims to build Falco eBPF Probes for Kernel Packages from different Operating Systems (i.e. Interoperability). To meet this goal without increasing maintenance complexity, we can programmatically retrieve a list of Kernel Packages for an Operating System. In the Amazon Linux 2 example, we achieved this by running the yum --showduplicates list kernel-devel command. Again, this command does not work in other Operating Systems such as Ubuntu Linux or Google Container OS and thus needs abstracting as well.

The Interface below abstracts these 2 Operating System requirements as 2 functions.

// OperatingSystem abstracts the implementation of determining which
// kernel packages are available and the retrieval of them.
// Note: This interface is provided as an example.
// It will be different in the implementation of this design
// to include scope such as error handling.
type OperatingSystem interface {
    // GetKernelPackageNames returns a list of all available Kernel Package names.
    GetKernelPackageNames() []string
    // GetKernalPackageByName returns a hydrated KernelPackage for the given Kernel Package name.
    GetKernelPackageByName(name string) KernelPackage
}

// e.g. for the `4.14.232-176.381.amzn2 kernel` on `Amazon Linux 2`:
// GetKernelPackageNames() returns []string{"4.14.232-176.381.amzn2", ...}.
// GetKernelPackageByName("4.14.232-176.381.amzn2") returns the example KernelPackage above.

Note: hydrated means that the values are retrieved, i.e. the GetKernelPackageByName function performs the fetching of Kernel Sources, Kernel Configuration, etc.

Operating Systems without Package Managers

Not all Operating Systems feature a Package Manager (e.g. yum, apt, pacman, etc.) thus KernelPackage may be seen as misnamed in the context of those Operating Systems. An example of this is Google Container OS which features security measures such as immutable filesystems. In order to fit these types of Operating Systems, we can use their Image Name (an amalgamation of Milestone and Build Number) as a KernelPackage where multiple KernelPackages may output the same Falco eBPF Probe in the likely event that images share Kernels.

Building Falco eBPF Probes at Scale

Now that we have our Interfaces which abstract the Operating Systems and their Kernels (Interoperability), we need to design our implementation for using these to build Falco eBPF probes at Scale.

The below binaries are currently separated to enforce separation of concerns, but it is plausible that these may be merged into a single binary in a future maturity.

A note on DRIVER_VERSIONs

In the falco-driver-loader script, Falco appear to organize their driver repository by a DRIVER_VERSION variable which implies that Falco probes will only be compatible with a specific DRIVER_VERSION.

With this in mind, we should consider which DRIVER_VERSIONs to actively support and build new kernel probes for. Due to the young nature of both Falco and this project, for now we will endeavour to at least support DRIVER_VERSIONs which are compatible with the latest version of Falco and may drop support for DRIVER_VERSIONs at any given time. In the future, once both projects are more mature we may commit to a policy of producing probes for the last n versions of Falco.

The DRIVER_VERSION gets hardcoded into the falco-driver-loader script in the Docker Image. We can obtain its value by performing:

$ docker run --rm --entrypoint="" docker.io/falcosecurity/falco-driver-loader:$FALCO_VERSION grep -w "^DRIVER_VERSION" /usr/bin/falco-driver-loader | cut -f2 -d\"
# e.g. 
# $ docker run --rm --entrypoint="" docker.io/falcosecurity/falco-driver-loader:0.28.1 grep -w "^DRIVER_VERSION" /usr/bin/falco-driver-loader | cut -f2 -d\"
# 5c0b863ddade7a45568c0ac97d037422c9efb750

//cmd/build-falco-ebpf-probe

In Building a Falco eBPF Probe, we used Bash to orchestrate the building of an eBPF probe via Docker. Bash can be replaced by a Go binary which utilises the above Interfaces to perform the build steps via the Docker SDK.

$ plz run //cmd/build-falco-ebpf-probe -- --falco_version=<falco-version> <operating-system> <kernel-package-name>
# $ plz run //cmd/build-falco-ebpf-probe -- --falco_version=0.28.1 amazonlinux2 4.14.232-176.381.amzn2
# output file: dist/falco-probes/<falco-driver-version>/<built-probe>
# e.g. `dist/falco-probes/5c0b863ddade7a45568c0ac97d037422c9efb750/falco_amazonlinux2_4.14.181-142.260.amzn2.x86_64_1.o`

//cmd/list-kernel-packages

We will also require a Go binary which can list the available Kernel Packages for a given Operating System.

$ plz run //cmd/list-kernel-packages -- <operating-system>
# $ plz run //cmd/list-kernel-packages -- amazonlinux2
# 4.14.232-176.381.amzn2
# ...

This binary's output will be used in CI/CD to build Falco eBPF probes for all available kernels.

//cmd/is-falco-ebpf-probe-uploaded

This Go binary will be used in CI/CD to determine whether or not a Falco eBPF probe has already been uploaded to our Probe Repository. The implementation of this depends on docs/REPOSITORY_DESIGN.md.

$ plz run //cmd/is-falco-ebpf-probe-uploaded -- --falco_version=<falco-version> <operating-system> <kernel-package-name>
# $ plz run //cmd/is-falco-ebpf-probe-uploaded -- --falco_version=0.28.1 amazonlinux2 4.14.232-176.381.amzn2
# (exit 0 - probe exists in repository)
# (exit 1 - probe does not exist in repository)

Volumes

In the KernelPackage Interface, we have referenced a Volume datatype. This is used to abstract from the implementation of different file storage mechanisms. For the first implementation of this design, we recommend that Docker Volumes are used.

GitHub Actions

GitHub Actions offers us a "free" and transparent way for us to build our eBPF probes as well as integrate with the GitHub Repository directly. There is an undocumented parallel worker limit of 256 on GitHub Actions, which means that we cannot build all available Kernels in parallel which also results in a very noisy GitHub Actions UI as there would be a "Job" for each Kernel.

Instead, we suggest that we use a GitHub Actions worker per Operating System, which falls well within our scaling needs as jobs can run for up to 2 hours.

In order to automatically build new eBPF probes, we propose to initially run these tools on a nightly cron-job. We will only build Falco eBPF probes for Kernels which do not already exist in the Probe Repository, so this will be quiet after the initial builds that populate the Probe Repository.

Future Considerations

This design aims to be representative of what a 1st iteration of maturity for this project could look like. There are certainly further improvements that can be made and should be considered in future maturities.

Operating System Implementation Specific Optimisations

  • Fetching Kernel Sources and Configuration: Currently, we've demonstrated the use of Operating System Package Managers to fetch Kernel Sources and Configuration which is inefficient as the package managers fetch additional packages. We could improve this by directly fetching from the repositories via HTTP.
  • Listing Kernel Packages: Currently, we've demonstrated the use of Operating System Package Managers to list available Kernel Packages which is inefficient as it requires running a Docker command to do so. We could improve this by directly fetching from the repositories via HTTP.

Building Entirely in Please without Docker

Currently, we're advising to build entirely in Docker via the docker.io/falcosecurity/falco-driver-loader base image which comes with all the build dependencies required. However, this requires running and depending on Docker which can be inefficient, where building entirely with Please would be much cleaner.

Automated verification of built eBPF Probes

It's possible that our build process may produce incompatible probes, we could build an E2E style test which tests our built eBPF probes against their respective kernels.

Supply-chain security of built eBPF Probes

As we're building eBPF programs which run inside the Linux Kernel, it is desirable for us to provide a way for consumers of this project to verify that the probe they have downloaded was built by us.