WIP: Add more testing + memory stats from cgroups #10
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,4 @@ | ||
| /systemd_exporter | ||
| /bin/golangci-lint | ||
| .idea | ||
| coverage.txt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| issues: | ||
| exclude: | ||
| - "not declared by package utf8" | ||
| - "unicode/utf8/utf8.go" | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,36 @@ | ||
| # TODO Use CodeCov 'Flags' to separate coverage for integration tests and unit | ||
| # tests. Near 100% on integration is expected - these tests cast a wide net to | ||
| # catch many issues (but do not easily tell you where the issue is). Near 100% | ||
| # on unit tests is a feat of heroism - these tests identify a specific code | ||
| # chunk with an issue. We mainly care about 100% on unit tests, but 100% on | ||
| # integration is an easy win and a nice to have | ||
| # | ||
| # See https://docs.codecov.io/docs/flags | ||
| # Use go get github.com/stristr/go-acc && go-acc ./... | ||
| # Or use coverpkg=github.com/povilasv/systemd_exporter,github.com/povilasv/systemd_exporter/systemd | ||
|
|
||
| # This defines the script for us automatically. By default it installs | ||
| # to requested go version then runs make | ||
| language: go | ||
| go: | ||
| - "1.x" | ||
|
|
||
| before_script: systemd --version | ||
| os: linux | ||
|
|
||
| go: | ||
| - 1.x | ||
|
|
||
| before_script: systemd --version && systemctl list-units | ||
|
|
||
| after_success: | ||
| - bash <(curl -s https://codecov.io/bash) | ||
|
|
||
| jobs: | ||
| include: | ||
| - dist: xenial | ||
| name: xenial-229 | ||
| - dist: bionic | ||
| name: bionic-237 | ||
|
|
||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
|
|
||
| This package provides functions to retrieve control group metrics from the pseudo-filesystem `/sys/cgroup/`. | ||
|
|
||
| **WARNING:** This package is a work in progress. Its API may still break in backwards-incompatible ways without warnings. Use it at your own risk. | ||
|
|
||
| The Linux kernel supports two APIs for userspace to interact with control groups, the v1 API and the v2 API. See | ||
| [this LWN Article](https://lwn.net/Articles/679786/) or | ||
| [this kernel documentation](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#deprecated-v1-core-features) | ||
| for background on the two APIs. This package will interact with both v1 and v2 APIs. | ||
|
|
||
|
|
||
| ### Focus on Systemd | ||
|
|
||
| This package is initially focused on reading metrics for systemd units. Therefore, | ||
| the following systemd documentation is relevant. | ||
|
|
||
| #### Systemd cgroup mount mode | ||
|
|
||
| The kernel can mount the cgroupfs in any manner it chooses. However, anyone wanting to use that cgroupfs must know | ||
| where/how it is mounted. When there was only one cgroup API, it was always mounted at `/sys/fs/cgroup`. With the | ||
| transition from v1 to v2, the mounting approach differs per-distro, with some mounting only v2, some mounting only | ||
| v1(all hierarchies), and some mounting a combination. For simplicity, this package initially focuses on the three | ||
| mount "modes" supported by systemd: | ||
|
|
||
| via [systemd.io](https://systemd.io/CGROUP_DELEGATION/#three-different-tree-setups-) | ||
|
|
||
| 1. Unified — this is the simplest mode, and exposes a pure cgroup v2 logic | ||
| 2. Legacy — this is the traditional cgroup v1 mode. In this mode the various controllers each get their own cgroup | ||
| file system mounted to `/sys/fs/cgroup/<controller>/` | ||
| 3. Hybrid — this is a hybrid between the unified and legacy mode. It’s set up mostly like legacy | ||
|
|
||
| #### Systemd Supported Controllers | ||
|
|
||
| The initial target controllers this package aims to read from are the controllers supported by systemd. Reading from | ||
| other controllers may be supported in the future. Systemd guarantees that all v1 hierarchies are kept in sync. | ||
|
|
||
| Via [systemd.io](https://systemd.io/CGROUP_DELEGATION/#controller-support): | ||
|
|
||
| Systemd supports a number of controllers (but not all). Specifically, supported are: | ||
|
|
||
| on cgroup v1: cpu, cpuacct, blkio, memory, devices, pids | ||
| on cgroup v2: cpu, io, memory, pids | ||
|
|
||
| It is our intention to natively support all cgroup v2 controllers as they are added | ||
| to the kernel. However, regarding cgroup v1: at this point we will not add support | ||
| for any other controllers anymore. This means systemd currently does not and will | ||
| never manage the following controllers on cgroup v1: freezer, cpuset, net_cls, | ||
| perf_event, net_prio, hugetlb | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| package cgroup | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "github.com/pkg/errors" | ||
| "github.com/prometheus/common/log" | ||
| "golang.org/x/sys/unix" | ||
| "os" | ||
| "path/filepath" | ||
| ) | ||
|
|
||
| // FS is the pseudo-filesystem cgroupfs, which provides an interface to | ||
| // kernel data structures | ||
| type FS struct { | ||
| mountPoint string | ||
|
|
||
| // WARNING: We only read this data once at process start, systemd updates | ||
| // may require restarting systemd-exporter | ||
| cgroupUnified MountMode | ||
| } | ||
|
|
||
| // DefaultMountPoint is the common mount point of the cgroupfs filesystem | ||
| const DefaultMountPoint = "/sys/fs/cgroup" | ||
|
|
||
| // NewDefaultFS returns a new cgroup FS mounted under the default mountPoint. | ||
| // It will error if cgroup hierarchies are not laid out in a manner understood | ||
| // by systemd. | ||
| func NewDefaultFS() (FS, error) { | ||
|
|
||
| mode, err := cgUnifiedCached() | ||
| if err != nil || mode == MountModeUnknown { | ||
| return FS{}, fmt.Errorf("could not determine cgroupfs mount mode: %s", err) | ||
| } | ||
|
|
||
| return NewFS(DefaultMountPoint, mode) | ||
| } | ||
|
|
||
| // NewFS returns a new cgroup FS mounted under the given mountPoint. It does not check | ||
| // the provided mount mode | ||
| func NewFS(mountPoint string, mountMode MountMode) (FS, error) { | ||
| info, err := os.Stat(mountPoint) | ||
| if err != nil { | ||
| return FS{}, fmt.Errorf("could not read %s: %s", mountPoint, err) | ||
| } | ||
| if !info.IsDir() { | ||
| return FS{}, fmt.Errorf("mount point %s is not a directory", mountPoint) | ||
| } | ||
| return FS{mountPoint, mountMode}, nil | ||
| } | ||
|
|
||
| // path appends the given path elements to the filesystem path, adding separators | ||
| // as necessary. | ||
| func (fs FS) path(p ...string) string { | ||
| return filepath.Join(append([]string{string(fs.mountPoint)}, p...)...) | ||
| } | ||
|
|
||
| // MountMode constants describe how the kernel has mounted various cgroup filesystems under /sys/fs/cgroup. | ||
| // Generally speaking, kernels using the cgroups-v1 API will have many cgroup controller hierarchies, each with | ||
| // their own fs and their own mount point. Kernels using cgroups-v2 API will only have the one unified hierarchy. | ||
| // To support back compatibility, kernels often mount both the v1 and v2 hierarchies at different points. Systemd | ||
| // has to know where the hierarchies are, so it inspects the mounts under /sys/fs/cgroup and decides what | ||
| // MountMode this kernel is using. See each constant for a description of that mode. This type corresponds to | ||
| // the unified_cache variable in systemd/src/basic/cgroup-util.c | ||
| type MountMode int8 | ||
|
|
||
| const ( | ||
| // MountModeUnknown indicates we do not recognize the mount pattern of the cgroup filesystems in /sys/fs/cgroup. | ||
| // systemd source calls this mode CGROUP_UNIFIED_UNKNOWN | ||
| MountModeUnknown MountMode = iota | ||
| // MountModeLegacy indicates both systemd and individual cgroups are using cgroup-v1 hierarchies. There is | ||
| // typically one mount point per hierarchy, and no usage of the cgroup-v2 unified hierarchy. | ||
| // systemd source calls this mode CGROUP_UNIFIED_NONE | ||
| MountModeLegacy MountMode = iota | ||
| // MountModeHybrid indicates the systemd controller is using cgroup-v2 unified hierarchy for organizing | ||
| // processes, but all other cgroups are using cgroup-v1 legacy hierarchies. | ||
| // systemd source calls this CGROUP_UNIFIED_SYSTEMD and also stores the unified_systemd_v232 flag | ||
| MountModeHybrid MountMode = iota | ||
| // MountModeUnified indicates cgroup-v2 API is in full usage and there are no cgroup-v1 hierarchies mounted. | ||
| // Non-updated programs (e.g. container orchestrators such as docker/runc) that rely on cgroup-v1 mounts will break. | ||
| // systemd source calls this CGROUP_UNIFIED_ALL | ||
| MountModeUnified MountMode = iota | ||
| ) | ||
| func (c MountMode) String() string { | ||
| return [...]string{"unknown", "none", "systemd", "all"}[c] | ||
| } | ||
|
|
||
|
|
||
| // Values copied from https://github.com/torvalds/linux/blob/master/include/uapi/linux/magic.h | ||
| const ( | ||
| tmpFsMagic = 0x01021994 | ||
| cgroupSuperMagic = 0x27e0eb | ||
| cgroup2SuperMagic = 0x63677270 | ||
| ) | ||
|
|
||
| // cgUnifiedCached checks the filesystem types mounted under /sys/fs/cgroup to determine | ||
| // which systemd layout (legacy/hybrid/unified) is in use. | ||
| // We do not bother to track unified_systemd_v232 as our usage does not | ||
| // depend on reading the systemd hierarchy directly, we only focus on reading | ||
| // the controllers. If you care if /sys/fs/cgroup/systemd is v1 or v2 you need | ||
| // to track this | ||
| // WARNING: We cache this data once at process start. Systemd updates | ||
| // may require restarting systemd-exporter | ||
| // Equivalent to systemd cgroup-util.c#cg_unified_cached | ||
| var statfsFunc = unix.Statfs | ||
| func cgUnifiedCached() (MountMode, error) { | ||
| // if cgroupUnified != MountModeUnknown { | ||
| // return cgroupUnified, nil | ||
| // } | ||
|
|
||
| var fs unix.Statfs_t | ||
| err := statfsFunc("/sys/fs/cgroup/", &fs) | ||
| if err != nil { | ||
| return MountModeUnknown, errors.Wrapf(err, "failed statfs(/sys/fs/cgroup)") | ||
| } | ||
|
|
||
| switch fs.Type { | ||
| case cgroup2SuperMagic: | ||
| log.Debugf("Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy") | ||
|
There was a problem hiding this comment. As this is a package, consider not logging or allowing users to choose their own logging pkg, more info https://dave.cheney.net/2015/11/05/lets-talk-about-logging |
||
| return MountModeUnified, nil | ||
| case tmpFsMagic: | ||
| err := statfsFunc("/sys/fs/cgroup/unified/", &fs) | ||
|
|
||
| // Ignore err, we expect path to be missing on v232 | ||
| if err == nil && fs.Type == cgroup2SuperMagic { | ||
| log.Debugf("Found cgroup2 on /sys/fs/cgroup/unified, unified hierarchy for systemd controller") | ||
| return MountModeHybrid, nil | ||
| } | ||
|
|
||
| err = statfsFunc("/sys/fs/cgroup/systemd/", &fs) | ||
| if err != nil { | ||
| return MountModeUnknown, errors.Wrapf(err, "failed statfs(/sys/fs/cgroup/systemd)") | ||
| } | ||
|
|
||
| switch fs.Type { | ||
| case cgroup2SuperMagic: | ||
| log.Debugf("Found cgroup2 on /sys/fs/cgroup/systemd, unified hierarchy for systemd controller (v232 variant)") | ||
| return MountModeHybrid, nil | ||
| case cgroupSuperMagic: | ||
| log.Debugf("Found cgroup on /sys/fs/cgroup/systemd, legacy hierarchy") | ||
| return MountModeLegacy, nil | ||
| default: | ||
| return MountModeUnknown, errors.Errorf("unknown magic number %x for fstype returned by statfs(/sys/fs/cgroup/systemd)", fs.Type) | ||
| } | ||
|
|
||
| default: | ||
| return MountModeUnknown, errors.Errorf("unknown magic number %x for fstype returned by statfs(/sys/fs/cgroup)", fs.Type) | ||
| } | ||
| } | ||
|
|
||
| // cgGetPath returns the absolute path for a specific file in a specific controller | ||
| // in the specific cgroup denoted by the passed subpath. | ||
| // Input examples: ("cpu", "/system.slice", "cpuacct.usage_all") | ||
| func (fs FS) cgGetPath(controller string, subpath string, suffix string) (string, error) { | ||
| // relevant systemd source code in cgroup-util.[h|c] specifically cg_get_path | ||
| // 2. Joins controller name with base path | ||
|
|
||
| if fs.cgroupUnified == MountModeUnknown { | ||
| return "", errors.Errorf("Cannot determine path with unknown mounting hierarchy") | ||
| } | ||
|
|
||
| // TODO Ensure controller name is valid | ||
| // TODO Convert controller name into guaranteed valid directory name | ||
| dn := controller | ||
|
|
||
| joined := "" | ||
| switch fs.cgroupUnified { | ||
| case MountModeLegacy, MountModeHybrid: | ||
| joined = fs.path(dn, subpath, suffix) | ||
| case MountModeUnified: | ||
| joined = fs.path(subpath, suffix) | ||
| default: | ||
| return "", errors.Errorf("unknown cgroup mount mode (e.g. unified mode) %d", fs.cgroupUnified) | ||
| } | ||
| return joined, nil | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a bit much, consider just using
flepath.Join(fs.MountPoint, subpath, suffix)directly in https://github.com/povilasv/systemd_exporter/pull/10/files#diff-966caeb64fc8c233e22788d6e3caccbbR171 andhttps://github.com/povilasv/systemd_exporter/pull/10/files#diff-966caeb64fc8c233e22788d6e3caccbbR173