c9s: afterburn hitting selinux denials when installing an OKD cluster #1555

Prashanth684 · 2024-07-18T21:28:21Z

When installing an OKD cluster, some nodes do not come up. It turns out they do not have a node name because the afterburn service does not run. It errors out due to selinux denials:

[core@ip-10-0-29-129 ~]$ systemctl status afterburn.service
× afterburn.service - Afterburn (Metadata)
     Loaded: loaded (/usr/lib/systemd/system/afterburn.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Thu 2024-07-18 06:32:31 UTC; 11h ago
       Docs: https://coreos.github.io/afterburn/usage/attributes/
   Main PID: 879 (code=exited, status=1/FAILURE)
        CPU: 42ms

Jul 18 06:32:30 localhost afterburn[879]: Jul 18 06:32:30.747 INFO Putting http://169.254.169.254/latest/api/token: Attempt #1
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]: Jul 18 06:32:31.765 INFO Putting http://169.254.169.254/latest/api/token: Attempt #2
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]: Error: failed to run
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]: Caused by:
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]:     0: writing metadata attributes
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]:     1: failed to create directory "/run/metadata"
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]:     2: Permission denied (os error 13)
Jul 18 06:32:31 ip-10-0-29-129 systemd[1]: afterburn.service: Main process exited, code=exited, status=1/FAILURE
Jul 18 06:32:31 ip-10-0-29-129 systemd[1]: afterburn.service: Failed with result 'exit-code'.
Jul 18 06:32:31 ip-10-0-29-129 systemd[1]: Failed to start Afterburn (Metadata).

Also, the denials in the audit logs:

time->Thu Jul 18 17:41:27 2024
type=PROCTITLE msg=audit(1721324487.450:7946): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D617474726962757465733D2F72756E2F6D657461646174612F61667465726275726E
type=PATH msg=audit(1721324487.450:7946): item=1 name=(null) inode=6892 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7946): item=0 name=(null) inode=1 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1721324487.450:7946): cwd="/"
type=SYSCALL msg=audit(1721324487.450:7946): arch=c000003e syscall=83 success=yes exit=0 a0=7ffd8fe9d9d0 a1=1ff a2=e a3=5635a2531097 items=2 ppid=1 pid=6572 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324487.450:7946): avc:  denied  { create } for  pid=6572 comm="afterburn" name="metadata" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1721324487.450:7946): avc:  denied  { add_name } for  pid=6572 comm="afterburn" name="metadata" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1721324487.450:7946): avc:  denied  { write } for  pid=6572 comm="afterburn" name="/" dev="tmpfs" ino=1 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir permissive=1
----
time->Thu Jul 18 17:41:27 2024
type=PROCTITLE msg=audit(1721324487.450:7947): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D617474726962757465733D2F72756E2F6D657461646174612F61667465726275726E
type=PATH msg=audit(1721324487.450:7947): item=3 name=(null) inode=6893 dev=00:18 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7947): item=2 name=(null) inode=6892 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7947): item=1 name=(null) nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7947): item=0 name=(null) inode=6892 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1721324487.450:7947): cwd="/"
type=SYSCALL msg=audit(1721324487.450:7947): arch=c000003e syscall=257 success=yes exit=6 a0=ffffff9c a1=7ffd8fe9d9d8 a2=80241 a3=1b6 items=4 ppid=1 pid=6572 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324487.450:7947): avc:  denied  { write open } for  pid=6572 comm="afterburn" path="/run/metadata/afterburn" dev="tmpfs" ino=6893 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
type=AVC msg=audit(1721324487.450:7947): avc:  denied  { create } for  pid=6572 comm="afterburn" name="afterburn" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
----
time->Thu Jul 18 17:42:45 2024
type=PROCTITLE msg=audit(1721324565.288:8006): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D7373682D6B6579733D636F7265
type=PATH msg=audit(1721324565.288:8006): item=0 name="/var/home/core/.ssh/authorized_keys.d/" inode=17825920 dev=103:04 mode=040700 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:ssh_home_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1721324565.288:8006): cwd="/"
type=SYSCALL msg=audit(1721324565.288:8006): arch=c000003e syscall=87 success=no exit=-2 a0=7fff03f1d1f8 a1=7fff03f1d1f8 a2=30 a3=55f52afed66a items=1 ppid=1 pid=6648 auid=4294967295 uid=0 gid=0 euid=1000 suid=0 fsuid=1000 egid=1000 sgid=0 fsgid=1000 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324565.288:8006): avc:  denied  { search } for  pid=6648 comm="afterburn" name=".ssh" dev="nvme0n1p4" ino=16777344 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=1
----
time->Thu Jul 18 17:42:45 2024
type=PROCTITLE msg=audit(1721324565.288:8007): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D7373682D6B6579733D636F7265
type=SYSCALL msg=audit(1721324565.288:8007): arch=c000003e syscall=257 success=yes exit=6 a0=ffffff9c a1=7fff03f1d1e8 a2=80000 a3=0 items=0 ppid=1 pid=6648 auid=4294967295 uid=0 gid=0 euid=1000 suid=0 fsuid=1000 egid=1000 sgid=0 fsgid=1000 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324565.288:8007): avc:  denied  { open } for  pid=6648 comm="afterburn" path="/var/home/core/.ssh/authorized_keys.d" dev="nvme0n1p4" ino=17825920 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1721324565.288:8007): avc:  denied  { read } for  pid=6648 comm="afterburn" name="authorized_keys.d" dev="nvme0n1p4" ino=17825920 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=1

This has started happening after #1552 where we had to use selinux version selinux-policy-38.1.36-1.el9 as selinux-policy-38.1.36-1.el9 is not available anymore (#1514).

The text was updated successfully, but these errors were encountered:

This commit implements a systemd unit to apply custom SELinux modules in SCOS shipped as CILs in the read-only /usr/lib/okd/selinux/ folder. Refers openshift#1555

Adds a systemd unit to load custom SELinux rules and workaround for afterburn failures the afterburn systemd units fail as the SELinux domain of the afterburn binary is restricted from changing the content of files in /run, /run/metadata and /home/$user/.ssh. This commit adds a afterburn-custom.cil SELinux module to allow the afterburn services to succeed and the nodes to properly join a cluster. The module is loaded by the okd-selinux.service implemented by 336013f Refers openshift#1555

the afterburn systemd units fail as the SELinux domain of the afterburn binary is restricted from changing the content of files in /run, /run/metadata and /home/$user/.ssh. This commit adds a afterburn-custom.cil SELinux module to allow the afterburn services to succeed and the nodes to properly join a cluster. The module is loaded by the okd-selinux.service implemented by 336013f Refers openshift#1555

This commit implements a systemd unit to apply custom SELinux modules in SCOS shipped as CILs in the read-only /usr/lib/okd/selinux/ folder. Refers openshift#1555

the afterburn systemd units fail as the SELinux domain of the afterburn binary is restricted from changing the content of files in /run, /run/metadata and /home/$user/.ssh. This commit adds a afterburn-custom.cil SELinux module to allow the afterburn services to succeed and the nodes to properly join a cluster. The module is loaded by the okd-selinux.service implemented by 336013f Refers openshift#1555

This commit implements a systemd unit to apply custom SELinux modules in SCOS shipped as CILs in the read-only /usr/lib/okd/selinux/ folder. Refers openshift#1555

the afterburn systemd units fail as the SELinux domain of the afterburn binary is restricted from changing the content of files in /run, /run/metadata and /home/$user/.ssh. This commit adds a afterburn-custom.cil SELinux module to allow the afterburn services to succeed and the nodes to properly join a cluster. The module is loaded by the okd-selinux.service implemented by 336013f Refers openshift#1555

jlebon · 2024-07-22T16:49:17Z

Can you file a bug on the RHEL board against the selinux-policy component and the version set to CentOS Stream 9?

Prashanth684 · 2024-07-22T17:05:21Z

@aleskandro already filed: https://issues.redhat.com/browse/RHEL-49735

the afterburn systemd units fail as the SELinux domain of the afterburn binary is restricted from changing the content of files in /run, /run/metadata and /home/$user/.ssh. This commit adds a afterburn-custom.cil SELinux module to allow the afterburn services to succeed and the nodes to properly join a cluster. The module is loaded by the okd-selinux.service implemented by 336013f Refers openshift#1555

openshift-bot · 2024-10-21T01:00:29Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2024-11-20T08:30:57Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

aleskandro mentioned this issue Jul 18, 2024

OKD-223: Load custom SELinux rules in SCOS and workaround afterburn failures #1556

Merged

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 21, 2024

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

c9s: afterburn hitting selinux denials when installing an OKD cluster #1555

c9s: afterburn hitting selinux denials when installing an OKD cluster #1555

Prashanth684 commented Jul 18, 2024

jlebon commented Jul 22, 2024

Prashanth684 commented Jul 22, 2024

openshift-bot commented Oct 21, 2024

openshift-bot commented Nov 20, 2024

c9s: afterburn hitting selinux denials when installing an OKD cluster #1555

c9s: afterburn hitting selinux denials when installing an OKD cluster #1555

Comments

Prashanth684 commented Jul 18, 2024

jlebon commented Jul 22, 2024

Prashanth684 commented Jul 22, 2024

openshift-bot commented Oct 21, 2024

openshift-bot commented Nov 20, 2024