Inconsistent "permissions denied" using secrets in nix systemd-nspawn containers

[JulianGodd]

I've been trying to use sops-nix to protect secrets used in containers. Unfortunately, I've run into an issue where they work in my nginx container but not the nextcloud example.

I added each key to its container using the method from these merged changes. I also mounted each key on the container to allow them access to the file.

Here's a nix file I import in my configuration.nix containing all relevant expressions:

{ pkgs, inputs, lib, config, stdenv, fetchurl, programs, virtualisation, ... }:
{
    config = {
        sops.secrets = {
            "nextcloud/admin/password" = {
                uid = config.containers."nextcloud".config.users.users."nextcloud".uid;
                gid = config.containers."nextcloud".config.users.groups."nextcloud".gid;
            };

            "reverseproxy/priv" = {
                uid = config.containers."reverseproxy".config.users.users."nginx".uid;
                gid = config.containers."reverseproxy".config.users.groups."nginx".gid;
            };

            "reverseproxy/pub" = {
                uid = config.containers."reverseproxy".config.users.users."nginx".uid;
                gid = config.containers."reverseproxy".config.users.groups."nginx".gid;
            };
        };

        networking.nat = {
            enable = true;
            internalInterfaces = ["ve-+"];
            externalInterface = "enp4s0";
        };

        containers = {
            nextcloud = {
                autoStart = true;
                privateNetwork = true;

                hostAddress = "10.100.0.1";
                localAddress = "10.100.0.2";

                bindMounts = {
                    "adminpass" = {
                        hostPath = config.sops.secrets."nextcloud/admin/password".path;
                        mountPoint = "/run/secrets/adminpass";
                        isReadOnly = true;
                    };
                };

                config = { pkgs, lib, ... }: {
                    services.nextcloud = {
                        enable = true;
                        package = pkgs.nextcloud30;
                        hostName = "localhost";
                        config.adminpassFile = "/run/secrets/adminpass";
                        settings = {
                            trusted_domains = ["10.100.0.2" ];
                        };
                    };

                    networking = {
                        firewall = {
                            enable = true;
                            allowedTCPPorts = [ 80 ];
                        };
                        useHostResolvConf = lib.mkForce false;
                    };

                    services.resolved.enable = true;
                    system.stateVersion = "23.11";
                };
            };


            reverseproxy = {
                autoStart = true;
                privateNetwork = true;

                hostAddress = "10.100.0.1";
                localAddress = "10.100.0.3";

                bindMounts = {
                    "pubkey" = {
                        hostPath = config.sops.secrets."reverseproxy/pub".path;
                        mountPoint = "/run/secrets/reverseproxy/pub";
                        isReadOnly = true;
                    };
                    "privkey" = {
                        hostPath = config.sops.secrets."reverseproxy/priv".path;
                        mountPoint = "/run/secrets/reverseproxy/priv";
                        isReadOnly = true;
                    };
                };

                config = { pkgs, lib, ... }: {
                    services.nginx = {
                        enable = lib.mkForce true;
                        virtualHosts."nextcloud.local" = {
                            forceSSL = true;
                            sslCertificate = "/run/secrets/reverseproxy/pub";
                            sslCertificateKey = "/run/secrets/reverseproxy/priv";
                            locations."/" = {
                                proxyPass = "http://10.100.0.2";
                                proxyWebsockets = true;                       
                            };
                        };
                    };

                    networking = {
                        firewall = {
                            enable = true;
                            allowedTCPPorts = [ 80 443 ];
                        };
                        useHostResolvConf = lib.mkForce false;
                    };

                    services.resolved.enable = true;
                    system.stateVersion = "23.11";
                };
            };
        };

        networking.extraHosts = ''
            10.100.0.3 nextcloud.local
        '';
    };
}

When I build and run the containers, while both succeed in running, only the nginx container can access its secrets.

To test, I enter a container with:
sudo nixos-container root-login reverseproxy

Inside the container I run:
sudo -H -u nginx bash -c 'cat /run/secrets/reverseproxy/pub'

Output:

-----BEGIN CERTIFICATE-----
... the key value I expect to see ...
-----END CERTIFICATE-----

However, when I run the corresponding commands on the nextcloud container, I get permisssion denied:
sudo nixos-container root-login nextcloud
sudo -H -u nextcloud bash -c 'cat /run/secrets/adminpass'

Output:

cat: /run/secrets/adminpass: Permission denied

The nginx secrets are ssl certificates while the nextcloud is a plaintext password.

I've seen alternative methods for this where you add files via environment.etc.<filename> but it seems like this might not leverage of some of sops-nix's security features. Let me know if this an incorrect assumption since I'm otherwise fine with this as a solution.

Thanks for any help!

[Mic92]

I am wondering if it wasn't easier to have one age key that gets shared across containers. Than you can just have normal sops-nix running inside each container instead of having to write all this boiler code.
An alternative is using the LoadSecret interface that systemd provides to inject secrets into nspawn and than propagated it from there. I forget how this worked by maybe @flokli remembers.

[Mic92]

Have you inspect the permissions of

uid = config.containers."nextcloud".config.users.users."nextcloud".uid;

Do look like in the container? Does this user have a fixed uid in the first place? Maybe it's null and gets not defined by nix but at runtime instead.

[JulianGodd]

Have you inspect the permissions of

uid = config.containers."nextcloud".config.users.users."nextcloud".uid;

Do look like in the container? Does this user have a fixed uid in the first place? Maybe it's null and gets not defined by nix but at runtime instead.

It appears as though some specific nixos services have hardcoded ids. This explains why nginx is the only service I've tried that's been able to access its secrets.

It looks like the user or uid doesn't exist when sops checks that variable. Files in /run/secrets in the nginx container are owned by the same uid as the nginx user. In the nextcloud container however, secret files are owned by root.

I tried setting users.users.nextcloud.uid = 399; inside the container, but while this sets the uid for the secrets to 399, the value seems to be overridden elsewhere as the id is 998 for the nextcloud user inside the container. It doesn't seem like this is affected by isSystemUser being true or false.

Is there a better way of setting a static uid that I'm missing? Could I just assign the id to 998 or is this value dynamic?

Thanks for the help

[JulianGodd]

I am wondering if it wasn't easier to have one age key that gets shared across containers. Than you can just have normal sops-nix running inside each container instead of having to write all this boiler code.

I'm tempted to get my partially working solution to work but if it turns out to be a dead end I think this is the best option.

An alternative is using the LoadSecret interface that systemd provides to inject secrets into nspawn and than propagated it from there. I forget how this worked by maybe @flokli remembers.

I think this is a working version of injecting secrets through systemd. I'm not familiar enough with this method to understand the security implications but it also seems quite promising.

[Mic92]

I think this is a working version of injecting secrets through systemd. I'm not familiar enough with this method to understand the security implications but it also seems quite promising.

There should be no issues with security. Systemd will load secrets into a tmpfs that never gets swapped and is only ever visible to the service - so it won't put data into unencrypted storage.

[Mic92]

I think there is little we can do if a user doesn't have uids that are visible from the nix on the host. Please use one of the other workarounds mentioned or assign fixed uids.