Skip to content

Commit

Permalink
Deployed 695c161 with MkDocs version: 1.3.0
Browse files Browse the repository at this point in the history
  • Loading branch information
Unknown committed Jan 24, 2025
1 parent 24218eb commit 9789f18
Show file tree
Hide file tree
Showing 3 changed files with 191 additions and 4 deletions.
2 changes: 1 addition & 1 deletion search/search_index.json
View file

Large diffs are not rendered by default.

Binary file modified sitemap.xml.gz
View file
Binary file not shown.
193 changes: 190 additions & 3 deletions worker-node/install-apptainer/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -706,6 +706,40 @@
Choosing whether or not to install Apptainer
</a>

</li>

<li class="md-nav__item">
<a href="#enabling-unprivileged-apptainer" class="md-nav__link">
Enabling Unprivileged Apptainer
</a>

<nav class="md-nav" aria-label="Enabling Unprivileged Apptainer">
<ul class="md-nav__list">

<li class="md-nav__item">
<a href="#configuring-docker-to-work-with-apptainer" class="md-nav__link">
Configuring Docker to work with Apptainer
</a>

</li>

<li class="md-nav__item">
<a href="#configuring-unprivileged-apptainer" class="md-nav__link">
Configuring Unprivileged Apptainer
</a>

</li>

<li class="md-nav__item">
<a href="#validating-unprivileged-apptainer-in-cvmfs" class="md-nav__link">
Validating Unprivileged Apptainer in CVMFS
</a>

</li>

</ul>
</nav>

</li>

<li class="md-nav__item">
Expand Down Expand Up @@ -2009,6 +2043,40 @@
Choosing whether or not to install Apptainer
</a>

</li>

<li class="md-nav__item">
<a href="#enabling-unprivileged-apptainer" class="md-nav__link">
Enabling Unprivileged Apptainer
</a>

<nav class="md-nav" aria-label="Enabling Unprivileged Apptainer">
<ul class="md-nav__list">

<li class="md-nav__item">
<a href="#configuring-docker-to-work-with-apptainer" class="md-nav__link">
Configuring Docker to work with Apptainer
</a>

</li>

<li class="md-nav__item">
<a href="#configuring-unprivileged-apptainer" class="md-nav__link">
Configuring Unprivileged Apptainer
</a>

</li>

<li class="md-nav__item">
<a href="#validating-unprivileged-apptainer-in-cvmfs" class="md-nav__link">
Validating Unprivileged Apptainer in CVMFS
</a>

</li>

</ul>
</nav>

</li>

<li class="md-nav__item">
Expand Down Expand Up @@ -2118,8 +2186,9 @@ <h1 id="install-apptainer">Install Apptainer<a class="headerlink" href="#install
or with a setuid-root assist program.
By default it does not install the setuid-root assist program
and it uses only unprivileged user namespaces.
Unprivileged user namespaces are available on all OS versions that
OSG supports.</p>
Unprivileged user namespaces are enabled by default on all OS versions that
OSG supports, although some system administrators may have disabled them.
Instructions to enable it are <a href="#enabling-unprivileged-apptainer">below</a>.</p>
<div class="admonition danger">
<p class="admonition-title">Kernel vs. Userspace Security</p>
<p>Enabling unprivileged user namespaces increases the risk to the
Expand Down Expand Up @@ -2158,7 +2227,8 @@ <h2 id="choosing-whether-or-not-to-install-apptainer">Choosing whether or not to
</ul>
<p>OSG VOs all support running apptainer directly from CVMFS, when CVMFS
is available and unprivileged user namespaces are enabled.
Unprivileged user namespaces are enabled by default on EL 8+.
Unprivileged user namespaces are enabled by default on all OS versions
that OSG supports.
When unprivileged user namespaces are enabled, OSG
recommends that sites not install Apptainer unless they have
non-OSG users that require it.</p>
Expand All @@ -2171,6 +2241,123 @@ <h2 id="choosing-whether-or-not-to-install-apptainer">Choosing whether or not to
and will need to install an additional apptainer-suid RPM if they
want a setuid installation that does not require unprivileged user
namespaces.</p>
<h2 id="enabling-unprivileged-apptainer">Enabling Unprivileged Apptainer<a class="headerlink" href="#enabling-unprivileged-apptainer" title="Permanent link">&para;</a></h2>
<p>The instructions in this section are for enabling Apptainer to run
unprivileged.</p>
<ol>
<li>
<p>Make sure unprivileged user namespaces are enabled. They are
enabled by default on EL 8 and EL 9, but some system administrators
may have disabled them, possibly because of worries about a need for
frequent kernel patching (regarding that see step 2). Look for a
<code>sysctl</code> setting for <code>user.max_user_namespaces</code> which needs to be
non-zero. Examples for when it was disabled often showed setting
it to a value of 15000.</p>
</li>
<li>
<p>(Recommended) Disable network namespaces:</p>
<div class="codehilite"><pre><span></span><code><span class="gp">root@host # </span><span class="nb">echo</span> <span class="s2">&quot;user.max_net_namespaces = 0&quot;</span> <span class="se">\</span>
&gt; /etc/sysctl.d/90-max_net_namespaces.conf
<span class="gp">root@host # </span>sysctl -p /etc/sysctl.d/90-max_net_namespaces.conf
</code></pre></div>

<p>OSG VOs do not need network namespaces with Apptainer, and
disabling them significantly lowers the risk profile of user
namespaces and reduces the frequency of needing to apply urgent updates.
Most of the kernel vulnerabilities related to unprivileged user
namespaces over the last several years have been in combination with
network namespaces.</p>
<p>Network namespaces are, however, utilized by other software,
such as Docker or Podman.
Disabling network namespaces may break other
software, or limit its capabilities (such as requiring the
<code>--net=host</code> option in Docker or Podman).</p>
<p>Disabling network namespaces blocks the systemd PrivateNetwork
feature, which is a feature that is used by some EL 8 &amp; 9 services
by default. To check them all, look for PrivateNetwork in
<code>/lib/systemd/system/*.service</code> and see which of those services are
enabled but failed to start. The only default such service on EL 8
is systemd-hostnamed, and a popular non-default such service is
mlocate-updatedb. The PrivateNetwork feature can be turned off for
a service without modifying an RPM-installed file through a
<code>&lt;service&gt;.d/*.conf</code> file, for example for systemd-hostnamed:</p>
<div class="codehilite"><pre><span></span><code><span class="gp">root@host # </span><span class="nb">cd</span> /etc/systemd/system
<span class="gp">root@host # </span>mkdir -p systemd-hostnamed.service.d
<span class="gp">root@host # </span><span class="o">(</span><span class="nb">echo</span> <span class="s2">&quot;[Service]&quot;</span><span class="p">;</span> <span class="nb">echo</span> <span class="s2">&quot;PrivateNetwork=no&quot;</span><span class="o">)</span> <span class="se">\</span>
&gt;systemd-hostnamed.service.d/no-private-network.conf
<span class="gp">root@host # </span>systemctl daemon-reload
<span class="gp">root@host # </span>systemctl start systemd-hostnamed
<span class="gp">root@host # </span>systemctl status systemd-hostnamed
</code></pre></div>

</li>
</ol>
<h3 id="configuring-docker-to-work-with-apptainer">Configuring Docker to work with Apptainer<a class="headerlink" href="#configuring-docker-to-work-with-apptainer" title="Permanent link">&para;</a></h3>
<p>If docker is being used to run jobs, the following options are
recommended to allow unprivileged Apptainer to run (it does not
need <code>--privileged</code> or any added capabilities):</p>
<div class="codehilite"><pre><span></span><code><span class="go">--security-opt seccomp=unconfined --security-opt systempaths=unconfined</span>
</code></pre></div>

<p><code>--security-opt seccomp=unconfined</code> enables unshare to be called
(which is needed to create namespaces),
and <code>--security-opt systempaths=unconfined</code> allows <code>/proc</code> to be mounted
in an unprivileged process namespace (as is done by apptainer exec -p).
<code>--security-opt systempaths=unconfined</code> requires Docker 19.03 or later.
The options are secure as long as the system administrator controls
the images and does not allow user code to run as root, and are
generally more secure than adding capabilities. If at this point no
setuid or setcap programs needs to be run within the container, adding the
following option will improve security by preventing any privilege
escalation (Apptainer uses the same feature on its containers):</p>
<div class="codehilite"><pre><span></span><code><span class="go">--security-opt no-new-privileges</span>
</code></pre></div>

<p>In addition, the following option is recommended for allowing
unprivileged fuse mounts:</p>
<div class="codehilite"><pre><span></span><code><span class="go">--device=/dev/fuse</span>
</code></pre></div>

<h3 id="configuring-unprivileged-apptainer">Configuring Unprivileged Apptainer<a class="headerlink" href="#configuring-unprivileged-apptainer" title="Permanent link">&para;</a></h3>
<p>When unprivileged user namespaces are enabled and VOs run apptainer from
CVMFS, the Apptainer configuration file also comes from CVMFS so local
sites have no control over changing the configuration. However, the
most common local configuration change to the apptainer RPM is to add
additional local "bind path" options to map extra local file paths into
containers. This can instead be accomplished by setting the
<code>APPTAINER_BINDPATH</code> variable in the environment of jobs, for
example through
<a href="../../other/configuration-with-osg-configure/#local-settings">configuration</a>
on your compute entrypoint.
This is a comma-separated list of paths to bind, following the syntax of the
<code>apptainer exec --bind</code> option.
In order to be backward compatible with Singularity, also set
<code>SINGULARITY_BINDPATH</code> to the same value.
Apptainer also recognizes that variable but it prints a deprecation
warning if only a <code>SINGULARITY_</code> variable is set without the
corresponding <code>APPTAINER_</code> variable.</p>
<p>There are also other environment variables that can affect Apptainer
operation; see the
<a href="https://apptainer.org/docs/user/main/appendix.html">Apptainer documentation</a>
for details.</p>
<h3 id="validating-unprivileged-apptainer-in-cvmfs">Validating Unprivileged Apptainer in CVMFS<a class="headerlink" href="#validating-unprivileged-apptainer-in-cvmfs" title="Permanent link">&para;</a></h3>
<p>If you will not be installing Apptainer locally and
you haven't yet installed <a href="../install-cvmfs/">CVMFS</a>, please do so.
Alternatively, use the
<a href="https://github.com/cvmfs-contrib/cvmfsexec">cvmfsexec package</a>
configured for osg as an unprivileged user and mount the
oasis.opensciencegrid.org and singularity.opensciencegrid.org
repositories.</p>
<p>Then as an unprivileged user verify that Apptainer in CVMFS works with this
command:</p>
<div class="highlight"><pre><span></span><code><span class="gp">user@host $ </span>/cvmfs/oasis.opensciencegrid.org/mis/apptainer/bin/apptainer <span class="se">\</span>
<span class="nb">exec</span> --contain --ipc --pid --bind /cvmfs <span class="se">\</span>
/cvmfs/singularity.opensciencegrid.org/opensciencegrid/osgvo-el8:latest <span class="se">\</span>
ps -ef
<span class="go">UID PID PPID C STIME TTY TIME CMD</span>
<span class="go">user 1 0 0 10:51 console 00:00:00 appinit</span>
<span class="go">user 11 1 0 10:51 console 00:00:00 /usr/bin/ps -ef</span>
</code></pre></div>
<h2 id="installing-apptainer">Installing Apptainer<a class="headerlink" href="#installing-apptainer" title="Permanent link">&para;</a></h2>
<p>The instructions in this section are for installing a local copy
of Apptainer, either an unprivileged installation or an RPM installation.</p>
Expand Down

0 comments on commit 9789f18

Please sign in to comment.