/usr/local Removed After POUDRIERE JAIL Command

[vermaden]

Prerequisites

• [v] Have you checked for an existing issue describing your problem?
• [v] Are you running the latest version?
• [v] Is your ports tree recent?
• [v] Is your FreeBSD Host on a supported release?

Describe

The /usr/local is (mostly) deleted after executing this command:

# poudriere jail -c -j 14-1-B1-amd64 -v 14.1-BETA1

Reproduce

List of commands I have typed in fresh FreeBSD 14.1-BETA1 system inside Bhyve VM:

# mkdir -p /usr/local/etc/pkg/repos

# sed -e s/quarterly/latest/g /etc/pkg/FreeBSD.conf /usr/local/etc/pkg/repos/FreeBSD.conf

# pkg install -y  \
  beadm           \
  lsblk           \
  poudriere-devel \
  nginx           \
  git-lite        \
  ccache4         \
  tree

# export SSL=/usr/local/etc/ssl

# mkdir -p \
  /usr/ports/distfiles \
  ${SSL}/keys \
  ${SSL}/certs

# chmod 0600 ${SSL}/keys

# openssl genrsa -out ${SSL}/keys/poudriere.key 4096

# openssl rsa \
  -in  ${SSL}/keys/poudriere.key -pubout \
  -out ${SSL}/certs/poudriere.cert

# export IP=10.0.0.123

# cat << EOF > /usr/local/etc/poudriere.conf
ZPOOL=zroot
BASEFS=/usr/local/poudriere
ZROOTFS=/usr/local/poudriere
FREEBSD_HOST=ftp://ftp.freebsd.org
POUDRIERE_DATA=/usr/local/poudriere/data
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
URL_BASE=http://${IP}/
USE_TMPFS=no
TMPFS_LIMIT=12
MAX_MEMORY=12
PARALLEL_JOBS=8
PREPARE_PARALLEL_JOBS=8
MAX_FILES=4096
DISTFILES_CACHE=/usr/ports/distfiles
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=3
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/ccache
RESTRICT_NETWORKING=no
EOF

# mkdir -p /usr/local/poudriere/data/logs/bulk

# ln -s \
  /usr/local/etc/ssl/certs/poudriere.cert \
  /usr/local/poudriere/data/logs/bulk/poudriere.cert

# poudriere jail -c -j 14-1-B1-amd64 -v 14.1-BETA1

Messages on the screen:

root@poudriere:~ # poudriere jail -c -j 14-1-B1-amd64 -v 14.1-BETA1
[00:00:00] Creating 14-1-B1-amd64 fs at /usr/local/poudriere/jails/14-1-B1-amd64... done
[00:00:00] Fetching MANIFEST for FreeBSD 14.1-BETA1 amd64
MANIFEST                                              1046  B 5877 kBps    00s
[00:00:01] Fetching base for FreeBSD 14.1-BETA1 amd64
base.txz                                               197 MB 1682 kBps 02m01s
[00:02:04] Extracting base... done
[00:02:38] Fetching src for FreeBSD 14.1-BETA1 amd64
src.txz                                                205 MB 1991 kBps 01m45s
[00:04:27] Extracting src... done
[00:05:10] Fetching lib32 for FreeBSD 14.1-BETA1 amd64
lib32.txz                                               60 MB 2091 kBps    30s
[00:05:41] Extracting lib32... done
[00:05:50] Cleaning up... done
[00:05:50] Recording filesystem state for clean... done
[00:05:50] Jail 14-1-B1-amd64 14.1-BETA1 amd64 is ready to be used

root@poudriere:~ # poudriere ports -c -p idm
-sh: poudriere: not found

root@poudriere:~ # ls -l /usr/local
total 1
drwxr-xr-x  3 root wheel 3 May  9 14:25 etc
drwxr-xr-x  3 root wheel 3 May  9 14:19 poudriere

root@poudriere:~ # ls -l /usr/local/etc
total 1
drwxr-xr-x  3 root wheel 3 May  9 14:25 poudriere.d

root@poudriere:~ # ls -l /usr/local/poudriere/
total 1
drwxr-xr-x  3 root wheel 3 May  9 14:19 jails

root@poudriere:~ # ls -l /usr/local/bin
ls: /usr/local/bin: No such file or directory

After that the /usr/local is mostly done.

Environment

• Host OS: FreeBSD 14.1-BETA1 amd64
• Poudriere Version: poudriere-devel-3.4.99.20240424

Other

I have gathered all executed commands during the poudriere jail with execsnoop from dtrace-toolkit like that.

# /usr/local/share/dtrace-toolkit/execsnoop 1> DTRACE.log 2>&1

I also executed the poudriere jail with sh -x prefix.

# sh -x /usr/local/bin/poudriere jail -c -j 14-1-B1-amd64 -v 14.1-BETA1 1> POUDRIERE.log 2>&1

Both files attached below.
DTRACE.log
POUDRIERE.log

[vermaden]

Same effect with regular poudriere-3.4.1 from 'latest' pkg(8) branch.

[vermaden]

Only I have that problem?

[vermaden]

Other interesting behavior ... I did the following:

1. Install fresh FreeBSD 14.1-BETA1.
2. Install poudriere-devel package.
3. Set some basic Poudriere config.
4. Fetch 14.1-BETA1 Poudriere Jail - /usr/local is gone.
5. Delete /var/db/pkg dir.
6. Install poudriere-devel package again.

Now after these steps I deleted the 14.1-BETA1 Poudriere Jail and added it again - and this time /usr/local was OK.

Investigating more ...

[okalm]

I can confirm!

I followed your guide and have been in the same situation, I thought I was doing something wrong at first so I tested with 13.3-RELEASE and 14.1-RELEASE just to be sure, but the result is the same.
Once the command poudriere jail -c -j ... is invoked, everything in /usr/local is vanished except the dir zroot/usr/local/poudriere and the sub directories, so the jail is correctly installed at least.
All packages installed previously are unusable, poudriere included, so in the end the system is messed up.

I did tried to set a VM with more space 50GB and more RAM 25G, but it didn't change anything.
All I can say is I am glad the tests were only in VM.

Host:
Filesystem is ZFS
$: freebsd-version -kru ; uname -rms
13.3-RELEASE-p1
13.3-RELEASE-p1
13.3-RELEASE-p2
FreeBSD 13.3-RELEASE-p1 amd64

poudriere version: poudriere-3.4.1_1

[okalm]

@vermaden
Following your guide, in the file /usr/local/etc/poudriere.conf both variables BASEFS and ZROOTFS point to the same location, that's where the problem is I think.
When ZROOTFS=/poudriere ,as it is by default, poudriere works normally.
I skipped that part too: zfs create -o mountpoint=/usr/local/poudriere zroot/poudriere
I came to that conclusion because I didn't notice any issue while using UFS and your guide minus the ZFS parts.

PS:
Just a quick note regarding your guide, the package ccache-memcached isn't available anymore apparently, I install ccache instead. Apart from that it was very helpful and well detailed, thank you!

[vermaden]

Can you share Your /usr/local/etc/poudriere.conf file? The one that DOES NOT make the /usr/local deletion?

[okalm]

Sure, here it is:

root@vmp ~ : cat /usr/local/etc/poudriere.conf
ZPOOL=zroot
BASEFS=/usr/local/poudriere
ZROOTFS=/poudriere
FREEBSD_HOST=https://download.freebsd.org
POUDRIERE_DATA=/usr/local/poudriere/data
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
URL_BASE=http://192.168.3.129/
USE_TMPFS=yes
TMPFS_LIMIT=8
MAX_MEMORY=22
MAX_FILES=2048
DISTFILES_CACHE=/usr/ports/distfiles
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=3
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/ccache
RESTRICT_NETWORKING=no

Also zfs datasets:

root@vmp ~ : zfs list
NAME                                        USED  AVAIL     REFER  MOUNTPOINT
zroot                                      5.11G  31.2G       96K  /zroot
zroot/ROOT                                 2.30G  31.2G       96K  none
zroot/ROOT/13.3-RELEASE_2024-06-06_021114     8K  31.2G     1.38G  /
zroot/ROOT/default                         2.30G  31.2G     2.05G  /
zroot/poudriere                            2.59G  31.2G       96K  /zroot/poudriere
zroot/poudriere/jails                      1.69G  31.2G       96K  /zroot/poudriere/jails
zroot/poudriere/jails/13-3-R-amd64         1.69G  31.2G     1.69G  /usr/local/poudriere/jails/13-3-R-amd64
zroot/poudriere/ports                       924M  31.2G       96K  /zroot/poudriere/ports
zroot/poudriere/ports/default               924M  31.2G      924M  /usr/local/poudriere/ports/default
zroot/tmp                                   104K  31.2G      104K  /tmp
zroot/usr                                   220M  31.2G       96K  /usr
zroot/usr/home                              128K  31.2G      128K  /usr/home
zroot/usr/ports                             219M  31.2G      219M  /usr/ports
zroot/usr/src                                96K  31.2G       96K  /usr/src
zroot/var                                   768K  31.2G       96K  /var
zroot/var/audit                              96K  31.2G       96K  /var/audit
zroot/var/crash                              96K  31.2G       96K  /var/crash
zroot/var/log                               240K  31.2G      240K  /var/log
zroot/var/mail                              128K  31.2G      128K  /var/mail
zroot/var/tmp                               112K  31.2G      112K  /var/tmp

[vermaden]

Thank You - will look into that and test.

[vermaden]

So ... it seams that this bug exists only if ZROOTFS is set to the same value as BASEFS in the /usr/local/etc/poudriere.conf file.

[okalm]

Yes, it looks like the root of the issue.
Not a big deal once you aware of it, the workaround is pretty simple.
But the damage caused by this "issue" is surprising tough, I didn't expect that since it's my first round with poudriere, so I am glad we found how to fix this :-)

[vermaden]

Should I submit it somewhere else? Like submit a FreeBSD BUG also?

[okalm]

I don't know mate, I think we are in the right place, at the bottom of the manpage it refers to this very git repository for reporting issue.
That being said there are few issues reported on bugzilla too

I am under the impression that everybody just go with the default variables ZROOTFS=/poudriere and BASEFS=/usr/local/poudriere , that's why no one is really affected by this issue yet.
But if it was working before (which I can't tell since I am new to this), and not anymore then I think it deserves to get a little attention, if it happened to us it will touch others too at some point.

[vermaden]

Not sure also - then added PR here:

• https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279911

Regards,
vermaden

[okalm]

Good job thank you ;)