Delete of the fetched package caused by a missing dependency

[thierry-FreeBSD]

Using poudriere-devel-3.4.99.20240105 on FreeBSD-CURRENT.
I'm not the maintainer of rust, but I need it for some dependencies and I prefer to avoid building it, therefore after refreshing a jail, with an up-to-date ports tree, I run the following command poudriere bulk -j 15amd64 -b latest lang/rust.
Hereunder is the resulting log: a package for rust has been fetched, but it has been deleted because no package has been found for curl!

1. This shouldn't be possible: pkg-fallout is a coherent repository, and if a package exists for rust, a package for curl certainly exists too.

• if for some reason it happens, a detailed message must be displayed;
• if this is caused by a network error, it would be great to retry at least 2 times.

2. However, in this case if a package for rust has been fetched, it should be kept preciously, and only the missing dependency should be built.

# poudriere bulk -j 15amd64 -b latest lang/rust
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for 15amd64-default
[00:00:01] Stashing existing package repository
[00:00:01] Mounting ccache from: /var/ccache
[00:00:01] Mounting ports from: /usr/local/poudriere/ports/default
[00:00:01] Mounting packages from: /usr/local/poudriere/data/packages/15amd64-default
[00:00:01] Mounting distfiles from: /usr/ports/distfiles
/etc/resolv.conf -> /usr/local/poudriere/data/.m/15amd64-default/ref/etc/resolv.conf
[00:00:01] Starting jail 15amd64-default
[00:00:01] Will build as nobody:nobody (65534:65534)
[00:00:03] Logs: /usr/local/poudriere/data/logs/bulk/15amd64-default/2024-01-14_00h19m57s
[00:00:03] Loading MOVED for /usr/local/poudriere/data/.m/15amd64-default/ref/usr/ports
[00:00:04] Ports supports: FLAVORS SUBPACKAGES SELECTED_OPTIONS
[00:00:04] Inspecting ports tree for modifications to git checkout... no
[00:00:05] Ports top-level git hash: cdd499a76d 
[00:00:05] Gathering ports metadata
[00:00:07] Calculating ports order and dependencies
[00:00:07] newer version of jail, cleaning all packages... done
[00:00:25] Trimming IGNORED and blacklisted ports
[00:00:25] Package fetch: Looking for missing packages to fetch from pkg+http://pkg.FreeBSD.org/${ABI}/latest
[00:00:25] Packge fetch: bootstrapping pkg
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:15:amd64/latest, please wait...
[15amd64-default] Installing pkg-1.20.9...
[15amd64-default] Extracting pkg-1.20.9: 100%
Updating FreeBSD repository catalogue...
[15amd64-default] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
[15amd64-default] Fetching packagesite.pkg: 100%    7 MiB   7.4MB/s    00:01    
Processing entries: 100%
FreeBSD repository update completed. 34352 packages processed.
All repositories are up to date.
[00:00:50] Package fetch: Will fetch 38 packages from remote or local pkg cache
Updating database digests format: 100%
The following packages will be fetched:

New packages to be FETCHED:
        ccache: 3.7.12_5 (133 KiB: 0.07% of the 187 MiB to download)
        gettext-runtime: 0.22.3 (231 KiB: 0.12% of the 187 MiB to download)
        gettext-tools: 0.22.3 (3 MiB: 1.48% of the 187 MiB to download)
        gmake: 4.3_2 (450 KiB: 0.23% of the 187 MiB to download)
        help2man: 1.49.3 (174 KiB: 0.09% of the 187 MiB to download)
        indexinfo: 0.3.1 (6 KiB: 0.00% of the 187 MiB to download)
        jsoncpp: 1.9.5 (137 KiB: 0.07% of the 187 MiB to download)
        libffi: 3.4.4 (45 KiB: 0.02% of the 187 MiB to download)
        libiconv: 1.17 (732 KiB: 0.38% of the 187 MiB to download)
        libidn2: 2.3.4 (155 KiB: 0.08% of the 187 MiB to download)
        libnghttp2: 1.58.0 (139 KiB: 0.07% of the 187 MiB to download)
        libssh2: 1.11.0,3 (281 KiB: 0.15% of the 187 MiB to download)
        libtextstyle: 0.22.3 (448 KiB: 0.23% of the 187 MiB to download)
        libunistring: 1.1 (679 KiB: 0.35% of the 187 MiB to download)
        libuv: 1.47.0 (134 KiB: 0.07% of the 187 MiB to download)
        mpdecimal: 2.5.1 (332 KiB: 0.17% of the 187 MiB to download)
        ninja: 1.11.1,2 (123 KiB: 0.06% of the 187 MiB to download)
        p5-Locale-gettext: 1.07 (17 KiB: 0.01% of the 187 MiB to download)
        p5-Locale-libintl: 1.33 (717 KiB: 0.37% of the 187 MiB to download)
        p5-Text-Unidecode: 1.30 (102 KiB: 0.05% of the 187 MiB to download)
        p5-Unicode-EastAsianWidth: 12.0 (10 KiB: 0.01% of the 187 MiB to download)
        perl5: 5.36.3_1 (15 MiB: 8.21% of the 187 MiB to download)
        pkgconf: 2.0.3_1,1 (81 KiB: 0.04% of the 187 MiB to download)
        public_suffix_list: 20231128 (80 KiB: 0.04% of the 187 MiB to download)
        py39-build: 0.10.0_2 (29 KiB: 0.01% of the 187 MiB to download)
        py39-flit-core: 3.9.0 (79 KiB: 0.04% of the 187 MiB to download)
        py39-installer: 0.7.0 (236 KiB: 0.12% of the 187 MiB to download)
        py39-packaging: 23.2 (77 KiB: 0.04% of the 187 MiB to download)
        py39-pyproject_hooks: 1.0.0_2 (15 KiB: 0.01% of the 187 MiB to download)
        py39-setuptools: 63.1.0_1 (1 MiB: 0.60% of the 187 MiB to download)
        py39-tomli: 2.0.1_2 (20 KiB: 0.01% of the 187 MiB to download)
        py39-wheel: 0.42.0 (100 KiB: 0.05% of the 187 MiB to download)
        python39: 3.9.18 (19 MiB: 9.91% of the 187 MiB to download)
        readline: 8.2.7 (398 KiB: 0.21% of the 187 MiB to download)
        rhash: 1.4.4 (233 KiB: 0.12% of the 187 MiB to download)
        rust: 1.75.0 (142 MiB: 75.66% of the 187 MiB to download)
        texinfo: 7.1,1 (1 MiB: 0.79% of the 187 MiB to download)

Number of packages to be fetched: 37

The process will require 187 MiB more space.
187 MiB to be downloaded.
[15amd64-default] Fetching py39-setuptools-63.1.0_1.pkg: 100%    1 MiB   1.2MB/s    00:01    
[15amd64-default] Fetching ccache-3.7.12_5.pkg: 100%  133 KiB 136.4kB/s    00:01    
[15amd64-default] Fetching indexinfo-0.3.1.pkg: 100%    6 KiB   6.0kB/s    00:01    
[15amd64-default] Fetching rust-1.75.0.pkg: 100%  142 MiB   9.3MB/s    00:16    
[15amd64-default] Fetching libnghttp2-1.58.0.pkg: 100%  139 KiB 142.1kB/s    00:01    
[15amd64-default] Fetching p5-Text-Unidecode-1.30.pkg: 100%  102 KiB 104.0kB/s    00:01    
[15amd64-default] Fetching mpdecimal-2.5.1.pkg: 100%  332 KiB 339.8kB/s    00:01    
[15amd64-default] Fetching python39-3.9.18.pkg: 100%   19 MiB   9.7MB/s    00:02    
[15amd64-default] Fetching libidn2-2.3.4.pkg: 100%  155 KiB 158.6kB/s    00:01    
[15amd64-default] Fetching py39-tomli-2.0.1_2.pkg: 100%   20 KiB  20.6kB/s    00:01    
[15amd64-default] Fetching ninja-1.11.1,2.pkg: 100%  123 KiB 126.0kB/s    00:01    
[15amd64-default] Fetching p5-Locale-gettext-1.07.pkg: 100%   17 KiB  16.9kB/s    00:01    
[15amd64-default] Fetching jsoncpp-1.9.5.pkg: 100%  137 KiB 140.3kB/s    00:01    
[15amd64-default] Fetching public_suffix_list-20231128.pkg: 100%   80 KiB  81.6kB/s    00:01    
[15amd64-default] Fetching py39-wheel-0.42.0.pkg: 100%  100 KiB 102.8kB/s    00:01    
[15amd64-default] Fetching libuv-1.47.0.pkg: 100%  134 KiB 137.0kB/s    00:01    
[15amd64-default] Fetching perl5-5.36.3_1.pkg: 100%   15 MiB   8.1MB/s    00:02    
[15amd64-default] Fetching gmake-4.3_2.pkg: 100%  450 KiB 460.8kB/s    00:01    
[15amd64-default] Fetching py39-pyproject_hooks-1.0.0_2.pkg: 100%   15 KiB  15.2kB/s    00:01    
[15amd64-default] Fetching libunistring-1.1.pkg: 100%  679 KiB 695.0kB/s    00:01    
[15amd64-default] Fetching py39-installer-0.7.0.pkg: 100%  236 KiB 241.6kB/s    00:01    
[15amd64-default] Fetching libtextstyle-0.22.3.pkg: 100%  448 KiB 459.3kB/s    00:01    
[15amd64-default] Fetching p5-Unicode-EastAsianWidth-12.0.pkg: 100%   10 KiB  10.6kB/s    00:01    
[15amd64-default] Fetching libffi-3.4.4.pkg: 100%   45 KiB  45.9kB/s    00:01    
[15amd64-default] Fetching readline-8.2.7.pkg: 100%  398 KiB 407.4kB/s    00:01    
[15amd64-default] Fetching texinfo-7.1,1.pkg: 100%    1 MiB   1.6MB/s    00:01    
[15amd64-default] Fetching libssh2-1.11.0,3.pkg: 100%  281 KiB 287.5kB/s    00:01    
[15amd64-default] Fetching py39-packaging-23.2.pkg: 100%   77 KiB  78.8kB/s    00:01    
[15amd64-default] Fetching py39-flit-core-3.9.0.pkg: 100%   79 KiB  81.3kB/s    00:01    
[15amd64-default] Fetching help2man-1.49.3.pkg: 100%  174 KiB 178.7kB/s    00:01    
[15amd64-default] Fetching rhash-1.4.4.pkg: 100%  233 KiB 238.5kB/s    00:01    
[15amd64-default] Fetching pkgconf-2.0.3_1,1.pkg: 100%   81 KiB  82.7kB/s    00:01    
[15amd64-default] Fetching libiconv-1.17.pkg: 100%  732 KiB 749.7kB/s    00:01    
[15amd64-default] Fetching gettext-tools-0.22.3.pkg: 100%    3 MiB   2.9MB/s    00:01    
[15amd64-default] Fetching gettext-runtime-0.22.3.pkg: 100%  231 KiB 236.1kB/s    00:01    
[15amd64-default] Fetching p5-Locale-libintl-1.33.pkg: 100%  717 KiB 733.9kB/s    00:01    
[15amd64-default] Fetching py39-build-0.10.0_2.pkg: 100%   29 KiB  29.4kB/s    00:01    
[00:01:17] Package fetch: Using cached copy of ccache-3.7.12_5
[00:01:17] Package fetch: Using cached copy of ninja-1.11.1,2
[00:01:17] Package fetch: Using cached copy of libuv-1.47.0
[00:01:17] Package fetch: Using cached copy of jsoncpp-1.9.5
[00:01:17] Package fetch: Using cached copy of libffi-3.4.4
[00:01:17] Package fetch: Using cached copy of pkgconf-2.0.3_1,1
[00:01:17] Package fetch: Using cached copy of readline-8.2.7
[00:01:17] Package fetch: Using cached copy of python39-3.9.18
[00:01:17] Package fetch: Using cached copy of rust-1.75.0
[00:01:17] Package fetch: Using cached copy of gettext-tools-0.22.3
[00:01:17] Package fetch: Using cached copy of gettext-runtime-0.22.3
[00:01:17] Package fetch: Using cached copy of perl5-5.36.3_1
[00:01:17] Package fetch: Using cached copy of libssh2-1.11.0,3
[00:01:17] Package fetch: Using cached copy of libnghttp2-1.58.0
[00:01:17] Package fetch: Using cached copy of gmake-4.3_2
[00:01:17] Package fetch: Using cached copy of mpdecimal-2.5.1
[00:01:17] Package fetch: Using cached copy of libtextstyle-0.22.3
[00:01:17] Package fetch: Using cached copy of libidn2-2.3.4
[00:01:17] Package fetch: Using cached copy of libunistring-1.1
[00:01:17] Package fetch: Using cached copy of meson-1.3.1
[00:01:17] Package fetch: Using cached copy of help2man-1.49.3
[00:01:17] Package fetch: Using cached copy of indexinfo-0.3.1
[00:01:17] Package fetch: Using cached copy of rhash-1.4.4
[00:01:17] Package fetch: Using cached copy of public_suffix_list-20231128
[00:01:17] Package fetch: Using cached copy of p5-Locale-libintl-1.33
[00:01:17] Package fetch: Using cached copy of p5-Locale-gettext-1.07
[00:01:17] Package fetch: Using cached copy of p5-Unicode-EastAsianWidth-12.0
[00:01:17] Package fetch: Using cached copy of texinfo-7.1,1
[00:01:17] Package fetch: Using cached copy of libiconv-1.17
[00:01:17] Package fetch: Using cached copy of p5-Text-Unidecode-1.30
[00:01:17] Package fetch: Using cached copy of py39-build-0.10.0_2
[00:01:17] Package fetch: Using cached copy of py39-installer-0.7.0
[00:01:17] Package fetch: Using cached copy of py39-wheel-0.42.0
[00:01:17] Package fetch: Using cached copy of py39-packaging-23.2
[00:01:17] Package fetch: Using cached copy of py39-flit-core-3.9.0
[00:01:17] Package fetch: Using cached copy of py39-setuptools-63.1.0_1
[00:01:17] Package fetch: Using cached copy of py39-pyproject_hooks-1.0.0_2
[00:01:17] Package fetch: Using cached copy of py39-tomli-2.0.1_2
[00:01:17] Sanity checking the repository
[00:01:17] Checking packages for incremental rebuild needs
[00:01:18] Deleting rust-1.75.0.pkg: missing dependency: curl-8.5.0
[00:01:18] Deleting stale symlinks... done
[00:01:18] Deleting empty directories... done
[00:01:18] Package fetch: Generating logs for fetched packages
[00:01:31] Unqueueing existing packages
[00:01:31] Unqueueing orphaned build dependencies
[00:01:31] Sanity checking build queue
[00:01:31] Processing PRIORITY_BOOST
[00:01:31] Balancing pool
[15amd64-default] [2024-01-14_00h19m57s] [balancing_pool] Queued: 43 Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 37 Tobuild: 6   Time: 00:01:29
[00:01:31] Recording filesystem state for prepkg... done
[00:01:31] Building 6 packages using up to 6 builders
[00:01:31] Hit CTRL+t at any time to see build progress and stats
[00:01:31] [01] [00:00:00] Builder starting
[00:01:32] [01] [00:00:01] Builder started
[00:01:32] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.20.9
[00:02:48] [01] [00:01:16] Finished ports-mgmt/pkg | pkg-1.20.9: Success
[00:02:48] [01] [00:00:00] Building textproc/expat2 | expat-2.5.0_1
[00:02:48] [02] [00:00:00] Builder starting
[00:02:50] [02] [00:00:02] Builder started
[00:02:50] [02] [00:00:00] Building dns/libpsl | libpsl-0.21.2_5
[00:03:11] [02] [00:00:21] Finished dns/libpsl | libpsl-0.21.2_5: Success
[00:03:11] [02] [00:00:00] Building ftp/curl | curl-8.5.0
[00:03:14] [01] [00:00:26] Finished textproc/expat2 | expat-2.5.0_1: Success
[00:03:14] [01] [00:00:00] Building devel/cmake-core | cmake-core-3.28.1
[00:04:58] [02] [00:01:47] Finished ftp/curl | curl-8.5.0: Success
[00:47:24] [01] [00:44:10] Finished devel/cmake-core | cmake-core-3.28.1: Success
[00:47:24] [01] [00:00:00] Building lang/rust | rust-1.75.0

[dsh2dsh]

JFYI my PR #1101 fixes this situation:

Also I moved download_from_repo below sanity_checks_pkgs, because sanity_check_pkgs removes packages which don't have builded dependencies and if we'll download_from_repo before sanity_check_pkgs, we will rebuild it. But if we'll download_from_repo after sanity_check_pkgs, we could download it instead of rebuild, which is faster.

[svmhdvn]

What is the current proposed solution to this bug, now that PR #1101 is closed? I'm also running into the exact same situation with attempting to fetch lang/rust (due to how enormous it is). After reading issue #1128, I'm not sure this particular bug is related.

Is there any workaround that can be done now?

[bdrewery]

Dupe of #954. The fix coming for it will assert that no fetched package is deleted (or it is a bug to be fixed). It will be merged in the next few days.

[bdrewery]

#1128 isn't related. Why do you mention that?

[svmhdvn]

Dupe of #954. The fix coming for it will assert that no fetched package is deleted (or it is a bug to be fixed). It will be merged in the next few days.

Awesome, thanks!

#1128 isn't related. Why do you mention that?

Yes, I was just confirming that #1128 isn't related. I mentioned it because that issue was listed as the reason for closing #1101.

[bdrewery]

#822 is the reason for closing that PR.

[dsh2dsh]

It will be merged in the next few days.

A month later. Was it merged? Is it ready to use?