wip

doc/build-helpers/testers.chapter.md

@@ -165,6 +165,59 @@ testers.shellcheck {
 A derivation that runs `shellcheck` on the given script(s).
 The build will fail if `shellcheck` finds any issues.
 
+## `testLibs` {#tester-testLibs}
+
+Some packages are sensitive to the version of the libraries they link against. The `testLibs` tester allows you to run an arbitrary command with the `LD_DEBUG` environment variable set to `libs`, which will cause the dynamic linker to print out the libraries it is loading. Through the `testLibs` tester, you can run assertions of the form:
+
+- soname was or was not searched for
+- soname was or was not loaded, optionally from a path matching a pattern
+
+:::{.example #ex-testLibs}
+# Run `testers.testLibs`
+
+```nix
+testers.testLibs {
+  name = "TODO";
+  bin = {
+    path = ./script.sh;
+    args = [ "arg1" "arg2" ];
+  };
+  assertions = [
+    { soname = "libfoo.so.1"; loaded = true; description = "Required library"; }
+    { soname = "libbar.so.2"; loaded = false; description = "Unnecessary library"; }
+    { soname = "libbaz.so.3"; loaded = true; path = "/path/to/libbaz.so.3"; }
+    { soname = "libqux.so.4"; loaded = false; path = "/path/to/libqux.so.4"; description = "Forbidden library"; }
+  ];
+}
+```
+
+:::
+
+### Inputs {#tester-testLibs-inputs}
+
+[`bin.path` (path or string)]{#tester-testLibs-param-bin-path}
+
+: The path to the executable to be run.
+  This is expected to be a single executable file.
+
+[`bin.args` (list of strings)]{#tester-testLibs-param-bin-args}
+
+: The arguments to pass to the executable.
+
+[`assertions` (list of attribute sets)]{#tester-testLibs-param-assertions}
+
+: A list of assertions to check.
+  Each assertion is an attribute set with the following attributes:
+
+  - `soname` (string): The soname of the library to check.
+  - `loaded` (boolean): Whether the library should be loaded.
+  - `path` (string, optional): The path the library should be loaded from.
+  - `description` (string, optional): A description of the assertion.
+
+### Return value {#tester-shellcheck-return}
+
+A derivation that runs `testLibs` on the given binary.
+
 ## `testVersion` {#tester-testVersion}
 
 Checks that the output from running a command contains the specified version string in it as a whole word.

pkgs/build-support/testers/default.nix

@@ -181,4 +181,6 @@
   testMetaPkgConfig = callPackage ./testMetaPkgConfig/tester.nix { };
 
   shellcheck = callPackage ./shellcheck/tester.nix { };
+
+  testLibs = callPackage ./testLibs { };
 }

pkgs/build-support/testers/testLibs/NOTES.md

@@ -0,0 +1,190 @@
+# NOTES
+
+Here's ChatGPT-4o's response when asked about
+
+## Question
+
+Here's output I got from running `LD_DEBUG=libs python3 --version`. Please explain the contents and structure of the output, breaking down the information on each line.
+
+```console
+   2753420:     find library=libpthread.so.0 [0]; searching
+   2753420:      search path=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/glibc-hwcaps/x86-64-v3:/home/connorbaker/micromamba/envs/testLibs/bin/../lib/glibc-hwcaps/x86-64-v2:/home/connorbaker/micromamba/envs/testLibs/bin/../lib          (RPATH from file /home/connorbaker/micromamba/envs/testLibs/bin/python3)
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/glibc-hwcaps/x86-64-v3/libpthread.so.0
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/glibc-hwcaps/x86-64-v2/libpthread.so.0
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/libpthread.so.0
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib/glibc-hwcaps/x86-64-v3:/run/current-system/sw/share/nix-ld/lib/glibc-hwcaps/x86-64-v2:/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/glibc-hwcaps/x86-64-v3/libpthread.so.0
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/glibc-hwcaps/x86-64-v2/libpthread.so.0
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/libpthread.so.0
+   2753420:      search cache=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/etc/ld.so.cache
+   2753420:      search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v3:/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v2:/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v3:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v2:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib            (system search path)
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v3/libpthread.so.0
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v2/libpthread.so.0
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libpthread.so.0
+   2753420:
+   2753420:     find library=libdl.so.2 [0]; searching
+   2753420:      search path=/home/connorbaker/micromamba/envs/testLibs/bin/../lib        (RPATH from file /home/connorbaker/micromamba/envs/testLibs/bin/python3)
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/libdl.so.2
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/libdl.so.2
+   2753420:      search cache=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/etc/ld.so.cache
+   2753420:      search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v3:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v2:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib                (system search path)
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libdl.so.2
+   2753420:
+   2753420:     find library=libutil.so.1 [0]; searching
+   2753420:      search path=/home/connorbaker/micromamba/envs/testLibs/bin/../lib        (RPATH from file /home/connorbaker/micromamba/envs/testLibs/bin/python3)
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/libutil.so.1
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/libutil.so.1
+   2753420:      search cache=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/etc/ld.so.cache
+   2753420:      search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v3:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v2:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib                (system search path)
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libutil.so.1
+   2753420:
+   2753420:     find library=libm.so.6 [0]; searching
+   2753420:      search path=/home/connorbaker/micromamba/envs/testLibs/bin/../lib        (RPATH from file /home/connorbaker/micromamba/envs/testLibs/bin/python3)
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/libm.so.6
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/libm.so.6
+   2753420:      search cache=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/etc/ld.so.cache
+   2753420:      search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v3:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v2:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib                (system search path)
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libm.so.6
+   2753420:
+   2753420:     find library=libc.so.6 [0]; searching
+   2753420:      search path=/home/connorbaker/micromamba/envs/testLibs/bin/../lib        (RPATH from file /home/connorbaker/micromamba/envs/testLibs/bin/python3)
+   2753420:       trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/libc.so.6
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/libc.so.6
+   2753420:      search cache=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/etc/ld.so.cache
+   2753420:      search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v3:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib/glibc-hwcaps/x86-64-v2:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib                (system search path)
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libc.so.6
+   2753420:
+   2753420:     find library=librt.so.1 [0]; searching
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/librt.so.1
+   2753420:      search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib(system search path)
+   2753420:       trying file=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/librt.so.1
+   2753420:
+   2753420:     find library=libatomic.so.1 [0]; searching
+   2753420:      search path=/run/current-system/sw/share/nix-ld/lib            (LD_LIBRARY_PATH)
+   2753420:       trying file=/run/current-system/sw/share/nix-ld/lib/libatomic.so.1
+   2753420:
+   2753420:
+   2753420:     calling init: /lib64/ld-linux-x86-64.so.2
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libc.so.6
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libpthread.so.0
+   2753420:
+   2753420:
+   2753420:     calling init: /run/current-system/sw/share/nix-ld/lib/libatomic.so.1
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/librt.so.1
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libm.so.6
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libutil.so.1
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libdl.so.2
+   2753420:
+   2753420:
+   2753420:     calling init: /nix/store/dbjkr6mxbfgn1s8i5bz9mj3zkfcvlzkp-malloc-provider-mimalloc/lib/libmimalloc.so
+   2753420:
+   2753420:
+   2753420:     initialize program: /home/connorbaker/micromamba/envs/testLibs/bin/python3
+   2753420:
+   2753420:
+   2753420:     transferring control: /home/connorbaker/micromamba/envs/testLibs/bin/python3
+   2753420:
+   2753420:
+   2753420:     calling fini:  [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbjkr6mxbfgn1s8i5bz9mj3zkfcvlzkp-malloc-provider-mimalloc/lib/libmimalloc.so [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libdl.so.2 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libutil.so.1 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libm.so.6 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/librt.so.1 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /run/current-system/sw/share/nix-ld/lib/libatomic.so.1 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libpthread.so.0 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libc.so.6 [0]
+   2753420:
+   2753420:
+   2753420:     calling fini: /lib64/ld-linux-x86-64.so.2 [0]
+   2753420:
+```
+
+## Answer
+
+This output is the result of setting `LD_DEBUG=libs` while running the `python3` command, which provides detailed information about how the dynamic linker resolves and loads shared libraries during program execution. Here's a breakdown of the content and structure:
+
+### General Format:
+Each line has the following format:
+```plaintext
+   <process id>: <debug message>
+```
+Here, the process ID is `2753420` for all lines. The message contains information about dynamic linking, such as searching for libraries and initializing or finalizing them.
+
+### Key Sections of the Output:
+
+1. **Finding and Loading Shared Libraries:**
+   - Example: `find library=libpthread.so.0 [0]; searching`
+     - The linker is trying to find the shared library `libpthread.so.0`. The `[0]` indicates that no previous instance of this library has been found.
+
+2. **Search Paths for Libraries:**
+   - Example: `search path=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/glibc-hwcaps/x86-64-v3`
+     - The linker searches various directories for the specified library. It looks through paths derived from the environment variables and default system paths:
+     - **RPATH**: Directories specified at the time the binary was built.
+     - **LD_LIBRARY_PATH**: User-defined library paths.
+     - **ld.so.cache**: A cache of libraries that the dynamic linker maintains for faster lookup.
+
+   - Example:
+     ```plaintext
+     search path=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib:/nix/store/5k2fkrnnp9w7kjvf7s0ancb5rlnml100-xgcc-13.3.0-libgcc/lib
+     ```
+     This indicates the dynamic linker is also searching within the Nix store paths.
+
+3. **Trying Specific Files:**
+   - Example: `trying file=/home/connorbaker/micromamba/envs/testLibs/bin/../lib/libpthread.so.0`
+     - The linker attempts to load the file from the specified path. This shows whether the library is present at a particular path.
+
+4. **Using the Cache:**
+   - Example: `search cache=/nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/etc/ld.so.cache`
+     - If the library cannot be found in the searched paths, the linker consults the cache (`ld.so.cache`) to resolve the location of the library.
+
+5. **Calling Initialization Functions:**
+   - Example: `calling init: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libpthread.so.0`
+     - Once a library is found and loaded, the linker invokes its initialization functions (e.g., `init` sections in libraries), which set up the necessary state for that library to be used.
+
+6. **Finalizing Loaded Libraries:**
+   - Example: `calling fini: /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib/libpthread.so.0 [0]`
+     - The linker invokes the finalization functions of each library when the program finishes, ensuring proper cleanup of resources.
+
+7. **Control Transfer:**
+   - Example: `transferring control: /home/connorbaker/micromamba/envs/testLibs/bin/python3`
+     - This indicates that after resolving all necessary libraries, the linker transfers control to the main program (`python3`), allowing it to start execution.
+
+### Special Points:
+- **glibc-hwcaps**: The search paths include "hwcaps" subdirectories (hardware capabilities). These directories contain optimized versions of libraries for specific CPU architectures (e.g., x86-64-v3, x86-64-v2).
+- **Nix Store Paths**: Many of the libraries are being loaded from the Nix package manager's store paths, indicating that the environment is using Nix to manage system libraries.
+
+This output shows how the dynamic linker is methodically finding, loading, initializing, and finalizing shared libraries as `python3` is executed in a Nix and Micromamba-managed environment.