diff --git a/previews/PR1360/.documenter-siteinfo.json b/previews/PR1360/.documenter-siteinfo.json index 7f1281a0..97965fb2 100644 --- a/previews/PR1360/.documenter-siteinfo.json +++ b/previews/PR1360/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.7.3","generation_timestamp":"2025-01-03T19:06:23","documenter_version":"1.8.0"}} \ No newline at end of file +{"documenter":{"julia_version":"1.7.3","generation_timestamp":"2025-01-03T19:14:06","documenter_version":"1.8.0"}} \ No newline at end of file diff --git a/previews/PR1360/FAQ/index.html b/previews/PR1360/FAQ/index.html index 09addb52..9432c78a 100644 --- a/previews/PR1360/FAQ/index.html +++ b/previews/PR1360/FAQ/index.html @@ -2,4 +2,4 @@ FAQ · BinaryBuilder.jl
GitHub

Frequently Asked Questions

I'm having trouble compiling <project name here>

First, make sure that you can compile that project natively on whatever platform you're attempting to compile it on. Once you are assured of that, search around the internet to see if anyone else has run into issues cross-compiling that project for that platform. In particular, most smaller projects should be just fine, but larger projects (and especially anything that does any kind of bootstrapping) may need some extra smarts smacked into their build system to support cross-compiling. Finally, if you're still stuck, try reaching out for help on the #binarybuilder channel in the JuliaLang slack.

How do I use this to compile my Julia code?

This package does not compile Julia code; it compiles C/C++/Fortran dependencies. Think about that time you wanted to use IJulia and you needed to download/install libnettle. The purpose of this package is to make generated tarballs that can be downloaded/installed painlessly as possible.

What is this I hear about the macOS SDK license agreement?

Apple restricts distribution and usage of the macOS SDK, a necessary component to build software for macOS targets. Please read the Apple and Xcode SDK agreement for more information on the restrictions and legal terms you agree to when using the SDK to build software for Apple operating systems. Copyright law is a complex area and you should not take legal advice from FAQs on the internet. This toolkit is designed to primarily run on Linux, though it can of course be used within a virtualized environment on a macOS machine or directly by running Linux Apple hardware. The Docker runner implements the virtualization approach on macOS machines. BinaryBuilder.jl, by default, will not automatically download or use the macOS SDK on non-apple host operating systems, unless the BINARYBUILDER_AUTOMATIC_APPLE environment variable is set to true.

Are there other environment variables I can use?

Yes, take a look.

Absolutely! There's nothing Julia-specific about the binaries generated by the cross-compilers used by BinaryBuilder.jl. Although the best interface for interacting with this software will always be the Julia interface defined within this package, you are free to use these software tools for other projects as well. Note that the cross-compiler image is built through a multistage bootstrapping process, see this repository for more information. Further note the macOS SDK license agreement tidbit above.

At line XXX, ABORTED (Operation not permitted)!

Some linux distributions have a bug in their overlayfs implementation that prevents us from mounting overlay filesystems within user namespaces. See this Ubuntu kernel bug report for a description of the situation and how Ubuntu has patched it in their kernels. To work around this, you can launch BinaryBuilder.jl in "privileged container" mode. BinaryBuilder should auto-detect this situation, however if the autodetection is not working or you want to silence the warning, you can set the BINARYBUILDER_RUNNER environment variable to privileged. Unfortunately, this involves running sudo every time you launch into a BinaryBuilder session, but on the other hand, this successfully works around the issue on distributions such as Arch linux.

I have to build a very small project without a Makefile, what do I have to do?

What BinaryBuilder needs is to find the relevant file (shared libraries, or executables, etc...) organised under the $prefix directory: libraries should go to ${libdir}, executables to ${bindir}. You may need to create those directories. You are free to choose whether to create a simple Makefile to build the project or to do everything within the build_tarballs.jl script. When the script completes, BinaryBuilder expects to find at least one artifact built for the expected architecture in either ${libdir} or ${bindir}. Remember also that you should use the standard environment variables like CC, CXX, CFLAGS, LDFLAGS as appropriate in order to cross compile. See the list of variables in the Tips for Building Packages section.

Can I open a shell in a particular build environment for doing some quick tests?

Yes! You can use BinaryBuilder.runshell(platform) to quickly start a shell in the current directory, without having to set up a working build_tarballs.jl script. For example,

julia -e 'using BinaryBuilder; BinaryBuilder.runshell(Platform("i686", "windows"))'

will open a shell in a Windows 32-bit build environment, without any source loaded. The current working directory of your system will be mounted on ${WORKSPACE} within this BinaryBuilder environment.

Can I publish a JLL package locally without going through Yggdrasil?

You can always build a JLL package on your machine with the --deploy flag to the build_tarballs.jl script. Read the help (--help) for more information.

A common use case is that you want to build a JLL package for, say, Libfoo, that will be used as dependency to build Quxlib, and you want to make sure that building both Libfoo and Quxlib will work before submitting all the pull requests to Yggdrasil. You can prepare the build_tarballs.jl script for Libfoo and then build and deploy it with

julia build_tarballs.jl --debug --verbose --deploy="MY_USERNAME/Libfoo_jll.jl"

replacing MY_USERNAME with your GitHub username: this will build the tarballs for all the platforms requested and upload them to a release of the MY_USERNAME/Libfoo_jll.jl, where the JLL package will also be created. As explained above, you can pass argument the list of triplets of the platforms for you which you want to build the tarballs, in case you want to compile only some of them. In the Julia REPL, you can install this package as any unregistered package with

]add https://github.com/MY_USERNAME/Libfoo_jll.jl.git

or develop it with

]dev https://github.com/MY_USERNAME/Libfoo_jll.jl.git

Since this package is unregistered, you have to use the full PackageSpec specification to add it as dependency of the local builder for Quxlib:

    Dependency(PackageSpec(; name = "Libfoo_jll",  uuid = "...", url = "https://github.com/MY_USERNAME/Libfoo_jll.jl.git"))

You can of course in turn build and deploy this package with

julia build_tarballs.jl --debug --verbose --deploy="MY_USERNAME/Quxlib_jll.jl"

Note that PackageSpec can also point to a local path: e.g., PackageSpec(; name="Libfoo_jll", uuid="...", path="/home/myname/.julia/dev/Libfoo_jll"). This is particularly useful when Building a custom JLL package locally, instead of deploying it to a remote Git repository.

What are those numbers in the list of sources? How do I get them?

The list of sources is a vector of BinaryBuilder.AbstractSources. What the hash is depends on what the source is:

  • For a FileSource or an ArchiveSource, the hash is a 64-character SHA256 checksum. If you have a copy of that file, you can compute the hash in Julia with

    using SHA
 open(path_to_the_file, "r") do f
      bytes2hex(sha256(f))
-end

    where path_to_the_file is a string with the path to the file. Alternatively, you can use the command line utilities curl and shasum to compute the hash of a remote file:

    $ curl -L http://example.org/file.tar.gz | shasum -a 256

    replacing http://example.org/file.tar.gz with the actual URL of the file you want to download.

  • For a GitSource, the hash is the 40-character SHA1 hash of the revision you want to checkout. For reproducibility you must indicate a specific revision, and not a branch or tag name, which are moving targets.

Now that I have a published and versioned jll package, what compat bounds do I put in its dependents? What if the upstream does not follow SemVer?

Imagine there is a package CoolFortranLibrary_jll that is a build of an upstream Fortran library CoolFortranLibrary. We will abbreviate these to CFL_jll and CFL.

Once you have CFL_jll you might want to have a Julia project that depends on it. As usual you put a compat bound for CFL_jll (the version number of upstream CFL and the jll version of CFL_jll are typically set equal during the jll registration). If you know for a fact that upstream CFL follows SemVer, then you just set compat bounds as if it was any other Julia project. However, not all ecosystems follow SemVer. The following two cases are quite common:

  1. CFL releases version 1.1.1 and version 1.1.2 that are incompatible. A real world example is Boost (which breaks the ABI in every single release because they embed the full version number in the soname of libraries). If you have a typical permissive semver-style compat section in a package that depends on CFL_jll, then your package will break whenever CFL_jll gets a new release. To solve this issue you have to use "hyphen style" compat bounds like "0.9.0 - 1.1.2". This leads to a separate problem: you need to change the compat bound every time there is a new CFL_jll release: this is the least bad option though – it causes more annoyance for developers but it ensures users never end up with broken installs. And bots like CompatHelper can mostly automate that issue.
  2. CFL releases versions 1.0.0 and 2.0.0 that are perfectly compatible. The Linux kernel, Chrome, Firefox, and curl are such examples. This causes annoying churn, as the developer still needs to update compat bounds in packages that depend on CFL_jll. Or, if you have a very strong belief in CFL's commitment to backward compatibility, you can put an extremely generous compat bound like ">= 1.0.0".

While the SemVer (and Julia's) "conservative" approach to compatibility ensures there will never be runtime crashes due to installed incompatible libraries, you might still end up with systems that refuse to install in the first place (which the Julia ecosystem considers the lesser evil). E.g., package A.jl that depends on newer versions of CLF and package B.jl that depends on older versions can not be installed at the same time. This happens less often in ecosystems that follow semver, but might happen relatively often in an ecosystem that does not. Thus developers that rely on jll packages that do not follow semver should be proactive in updating their compat bonds (and are strongly encouraged to heavily use the CompatHelper bot).

+end

where path_to_the_file is a string with the path to the file. Alternatively, you can use the command line utilities curl and shasum to compute the hash of a remote file:

$ curl -L http://example.org/file.tar.gz | shasum -a 256

replacing http://example.org/file.tar.gz with the actual URL of the file you want to download.

  • For a GitSource, the hash is the 40-character SHA1 hash of the revision you want to checkout. For reproducibility you must indicate a specific revision, and not a branch or tag name, which are moving targets.

    • Now that I have a published and versioned jll package, what compat bounds do I put in its dependents? What if the upstream does not follow SemVer?

    Imagine there is a package CoolFortranLibrary_jll that is a build of an upstream Fortran library CoolFortranLibrary. We will abbreviate these to CFL_jll and CFL.

    Once you have CFL_jll you might want to have a Julia project that depends on it. As usual you put a compat bound for CFL_jll (the version number of upstream CFL and the jll version of CFL_jll are typically set equal during the jll registration). If you know for a fact that upstream CFL follows SemVer, then you just set compat bounds as if it was any other Julia project. However, not all ecosystems follow SemVer. The following two cases are quite common:

    1. CFL releases version 1.1.1 and version 1.1.2 that are incompatible. A real world example is Boost (which breaks the ABI in every single release because they embed the full version number in the soname of libraries). If you have a typical permissive semver-style compat section in a package that depends on CFL_jll, then your package will break whenever CFL_jll gets a new release. To solve this issue you have to use "hyphen style" compat bounds like "0.9.0 - 1.1.2". This leads to a separate problem: you need to change the compat bound every time there is a new CFL_jll release: this is the least bad option though – it causes more annoyance for developers but it ensures users never end up with broken installs. And bots like CompatHelper can mostly automate that issue.
    2. CFL releases versions 1.0.0 and 2.0.0 that are perfectly compatible. The Linux kernel, Chrome, Firefox, and curl are such examples. This causes annoying churn, as the developer still needs to update compat bounds in packages that depend on CFL_jll. Or, if you have a very strong belief in CFL's commitment to backward compatibility, you can put an extremely generous compat bound like ">= 1.0.0".

    While the SemVer (and Julia's) "conservative" approach to compatibility ensures there will never be runtime crashes due to installed incompatible libraries, you might still end up with systems that refuse to install in the first place (which the Julia ecosystem considers the lesser evil). E.g., package A.jl that depends on newer versions of CLF and package B.jl that depends on older versions can not be installed at the same time. This happens less often in ecosystems that follow semver, but might happen relatively often in an ecosystem that does not. Thus developers that rely on jll packages that do not follow semver should be proactive in updating their compat bonds (and are strongly encouraged to heavily use the CompatHelper bot).

    diff --git a/previews/PR1360/build_tips/index.html b/previews/PR1360/build_tips/index.html index 802074cb..d402934a 100644 --- a/previews/PR1360/build_tips/index.html +++ b/previews/PR1360/build_tips/index.html @@ -10,4 +10,4 @@ ninja install

    The wizard automatically suggests using Meson if the meson.build file is present.

    Examples of builds performed with Meson include:

    Go builds

    The Go toolchain provided by BinaryBuilder can be requested by adding :go to the compilers keyword argument to build_tarballs: compilers=[:c, :go], and a specific version of the toolchain can be selected by adding the preferred_go_version keyword argument to build_tarballs. Go-based packages can usually be built and installed with go:

    go build -o ${bindir}

    The Go toolchain provided by BinaryBuilder automatically selects the appropriate target.

    Example of packages using Go:

    Rust builds

    The Rust toolchain provided by BinaryBuilder can be requested by adding :rust to the compilers keyword argument to build_tarballs: compilers=[:c, :rust], and a specific version of the toolchain can be selected by adding the preferred_rust_version keyword argument to build_tarballs. Rust-based packages can usually be built with cargo:

    cargo build --release

    The Rust toolchain provided by BinaryBuilder automatically selects the appropriate target and number of parallel jobs to be used. Note, however, that you may have to manually install the product in the ${prefix}. Read the installation instructions of the package in case they recommend a different build procedure.

    Example of packages using Rust:

    Warn

    The Rust toolchain currently used does not work with the i686-w64-mingw32 (32-bit Windows) platform.

    Editing files in the wizard

    In the wizard, the vim editor is available for editing files. But, it doesn't leave any record in the build script. One generally needs to provide patch files or use something like sed. If a file needs patching, we suggest using git to add the entire worktree to a new repo, make the changes you need, then use git diff -p to output a patch that can be included alongside your build recipe.

    You can include local files like patches very easily by placing them within a bundled/patches nested directory, and then providing "./bundled" as one of the sources for your build. See, for example, OpenBLAS.

    Automatic environment variables

    The following environment variables are automatically set in the build environment and should be used to build the project. Occasionally, you may need to tweak them (e.g., when Using GCC on macOS and FreeBSD).

    The above variables point to utilities for the target environment. To reference the utilities for the host environment either prepend HOST or append _HOST. For example, HOSTCC and CC_HOST point to the native C compiler.

    These are other environment variables that you may occasionally need to set during a build

    The following variables are useful to control the build script over different target systems, but are not intended to be modified by the users:

    Using GCC on macOS and FreeBSD

    For these target systems Clang is the default compiler, however some programs may not be compatible with Clang.

    For programs built with CMake (see the CMake build section) you can use the GCC toolchain file that is in ${CMAKE_TARGET_TOOLCHAIN%.*}_gcc.cmake.

    For programs built with Meson (see the Meson build section) you can use the GCC toolchain file that is in ${MESON_TARGET_TOOLCHAIN%.*}_gcc.meson.

    If the project that you want to build uses the GNU Build System (also known as the Autotools), there isn't an automatic switch to use GCC, but you have to set the appropriate variables. For example, this setting can be used to build most C/C++ programs with GCC for FreeBSD and macOS:

    if [[ "${target}" == *-freebsd* ]] || [[ "${target}" == *-apple-* ]]; then
     CC=gcc
     CXX=g++
-fi

    Many numerical libraries link to BLAS/LAPACK libraries to execute optimised linear algebra routines. It is important to understand that the elements of the arrays manipulated by these libraries can be indexed by either 32-bit integer numbers (LP64), or 64-bit integers (ILP64). For example, Julia itself employs BLAS libraries for linear algebra, and it expects ILP64 model on 64-bit platforms (e.g. the x86_64 and aarch64 architectures) and LP64 on LP64 on 32-bit platforms (e.g. the i686 and armv7l architectures). Furthermore, Julia comes by default with libblastrampoline, a library which doesn't implement itself any BLAS/LAPACK routine but it forwards all BLAS/LAPACK function calls to another library (by default OpenBLAS) which can be designated at runtime, allowing you to easily switch between different backends if needed. libblastrampoline provides both ILP64 and LP64 interfaces on 64-bit platforms, in the former case BLAS function calls are expected to have the _64 suffix to the standard BLAS names.

    If in your build you need to use a package to a BLAS/LAPACK library you have the following options:

    Dependencies for the target system vs host system

    BinaryBuilder provides a cross-compilation environment, which means that in general there is a distinction between the target platform (where the build binaries will eventually run) and the host platform (where compilation is currently happening). In particular, inside the build environment in general you cannot run binary executables built for the target platform.

    For a build to work there may be different kinds of dependencies, for example:

    You need to understand the build process of package you want to compile in order to know what of these classes a dependency belongs to.

    Installing the license file

    Generated tarballs should come with the license of the library that you want to install. If at the end of a successful build there is only one directory inside ${WORKSPACE}/srcdir, BinaryBuilder will look into it for files with typical names for license (like LICENSE, COPYRIGHT, etc... with some combinations of extensions) and automatically install them to ${prefix}/share/licenses/${SRC_NAME}/. If in the final tarball there are no files in this directory a warning will be issued, to remind you to provide a license file.

    If the license file is not automatically installed (for example because there is more than one directory in ${WORKSPACE}/srcdir or because the file name doesn't match the expected pattern) you have to manually install the file. In the build script you can use the install_license command. See the Utilities in the build environment section below.

    Utilities in the build environment

    In addition to the standard Unix tools, in the build environment there are some extra commands provided by BinaryBuilder. Here is a list of some of these commands:

    +fi

    Many numerical libraries link to BLAS/LAPACK libraries to execute optimised linear algebra routines. It is important to understand that the elements of the arrays manipulated by these libraries can be indexed by either 32-bit integer numbers (LP64), or 64-bit integers (ILP64). For example, Julia itself employs BLAS libraries for linear algebra, and it expects ILP64 model on 64-bit platforms (e.g. the x86_64 and aarch64 architectures) and LP64 on LP64 on 32-bit platforms (e.g. the i686 and armv7l architectures). Furthermore, Julia comes by default with libblastrampoline, a library which doesn't implement itself any BLAS/LAPACK routine but it forwards all BLAS/LAPACK function calls to another library (by default OpenBLAS) which can be designated at runtime, allowing you to easily switch between different backends if needed. libblastrampoline provides both ILP64 and LP64 interfaces on 64-bit platforms, in the former case BLAS function calls are expected to have the _64 suffix to the standard BLAS names.

    If in your build you need to use a package to a BLAS/LAPACK library you have the following options:

    Dependencies for the target system vs host system

    BinaryBuilder provides a cross-compilation environment, which means that in general there is a distinction between the target platform (where the build binaries will eventually run) and the host platform (where compilation is currently happening). In particular, inside the build environment in general you cannot run binary executables built for the target platform.

    For a build to work there may be different kinds of dependencies, for example:

    You need to understand the build process of package you want to compile in order to know what of these classes a dependency belongs to.

    Installing the license file

    Generated tarballs should come with the license of the library that you want to install. If at the end of a successful build there is only one directory inside ${WORKSPACE}/srcdir, BinaryBuilder will look into it for files with typical names for license (like LICENSE, COPYRIGHT, etc... with some combinations of extensions) and automatically install them to ${prefix}/share/licenses/${SRC_NAME}/. If in the final tarball there are no files in this directory a warning will be issued, to remind you to provide a license file.

    If the license file is not automatically installed (for example because there is more than one directory in ${WORKSPACE}/srcdir or because the file name doesn't match the expected pattern) you have to manually install the file. In the build script you can use the install_license command. See the Utilities in the build environment section below.

    Utilities in the build environment

    In addition to the standard Unix tools, in the build environment there are some extra commands provided by BinaryBuilder. Here is a list of some of these commands:

    diff --git a/previews/PR1360/building/index.html b/previews/PR1360/building/index.html index 48c74fb6..b6d0433a 100644 --- a/previews/PR1360/building/index.html +++ b/previews/PR1360/building/index.html @@ -92,4 +92,4 @@ dependencies = [ Dependency("Xorg_libxkbfile_jll"), BuildDependency("Xorg_util_macros_jll"), -]

    The argument of Dependency, RuntimeDependency, BuildDependency, and HostBuildDependency can also be a Pkg.PackageSpec, with which you can specify more details about the dependency, like a version number, or also a non-registered package. Note that in Yggdrasil only JLL packages in the General registry can be accepted.

    The dependencies for the target system (Dependency and BuildDependency) will be installed under ${prefix} within the build environment, while the dependencies for the host system (HostBuildDependency) will be installed under ${host_prefix}.

    In the wizard, dependencies can be specified with the prompt: Do you require any (binary) dependencies? [y/N].

    Examples of builders that depend on other binaries include:

    Platform-specific dependencies

    By default, all dependencies are used for all platforms, but there are some cases where a package requires some dependencies only on some platforms. You can specify the platforms where a dependency is needed by passing the platforms keyword argument to the dependency constructor, which is the vector of AbstractPlatforms where the dependency should be used.

    For example, assuming that the variable platforms holds the vector of the platforms for which to build your package, you can specify that Package_jl is required on all platforms excluding Windows one with

    Dependency("Package_jll"; platforms=filter(!Sys.iswindows, platforms))

    The information that a dependency is only needed on some platforms is transferred to the JLL package as well: the wrappers will load the platform-dependent JLL dependencies only when needed.

    Warning

    Julia's package manager doesn't have the concept of optional (or platform-dependent) dependencies: this means that when installing a JLL package in your environment, all of its dependencies will always be installed as well in any case. It's only at runtime that platform-specific dependencies will be loaded where necessary.

    For the same reason, even if you specify a dependency to be not needed on for a platform, the build recipe may still pull it in if that's also an indirect dependency required by some other dependencies. At the moment BinaryBuilder.jl isn't able to propagate the information that a dependency is platform-dependent when installing the artifacts of the dependencies.

    Examples:

    Version number of dependencies

    There are two different ways to specify the version of a dependency, with two different meanings:

    Building and testing JLL packages locally

    As a package developer, you may want to test JLL packages locally, or as a binary dependency developer you may want to easily use custom binaries. Through a combination of dev'ing out the JLL package and creating an override directory, it is easy to get complete control over the local JLL package state.

    Overriding a prebuilt JLL package's binaries

    After running pkg> dev LibFoo_jll, a local JLL package will be checked out to your depot's dev directory (on most installations this is ~/.julia/dev) and by default the JLL package will make use of binaries within your depot's artifacts directory. If an override directory is present within the JLL package directory, the JLL package will look within that override directory for binaries, rather than in any artifact directory. Note that there is no mixing and matching of binaries within a single JLL package; if an override directory is present, all products defined within that JLL package must be found within the override directory, none will be sourced from an artifact. Dependencies (e.g. found within another JLL package) may still be loaded from their respective artifacts, so dependency JLLs must themselves be dev'ed and have override directories created with files or symlinks created within them.

    Auto-populating the override directory

    To ease creation of an override directory, JLL packages contain a dev_jll() function, that will ensure that a ~/.julia/dev/<jll name> package is dev'ed out, and will copy the normal artifact contents into the appropriate override directory. This will result in no functional difference from simply using the artifact directory, but provides a template of files that can be replaced by custom-built binaries.

    Note that this feature is rolling out to new JLL packages as they are rebuilt; if a JLL package does not have a dev_jll() function, open an issue on Yggdrasil and a new JLL version will be generated to provide the function.

    Building a custom JLL package locally

    When building a new version of a JLL package, if --deploy is passed to build_tarballs.jl then a newly-built JLL package will be deployed to a GitHub repository. (Read the documentation in the Command Line section or given by passing --help to a build_tarballs.jl script for more on --deploy options). If --deploy=local is passed, the JLL package will still be built in the ~/.julia/dev/ directory, but it will not be uploaded anywhere. This is useful for local testing and validation that the built artifacts are working with your package.

    If you want to build a JLL package only for your current platform, you can use platforms = [HostPlatform()] in the build_tarball.jl script. You can also provide the target triplet Base.BinaryPlatforms.host_triplet() if you run the script in the command line.

    Deploying local builds without recreating the tarballs

    Sometimes all tarballs have already been created successfully locally but not deployed to GitHub. This can happen, e.g., if it is tricky to figure out the correct build script for all platforms, or if each platform build takes a long time. In this case, it is possible to skip the build process and just deploy the JLL package by providing the --skip-build flag to the build_tarballs.jl script. Read the help (--help) for more information.

    +]

    The argument of Dependency, RuntimeDependency, BuildDependency, and HostBuildDependency can also be a Pkg.PackageSpec, with which you can specify more details about the dependency, like a version number, or also a non-registered package. Note that in Yggdrasil only JLL packages in the General registry can be accepted.

    The dependencies for the target system (Dependency and BuildDependency) will be installed under ${prefix} within the build environment, while the dependencies for the host system (HostBuildDependency) will be installed under ${host_prefix}.

    In the wizard, dependencies can be specified with the prompt: Do you require any (binary) dependencies? [y/N].

    Examples of builders that depend on other binaries include:

    Platform-specific dependencies

    By default, all dependencies are used for all platforms, but there are some cases where a package requires some dependencies only on some platforms. You can specify the platforms where a dependency is needed by passing the platforms keyword argument to the dependency constructor, which is the vector of AbstractPlatforms where the dependency should be used.

    For example, assuming that the variable platforms holds the vector of the platforms for which to build your package, you can specify that Package_jl is required on all platforms excluding Windows one with

    Dependency("Package_jll"; platforms=filter(!Sys.iswindows, platforms))

    The information that a dependency is only needed on some platforms is transferred to the JLL package as well: the wrappers will load the platform-dependent JLL dependencies only when needed.

    Warning

    Julia's package manager doesn't have the concept of optional (or platform-dependent) dependencies: this means that when installing a JLL package in your environment, all of its dependencies will always be installed as well in any case. It's only at runtime that platform-specific dependencies will be loaded where necessary.

    For the same reason, even if you specify a dependency to be not needed on for a platform, the build recipe may still pull it in if that's also an indirect dependency required by some other dependencies. At the moment BinaryBuilder.jl isn't able to propagate the information that a dependency is platform-dependent when installing the artifacts of the dependencies.

    Examples:

    Version number of dependencies

    There are two different ways to specify the version of a dependency, with two different meanings:

    Building and testing JLL packages locally

    As a package developer, you may want to test JLL packages locally, or as a binary dependency developer you may want to easily use custom binaries. Through a combination of dev'ing out the JLL package and creating an override directory, it is easy to get complete control over the local JLL package state.

    Overriding a prebuilt JLL package's binaries

    After running pkg> dev LibFoo_jll, a local JLL package will be checked out to your depot's dev directory (on most installations this is ~/.julia/dev) and by default the JLL package will make use of binaries within your depot's artifacts directory. If an override directory is present within the JLL package directory, the JLL package will look within that override directory for binaries, rather than in any artifact directory. Note that there is no mixing and matching of binaries within a single JLL package; if an override directory is present, all products defined within that JLL package must be found within the override directory, none will be sourced from an artifact. Dependencies (e.g. found within another JLL package) may still be loaded from their respective artifacts, so dependency JLLs must themselves be dev'ed and have override directories created with files or symlinks created within them.

    Auto-populating the override directory

    To ease creation of an override directory, JLL packages contain a dev_jll() function, that will ensure that a ~/.julia/dev/<jll name> package is dev'ed out, and will copy the normal artifact contents into the appropriate override directory. This will result in no functional difference from simply using the artifact directory, but provides a template of files that can be replaced by custom-built binaries.

    Note that this feature is rolling out to new JLL packages as they are rebuilt; if a JLL package does not have a dev_jll() function, open an issue on Yggdrasil and a new JLL version will be generated to provide the function.

    Building a custom JLL package locally

    When building a new version of a JLL package, if --deploy is passed to build_tarballs.jl then a newly-built JLL package will be deployed to a GitHub repository. (Read the documentation in the Command Line section or given by passing --help to a build_tarballs.jl script for more on --deploy options). If --deploy=local is passed, the JLL package will still be built in the ~/.julia/dev/ directory, but it will not be uploaded anywhere. This is useful for local testing and validation that the built artifacts are working with your package.

    If you want to build a JLL package only for your current platform, you can use platforms = [HostPlatform()] in the build_tarball.jl script. You can also provide the target triplet Base.BinaryPlatforms.host_triplet() if you run the script in the command line.

    Deploying local builds without recreating the tarballs

    Sometimes all tarballs have already been created successfully locally but not deployed to GitHub. This can happen, e.g., if it is tricky to figure out the correct build script for all platforms, or if each platform build takes a long time. In this case, it is possible to skip the build process and just deploy the JLL package by providing the --skip-build flag to the build_tarballs.jl script. Read the help (--help) for more information.

    diff --git a/previews/PR1360/environment_variables/index.html b/previews/PR1360/environment_variables/index.html index e491a219..78bdb21b 100644 --- a/previews/PR1360/environment_variables/index.html +++ b/previews/PR1360/environment_variables/index.html @@ -1,2 +1,2 @@ -Environment Variables · BinaryBuilder.jl
    GitHub

    Environment Variables

    BinaryBuilder.jl supports multiple environment variables to modify its behavior globally:

    • BINARYBUILDER_AUTOMATIC_APPLE: when set to true, this automatically agrees to the Apple macOS SDK license agreement, enabling the building of binary objects for macOS systems.

    • BINARYBUILDER_USE_SQUASHFS: when set to true, this uses .squashfs images instead of tarballs to download cross-compiler shards. This consumes significantly less space on-disk and boasts a modest reduction in download size as well, but requires sudo on the local machine to mount the .squashfs images. This is the default when using the "privileged" runner.

    • BINARYBUILDER_RUNNER: When set to a runner string, alters the execution engine that BinaryBuilder.jl will use to wrap the build process in a sandbox. Valid values are one of "userns", "privileged" and "docker". If not given, BinaryBuilder.jl will do its best to guess.

    • BINARYBUILDER_ALLOW_ECRYPTFS: When set to true, this allows the mounting of rootfs/shard/workspace directories from within encrypted mounts. This is disabled by default, as at the time of writing, this triggers kernel bugs. To avoid these kernel bugs on a system where e.g. the home directory has been encrypted, set the BINARYBUILDER_ROOTFS_DIR and BINARYBUILDER_SHARDS_DIR environment variables to a path outside of the encrypted home directory.

    • BINARYBUILDER_USE_CCACHE: When set to true, this causes a /root/.ccache volume to be mounted within the build environment, and for the CC, CXX and FC environment variables to have ccache prepended to them. This can significantly accelerate rebuilds of the same package on the same host. Note that ccache will, by default, store 5G of cached data.

    • BINARYBUILDER_NPROC: Overrides the value of the environment variable ${nproc} set during a build, see Automatic environment variables.

    +Environment Variables · BinaryBuilder.jl
    GitHub

    Environment Variables

    BinaryBuilder.jl supports multiple environment variables to modify its behavior globally:

    • BINARYBUILDER_AUTOMATIC_APPLE: when set to true, this automatically agrees to the Apple macOS SDK license agreement, enabling the building of binary objects for macOS systems.

    • BINARYBUILDER_USE_SQUASHFS: when set to true, this uses .squashfs images instead of tarballs to download cross-compiler shards. This consumes significantly less space on-disk and boasts a modest reduction in download size as well, but requires sudo on the local machine to mount the .squashfs images. This is the default when using the "privileged" runner.

    • BINARYBUILDER_RUNNER: When set to a runner string, alters the execution engine that BinaryBuilder.jl will use to wrap the build process in a sandbox. Valid values are one of "userns", "privileged" and "docker". If not given, BinaryBuilder.jl will do its best to guess.

    • BINARYBUILDER_ALLOW_ECRYPTFS: When set to true, this allows the mounting of rootfs/shard/workspace directories from within encrypted mounts. This is disabled by default, as at the time of writing, this triggers kernel bugs. To avoid these kernel bugs on a system where e.g. the home directory has been encrypted, set the BINARYBUILDER_ROOTFS_DIR and BINARYBUILDER_SHARDS_DIR environment variables to a path outside of the encrypted home directory.

    • BINARYBUILDER_USE_CCACHE: When set to true, this causes a /root/.ccache volume to be mounted within the build environment, and for the CC, CXX and FC environment variables to have ccache prepended to them. This can significantly accelerate rebuilds of the same package on the same host. Note that ccache will, by default, store 5G of cached data.

    • BINARYBUILDER_NPROC: Overrides the value of the environment variable ${nproc} set during a build, see Automatic environment variables.

    diff --git a/previews/PR1360/index.html b/previews/PR1360/index.html index 626e8156..5ff3acbe 100644 --- a/previews/PR1360/index.html +++ b/previews/PR1360/index.html @@ -1,3 +1,3 @@ Home · BinaryBuilder.jl
    GitHub

    BinaryBuilder.jl

    The purpose of the BinaryBuilder.jl Julia package is to provide a system for compiling 3rd-party binary dependencies that should work anywhere the official Julia distribution does. In particular, using this package you will be able to compile your large pre-existing codebases of C, C++, Fortran, Rust, Go, etc... software into binaries that can be downloaded and loaded/run on a very wide range of machines. As it is difficult (and often expensive) to natively compile software packages across the growing number of platforms that this package will need to support, we focus on providing a set of Linux-hosted cross-compilers. This package will therefore set up an environment to perform cross-compilation for all of the major platforms, and will do its best to make the compilation process as painless as possible.

    Note that at this time, BinaryBuilder itself runs on Linux x86_64 and macOS x86_64 systems only, with Windows support under active development. On macOS and Windows, you must have docker installed as the backing virtualization engine. Note that Docker Desktop is the recommended version; if you have Docker Machine installed it may not work correctly or may need additional configuration.

    Warn

    This package currently requires Julia v1.7. Contribute to JuliaPackaging/JLLPrefixes.jl#6 if you care about supporting newer versions of Julia.

    Project flow

    Suppose that you have a Julia package Foo.jl which wants to use a compiled libfoo shared library. As your first step in writing Foo.jl, you may compile libfoo locally on your own machine with your system compiler, then using Libdl.dlopen() to open the library, and ccall() to call into the exported functions. Once you have written your C bindings in Julia, you will naturally desire to share the fruits of your labor with the rest of the world, and this is where BinaryBuilder can help you. Not only will BinaryBuilder aid you in constructing compiled versions of all your dependencies, but it will also build a wrapper Julia package (referred to as a JLL package) to aid in installation, versioning, and build product localization.

    The first step in the BinaryBuilder journey is to create a build recipe, usually named build_tarballs.jl. The Julia community curates a tree of build recipes, Yggdrasil, that already contains many examples of how to write a build_tarballs.jl file. These files contain information such as the name, version and source locations for a particular build, as well as the actual steps (in the form of a bash script) and the products that should be generated by the build.

    The result of a successful build is an autogenerated JLL package, typically uploaded to the JuliaBinaryWrappers github organization. Binaries for each version of every build are uploaded to the GitHub releases page of the relevant JLL package. Finally, a registration request is opened against the General Julia registry, so that packages such as the aforementioned Foo.jl can simply pkg> add libfoo_jll to download the binary artifacts as well as the autogenerated Julia wrapper code. See also the FAQ, build tips, build troubleshooting and tricksy gotchas for help with common problems.

    Wizard interface

    BinaryBuilder.jl supports an interactive method for building the binary dependencies and capturing the commands used to build it into a build_tarballs.jl file: the Wizard interface. To launch it, run

    using BinaryBuilder
-state = BinaryBuilder.run_wizard()

    and follow the instructions on-screen. You can watch an asciinema demo of the use of the wizard.

    Manually create or edit build_tarballs.jl

    The wizard is a great tool, especially to get started with BinaryBuilder and create your first simple recipes for new packages. However, it lacks the full control of all options you can use in a build_tarballs.jl script. To generate this file (explained in greater detail in Building Packages), one can clone Yggdrasil, copy an existing build recipe, modify it, and submit a new pull request. Manually editing the build_tarballs.jl script is also the recommended way when you want to update an existing recipe, rather then starting from scratch with the wizard.

    The build_tarballs.jl script can be used as a command line utility, it takes a few options and as argument the list of triplets of the targets. You can find more information about the syntax of the script in the Command Line section or by running

    julia build_tarballs.jl --help

    You can build the tarballs with

    julia build_tarballs.jl --debug --verbose

    The --debug option will drop you into the BinaryBuilder interactive shell if an error occurs. If the build fails, after finding out the steps needed to fix the build you have to manually update the script in build_tarballs.jl. You should run again the above command to make sure that everything is actually working.

    Since build_tarballs.jl takes as argument the comma-separated list of triplets for which to build the tarballs, you can select only a few of them. For example, with

    julia build_tarballs.jl --debug --verbose aarch64-linux-musl,arm-linux-musleabihf

    you'll run the build script only for the aarch64-linux-musl and arm-linux-musleabihf target platforms.

    If you decide to use this workflow, however, you will need to manually open pull requests for Yggdrasil.

    GitHub Codespaces

    If you already have access to the GitHub Codespaces service, you can use BinaryBuilder and all the workflows described above in your browser or with Visual Studio Code, on any operating system, including those not natively supported by the package! Head to Yggdrasil and create a new Codespace.

    How does this all work?

    BinaryBuilder.jl wraps a root filesystem that has been carefully constructed so as to provide the set of cross-compilers needed to support the wide array of platforms that Julia runs on. This RootFS is then used as the chroot jail for a sandboxed process which runs within the RootFS as if that were the whole world. The workspace containing input source code and (eventually) output binaries is mounted within the RootFS and environment variables are setup such that the appropriate compilers for a particular target platform are used by build tools.

    Reproducibility

    Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code.

    BinaryBuilder.jl puts into place many of the practices needed to achieve reproducible builds. For example, the building environment is sandboxed and uses a fixed tree structure, thus having a reproducible build path. The toolchain used by BinaryBuilder.jl also sets some environment variables and enforces certain compiler flags which help reproducibility.

    While BinaryBuilder.jl does not guarantee to always have reproducible builds, it achieves this goal in most cases. Reproducibility in BinaryBuilder.jl includes also the generated tarballs: they are created with Tar.jl, which takes a few measures to ensure reproducibility of tarballs with the same git tree hash. If you rebuild multiple times the same package, with the same version of BinaryBuilder, the generated tarball which contains the main products (i.e. not the log files which are known not to be reproducible) should always have the same git tree hash and SHA256 sum, information which are printed to screen at the end of the build process and stored in the Artifacts.toml file of the JLL package.

    There are however some caveats:

    • reproducibility can only be expected when using the toolchain offered by BinaryBuilder.jl;
    • there are very specific cases where the macOS C/C++ toolchain does not produce reproducible binaries. This happens when doing debug builds (-g flag) and not building object files with deterministic names separately (e.g. if directly building and linking a program or a shared library from the source file, letting the compiler create the intermediate object files automatically with random names). We have decided not to take action for this case because in practice most packages use build systems which compile intermediate object files with deterministic names (which is also the only way to take advantage of ccache, which BinaryBuilder.jl uses extensively) and typically do not do debug builds, thus sidestepping the issue entirely.

    Videos and tutorials

    BinaryBuilder has been covered in some videos, you may want to check them out if you want to know more about the framework (the date is specified in parentheses, to make it clear how old/new the videos are):

    +state = BinaryBuilder.run_wizard()

    and follow the instructions on-screen. You can watch an asciinema demo of the use of the wizard.

    Manually create or edit build_tarballs.jl

    The wizard is a great tool, especially to get started with BinaryBuilder and create your first simple recipes for new packages. However, it lacks the full control of all options you can use in a build_tarballs.jl script. To generate this file (explained in greater detail in Building Packages), one can clone Yggdrasil, copy an existing build recipe, modify it, and submit a new pull request. Manually editing the build_tarballs.jl script is also the recommended way when you want to update an existing recipe, rather then starting from scratch with the wizard.

    The build_tarballs.jl script can be used as a command line utility, it takes a few options and as argument the list of triplets of the targets. You can find more information about the syntax of the script in the Command Line section or by running

    julia build_tarballs.jl --help

    You can build the tarballs with

    julia build_tarballs.jl --debug --verbose

    The --debug option will drop you into the BinaryBuilder interactive shell if an error occurs. If the build fails, after finding out the steps needed to fix the build you have to manually update the script in build_tarballs.jl. You should run again the above command to make sure that everything is actually working.

    Since build_tarballs.jl takes as argument the comma-separated list of triplets for which to build the tarballs, you can select only a few of them. For example, with

    julia build_tarballs.jl --debug --verbose aarch64-linux-musl,arm-linux-musleabihf

    you'll run the build script only for the aarch64-linux-musl and arm-linux-musleabihf target platforms.

    If you decide to use this workflow, however, you will need to manually open pull requests for Yggdrasil.

    GitHub Codespaces

    If you already have access to the GitHub Codespaces service, you can use BinaryBuilder and all the workflows described above in your browser or with Visual Studio Code, on any operating system, including those not natively supported by the package! Head to Yggdrasil and create a new Codespace.

    How does this all work?

    BinaryBuilder.jl wraps a root filesystem that has been carefully constructed so as to provide the set of cross-compilers needed to support the wide array of platforms that Julia runs on. This RootFS is then used as the chroot jail for a sandboxed process which runs within the RootFS as if that were the whole world. The workspace containing input source code and (eventually) output binaries is mounted within the RootFS and environment variables are setup such that the appropriate compilers for a particular target platform are used by build tools.

    Reproducibility

    Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code.

    BinaryBuilder.jl puts into place many of the practices needed to achieve reproducible builds. For example, the building environment is sandboxed and uses a fixed tree structure, thus having a reproducible build path. The toolchain used by BinaryBuilder.jl also sets some environment variables and enforces certain compiler flags which help reproducibility.

    While BinaryBuilder.jl does not guarantee to always have reproducible builds, it achieves this goal in most cases. Reproducibility in BinaryBuilder.jl includes also the generated tarballs: they are created with Tar.jl, which takes a few measures to ensure reproducibility of tarballs with the same git tree hash. If you rebuild multiple times the same package, with the same version of BinaryBuilder, the generated tarball which contains the main products (i.e. not the log files which are known not to be reproducible) should always have the same git tree hash and SHA256 sum, information which are printed to screen at the end of the build process and stored in the Artifacts.toml file of the JLL package.

    There are however some caveats:

    Videos and tutorials

    BinaryBuilder has been covered in some videos, you may want to check them out if you want to know more about the framework (the date is specified in parentheses, to make it clear how old/new the videos are):

    diff --git a/previews/PR1360/jll/index.html b/previews/PR1360/jll/index.html index d489de26..4df30a3b 100644 --- a/previews/PR1360/jll/index.html +++ b/previews/PR1360/jll/index.html @@ -38,4 +38,4 @@ "LocalPreferences.toml", "Bzip2_jll", "libbzip2_path" => "/usr/local/lib/libbz2.so", -)

    Note that the product name is libbzip2, but we use libbzip2_path.

    Warning

    There are two common cases where this will not work:

    1. The JLL is part of the Julia stdlib, for example Zlib_jll
    2. The JLL has not been compiled with JLLWrappers.jl as a dependency. In this case, it means that the last build of the JLL pre-dates the introduction of the JLLWrappers package and needs a fresh build. Please open an issue on Yggdrasil requesting a new build, or make a pull request to update the relevant build_tarballs.jl script.
    +)

    Note that the product name is libbzip2, but we use libbzip2_path.

    Warning

    There are two common cases where this will not work:

    1. The JLL is part of the Julia stdlib, for example Zlib_jll
    2. The JLL has not been compiled with JLLWrappers.jl as a dependency. In this case, it means that the last build of the JLL pre-dates the introduction of the JLLWrappers package and needs a fresh build. Please open an issue on Yggdrasil requesting a new build, or make a pull request to update the relevant build_tarballs.jl script.
    diff --git a/previews/PR1360/reference/index.html b/previews/PR1360/reference/index.html index 9deb9b1f..47257854 100644 --- a/previews/PR1360/reference/index.html +++ b/previews/PR1360/reference/index.html @@ -4,7 +4,7 @@ dont_dlopen=false, dlopen_flags=Symbol[])

    Declares a LibraryProduct that points to a library located within the prefix. libname specifies the basename of the library, varname is the name of the variable in the JLL package that can be used to call into the library. By default, the library is searched in the libdir, but you can add other directories within the prefix to the dir_paths argument. You can specify the flags to pass to dlopen as a vector of Symbols with the dlopen_flags keyword argument. If the library should not be dlopen'ed automatically by the JLL package, set dont_dlopen=true.

    For example, if the libname is libnettle, this would be satisfied by the following paths:

    Libraries matching the search pattern are rejected if they are not dlopen()'able.

    If you are unsure what value to use for libname, you can use Base.BinaryPlatforms.parse_dl_name_version:

    julia> using Base.BinaryPlatforms
 
 julia> parse_dl_name_version("sfml-audio-2.dll", "windows")[1]
-"sfml-audio"

    If the library would have different basenames on different operating systems (e.g., libz.so on Linux and FreeBSD, libz.dylib on macOS, and zlib.dll on Windows), libname can be also a vector of Strings with the different alternatives:

    LibraryProduct(["libz", "zlib"], :libz)
    source
    BinaryBuilderBase.Product — Type

    A Product is an expected result after building or installation of a package.

    Examples of Products include LibraryProduct, FrameworkProduct, ExecutableProduct and FileProduct. All Product types must define the following minimum set of functionality:

    • locate(::Product): given a Product, locate it within the wrapped Prefix returning its location as a string

    • satisfied(::Product): given a Product, determine whether it has been successfully satisfied (e.g. it is locateable and it passes all callbacks)

    • variable_name(::Product): return the variable name assigned to a Product

    • repr(::Product): Return a representation of this Product, useful for auto-generating source code that constructs Products, if that's your thing.

    source
    BinaryBuilderBase.RuntimeDependency — Type
    RuntimeDependency(dep::Union{PackageSpec,String}; compat::String, platforms::Vector{<:AbstractPlatform}, top_level::Bool=false)

    Define a binary dependency that is only listed as dependency of the generated JLL package, but its artifact is not installed in the prefix during the build. The dep argument can be either a string with the name of the JLL package or a Pkg.PackageSpec.

    The optional keyword argument compat can be used to specify a string for use in the Project.toml of the generated Julia package.

    The optional keyword argument platforms is a vector of AbstractPlatforms which indicates for which platforms the dependency should be used. By default platforms=[AnyPlatform()], to mean that the dependency is compatible with all platforms.

    The optional keyword argument top_level specifies whether the dependency should be use only at the top-level of the generated JLL package, instead of inside each platform-specific wrapper. Using top_level=true is useful for packages needed for platform augmentation (e.g. MPIPreferences.jl).

    source
    BinaryBuilderBase.UserNSRunner — Type
    UserNSRunner

    A UserNSRunner represents an "execution context", an object that bundles all necessary information to run commands within the container that contains our crossbuild environment. Use run() to actually run commands within the UserNSRunner, and runshell() as a quick way to get an interactive shell within the crossbuild environment.

    source
    BinaryBuilder.Wizard.WizardState — Type
    WizardState

    Building large dependencies can take a lot of time. This state object captures all relevant state of this function. It can be passed back to the function to resume where we left off. This can aid debugging when code changes are necessary. It also holds all necessary metadata such as input/output streams.

    source

    Functions

    BinaryBuilderBase.abi_agnostic — Method
    abi_agnostic(p::AbstractPlatform)

    Strip out any tags that are not the basic annotations like libc and call_abi.

    source
    BinaryBuilderBase.accept_apple_sdk — Method
    accept_apple_sdk(ins::IO, outs::IO) -> Bool

    Ask the user whether they accept the terms of the macOS SDK, and return a boolean with their choice. Write messages to outs, read input from ins.

    source
    BinaryBuilderBase.artifact_name — Method
    artifact_name(cs::CompilerShard)

    Return the bound artifact name for a particular shard.

    source
    BinaryBuilderBase.bindir — Method
    bindir(prefix::Prefix)

    Returns the binary directory for the given prefix.

    source
    BinaryBuilderBase.choose_shards — Method
    choose_shards(p::AbstractPlatform; rootfs_build, ps_build, GCC_builds,
+"sfml-audio"

    If the library would have different basenames on different operating systems (e.g., libz.so on Linux and FreeBSD, libz.dylib on macOS, and zlib.dll on Windows), libname can be also a vector of Strings with the different alternatives:

    LibraryProduct(["libz", "zlib"], :libz)
    source
    BinaryBuilderBase.Product — Type

    A Product is an expected result after building or installation of a package.

    Examples of Products include LibraryProduct, FrameworkProduct, ExecutableProduct and FileProduct. All Product types must define the following minimum set of functionality:

    • locate(::Product): given a Product, locate it within the wrapped Prefix returning its location as a string

    • satisfied(::Product): given a Product, determine whether it has been successfully satisfied (e.g. it is locateable and it passes all callbacks)

    • variable_name(::Product): return the variable name assigned to a Product

    • repr(::Product): Return a representation of this Product, useful for auto-generating source code that constructs Products, if that's your thing.

    source
    BinaryBuilderBase.RuntimeDependency — Type
    RuntimeDependency(dep::Union{PackageSpec,String}; compat::String, platforms::Vector{<:AbstractPlatform}, top_level::Bool=false)

    Define a binary dependency that is only listed as dependency of the generated JLL package, but its artifact is not installed in the prefix during the build. The dep argument can be either a string with the name of the JLL package or a Pkg.PackageSpec.

    The optional keyword argument compat can be used to specify a string for use in the Project.toml of the generated Julia package.

    The optional keyword argument platforms is a vector of AbstractPlatforms which indicates for which platforms the dependency should be used. By default platforms=[AnyPlatform()], to mean that the dependency is compatible with all platforms.

    The optional keyword argument top_level specifies whether the dependency should be use only at the top-level of the generated JLL package, instead of inside each platform-specific wrapper. Using top_level=true is useful for packages needed for platform augmentation (e.g. MPIPreferences.jl).

    source
    BinaryBuilderBase.UserNSRunner — Type
    UserNSRunner

    A UserNSRunner represents an "execution context", an object that bundles all necessary information to run commands within the container that contains our crossbuild environment. Use run() to actually run commands within the UserNSRunner, and runshell() as a quick way to get an interactive shell within the crossbuild environment.

    source
    BinaryBuilder.Wizard.WizardState — Type
    WizardState

    Building large dependencies can take a lot of time. This state object captures all relevant state of this function. It can be passed back to the function to resume where we left off. This can aid debugging when code changes are necessary. It also holds all necessary metadata such as input/output streams.

    source

    Functions

    BinaryBuilderBase.abi_agnostic — Method
    abi_agnostic(p::AbstractPlatform)

    Strip out any tags that are not the basic annotations like libc and call_abi.

    source
    BinaryBuilderBase.accept_apple_sdk — Method
    accept_apple_sdk(ins::IO, outs::IO) -> Bool

    Ask the user whether they accept the terms of the macOS SDK, and return a boolean with their choice. Write messages to outs, read input from ins.

    source
    BinaryBuilderBase.artifact_name — Method
    artifact_name(cs::CompilerShard)

    Return the bound artifact name for a particular shard.

    source
    BinaryBuilderBase.bindir — Method
    bindir(prefix::Prefix)

    Returns the binary directory for the given prefix.

    source
    BinaryBuilderBase.choose_shards — Method
    choose_shards(p::AbstractPlatform; rootfs_build, ps_build, GCC_builds,
                            LLVM_builds, archive_type)

    This method chooses, given a Platform, which shards to download, extract and mount, returning a list of CompilerShard objects. At the moment, this always consists of four shards, but that may not always be the case.

    source
    BinaryBuilderBase.chown_cleanup — Method
    chown_cleanup(dr::DockerRunner)

    On Linux, the user id inside of the docker container doesn't correspond to ours on the outside, so permissions get all kinds of screwed up. To fix this, we have to chown -R $(id -u):$(id -g) $prefix, which really sucks, but is still better than nothing. This is why we prefer the UserNSRunner on Linux.

    source
    BinaryBuilderBase.collect_jll_uuids — Method
    collect_jll_uuids(manifest::Pkg.Types.Manifest, dependencies::Set{Base.UUID})

    Return a Set of all JLL packages in the manifest with dependencies being the list of direct dependencies of the environment.

    source
    BinaryBuilderBase.compress_dir — Method
    compress_dir(dir::AbstractString;
              compressor_stream = GzipCompressorStream,
              level::Int = 9,
@@ -128,9 +128,9 @@
           skip_audit = false, ignore_audit_errors = true,
           autofix = true, code_dir = nothing,
           meta_json_file = nothing, require_license = true,
-          dont_dlopen = false, kwargs...)

    Runs the boiler plate code to download, build, and package a source package for a list of platforms. This method takes a veritable truckload of arguments, here are the relevant actors, broken down in brief:

    • dir: the root of the build; products will be placed within dir/products, and mountpoints will be placed within dir/build/.

    • src_name: the name of the source package being built and will set the name of the built tarballs.

    • src_version: the version of the source package.

    • platforms: a list of platforms to build for.

    • sources: a vector of all sources to download and unpack before building begins, as AbstractSources.

    • script: a string representing a shell script to run as the build.

    • products: the list of Products which shall be built.

    • dependencies: a vector of JLL dependency packages as AbstractDependency that should be installed before building begins.

    • verbose: Enable verbose mode. What did you expect?

    • debug: cause a failed build to drop into an interactive shell so that the build can be inspected easily.

    • skip_audit: disable the typical audit that occurs at the end of a build.

    • ignore_audit_errors: do not kill a build even if a problem is found.

    • autofix: give BinaryBuilder permission to automatically fix issues it finds during audit passes. Highly recommended.

    • code_dir: sets where autogenerated JLL packages will be put.

    • require_license enables a special audit pass that requires licenses to be installed by all packages.

    • dont_dlopen: don't try to dlopen library products. This is separate from specifying dont_dlopen on a LibraryProduct in that it still results in the generated JLL loading the library at run time, and only prevents BinaryBuilder from doing so during JLL generation.

    source
    BinaryBuilder.get_compilers_versions — Method
    get_compilers_versions(; compilers = [:c])

    Return the script string that is used to print the versions of the given compilers.

    source
    BinaryBuilder.preferred_platform_compiler — Function
    preferred_platform_compiler(platforms::Vector{<:Platform}, version::Union{Nothing,VersionNumber}=nothing)

    Initializes a dictionary with an entry for every platform given that will store platform-specific compiler versions (if any).

    source
    BinaryBuilder.set_preferred_compiler_version! — Method
    set_preferred_compiler_version!(compiler_versions::Dict{Platform, Union{Nothing,VersionNumber}},
+          dont_dlopen = false, kwargs...)

    Runs the boiler plate code to download, build, and package a source package for a list of platforms. This method takes a veritable truckload of arguments, here are the relevant actors, broken down in brief:

    • dir: the root of the build; products will be placed within dir/products, and mountpoints will be placed within dir/build/.

    • src_name: the name of the source package being built and will set the name of the built tarballs.

    • src_version: the version of the source package.

    • platforms: a list of platforms to build for.

    • sources: a vector of all sources to download and unpack before building begins, as AbstractSources.

    • script: a string representing a shell script to run as the build.

    • products: the list of Products which shall be built.

    • dependencies: a vector of JLL dependency packages as AbstractDependency that should be installed before building begins.

    • verbose: Enable verbose mode. What did you expect?

    • debug: cause a failed build to drop into an interactive shell so that the build can be inspected easily.

    • skip_audit: disable the typical audit that occurs at the end of a build.

    • ignore_audit_errors: do not kill a build even if a problem is found.

    • autofix: give BinaryBuilder permission to automatically fix issues it finds during audit passes. Highly recommended.

    • code_dir: sets where autogenerated JLL packages will be put.

    • require_license enables a special audit pass that requires licenses to be installed by all packages.

    • dont_dlopen: don't try to dlopen library products. This is separate from specifying dont_dlopen on a LibraryProduct in that it still results in the generated JLL loading the library at run time, and only prevents BinaryBuilder from doing so during JLL generation.

    source
    BinaryBuilder.get_compilers_versions — Method
    get_compilers_versions(; compilers = [:c])

    Return the script string that is used to print the versions of the given compilers.

    source
    BinaryBuilder.preferred_platform_compiler — Function
    preferred_platform_compiler(platforms::Vector{<:Platform}, version::Union{Nothing,VersionNumber}=nothing)

    Initializes a dictionary with an entry for every platform given that will store platform-specific compiler versions (if any).

    source
    BinaryBuilder.set_preferred_compiler_version! — Method
    set_preferred_compiler_version!(compiler_versions::Dict{Platform, Union{Nothing,VersionNumber}},
                                 version::VersionNumber,
-                                selector::Union{Vector{<:Platform},Function})

    Set the preferred compiler version for the platforms matching selector to version.

    source
    Base.BinaryPlatforms.detect_cxxstring_abi — Method
    detect_cxxstring_abi(oh::ObjectHandle, platform::AbstractPlatform)

    Given an ObjectFile, examine its symbols to discover which (if any) C++11 std::string ABI it's using. We do this by scanning the list of exported symbols, triggering off of instances of St7__cxx11 or _ZNSs to give evidence toward a constraint on cxx11, cxx03 or neither.

    source
    Base.BinaryPlatforms.detect_libstdcxx_version — Method
    detect_libstdcxx_version(oh::ObjectHandle, platform::AbstractPlatform)

    Given an ObjectFile, examine its dynamic linkage to discover which (if any) libgfortran it's linked against. The major SOVERSION will determine which GCC version we're restricted to.

    source
    BinaryBuilder.Auditor.analyze_instruction_set — Method
    analyze_instruction_set(oh::ObjectHandle, platform::AbstractPlatform; verbose::Bool = false)

    Analyze the instructions within the binary located at the given path for which minimum instruction set it requires, taking note of groups of instruction sets used such as avx, sse4.2, i486, etc....

    Some binary files (such as libopenblas) contain multiple versions of functions, internally determining which version to call by using the cpuid instruction to determine processor support. In an effort to detect this, we make note of any usage of the cpuid instruction, disabling our minimum instruction set calculations if such an instruction is found, and notifying the user of this if verbose is set to true.

    Note that this function only really makes sense for x86/x64 binaries. Don't run this on armv7l, aarch64, ppc64le etc... binaries and expect it to work.

    source
    BinaryBuilder.Auditor.audit — Function
    audit(prefix::Prefix, src_name::AbstractString = "";
+                                selector::Union{Vector{<:Platform},Function})

    Set the preferred compiler version for the platforms matching selector to version.

    source
    Base.BinaryPlatforms.detect_cxxstring_abi — Method
    detect_cxxstring_abi(oh::ObjectHandle, platform::AbstractPlatform)

    Given an ObjectFile, examine its symbols to discover which (if any) C++11 std::string ABI it's using. We do this by scanning the list of exported symbols, triggering off of instances of St7__cxx11 or _ZNSs to give evidence toward a constraint on cxx11, cxx03 or neither.

    source
    Base.BinaryPlatforms.detect_libstdcxx_version — Method
    detect_libstdcxx_version(oh::ObjectHandle, platform::AbstractPlatform)

    Given an ObjectFile, examine its dynamic linkage to discover which (if any) libgfortran it's linked against. The major SOVERSION will determine which GCC version we're restricted to.

    source
    BinaryBuilder.Auditor.analyze_instruction_set — Method
    analyze_instruction_set(oh::ObjectHandle, platform::AbstractPlatform; verbose::Bool = false)

    Analyze the instructions within the binary located at the given path for which minimum instruction set it requires, taking note of groups of instruction sets used such as avx, sse4.2, i486, etc....

    Some binary files (such as libopenblas) contain multiple versions of functions, internally determining which version to call by using the cpuid instruction to determine processor support. In an effort to detect this, we make note of any usage of the cpuid instruction, disabling our minimum instruction set calculations if such an instruction is found, and notifying the user of this if verbose is set to true.

    Note that this function only really makes sense for x86/x64 binaries. Don't run this on armv7l, aarch64, ppc64le etc... binaries and expect it to work.

    source
    BinaryBuilder.Auditor.audit — Function
    audit(prefix::Prefix, src_name::AbstractString = "";
                       io=stderr,
                       platform::AbstractPlatform = HostPlatform(),
                       verbose::Bool = false,
@@ -138,17 +138,17 @@
                       autofix::Bool = false,
                       has_csl::Bool = true,
                       require_license::Bool = true,
-      )

    Audits a prefix to attempt to find deployability issues with the binary objects that have been installed within. This auditing will check for relocatability issues such as dependencies on libraries outside of the current prefix, usage of advanced instruction sets such as AVX2 that may not be usable on many platforms, linkage against newer glibc symbols, etc...

    This method is still a work in progress, only some of the above list is actually implemented, be sure to actually inspect Auditor.jl to see what is and is not currently in the realm of fantasy.

    source
    BinaryBuilder.Auditor.check_license — Function
    check_license(prefix, src_name; verbose::Bool = false,, silent::Bool = false)

    Check that there are license files for the project called src_name in the prefix.

    source
    BinaryBuilder.Auditor.collapse_symlinks — Method
    collapse_symlinks(files::Vector{String})

    Given a list of files, prune those that are symlinks pointing to other files within the list.

    source
    BinaryBuilder.Auditor.collect_files — Function
    collect_files(path::AbstractString, predicate::Function = f -> true)

    Find all files that satisfy predicate() when the full path to that file is passed in, returning the list of file paths.

    source
    BinaryBuilder.Auditor.compatible_marchs — Method
    compatible_marchs(p::AbstractPlatform)

    Return a (sorted) list of compatible microarchitectures, starting from the most compatible to the most highly specialized. If no microarchitecture is specified within p, returns the most generic microarchitecture possible for the given architecture.

    source
    BinaryBuilder.Auditor.detect_libgfortran_version — Method
    detect_libgfortran_version(oh::ObjectHandle, platform::AbstractPlatform)

    Given an ObjectFile, examine its dynamic linkage to discover which (if any) libgfortran it's linked against. The major SOVERSION will determine which GCC version we're restricted to.

    source
    BinaryBuilder.Auditor.instruction_mnemonics — Method
    instruction_mnemonics(path::AbstractString, platform::AbstractPlatform)

    Dump a binary object with objdump, returning a list of instruction mnemonics for further analysis with analyze_instruction_set().

    Note that this function only really makes sense for x86/x64 binaries. Don't run this on armv7l, aarch64, ppc64le etc... binaries and expect it to work.

    This function returns the list of mnemonics as well as the counts of each, binned by the mapping defined within instruction_categories.

    source
    BinaryBuilder.Auditor.is_for_platform — Method
    is_for_platform(h::ObjectHandle, platform::AbstractPlatform)

    Returns true if the given ObjectHandle refers to an object of the given platform; E.g. if the given platform is for AArch64 Linux, then h must be an ELFHandle with h.header.e_machine set to ELF.EM_AARCH64.

    In particular, this method and platform_for_object() both exist because the latter is not smart enough to deal with :glibc and :musl yet.

    source
    BinaryBuilder.Auditor.is_troublesome_library_link — Method
    is_troublesome_library_link(libname::AbstractString, platform::AbstractPlatform)

    Return true if depending on libname is known to cause problems at runtime, false otherwise.

    source
    BinaryBuilder.Auditor.minimum_march — Method
    minimum_march(counts::Dict, p::AbstractPlatform)

    This function returns the minimum instruction set required, depending on whether the object file being pointed to is a 32-bit or 64-bit one:

    • For 32-bit object files, this returns one of ["i686", "prescott"]

    • For 64-bit object files, this returns one of ["x86_64", "avx", "avx2", "avx512"]

    source
    BinaryBuilder.Auditor.platform_for_object — Method
    platform_for_object(oh::ObjectHandle)

    Returns the platform the given ObjectHandle should run on. E.g. if the given ObjectHandle is an x86_64 Linux ELF object, this function will return Platform("x86_64", "linux"). This function does not yet distinguish between different libc's such as :glibc and :musl.

    source
    BinaryBuilder.Auditor.symlink_soname_lib — Method
    symlink_soname_lib(path::AbstractString)

    We require that all shared libraries are accessible on disk through their SONAME (if it exists). While this is almost always true in practice, it doesn't hurt to make doubly sure.

    source
    BinaryBuilder.Auditor.translate_symlinks — Method
    translate_symlinks(root::AbstractString; verbose::Bool=false)

    Walks through the root directory given within root, finding all symlinks that point to an absolute path within root, and rewriting them to be a relative symlink instead, increasing relocatability.

    source
    BinaryBuilder.Auditor.update_linkage — Method
    update_linkage(prefix::Prefix, platform::AbstractPlatform, path::AbstractString,
-               old_libpath, new_libpath; verbose::Bool = false)

    Given a binary object located at path within prefix, update its dynamic linkage to point to new_libpath instead of old_libpath. This is done using a tool within the cross-compilation environment such as install_name_tool on MacOS or patchelf on Linux. Windows platforms are completely skipped, as they do not encode paths or RPaths within their executables.

    source
    BinaryBuilder.Auditor.warn_deadlinks — Method
    warn_deadlinks(root::AbstractString)

    Walks through the given root directory, finding broken symlinks and warning the user about them. This is used to catch instances such as a build recipe copying a symlink that points to a dependency; by doing so, it implicitly breaks relocatability.

    source
    BinaryBuilder.Wizard.canonicalize_file_url — Method

    Canonicalize URL to a file within a GitHub repo

    source
    BinaryBuilder.Wizard.canonicalize_source_url — Method

    Canonicalize a GitHub repository URL

    source
    BinaryBuilder.Wizard.change_script! — Method
    Change the script. This will invalidate all platforms to make sure we later
-verify that they still build with the new script.
    source
    BinaryBuilder.Wizard.download_source — Method
    download_source(state::WizardState)

    Ask the user where the source code is coming from, then download and record the relevant parameters, returning the source url, the local path it is stored at after download, and a hash identifying the version of the code. In the case of a git source URL, the hash will be a git treeish identifying the exact commit used to build the code, in the case of a tarball, it is the sha256 hash of the tarball itself.

    source
    BinaryBuilder.Wizard.edit_script — Method
    edit_script(state::WizardState, script::AbstractString)

    For consistency (and security), use the sandbox for editing a script, launching vi within an interactive session to edit a buildscript.

    source
    BinaryBuilder.Wizard.filter_object_files — Method
    filter_object_files(files)

    Given a list of files, filter out any that cannot be opened by readmeta() from ObjectFile.

    source
    BinaryBuilder.Wizard.interactive_build — Method
    interactive_build(state::WizardState, prefix::Prefix,
+      )

    Audits a prefix to attempt to find deployability issues with the binary objects that have been installed within. This auditing will check for relocatability issues such as dependencies on libraries outside of the current prefix, usage of advanced instruction sets such as AVX2 that may not be usable on many platforms, linkage against newer glibc symbols, etc...

    This method is still a work in progress, only some of the above list is actually implemented, be sure to actually inspect Auditor.jl to see what is and is not currently in the realm of fantasy.

    source
    BinaryBuilder.Auditor.check_license — Function
    check_license(prefix, src_name; verbose::Bool = false,, silent::Bool = false)

    Check that there are license files for the project called src_name in the prefix.

    source
    BinaryBuilder.Auditor.collapse_symlinks — Method
    collapse_symlinks(files::Vector{String})

    Given a list of files, prune those that are symlinks pointing to other files within the list.

    source
    BinaryBuilder.Auditor.collect_files — Function
    collect_files(path::AbstractString, predicate::Function = f -> true)

    Find all files that satisfy predicate() when the full path to that file is passed in, returning the list of file paths.

    source
    BinaryBuilder.Auditor.compatible_marchs — Method
    compatible_marchs(p::AbstractPlatform)

    Return a (sorted) list of compatible microarchitectures, starting from the most compatible to the most highly specialized. If no microarchitecture is specified within p, returns the most generic microarchitecture possible for the given architecture.

    source
    BinaryBuilder.Auditor.detect_libgfortran_version — Method
    detect_libgfortran_version(oh::ObjectHandle, platform::AbstractPlatform)

    Given an ObjectFile, examine its dynamic linkage to discover which (if any) libgfortran it's linked against. The major SOVERSION will determine which GCC version we're restricted to.

    source
    BinaryBuilder.Auditor.instruction_mnemonics — Method
    instruction_mnemonics(path::AbstractString, platform::AbstractPlatform)

    Dump a binary object with objdump, returning a list of instruction mnemonics for further analysis with analyze_instruction_set().

    Note that this function only really makes sense for x86/x64 binaries. Don't run this on armv7l, aarch64, ppc64le etc... binaries and expect it to work.

    This function returns the list of mnemonics as well as the counts of each, binned by the mapping defined within instruction_categories.

    source
    BinaryBuilder.Auditor.is_for_platform — Method
    is_for_platform(h::ObjectHandle, platform::AbstractPlatform)

    Returns true if the given ObjectHandle refers to an object of the given platform; E.g. if the given platform is for AArch64 Linux, then h must be an ELFHandle with h.header.e_machine set to ELF.EM_AARCH64.

    In particular, this method and platform_for_object() both exist because the latter is not smart enough to deal with :glibc and :musl yet.

    source
    BinaryBuilder.Auditor.is_troublesome_library_link — Method
    is_troublesome_library_link(libname::AbstractString, platform::AbstractPlatform)

    Return true if depending on libname is known to cause problems at runtime, false otherwise.

    source
    BinaryBuilder.Auditor.minimum_march — Method
    minimum_march(counts::Dict, p::AbstractPlatform)

    This function returns the minimum instruction set required, depending on whether the object file being pointed to is a 32-bit or 64-bit one:

    • For 32-bit object files, this returns one of ["i686", "prescott"]

    • For 64-bit object files, this returns one of ["x86_64", "avx", "avx2", "avx512"]

    source
    BinaryBuilder.Auditor.platform_for_object — Method
    platform_for_object(oh::ObjectHandle)

    Returns the platform the given ObjectHandle should run on. E.g. if the given ObjectHandle is an x86_64 Linux ELF object, this function will return Platform("x86_64", "linux"). This function does not yet distinguish between different libc's such as :glibc and :musl.

    source
    BinaryBuilder.Auditor.symlink_soname_lib — Method
    symlink_soname_lib(path::AbstractString)

    We require that all shared libraries are accessible on disk through their SONAME (if it exists). While this is almost always true in practice, it doesn't hurt to make doubly sure.

    source
    BinaryBuilder.Auditor.translate_symlinks — Method
    translate_symlinks(root::AbstractString; verbose::Bool=false)

    Walks through the root directory given within root, finding all symlinks that point to an absolute path within root, and rewriting them to be a relative symlink instead, increasing relocatability.

    source
    BinaryBuilder.Auditor.update_linkage — Method
    update_linkage(prefix::Prefix, platform::AbstractPlatform, path::AbstractString,
+               old_libpath, new_libpath; verbose::Bool = false)

    Given a binary object located at path within prefix, update its dynamic linkage to point to new_libpath instead of old_libpath. This is done using a tool within the cross-compilation environment such as install_name_tool on MacOS or patchelf on Linux. Windows platforms are completely skipped, as they do not encode paths or RPaths within their executables.

    source
    BinaryBuilder.Auditor.warn_deadlinks — Method
    warn_deadlinks(root::AbstractString)

    Walks through the given root directory, finding broken symlinks and warning the user about them. This is used to catch instances such as a build recipe copying a symlink that points to a dependency; by doing so, it implicitly breaks relocatability.

    source
    BinaryBuilder.Wizard.canonicalize_file_url — Method

    Canonicalize URL to a file within a GitHub repo

    source
    BinaryBuilder.Wizard.canonicalize_source_url — Method

    Canonicalize a GitHub repository URL

    source
    BinaryBuilder.Wizard.change_script! — Method
    Change the script. This will invalidate all platforms to make sure we later
+verify that they still build with the new script.
    source
    BinaryBuilder.Wizard.download_source — Method
    download_source(state::WizardState)

    Ask the user where the source code is coming from, then download and record the relevant parameters, returning the source url, the local path it is stored at after download, and a hash identifying the version of the code. In the case of a git source URL, the hash will be a git treeish identifying the exact commit used to build the code, in the case of a tarball, it is the sha256 hash of the tarball itself.

    source
    BinaryBuilder.Wizard.edit_script — Method
    edit_script(state::WizardState, script::AbstractString)

    For consistency (and security), use the sandbox for editing a script, launching vi within an interactive session to edit a buildscript.

    source
    BinaryBuilder.Wizard.filter_object_files — Method
    filter_object_files(files)

    Given a list of files, filter out any that cannot be opened by readmeta() from ObjectFile.

    source
    BinaryBuilder.Wizard.interactive_build — Method
    interactive_build(state::WizardState, prefix::Prefix,
                   ur::Runner, build_path::AbstractString)
 
 Runs the interactive shell for building, then captures bash history to save
-reproducible steps for building this source. Shared between steps 3 and 5
    source
    BinaryBuilder.Wizard.match_files — Method
    match_files(state::WizardState, prefix::Prefix,
-            platform::AbstractPlatform, files::Vector; silent::Bool = false)

    Inspects all binary files within a prefix, matching them with a given list of files, complaining if there are any files that are not properly matched and returning the set of normalized names that were not matched, or an empty set if all names were properly matched.

    source
    BinaryBuilder.Wizard.normalize_name — Method
    normalize_name(file::AbstractString)

    Given a filename, normalize it, stripping out extensions. E.g. the file path "foo/libfoo.tar.gz" would get mapped to "libfoo".

    source
    BinaryBuilder.Wizard.pick_preferred_platform — Method

    Pick the first platform for use to run on. We prefer Linux x86_64 because that's generally the host platform, so it's usually easiest. After that we go by the following preferences:

    • OS (in order): Linux, Windows, OSX
    • Architecture: x86_64, i686, aarch64, powerpc64le, armv7l, riscv64
    • The first remaining after this selection
    source
    BinaryBuilder.Wizard.print_autoconf_hint — Method
    print_autoconf_hint(state::WizardState)

    Print a hint for projects that use autoconf to have a good ./configure line.

    source
    BinaryBuilder.Wizard.provide_hints — Method
    provide_hints(state::WizardState, path::AbstractString)

    Given an unpacked source directory, provide hints on how a user might go about building the binary bounty they so richly desire.

    source
    BinaryBuilder.Wizard.step1 — Method
    step1(state::WizardState)

    It all starts with a single step, the unabashed ambition to leave your current stability and engage with the universe on a quest to create something new, beautiful and unforeseen. It all ends with compiler errors.

    This step selects the relevant platform(s) for the built binaries.

    source
    BinaryBuilder.Wizard.step2 — Method
    step2(state::WizardState)

    This step obtains the source code to be built and required binary dependencies.

    source
    BinaryBuilder.Wizard.step34 — Method
    step34(state::WizardState)

    Starts initial build for Linux x86_64, which is our initial test target platform. Sources that build properly for this platform continue on to attempt builds for more complex platforms.

    source
    BinaryBuilder.Wizard.step3_audit — Method
    step3_audit(state::WizardState, platform::AbstractPlatform, prefix::Prefix)

    Audit the prefix.

    source
    BinaryBuilder.Wizard.step3_interactive — Method
    step3_interactive(state::WizardState, prefix::Prefix, platform::AbstractPlatform,
-                  ur::Runner, build_path::AbstractString)

    The interactive portion of step3, moving on to either rebuild with an edited script or proceed to step 4.

    source
    BinaryBuilder.Wizard.step3_retry — Method
    step3_retry(state::WizardState)

    Rebuilds the initial Linux x86_64 build after things like editing the script file manually, etc...

    source
    BinaryBuilder.Wizard.step4 — Method
    step4(state::WizardState, ur::Runner, platform::AbstractPlatform,
-      build_path::AbstractString, prefix::Prefix)

    The fourth step selects build products after the first build is done

    source
    BinaryBuilder.Wizard.with_gitcreds — Method
    with_gitcreds(f, username::AbstractString, password::AbstractString)

    Calls f with an LibGit2.UserPasswordCredential object as an argument, constructed from the username and password values. with_gitcreds ensures that the credentials object gets properly shredded after it's no longer necessary. E.g.:

    julia with_gitcreds(user, token) do creds LibGit2.clone("https://github.com/foo/bar.git", "bar"; credentials=creds) end`

    source
    BinaryBuilder.Wizard.yggdrasil_build_tarballs_path — Method
    yggdrasil_build_tarballs_path(name::String)

    Return the relative path within an Yggdrasil clone where this project (given its name) would be stored. This is useful for things like generating the build_tarballs.jl file and checking to see if it already exists, etc...

    Note that we do not allow case-ambiguities within Yggdrasil, we check for this using the utility function case_insensitive_file_exists(path).

    source
    BinaryBuilder.Wizard.yggdrasil_deploy — Function
    yggdrasil_deploy(state::WizardState)

    Write out a WizardState to a build_tarballs.jl in an Yggdrasil clone, then open a pull request against Yggdrasil.

    source
    BinaryBuilder.Wizard.yn_prompt — Function
    yn_prompt(state::WizardState, question::AbstractString, default = :y)

    Perform a [Y/n] or [y/N] question loop, using default to choose between the prompt styles, and looping until a proper response (e.g. "y", "yes", "n" or "no") is received.

    source

    Command Line

    BinaryBuilder.build_tarballs — Function
    build_tarballs(ARGS, src_name, src_version, sources, script, platforms,
-               products, dependencies; kwargs...)

    This should be the top-level function called from a build_tarballs.jl file. It takes in the information baked into a build_tarballs.jl file such as the sources to download, the products to build, etc... and will automatically download, build and package the tarballs, generating a build.jl file when appropriate.

    Generally, ARGS should be the top-level Julia ARGS command-line arguments object. build_tarballs does some rudimentary parsing of the arguments. To see what it can do, you can call it with --help in the ARGS or see the Command Line section in the manual.

    The kwargs are passed on to autobuild, see there for a list of supported ones. A few additional keyword arguments are accept:

    • julia_compat can be set to a version string which is used to set the supported Julia version in the [compat] section of the Project.toml of the generated JLL package. The default value is "1.0".

    • lazy_artifacts sets whether the artifacts should be lazy.

    • init_block may be set to a string containing Julia code; if present, this code will be inserted into the initialization path of the generated JLL package. This can for example be used to invoke an initialization API of a shared library.

    • augment_platform_block may be set to a string containing Julia code; if present, this code will be inserted into the top-level of the generated JLL package. It must define a function augment_platform! that takes as a single argument, the target platform and returns the target platform, with amended tags. This augmented platform will then be used by the JLL wrapper to select the artifact. Note that this option requires the Julia compatibility julia_compat to be 1.6 or higher.

    • validate_name ensures that src_name constitutes a valid Julia identifier. Since the generated JLL package is named according to src_name, this should only be set to false if you really know what you're doing.

    Note

    The init_block and augment_platform_block keyword arguments are experimental and may be removed in a future version of this package. Please use them sparingly.

    source

    The build_tarballs function also parses command line arguments. The syntax is described in the --help output:

    Usage: build_tarballs.jl [target1,target2,...] [--help]
+reproducible steps for building this source. Shared between steps 3 and 5
    source
    BinaryBuilder.Wizard.match_files — Method
    match_files(state::WizardState, prefix::Prefix,
+            platform::AbstractPlatform, files::Vector; silent::Bool = false)

    Inspects all binary files within a prefix, matching them with a given list of files, complaining if there are any files that are not properly matched and returning the set of normalized names that were not matched, or an empty set if all names were properly matched.

    source
    BinaryBuilder.Wizard.normalize_name — Method
    normalize_name(file::AbstractString)

    Given a filename, normalize it, stripping out extensions. E.g. the file path "foo/libfoo.tar.gz" would get mapped to "libfoo".

    source
    BinaryBuilder.Wizard.pick_preferred_platform — Method

    Pick the first platform for use to run on. We prefer Linux x86_64 because that's generally the host platform, so it's usually easiest. After that we go by the following preferences:

    • OS (in order): Linux, Windows, OSX
    • Architecture: x86_64, i686, aarch64, powerpc64le, armv7l, riscv64
    • The first remaining after this selection
    source
    BinaryBuilder.Wizard.print_autoconf_hint — Method
    print_autoconf_hint(state::WizardState)

    Print a hint for projects that use autoconf to have a good ./configure line.

    source
    BinaryBuilder.Wizard.provide_hints — Method
    provide_hints(state::WizardState, path::AbstractString)

    Given an unpacked source directory, provide hints on how a user might go about building the binary bounty they so richly desire.

    source
    BinaryBuilder.Wizard.step1 — Method
    step1(state::WizardState)

    It all starts with a single step, the unabashed ambition to leave your current stability and engage with the universe on a quest to create something new, beautiful and unforeseen. It all ends with compiler errors.

    This step selects the relevant platform(s) for the built binaries.

    source
    BinaryBuilder.Wizard.step2 — Method
    step2(state::WizardState)

    This step obtains the source code to be built and required binary dependencies.

    source
    BinaryBuilder.Wizard.step34 — Method
    step34(state::WizardState)

    Starts initial build for Linux x86_64, which is our initial test target platform. Sources that build properly for this platform continue on to attempt builds for more complex platforms.

    source
    BinaryBuilder.Wizard.step3_audit — Method
    step3_audit(state::WizardState, platform::AbstractPlatform, prefix::Prefix)

    Audit the prefix.

    source
    BinaryBuilder.Wizard.step3_interactive — Method
    step3_interactive(state::WizardState, prefix::Prefix, platform::AbstractPlatform,
+                  ur::Runner, build_path::AbstractString)

    The interactive portion of step3, moving on to either rebuild with an edited script or proceed to step 4.

    source
    BinaryBuilder.Wizard.step3_retry — Method
    step3_retry(state::WizardState)

    Rebuilds the initial Linux x86_64 build after things like editing the script file manually, etc...

    source
    BinaryBuilder.Wizard.step4 — Method
    step4(state::WizardState, ur::Runner, platform::AbstractPlatform,
+      build_path::AbstractString, prefix::Prefix)

    The fourth step selects build products after the first build is done

    source
    BinaryBuilder.Wizard.with_gitcreds — Method
    with_gitcreds(f, username::AbstractString, password::AbstractString)

    Calls f with an LibGit2.UserPasswordCredential object as an argument, constructed from the username and password values. with_gitcreds ensures that the credentials object gets properly shredded after it's no longer necessary. E.g.:

    julia with_gitcreds(user, token) do creds LibGit2.clone("https://github.com/foo/bar.git", "bar"; credentials=creds) end`

    source
    BinaryBuilder.Wizard.yggdrasil_build_tarballs_path — Method
    yggdrasil_build_tarballs_path(name::String)

    Return the relative path within an Yggdrasil clone where this project (given its name) would be stored. This is useful for things like generating the build_tarballs.jl file and checking to see if it already exists, etc...

    Note that we do not allow case-ambiguities within Yggdrasil, we check for this using the utility function case_insensitive_file_exists(path).

    source
    BinaryBuilder.Wizard.yggdrasil_deploy — Function
    yggdrasil_deploy(state::WizardState)

    Write out a WizardState to a build_tarballs.jl in an Yggdrasil clone, then open a pull request against Yggdrasil.

    source
    BinaryBuilder.Wizard.yn_prompt — Function
    yn_prompt(state::WizardState, question::AbstractString, default = :y)

    Perform a [Y/n] or [y/N] question loop, using default to choose between the prompt styles, and looping until a proper response (e.g. "y", "yes", "n" or "no") is received.

    source

    Command Line

    BinaryBuilder.build_tarballs — Function
    build_tarballs(ARGS, src_name, src_version, sources, script, platforms,
+               products, dependencies; kwargs...)

    This should be the top-level function called from a build_tarballs.jl file. It takes in the information baked into a build_tarballs.jl file such as the sources to download, the products to build, etc... and will automatically download, build and package the tarballs, generating a build.jl file when appropriate.

    Generally, ARGS should be the top-level Julia ARGS command-line arguments object. build_tarballs does some rudimentary parsing of the arguments. To see what it can do, you can call it with --help in the ARGS or see the Command Line section in the manual.

    The kwargs are passed on to autobuild, see there for a list of supported ones. A few additional keyword arguments are accept:

    • julia_compat can be set to a version string which is used to set the supported Julia version in the [compat] section of the Project.toml of the generated JLL package. The default value is "1.0".

    • lazy_artifacts sets whether the artifacts should be lazy.

    • init_block may be set to a string containing Julia code; if present, this code will be inserted into the initialization path of the generated JLL package. This can for example be used to invoke an initialization API of a shared library.

    • augment_platform_block may be set to a string containing Julia code; if present, this code will be inserted into the top-level of the generated JLL package. It must define a function augment_platform! that takes as a single argument, the target platform and returns the target platform, with amended tags. This augmented platform will then be used by the JLL wrapper to select the artifact. Note that this option requires the Julia compatibility julia_compat to be 1.6 or higher.

    • validate_name ensures that src_name constitutes a valid Julia identifier. Since the generated JLL package is named according to src_name, this should only be set to false if you really know what you're doing.

    Note

    The init_block and augment_platform_block keyword arguments are experimental and may be removed in a future version of this package. Please use them sparingly.

    source

    The build_tarballs function also parses command line arguments. The syntax is described in the --help output:

    Usage: build_tarballs.jl [target1,target2,...] [--help]
                          [--verbose] [--debug]
                          [--deploy] [--deploy-bin] [--deploy-jll]
                          [--register] [--meta-json]
@@ -241,4 +241,4 @@
     x86_64-linux-musl
     x86_64-unknown-freebsd
     x86_64-w64-mingw32
-
    + diff --git a/previews/PR1360/rootfs/index.html b/previews/PR1360/rootfs/index.html index a54cc5a7..bb74f055 100644 --- a/previews/PR1360/rootfs/index.html +++ b/previews/PR1360/rootfs/index.html @@ -1,2 +1,2 @@ -RootFS · BinaryBuilder.jl
    GitHub

    RootFS

    The execution environment that all BinaryBuilder.jl builds are executed within is referred to as the "root filesystem" or RootFS. This RootFS is built using the builder scripts contained within the 0_Rootfs directory within Yggdrasil. The rootfs image is based upon the alpine docker image, and is used to build compilers for every target platform we support. The target platform compiler toolchains are stored within /opt/${triplet}, so the 64-bit Linux (using glibc as the backing libc) compilers would be found in /opt/x86_64-linux-gnu/bin.

    Each compiler "shard" is packaged separately, so that users do not have to download a multi-GB tarball just to build for a single platform. There is an overall "root" shard, along with platform support shards, GCC shards, an LLVM shard, Rust shards, etc... These are all embedded within the Artifacts.toml file in BinaryBuilderBase.jl, and BinaryBuilder.jl downloads them on-demand as necessary, making use of the new Pkg.Artifacts system within Julia 1.3+.

    Each shard is made available both as an unpacked directory tree, and as a .squashfs image. .squashfs images take up significantly less disk space, however they unfortunately require root privileges on the host machine, and only work on Linux. This will hopefully be fixed in a future Linux kernel release, but if you have sudo privileges, it is often desirable to use the .squashfs files to save network bandwidth and disk space. See the Environment Variables for instructions on how to do that.

    When launching a process within the RootFS image, BinaryBuilder.jl sets up a set of environment variables to enable a target-specific compiler toolchain, among other niceties. See the Build Tips doc page for more details on that, along with the src/Runner.jl file within this repository for the implementation.

    +RootFS · BinaryBuilder.jl
    GitHub

    RootFS

    The execution environment that all BinaryBuilder.jl builds are executed within is referred to as the "root filesystem" or RootFS. This RootFS is built using the builder scripts contained within the 0_Rootfs directory within Yggdrasil. The rootfs image is based upon the alpine docker image, and is used to build compilers for every target platform we support. The target platform compiler toolchains are stored within /opt/${triplet}, so the 64-bit Linux (using glibc as the backing libc) compilers would be found in /opt/x86_64-linux-gnu/bin.

    Each compiler "shard" is packaged separately, so that users do not have to download a multi-GB tarball just to build for a single platform. There is an overall "root" shard, along with platform support shards, GCC shards, an LLVM shard, Rust shards, etc... These are all embedded within the Artifacts.toml file in BinaryBuilderBase.jl, and BinaryBuilder.jl downloads them on-demand as necessary, making use of the new Pkg.Artifacts system within Julia 1.3+.

    Each shard is made available both as an unpacked directory tree, and as a .squashfs image. .squashfs images take up significantly less disk space, however they unfortunately require root privileges on the host machine, and only work on Linux. This will hopefully be fixed in a future Linux kernel release, but if you have sudo privileges, it is often desirable to use the .squashfs files to save network bandwidth and disk space. See the Environment Variables for instructions on how to do that.

    When launching a process within the RootFS image, BinaryBuilder.jl sets up a set of environment variables to enable a target-specific compiler toolchain, among other niceties. See the Build Tips doc page for more details on that, along with the src/Runner.jl file within this repository for the implementation.

    diff --git a/previews/PR1360/tricksy_gotchas/index.html b/previews/PR1360/tricksy_gotchas/index.html index 4ad7f6ea..5eee0d87 100644 --- a/previews/PR1360/tricksy_gotchas/index.html +++ b/previews/PR1360/tricksy_gotchas/index.html @@ -1,2 +1,2 @@ -Tricksy Gotchas · BinaryBuilder.jl
    GitHub

    Tricksy Gotchas

    There are a plethora of gotchas when it comes to binary compilation and distribution that must be appropriately addressed, or the binaries will only work on certain machines and not others. Here is an incomplete list of things that BinaryBuilder.jl takes care of for you:

    • Uniform compiler interface

    No need to worry about invoking compilers through weird names; just run gcc within the proper environment and you'll get the appropriate cross-compiler. Triplet-prefixed names (such as x86_64-linux-gnu-gcc) are, of course, also available, and the same version of gcc, g++ and gfortran is used across all platforms.

    • glibc versioning

    On Linux platforms that use glibc as the C runtime library (at the time of writing, this is the great majority of most desktop and server distros), it is necessary to compile code against a version of glibc that is older than any glibc version it will be run on. E.g. if your code is compiled against glibc v2.5, it will run on glibc v2.6, but it will not run on glibc v2.4. Therefore, to maximize compatibility, all code should be compiled against as old a version of glibc as possible.

    • gfortran versioning

    When compiling FORTRAN code, the gfortran compiler has broken ABI compatibility in the 6.X -> 7.X transition, and the 7.X -> 8.X transition. This means that code built with gfortran 6.X cannot be linked against code built with gfortran 7.X. We therefore compile all gfortran code against multiple different gfortran versions, then at runtime decide which to download based upon the currently running process' existing linkage.

    • cxx11 string ABI

    When switching from the cxx03 standard to cxx11 in GCC 5, the internal layout of std::string objects changed. This causes incompatibility between C++ code passing strings back and forth across the public interface if they are not built with the same C++ string ABI. We therefore detect when std::string objects are being passed around, and warn that you need to build two different versions, one with cxx03-style strings (doable by setting -D_GLIBCXX_USE_CXX11_ABI=0 for newer GCC versions) and one with cxx11-style strings.

    • Library Dependencies

    A large source of problems in binary distribution is improper library linkage. When building a binary object that depends upon another binary object, some operating systems (such as macOS) bake the absolute path to the dependee library into the dependent, whereas others rely on the library being present within a default search path. BinaryBuilder.jl takes care of this by automatically discovering these errors and fixing them by using the RPATH/RUNPATH semantics of whichever platform it is targeting. Note that this is technically a build system error, and although we will fix it automatically, it will raise a nice yellow warning during build prefix audit time.

    • Embedded absolute paths

    Similar to library dependencies, plain files (and even symlinks) can have the absolute location of files embedded within them. BinaryBuilder.jl will automatically transform symlinks to files within the build prefix to be the equivalent relative path, and will alert you if any files within the prefix contain absolute paths to the build prefix within them. While the latter cannot be automatically fixed, it may help in tracking down problems with the software later on.

    • Instruction Set Differences

    When compiling for architectures that have evolved over time (such as x86_64), it is important to target the correct instruction set, otherwise a binary may contain instructions that will run on the computer it was compiled on, but will fail rather ungracefully when run on a machine that does not have as new a processor. BinaryBuilder.jl will automatically disassemble every built binary object and inspect the instructions used, warning the user if a binary is found that does not conform to the agreed-upon minimum instruction set architecture. It will also notice if the binary contains a cpuid instruction, which is a good sign that the binary is aware of this issue and will internally switch itself to use only available instructions.

    +Tricksy Gotchas · BinaryBuilder.jl
    GitHub

    Tricksy Gotchas

    There are a plethora of gotchas when it comes to binary compilation and distribution that must be appropriately addressed, or the binaries will only work on certain machines and not others. Here is an incomplete list of things that BinaryBuilder.jl takes care of for you:

    • Uniform compiler interface

    No need to worry about invoking compilers through weird names; just run gcc within the proper environment and you'll get the appropriate cross-compiler. Triplet-prefixed names (such as x86_64-linux-gnu-gcc) are, of course, also available, and the same version of gcc, g++ and gfortran is used across all platforms.

    • glibc versioning

    On Linux platforms that use glibc as the C runtime library (at the time of writing, this is the great majority of most desktop and server distros), it is necessary to compile code against a version of glibc that is older than any glibc version it will be run on. E.g. if your code is compiled against glibc v2.5, it will run on glibc v2.6, but it will not run on glibc v2.4. Therefore, to maximize compatibility, all code should be compiled against as old a version of glibc as possible.

    • gfortran versioning

    When compiling FORTRAN code, the gfortran compiler has broken ABI compatibility in the 6.X -> 7.X transition, and the 7.X -> 8.X transition. This means that code built with gfortran 6.X cannot be linked against code built with gfortran 7.X. We therefore compile all gfortran code against multiple different gfortran versions, then at runtime decide which to download based upon the currently running process' existing linkage.

    • cxx11 string ABI

    When switching from the cxx03 standard to cxx11 in GCC 5, the internal layout of std::string objects changed. This causes incompatibility between C++ code passing strings back and forth across the public interface if they are not built with the same C++ string ABI. We therefore detect when std::string objects are being passed around, and warn that you need to build two different versions, one with cxx03-style strings (doable by setting -D_GLIBCXX_USE_CXX11_ABI=0 for newer GCC versions) and one with cxx11-style strings.

    • Library Dependencies

    A large source of problems in binary distribution is improper library linkage. When building a binary object that depends upon another binary object, some operating systems (such as macOS) bake the absolute path to the dependee library into the dependent, whereas others rely on the library being present within a default search path. BinaryBuilder.jl takes care of this by automatically discovering these errors and fixing them by using the RPATH/RUNPATH semantics of whichever platform it is targeting. Note that this is technically a build system error, and although we will fix it automatically, it will raise a nice yellow warning during build prefix audit time.

    • Embedded absolute paths

    Similar to library dependencies, plain files (and even symlinks) can have the absolute location of files embedded within them. BinaryBuilder.jl will automatically transform symlinks to files within the build prefix to be the equivalent relative path, and will alert you if any files within the prefix contain absolute paths to the build prefix within them. While the latter cannot be automatically fixed, it may help in tracking down problems with the software later on.

    • Instruction Set Differences

    When compiling for architectures that have evolved over time (such as x86_64), it is important to target the correct instruction set, otherwise a binary may contain instructions that will run on the computer it was compiled on, but will fail rather ungracefully when run on a machine that does not have as new a processor. BinaryBuilder.jl will automatically disassemble every built binary object and inspect the instructions used, warning the user if a binary is found that does not conform to the agreed-upon minimum instruction set architecture. It will also notice if the binary contains a cpuid instruction, which is a good sign that the binary is aware of this issue and will internally switch itself to use only available instructions.

    diff --git a/previews/PR1360/troubleshooting/index.html b/previews/PR1360/troubleshooting/index.html index 688c6c2b..c1ed707e 100644 --- a/previews/PR1360/troubleshooting/index.html +++ b/previews/PR1360/troubleshooting/index.html @@ -41,4 +41,4 @@ "Clang" - version 12.0.0.

    This issue is caused by not setting CMake policy CMP0025. This policy is supposed to only affect the CompilerId for AppleClang, but it also has the effect of turning off feature detection for upstream clang (which is what we're using here) on CMake versions prior to CMake 3.18. Add

    cmake_policy(SET CMP0025 NEW)

    At the very top of your CMakeLists.txt, before the project definition (or get an updated version of CMake).

    + version 12.0.0.

    This issue is caused by not setting CMake policy CMP0025. This policy is supposed to only affect the CompilerId for AppleClang, but it also has the effect of turning off feature detection for upstream clang (which is what we're using here) on CMake versions prior to CMake 3.18. Add

    cmake_policy(SET CMP0025 NEW)

    At the very top of your CMakeLists.txt, before the project definition (or get an updated version of CMake).