-
Notifications
You must be signed in to change notification settings - Fork 17
Build from Source (Developer Workflow)
Normally to build the tools for release a developer would follow the conda-build
procedures to automatically handle all that follows. However this method is redundant, slow and poorly suited for an iterative development and debugging style. Consequently we break the procedure into its component parts and enable each of them to be performed independently.
The Fermitools is a compiled conda package written in C++ and python for MacOSX and Linux operating systems. To build the tools you will need:
- A Linux or MacOSX based system
- An anaconda distribution (Miniconda 2 or 3 will suffice)
- fermi-repoman: `conda install -c fermi fermi-repoman
- If using MacOSX you will need to install Xcode, or acquire the OSX SDK.
Here is a quick overview example of the steps involved in preparing, downloading and building the tools from source. This example applies to linux only, there is an additional complication for macosx systems.
- Prepare your build environment, download anaconda and install fermi-repoman
conda install -c fermi fermi-repoman
git clone https://github.com/fermi-lat/fermi_dev_scripts.git
- create a conda environment with necessary dependencies
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
./fermi_dev_scripts/conda_fermi_build_deps_explicit.sh
conda activate my_fermi_env
- Download the source code and build the tools
export CPU_COUNT=4
source fermi_dev_scripts/build.sh
Before building the tools the user must create a special conda environment for development and install into that environment all such external packages as are dependencies for the fermitools. These "Build dependencies" or "Build externals" can be acquired in various ways, but the most straightforward is to copy them explicitly from a successful past build of the tools. The exact mechanism for which has been coded into a suite of scripts available at fermi_dev_scripts.
In git a ref is a commit SHA, branch name, or git tag. Our system is designed to share branches and tags across multiple repositories.
Each release and official beta of the Fermitools is automatically assigned a git tag in the fermi-lat github organization by our automated build pipeline. These tags follow the format Fermitools-X.X.X
. Choose a common Tag or branch name and use that as your base for building dependencies.
The development scripts read variables from the user's environment. These may be customized for your target REF, environment and hardware, etc.
# Linux
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
export CPU_COUNT=4
# MacOSX
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
export CPU_COUNT=4
export CONDA_BUILD_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
export MACOSX_DEPLOYMENT_TARGET=10.9
On MacOSX you must set your CONDA_BUILD_SYSROOT
and MACOSX_DEPLOYMENT_TARGET
variables. If left unset they will be defaulted to the values below.
MacOSX builds require an Xcode SDK file.
Create a conda environment from the exact dependency tarballs used to create the original build environment.
./conda_fermi_deps_explicit.sh
conda activate my_fermi_env
The scripts avail themselves of the conda package manager's --explicit
flag to download tarballs from the URLs stored in a text file. This file is located in the fermi-lat github organization under one of 2 repositories: Fermitools-explicit-build-deps-linux Fermitools-explicit-build-deps-macosx with a git tag indicating which release it was compiled for.
The files and tags are uploaded automatically by the Azure Pipelines automated build pipeline.
Sources for the fermitools are spread across multiple repositories in the fermi-lat github organization. In order to obtain them we have a repoman stage of the build script that optionally checks out the sources with a branch, tag, or commit hash taken from the $FERMI_REF
environment variable. The exact list of packages to checkout is stored in Fermitools-conda under the packageList.txt file.
./build.sh
export FERMI_NO_CHECKOUT=true
./build.sh
export FERMI_NO_INSTALL=true
./build.sh