Skip to content

Build from Source (Developer Workflow)

Alex Reustle edited this page Sep 17, 2020 · 10 revisions

Developing without conda-build

Normally to build the tools for release a developer would follow the conda-build procedures to automatically handle all that follows. However this method is redundant, slow and poorly suited for an iterative development and debugging style. Consequently we break the procedure into its component parts and enable each of them to be performed independently.

Development Requirements

The Fermitools is a compiled conda package written in C++ and python for MacOSX and Linux operating systems. To build the tools you will need:

  • A Linux or MacOSX based system
  • An anaconda distribution (Miniconda 2 or 3 will suffice)
  • fermi-repoman: `conda install -c fermi fermi-repoman
  • If using MacOSX you will need to install Xcode, or acquire the OSX SDK.

Quick Linux build example

Here is a quick overview example of the steps involved in preparing, downloading and building the tools from source. This example applies to linux only, there is an additional complication for macosx systems.

  1. Prepare your build environment, download anaconda and install fermi-repoman
conda install -c fermi fermi-repoman
git clone https://github.com/fermi-lat/fermi_dev_scripts.git
  1. create a conda environment with necessary dependencies
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
./fermi_dev_scripts/conda_fermi_build_deps_explicit.sh
conda activate my_fermi_env
  1. Download the source code and build the tools
export CPU_COUNT=4
source fermi_dev_scripts/build.sh

Conda Build-time Externals

Before building the tools the user must create a special conda environment for development and install into that environment all such external packages as are dependencies for the fermitools. These "Build dependencies" or "Build externals" can be acquired in various ways, but the most straightforward is to copy them explicitly from a successful past build of the tools. The exact mechanism for which has been coded into a suite of scripts available at fermi_dev_scripts.

FERMI_REF

In git a ref is a commit SHA, branch name, or git tag. Our system is designed to share branches and tags across multiple repositories. Each release and official beta of the Fermitools is automatically assigned a git tag in the fermi-lat github organization by our automated build pipeline. These tags follow the format Fermitools-X.X.X. Choose a common Tag or branch name and use that as your base for building dependencies.

Set needed Environment Variables

The development scripts read variables from the user's environment. These may be customized for your target REF, environment and hardware, etc.

# Linux
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
export CPU_COUNT=4
# MacOSX
export FERMI_REF=Fermitools-2.0.0
export FERMI_CONDA_ENV=my_fermi_env
export CPU_COUNT=4
export CONDA_BUILD_SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk  
export MACOSX_DEPLOYMENT_TARGET=10.9

On MacOSX you must set your CONDA_BUILD_SYSROOT and MACOSX_DEPLOYMENT_TARGET variables. If left unset they will be defaulted to the values below. MacOSX builds require an Xcode SDK file.

Create a conda environment populated with build-time dependencies (Explicit).

Create a conda environment from the exact dependency tarballs used to create the original build environment.

./conda_fermi_deps_explicit.sh
conda activate my_fermi_env

The scripts avail themselves of the conda package manager's --explicit flag to download tarballs from the URLs stored in a text file. This file is located in the fermi-lat github organization under one of 2 repositories: Fermitools-explicit-build-deps-linux Fermitools-explicit-build-deps-macosx with a git tag indicating which release it was compiled for.

The files and tags are uploaded automatically by the Azure Pipelines automated build pipeline.

Obtain Sources and Build the Tools

Sources for the fermitools are spread across multiple repositories in the fermi-lat github organization. In order to obtain them we have a repoman stage of the build script that optionally checks out the sources with a branch, tag, or commit hash taken from the $FERMI_REF environment variable. The exact list of packages to checkout is stored in Fermitools-conda under the packageList.txt file.

Optional Repoman checkout and scons build

./build.sh

Disable Repoman checkout during the build

export FERMI_NO_CHECKOUT=true
./build.sh

Disable SCONS Install step during the build

export FERMI_NO_INSTALL=true
./build.sh