boost_python_compile_notes

My travels with building Boost.Python

Objective: Build Boost.Python on Ubuntu Linux against an existing, specific conda environment that is using Python3.11. I had a few system libs already installed, and as they are needed in this hocus pocus, you will need to make sure your "local" system deb install of the python version matches the conda python version. This is due to some "dev" libraries that I'm not sure how to install into a conda env, but I know with certainty how to install into a system with deb files using apt install. In my case, my conda version is python 3.11, and my system version is python 3.11. On my system I had run these commands in my history, and having them installed made a difference when I tried this on a new machine:
sudo apt install python3.11-dev libpython3.11-dev
sudo apt install python3.11-full

Steps:

1. Download boost. As of this writing: boost_1_82_0.tar.gz
2. tar -xzvf boost_1_82_0.tar.gz
3. sudo mv boost_1_82_0 /usr/local/boost

Decide which python environment you want to build against. In this case, I want to build against my conda xxx environment, which uses Python 3.11.

4. Find these files within the specified conda environment, and record their paths:
   libpython3.11.so : ~/miniconda3/envs/xxx/lib/libpython3.11.so

   Python.h : ~/miniconda3/envs/xxx/include/python3.11/Python.h

   python3.11 executable : ~/miniconda3/envs/xxx/bin/python3.11

5. Create a directory to hold all your freshly-built Boost Python stuff:

mkdir ~/3Boost

Prepare your bootstrap.sh arguments (adjust all paths for your own environment):

./bootstrap.sh --with-python=/home/rl/miniconda3/envs/xxx/bin/python3.11 --with-python-root=/home/rl/miniconda3/envs/xxx --with-python-version=3.11 --with-libraries=python --with-icu --with-icu=/home/rl/miniconda3/envs/xxx/lib --prefix=/home/rl/3Boost --exec-prefix=/home/rl/3Boost --libdir=/home/rl/3Boost/lib --includedir=/home/rl/3Boost/include

// bootstrap overwrites your project-config.jam.  But a lot of the arguments I passed to bootstrap were to build the project-config.jam file properly anyway.  Nevertheless, always cat project-config.jam to assert that it contains what you need it to contain.

NOTE:

Don't shortcut paths with ~/ for home. In any command or config, write out the full path: /home/rl/whatever/... instead of ~/

When you run bootstrap, it will backup but also write over any work you may have done to project-config.jam. If you already made the changes below to project-config.jam, it will likely now be renamed project-config.jam.1. Or some other number. In all regards, at this point, you must make sure that the file, project-config.jam is the file with your paths and notes. You know what to do....

File: project-config.jam Location: /usr/local/boost/project-config.jam

project-config.jam file contents:

cat project-config.jam
# B2 Configuration
# Automatically generated by bootstrap.sh, but we make changes below before running ./b2 ....

import option ;
import feature ;

# Compiler configuration. This definition will be used unless
# you already have defined some toolsets in your user-config.jam
# file.
if ! gcc in [ feature.values <toolset> ]
{
    using gcc ; 
}

project : default-build <toolset>gcc ;

# Python configuration  (I hard-coded my home/rl path.  Shortcutting with ~/ does not work.  Fix path to suit your proper /home/whatever/ path.
import python ;
if ! [ python.configured ]
{
    using python 
    : 3.11 
    : /home/rl/miniconda3/envs/xxx/bin/python3.11
    : /home/rl/miniconda3/envs/xxx/include/python3.11
    : /home/rl/miniconda3/envs/xxx/lib
    ;
}

path-constant ICU_PATH : /usr ;   #not sure, but I think this is why I needed the apt install .. system files mentioned earlier.


# List of --with-<library> and --without-<library>
# options. If left empty, all libraries will be built.
# Options specified on the command line completely
# override this variable.
libraries = --with-python ;

# These settings are equivalent to corresponding command-line
# options.
option.set prefix : /home/rl/3Boost ;
option.set exec-prefix : /home/rl/3Boost ;
option.set libdir : /home/rl/3Boost/lib ;
option.set includedir : /home/rl/3Boost/include ;

# Stop on first error
option.set keep-going : false ;
##### END of file 

Run ./b2 --help. To get a feel for things. When ready, run your own b2 with your own arguments:

cd /usr/local/boost
./b2 --with-python --prefix=/home/rl/3Boost  --stagedir=/home/rl/3Boost/stage  stage --build-type=complete  --build-dir=/home/rl/3Boost-build --layout=versioned --variant=release --link=shared threading=single,multi runtime-link=static,shared

I recommend adding this to the tail of the b2 command: > /home/rl/3Boost/b2_output.txt You can read that file to look for any errors in the build.

Boost.Python .so files will now be ready for use in your c++ library build location you specified earlier.

the super-important makefile:

command: make -f makefile_name.mak

File Contents:

PYTHON_VERSION = 3.11
PYTHON_INCLUDE = /home/rl/miniconda3/envs/xxx/include/python3.11

BOOST_INC = /usr/local/boost
BOOST_LIB = /home/rl/3Boost/stage/lib/

# compile mesh class
TARGET = calc_dist
SOURCES = calc_dist.cpp options.cpp
O_FILES = calc_dist.o options.o

# CRITICAL:  see the -lboost_python311-gcc11-x64-1_82 below?  if you use a more recent Boost library, adjust this to the correct version for you.   

# all commands on the 2nd line must be TAB-INDENTED
$(TARGET).so: $(O_FILES)
	g++ -shared -Wl,-rpath=/home/rl/3Boost/stage/lib -Wall -export-dynamic $(O_FILES) -L$(BOOST_LIB) -lboost_python311-gcc11-x64-1_82 -L/usr/lib/python3.11/config-3.11-x86_64-linux-gnu -lpython$(PYTHON_VERSION) -o $(TARGET).so

calc_dist.o: calc_dist.cpp
	g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c calc_dist.cpp

options.o: options.cpp
	g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c options.cpp

clean:
	rm -f *.o *.so

Note the use of rpath in the makefile build command. This is critical to do. In your python script, when you import calc_dist (or your own library name), the import will fail with an error message indicating it can't find one of the more arcane files in your Boost.Python .so files. Using rpath in your build of your own .so library fixes this.

Finally, don't ask me. I'm not a Boost expert. I was just struggling through the Boost.Python build process like most everyone, and documented my navigation of the process here, mostly for me, the next time I find myself needing to do this again.