boost-python-stdcon: Boost.Python Extension for Containers Automatic Conversion

Introduction

Often we use Python to collect some data and pass them to C++ functions for further process. Data is organized in Python builtin data structures (list, dict and etc), while C++ functions accept some standard containers or data structure as arguments. To bridge such data path from Python to C++ by using boost::python, we should either expose the C++ standard containers or data structures as Python extension classes or create wrapper functions that accept Python objects as arguments and in those wapper functions prepare the std containers with data extracted from the Python objects. Either way will involve boilplate and tedious coding jobs as the C++ interface grows.

boost-python-stdcon is a header-only library that automates the conversion between C++ standard containers/data structures and their Python counterparts. It embeds the automatic conversion code into boost::python namespace to enable it to recognize function arguments and results of std container types and to do the conversion in boost::python’s own wrapper functions.

Here is a simple example. Let’s expose the below simple C++ function to Python so that it can accept data in a Python list.

int sum(std::vector<int> const &vec);

With the stock boost::python library, we should create a wrapper function that accepts one Python list object and prepare the vector argument in that wrapper function manually.

#include <boost/python/extract.hpp>

int pySum(PyObject* obj)
{
  std::vector<int> vec;
  // Assuming obj is a list. Exact int values from obj and prepapre
  // vec.
  vec.reserve(PyList_Size(obj));
  for (Py_ssize_t i = 0; i < PyList_Size(obj); ++i) {
    vec.push_back(boost::python::extract<int>(PyList_GetItem(obj, i)));
  }
  return sum(vec); // Call sum
}

And finally, register the wrapper function to Python.

#include <boost/python.hpp>

BOOST_PYTHON_MODULE(foo) {
  boost::python::def("sum", &pySum);
}

But with boost-python-stdcon, the wrapper function is no longer required and sum can be registered to Python directly. All is done by including a provided header file vector_arg.hpp.

#include <vector_arg.hpp> // Provided by boost-python-stdcon

BOOST_PYTHON_MODULE(foo) {
  boost::python::def("sum", &sum);
}

Usage

This library is header-only. Download the code package and unpack it to somewhere in your local directory. Add the include directory to your C++ compiler’s include file search path. Then add the proper header file to enable boost::python for specific container types.

Argument Conversion

For converting python containers to C++ containers as function arguments, please include the proper header file listed in the table below.

C++ Container	Expecting	Header file for
	Python object type	argument conversion
vector	list	vector_arg.hpp
list	list	list_arg.hpp
forward_list	list	forward_list_arg.hpp
map	dict	map_arg.hpp
unordered_map	dict	unordered_map_arg.hpp
set	set or frozenset	set_arg.hpp
unordered_set	set or frozenset	unordered_set_arg.hpp
pair	tuple of size 2	pair_arg.hpp
tuple	tuple	tuple_arg.hpp
optional	Counterpart types or None	optional_arg.hpp
variant	Counterpart of any of the variant’s alternative types	variant_arg.hpp

Restriction on Rvalues

The argument conversion is done by creating temporary C++ container object and COPYING each element of Python container to that temporary C++ object. That means any change to the container itself in C++ side is discarded and cannot affect the original Python container. To reflect such semantic, the conversion is restricted to rvalue types as container value type, constant reference type and rvalue reference type. The following code shows some example of acceptable arguments.

void foo_0(std::vector<int> v); // A vector value, convertible
void foo_1(std::vector<int> const &v); // Constant reference to a
                                       // vector, convertible
void foo_2(std::vector<int> &&v); // rvalue reference to a vector,
                                  // convertible

void foo_3(std::vector<int> &v); // Reference to a vector, a lvalue
                                 // type, NOT convertible

Effective Range

In short, once an *_arg.hpp* file is included, it forces all arguments of its type of all registered functions to be expecting the corresponding Python containers, regardless whether the argument type has been registered as Python extended class, in the whole current compilation unit(the current C++ code file).

The reason is, boost-python-stdcon embeds the auto-conversion logics into boost.python by adding template specializations to its meta-functions. Such specializations preempt boost.python’s default behavior on extended classes and are effective in the current compilation unit. However, boost-python-stdcon doesn’t preempt those arguments of lvalues (a non-const reference or pointer) which are still registered to accept Python extended classes. The following code is an demostration of feasible mixing of auto-converted arguments with extended classes.

#include <list>
#include <vector_arg.hpp>

using namespace std;
namespace py = boost::python;

template<typename SEQ>
size_t seq_size(SEQ v){return v.size();}

void def_mixed()
{
  py::class_<vector<float>>("float_vector");
  py::class_<list<float>>("float_list");

  // These two functions are both registered as accepting
  // Python list, even if vector<float> is already registered as a Python
  // extension class, due to the included file <vector_arg.hpp>
  py::def("int_vector_size", &seq_size<vector<int>>);
  py::def("float_vector_size", &seq_size<vector<float>>);

  // However, this function is registered as accepting the Python
  // extension class of type vector<float>, as its argument is of
  // lvalue type that is not overwritten by boost-std-con
  py::def("float_vector_size_lvalue", &seq_size<vector<float>&>);

  // This function is also registered as accepting the Python
  // extension class of type list<float>, unless <list_arg.hpp> is
  // included
  py::def("float_list_size", &seq_size<list<float>>);
}

Combined Conversion

Combined containers and data structures are also supported by including header files of all the used container and data structure types. For example, to convert argument of type std::vector<std::pair<int, float>>, including <vector_arg.hpp> and <pair_arg.hpp> would be sufficient. The following code shows more examples.

#include <vector_arg.hpp>
#include <pair_arg.hpp>

using namespace std;
namespace py = boost::python;

int first_at(vector<pair<int, float>> const &v, size_t idx)
{
  return v[idx].first;
}

int size_of_first(pair<vector<int>, float> const &v)
{
  return v.first.size();
}

void def_combined()
{
  py::def("first_at", &first_at);
  py::def("size_of_first", &size_of_first);
}

Return Value Conversion

boost-python-stdcon provides a boost.python CallPolicy class template copy_return_value (defined in <copy_return_value.hpp>) that enable automatic copying of function return values of standard container and data structure type to corrresponding python data structures. It follows boost.python’s suggestion to support policy chaining by providing other policies as its template argument. Its definition is as:

template<typename Base = default_call_policies>
struct copy_return_value : Base
{
  typedef copy_return_value_dispatcher<Base> result_converter;
};

The below code is an example of basic usage of copy_return_value.

#include <copy_return_value.hpp>
#include <vector>

using namespace std;
namespace py = boost::python;

vector<int> lift(int v)
{
  vector<int> vec;
  vec.push_back(v);
  return vec;
}

void def_return_value()
{
  py::def("lift", &lift, py::copy_return_value<>());
}

As the name indicates, copy_return_value copies the elements a returned container or data structure holds to a new Python data structure and returns.

Supported Containers and Data Structures

It supports the conversion in ConvertionTable with the following exceptions.

• multimap and multiset are not convertible as there is no equivalent data structure in Python
• set and unordered_set are always converted to Python set at now. If conversion to frozenset is indeed necessary, please file an issue.

Restriction on Rvalues

Just like argument conversion, return value conversion supports rvalue types only, which includes value types and constant references. The example below show its usage.

#include <copy_return_value.hpp>

using namespace std;
namespace py = boost::python;

typedef vector<int> ivector;

ivector make_vec(size_t sz)
{
  ivector vec;
  for (size_t i = 0; i < sz; ++i) {
    vec.push_back(i);
  }
  return vec;
}

ivector const & update_vec(int v)
{
  static ivector vec;
  vec.push_back(v);
  return vec;
}


void def_return_value() {
  py::def("make_vec", &make_vec, py::copy_return_value<>());
  py::def("update_vec", &update_vec, py::copy_return_value<>());
}

Effective Range

Unlike argument conversion, return value conversion is specified as callpolicy at registering specific function. Auto-coverted functions can be mixed with those returning registered extension classes without conflicts.

Combined Conversion

Combined types of standard containers and data structures are also supported, without any extra setup, as the following example shows.

#include <copy_return_value.hpp>
using namespace std;
namespace py = boost::python;

optional<vector<pair<int, int>>> return_combined(size_t sz)
{
  if (sz > 0) {
    vector<pair<int, int>> vec;
    for (size_t i = 0; i < sz; ++i)
      vec.emplace_back(i, i);
    return vec;
  } else {
    return nullopt;
  }
}

void def_return_value() {
  py::def("return_combined", &return_combined, py::copy_return_value<>());
}