REP: 111 Title: Multiple Package Manager Support for Rosdep Author: Tully Foote, Dirk Thomas Status: Final Type: Standards Track Content-Type: text/x-rst Created: 28-June-2011 ROS-Version: 1.6 Post-History: 30-Jun-2011, 24-Jan-2018
This REP add support to the rosdep YAML format [1] to support multiple package managers on an individual platform. This includes a a new 'source' package manager that replacees the previous mechanisms for performing source-based installs.
Previously, the rosdep format has enabled two modes of installation of system dependencies: the platform's preferred package manager, or a custom bash script. Both of these installation mechanisms were limiting.
Using only a single package manager limits integration options on platforms like OS X, where there are multiple mechanisms available for installing system libraries (e.g. MacPorts, distribute/easy_install). For example, for Python libraries, it can be beneficial to use distribute, which works with the default Python installation on OS X 10.6, instead of using the MacPorts Python libraries, which require additional setup by the user.
Similarly, the bash script specification was brittle and provided limited information to rosdep itself. The scripts were required to be idempotent because rosdep was unable to detect whether or not the dependency was already installed.
The goal of this REP is to make rosdep more robust and flexible, as well as enable better support of platforms other than Ubuntu.
The current implementation of rosdep in ROS Diamondback [2] supports the following structures:
Simple
ROSDEP_NAME: OS_NAME1: PACKAGE_ARGUMENTS_A OS_NAME2: PACKAGE_ARGUMENTS_AVersioned
ROSDEP_NAME: OS_NAME1: OS_VERSION1: PACKAGE_ARGUMENTS_A OS_VERSION2: PACKAGE_ARGUMENTS_A2
The names above resolve as follows:
ROSDEP_NAMEis the name referred to by manifest files. Examples:log4cxx,gtest.OS_NAMEis the name of an OS. Examples:ubuntu,fedora,debian,windows,macports.OS_VERSION(''optional'') is the name of specific versions in the OS. Examples:'10.04',squeeze. If noOS_VERSIONis specified, the rule is assumed to apply to all versions.PACKAGE_ARGUMENThas two interpretations. If it is a single line, it is expected to be a list of packages to pass to the package manager for the OS. If this field is a multi-line script, it is an idempotent bash script that is executed to install the dependency.
Example:
python:
ubuntu: python-dev
fedora: python-devel
macports: py26-scipy
boost:
ubuntu:
'11.04': libboost.42-all-dev
'10.10': libboost.42-all-dev
For many Linux-based platforms, there are dominant package managers that are the main mechanism for deploying software, such as deb/apt for Debian-based systems and rpm/yum for RedHat-based systems. However, for non-Linux platforms, as well as for cross-platform language environments (e.g. Python, Java, Ruby), there exist other software deployment mechanisms that are commonly used. There are also platforms like OS X and Windows, where no package manager has a dominant role.
On OS X, rosdep currently uses MacPorts as the default package manager. While MacPorts does provide most of the libraries necessary for integrating ROS on an OS X system, the current implementation limits the flexibility of integration options. One area of integration we wish to improve is Python. The MacPorts Python integration requires a separate Python tree from the default OS X Python interpreter. This has proven to be very brittle for users and a common are of misconfiguration. Now that OS X now deploys with a Python version that is compatible with ROS, we wish to explore using distribute/easy_install for Python libraries instead.
The current support for bash scripts requires these scripts to be idempotent: they must detect the presence of the required dependency and install it if it is not present. There is no mechanism for rosdep to determine for itself whether the required dependency is installed, nor is there any mechanism for removing dependencies installed via this mechanism. This creates confusion on the part of the user as there is no mechanism for analyzing whether or not these dependencies are installed correctly. Creating more structure for source-based installations will enable simpler installation scripts, better error reporting, and more advanced features (e.g. uninstall) in the future.
OS X dependencies are specified generally as macports and are not
versioned to a particular OS X release. OS X releases frequently have
significant differences as Apple does not maintain much backwards
compatibility.
The current ROS OS X integration suffers from 32-bit/64-bit compatibility problems due to the OS X 10.6 release. The current rosdep specification also does not allow integrators to take advantage of the builtin Python 2.5+ interpeter that was introduced in the OS X 10.5 release.
Another limitation of the OS X integration is that the rosdep syntax did not allow for specification of MacPorts "variants" for individual packages. The rosdep syntax interpreted the package argument value as a list of packages, which was not flexible enough for specifying variants on specific packages.
The current OS version identifiers use the release number (e.g. 11.04)
instead of the release codename (e.g. lucid). Using the release
number has been a frequent source of bugs as YAML interprets 11.04
as a floating point number instead of a string. This requires
explicit string escaping when specifying rules (e.g. '11.04').
The new specification for rosdep introduces a PACKAGE_MANAGER
key in the YAML specification, which is a child key of the OS_NAME.
rosdep no longer interprets the PACKAGE_MANAGER_ARGUMENTS and instead
passes them to the specified PACKAGE_MANAGER handler.
- Simple
ROSDEP_NAME: OS_NAME1: PACKAGE_MANAGER1: PACKAGE_ARGUMENTS_A OS_NAME2: PACKAGE_MANAGER2: PACKAGE_ARGUMENTS_A
- Versioned
ROSDEP_NAME: OS_NAME1: OS_VERSION1: PACKAGE_MANAGER1: PACKAGE_ARGUMENTS_A OS_VERSION2: PACKAGE_MANAGER2: PACKAGE_ARGUMENTS_A2
The names above resolve as follows:
ROSDEP_NAMEis the name referred to by manifest files. Examples:log4cxxorgtest.OS_NAMEis the name of an OS. Examples:ubuntu,osx,fedora,debian, orwindows.OS_VERSION(optional) is the name of specific versions in the OS. Examples:lucidorsqueeze. If noOS_VERSIONis specified, the rule is assumed to apply to all versions.PACKAGE_MANAGER(optional in ROS Electric, required in ROS Fuerte) is a key to select which package manager to use for this rosdep. Examples:apt,easy_install,macports.PACKAGE_ARGUMENTis free-form YAML that is be passed to the handler for the specifiedPACKAGE_MANAGER.
For Ubuntu the default package manager is apt. The new syntax is:
rosdep_name:
ubuntu:
apt:
packages: [ debian-package-name, other-debian-package-name]
or versioned as follows:
rosdep_name:
ubuntu:
lucid:
apt:
packages: [debian-package-name, other-debian-package-name]
OS version identifiers will be standardized onto one-word codenames for each distro.
For example:
- debian: squeeze
- ubuntu: lucid, maverick, natty, oneiric
- osx: snow, lion
The OS_NAME key for OS X will be changed to osx and will start
including OS_VERSION keys that represent the version number of the
OS X releases (e.g. lion). The default package is macports,
which can be used as a PACKAGE_MANAGER field.
osx:
lion:
macports: bzip2
For backwards compatibility, the PACKAGE_MANAGER is allowed to be
optional in the ROS Electric case. As both PACKAGE_MANAGER and
OS_VERSION are optional, this creates an ambiguious case where
either OS_VERSION or PACKAGE_MANAGER is specified, but not
both.
In this ambiguous case, rosdep first interprets the key as a
PACKAGE_MANAGER. If this test fails, it will be interpreted as an
OS_VERSION. Developers should exercise caution in keeping
OS_VERSION and PACKAGE_MANAGER keys globally distinct.
Many rosdep keys exist with a generic mapping independent of the OS version. This is great since the rule doesn't have to be updated for new OS versions as long as the system package name stays the same. But when at some point the system package name does change there are only two choices which are both not optimal:
- Either all OS versions need to be explicitly enumerated and from that point on forward the entry needs to be updated for every new OS version or
- at some point the "old" OS versions before the system package was renamed are being dropped to collapse the mapping back into a generic rule again (in order to not require to update the entry in the future).
To avoid either of these downsides the OS_VERSION can be specified as *. When a specific rule is being looked up and the desired OS version isn't in the dictionary but an OS_VERSION * is present it will be used as a fallback (instead of failing the lookup).
This allows "future proofing" rosdep rules and avoids the need to update them for every future OS version without requiring an order on those to be defined (which would need to be maintained too).
Since the wildcard matches every not explicitly defined OS_VERSION it is not usable as-is in the following example:
rosdep_name:
debian:
jessie: [other-name]
wheezy: [some-name]
Using the wildcard to use other-name also for newer OS versions would look like this:
rosdep_name:
debian:
'*': [other-name]
wheezy: [some-name]
While other-name will now also be used for newer OS versions like stretch the wildcard will also be used for older releases which are not explicitly defined, like squeeze.
In order to declare that there is no valid mapping for squeeze and still be able to use the wildcard OS version it is allowed to use null as a PACKAGE_ARGUMENT:
rosdep_name:
debian:
'*': [other-name]
squeeze: null
wheezy: [some-name]
Although not specified in this REP, developers of new package managers
for rosdep are encouraged to include a dependency specification in
their PACKAGE_ARGUMENTS values. This dependency specification
should enable rosdep rules to specify dependencies on other rosdep
rules. This, for example, would enable an easy_install package to
also depend on a apt-based install. One use case of this is
rosinstall, which can be installed via easy_install, but also
implicitly requires non-Python libraries for Git, Subversion,
Mercurial, and other non-easy_install packages.
This advisory does not apply to package managers that are builtin to a particular platform, such as apt and yum.
Internally, the rosdep library will provide APIs for developers to trigger installation of these dependencies.
A new source-based package manager is discussed in the motivation for this REP. This specification of this new source-based package manager will be described in a separate forthcoming REP.
rosdep rules are only allowed to specify a single package manager to fulfill them.
All current rosdep.yaml files are compatible with the new proposed syntax, though some features will be deprecated and the files should be upgraded during the ROS Electric cycle to maintain compatibility.
If no package manager is specified, the previous default package manager will be used.
Defaults:
- ubuntu: apt
- debian: apt
- mint: apt
- osx: macports
- freebsd: port
Support for the default package manger will be maintained upto and including ROS Fuerte.
Multi-line values will still be treated as bash scripts in ROS Electric. Support for this will be removed in ROS Fuerte.
The macports rules will be supported during ROS Electric.
For the following codenames backwards compatabilty will be built into the tool to automatically translate rules:
- '10.04' -> lucid
- '10.10' -> maverick
- '11.04' -> natty
It is expected that all future releases will refer to the correct codename, e.g. 'oneiric'.
This will be supported through ROS Fuerte.
The sections Wildcard OS_VERSION and Allow PACKAGE_ARGUMENT to be explicitly null have been added.
| [1] | rosdep documentation (http://www.ros.org/wiki/rosdep) |
| [2] | rosdep.yaml format in ROS Diamondback (http://www.ros.org/wiki/rosdep/rosdep.yaml/diamondback) |
This document has been placed in the public domain.