[REP Index] [REP Source]

REP:111
Title:Multiple Package Manager Support for Rosdep
Author:Tully Foote, Dirk Thomas, Scott K Logan
Status:Final
Type:Standards Track
Content-Type:text/x-rst
Created:28-June-2011
ROS-Version:1.6
Post-History:30-Jun-2011, 24-Jan-2018, 12-Nov-2021

Abstract

This REP add support to the rosdep YAML format [1] to support multiple package managers on an individual platform. This includes a a new 'source' package manager that replacees the previous mechanisms for performing source-based installs.

Rationale

Previously, the rosdep format has enabled two modes of installation of system dependencies: the platform's preferred package manager, or a custom bash script. Both of these installation mechanisms were limiting.

Using only a single package manager limits integration options on platforms like OS X, where there are multiple mechanisms available for installing system libraries (e.g. MacPorts, distribute/easy_install). For example, for Python libraries, it can be beneficial to use distribute, which works with the default Python installation on OS X 10.6, instead of using the MacPorts Python libraries, which require additional setup by the user.

Similarly, the bash script specification was brittle and provided limited information to rosdep itself. The scripts were required to be idempotent because rosdep was unable to detect whether or not the dependency was already installed.

The goal of this REP is to make rosdep more robust and flexible, as well as enable better support of platforms other than Ubuntu.

Motivation

Existing Implementation

The current implementation of rosdep in ROS Diamondback [2] supports the following structures:

  • Simple

    ROSDEP_NAME:
  OS_NAME1: PACKAGE_ARGUMENTS_A
  OS_NAME2: PACKAGE_ARGUMENTS_A

  • Versioned

    ROSDEP_NAME:
  OS_NAME1:
    OS_VERSION1:  PACKAGE_ARGUMENTS_A
    OS_VERSION2:  PACKAGE_ARGUMENTS_A2

The names above resolve as follows:

  • ROSDEP_NAME is the name referred to by manifest files. Examples: log4cxx, gtest.
  • OS_NAME is the name of an OS. Examples: ubuntu, fedora, debian, windows, macports.
  • OS_VERSION (''optional'') is the name of specific versions in the OS. Examples: '10.04', squeeze. If no OS_VERSION is specified, the rule is assumed to apply to all versions.
  • PACKAGE_ARGUMENT has two interpretations. If it is a single line, it is expected to be a list of packages to pass to the package manager for the OS. If this field is a multi-line script, it is an idempotent bash script that is executed to install the dependency.

Example:

python:
  ubuntu: python-dev
  fedora: python-devel
  macports: py26-scipy
boost:
  ubuntu:
    '11.04': libboost.42-all-dev
    '10.10': libboost.42-all-dev

Support for Multiple Package Managers

For many Linux-based platforms, there are dominant package managers that are the main mechanism for deploying software, such as deb/apt for Debian-based systems and rpm/yum for RedHat-based systems. However, for non-Linux platforms, as well as for cross-platform language environments (e.g. Python, Java, Ruby), there exist other software deployment mechanisms that are commonly used. There are also platforms like OS X and Windows, where no package manager has a dominant role.

On OS X, rosdep currently uses MacPorts as the default package manager. While MacPorts does provide most of the libraries necessary for integrating ROS on an OS X system, the current implementation limits the flexibility of integration options. One area of integration we wish to improve is Python. The MacPorts Python integration requires a separate Python tree from the default OS X Python interpreter. This has proven to be very brittle for users and a common are of misconfiguration. Now that OS X now deploys with a Python version that is compatible with ROS, we wish to explore using distribute/easy_install for Python libraries instead.

Richer information from scripting interface

The current support for bash scripts requires these scripts to be idempotent: they must detect the presence of the required dependency and install it if it is not present. There is no mechanism for rosdep to determine for itself whether the required dependency is installed, nor is there any mechanism for removing dependencies installed via this mechanism. This creates confusion on the part of the user as there is no mechanism for analyzing whether or not these dependencies are installed correctly. Creating more structure for source-based installations will enable simpler installation scripts, better error reporting, and more advanced features (e.g. uninstall) in the future.

Integration with OS X

OS X dependencies are specified generally as macports and are not versioned to a particular OS X release. OS X releases frequently have significant differences as Apple does not maintain much backwards compatibility.

The current ROS OS X integration suffers from 32-bit/64-bit compatibility problems due to the OS X 10.6 release. The current rosdep specification also does not allow integrators to take advantage of the builtin Python 2.5+ interpeter that was introduced in the OS X 10.5 release.

Another limitation of the OS X integration is that the rosdep syntax did not allow for specification of MacPorts "variants" for individual packages. The rosdep syntax interpreted the package argument value as a list of packages, which was not flexible enough for specifying variants on specific packages.

OS version identifiers

The current OS version identifiers use the release number (e.g. 11.04) instead of the release codename (e.g. lucid). Using the release number has been a frequent source of bugs as YAML interprets 11.04 as a floating point number instead of a string. This requires explicit string escaping when specifying rules (e.g. '11.04').

Specification

The new specification for rosdep introduces a PACKAGE_MANAGER key in the YAML specification, which is a child key of the OS_NAME. rosdep no longer interprets the PACKAGE_MANAGER_ARGUMENTS and instead passes them to the specified PACKAGE_MANAGER handler.

  • Simple
    ROSDEP_NAME:
  OS_NAME1:
    PACKAGE_MANAGER1:
      PACKAGE_ARGUMENTS_A
  OS_NAME2:
    PACKAGE_MANAGER2:
      PACKAGE_ARGUMENTS_A
  • Versioned
    ROSDEP_NAME:
  OS_NAME1:
    OS_VERSION1:
      PACKAGE_MANAGER1:
        PACKAGE_ARGUMENTS_A
    OS_VERSION2:
      PACKAGE_MANAGER2:
        PACKAGE_ARGUMENTS_A2

The names above resolve as follows:

  • ROSDEP_NAME is the name referred to by manifest files. Examples: log4cxx or gtest.
  • OS_NAME is the name of an OS. Examples: ubuntu, osx, fedora, debian, or windows.
  • OS_VERSION (optional) is the name of specific versions in the OS. Examples: lucid or squeeze. If no OS_VERSION is specified, the rule is assumed to apply to all versions.
  • PACKAGE_MANAGER (optional in ROS Electric, required in ROS Fuerte) is a key to select which package manager to use for this rosdep. Examples: apt, easy_install, macports.
  • PACKAGE_ARGUMENT is free-form YAML that is be passed to the handler for the specified PACKAGE_MANAGER.

Example

For Ubuntu the default package manager is apt. The new syntax is:

rosdep_name:
  ubuntu:
    apt:
      packages: [ debian-package-name, other-debian-package-name]

or versioned as follows:

rosdep_name:
  ubuntu:
    lucid:
      apt:
        packages: [debian-package-name, other-debian-package-name]

OS version identifiers

OS version identifiers will be standardized onto one-word codenames for each distro.

For example:

  • debian: squeeze
  • ubuntu: lucid, maverick, natty, oneiric
  • osx: snow, lion

Changes to OS X

The OS_NAME key for OS X will be changed to osx and will start including OS_VERSION keys that represent the version number of the OS X releases (e.g. lion). The default package is macports, which can be used as a PACKAGE_MANAGER field.

osx:
  lion:
    macports: bzip2

Disambiguation of OS_VERSION and PACKAGE_MANAGER

For backwards compatibility, the PACKAGE_MANAGER is allowed to be optional in the ROS Electric case. As both PACKAGE_MANAGER and OS_VERSION are optional, this creates an ambiguious case where either OS_VERSION or PACKAGE_MANAGER is specified, but not both.

In this ambiguous case, rosdep first interprets the key as a PACKAGE_MANAGER. If this test fails, it will be interpreted as an OS_VERSION. Developers should exercise caution in keeping OS_VERSION and PACKAGE_MANAGER keys globally distinct.

Wildcard OS_VERSION

Many rosdep keys exist with a generic mapping independent of the OS version. This is great since the rule doesn't have to be updated for new OS versions as long as the system package name stays the same. But when at some point the system package name does change there are only two choices which are both not optimal:

  • Either all OS versions need to be explicitly enumerated and from that point on forward the entry needs to be updated for every new OS version or
  • at some point the "old" OS versions before the system package was renamed are being dropped to collapse the mapping back into a generic rule again (in order to not require to update the entry in the future).

To avoid either of these downsides the OS_VERSION can be specified as *. When a specific rule is being looked up and the desired OS version isn't in the dictionary but an OS_VERSION * is present it will be used as a fallback (instead of failing the lookup).

This allows "future proofing" rosdep rules and avoids the need to update them for every future OS version without requiring an order on those to be defined (which would need to be maintained too).

Allow PACKAGE_ARGUMENT to be explicitly null

Since the wildcard matches every not explicitly defined OS_VERSION it is not usable as-is in the following example:

rosdep_name:
  debian:
    jessie: [other-name]
    wheezy: [some-name]

Using the wildcard to use other-name also for newer OS versions would look like this:

rosdep_name:
  debian:
    '*': [other-name]
    wheezy: [some-name]

While other-name will now also be used for newer OS versions like stretch the wildcard will also be used for older releases which are not explicitly defined, like squeeze.

In order to declare that there is no valid mapping for squeeze and still be able to use the wildcard OS version it is allowed to use null as a PACKAGE_ARGUMENT:

rosdep_name:
  debian:
    '*': [other-name]
    squeeze: null
    wheezy: [some-name]

Wildcard OS_NAME

Some package managers are supported and function on more than one platform, and the names of packages in those package managers are typically the same between platforms. To avoid duplicating the rules under several OS_NAME stanzas the OS_NAME can be specified as * as long as the rule under it specifies the PACKAGE_MANAGER explicitly.

Similar to rule lookups regarding a Wildcard OS_VERSION, an explicit OS_NAME entry will take precedence over * rules entirely and an explicitly null PACKAGE_ARGUMENT for an OS_NAME will omit it from the wildcard.

Dependencies

Although not specified in this REP, developers of new package managers for rosdep are encouraged to include a dependency specification in their PACKAGE_ARGUMENTS values. This dependency specification should enable rosdep rules to specify dependencies on other rosdep rules. This, for example, would enable an easy_install package to also depend on a apt-based install. One use case of this is rosinstall, which can be installed via easy_install, but also implicitly requires non-Python libraries for Git, Subversion, Mercurial, and other non-easy_install packages.

This advisory does not apply to package managers that are builtin to a particular platform, such as apt and yum.

Internally, the rosdep library will provide APIs for developers to trigger installation of these dependencies.

New source-based package manager

A new source-based package manager is discussed in the motivation for this REP. This specification of this new source-based package manager will be described in a separate forthcoming REP.

Single package manager per rule

rosdep rules are only allowed to specify a single package manager to fulfill them.

Backwards Compatibility

All current rosdep.yaml files are compatible with the new proposed syntax, though some features will be deprecated and the files should be upgraded during the ROS Electric cycle to maintain compatibility.

Default Package Manager

If no package manager is specified, the previous default package manager will be used.

Defaults:

  • ubuntu: apt
  • debian: apt
  • mint: apt
  • osx: macports
  • freebsd: port

Support for the default package manger will be maintained upto and including ROS Fuerte.

Bash scripts

Multi-line values will still be treated as bash scripts in ROS Electric. Support for this will be removed in ROS Fuerte.

Macports

The macports rules will be supported during ROS Electric.

Ubuntu Codename Usage

For the following codenames backwards compatabilty will be built into the tool to automatically translate rules:

  • '10.04' -> lucid
  • '10.10' -> maverick
  • '11.04' -> natty

It is expected that all future releases will refer to the correct codename, e.g. 'oneiric'.

This will be supported through ROS Fuerte.