rosdep keys

Some Python rosdep keys have <platform>_python3 entries. This was an experiment at having a rosdep key conditionally evaluate to either a Python 2 or Python 3 system dependency. No new <platform>_python3 entries should be added, and the entries should not be relied on.

All existing Python rosdep keys including <platform>_python3 entries will stay the same to avoid breaking unknown existing use cases. Even if a key refers to a Python 2 package on one platform and Python 3 on another, it should stay that way.

For new Python keys, when a platform has separate Python 2 and Python 3 versions of a package, there should be a rosdep key for each. For example, consider the two keys python-catkin-pkg and python3-catkin-pkg. The python-catkin-pkg key should resolve to a Python 2 dependency on any platform where one exists, while python3-catkin-pkg should only resolve to a Python 3 dependency.

ROS_PYTHON_VERSION

ROS_PYTHON_VERSION is an environment variable that is set to either 2 or 3 to indicate the major version of Python supported by a ROS distro. The intent is to allow package.xml files to have conditional Python dependencies.

It is set by the package ros_environment when sourcing a ROS workspace. The version set will be dictated by REP 3 [4]. For example, in Melodic it defaults to 2, but in Noetic it must be 3.

Users building ROS from source will not have an existing ROS workspace set, so the variable won't be set, and conditional dependencies can't be evaluated when using rosdep to install them. To fix this, rosdep will set ROS_PYTHON_VERSION. It must choose a Python version to match the ROS distro while allowing platforms that already use Python 3 to override it

The choice is made using the following information:

• the command line option --rosdistro
• the environment variable ROS_DISTRO
• the environment variable ROS_PYTHON_VERSION
• the value python_version in the ROS distribution file [8]
• the version of Python used to invoke rosdep via sys.version[0]

If --rosdistro is set then the value of python_version in the ROS distro file is used. For the purpose of evaluating conditional dependencies, ROS_PYTHON_VERSION will be set to python_version, and ROS_DISTRO will be set to the value passed in via --rosdistro. This allows users who may be unaware they have an existing ROS workspace sourced to install dependencies for a different ROS distro.

Otherwise, if ROS_PYTHON_VERSION is set then that value is used. The value of ROS_DISTRO is unchanged whether it is set or unset. This allows users to install dependencies for a ROS distro with a different Python version than the ROS distro targets, such as Arch Linux users using Python 3 for ROS Melodic.

If ROS_PYTHON_VERSION is not set but ROS_DISTRO is then for the purpose of evaluating conditional dependencies ROS_PYTHON_VERSION will be set to the value of python_version. This makes sure ROS_PYTHON_VERSION dependencies get evaluated in a way that matches the desired distro. This case should not happen so long as the user has sourced a workspace with the ros_environment package, since that package sets both ROS_DISTRO and ROS_PYTHON_VERSION.

Finally, if none of --rosdistro, ROS_DISTRO or ROS_PYTHON_VERSION are set then for the purpose of evaluating conditional dependencies ROS_PYTHON_VERSION will be set to sys.version[0]. This makes sure ROS_PYTHON_VERSION dependencies are not silently ignored, though there is a risk of installing the wrong packages.

rosdep and pip packages

Unlike Debian packages, keys that resolve to packages on PyPI may refer to either Python 2 or Python 3 dependencies. The version that gets downloaded depends on the version of Python used to invoke pip. Rosdep currently calls the command pip. It might use Python 2 when using the Debian package python-pip, but it could use Python 3 inside a venv. rosdep should pick a version of pip that matches ROS_PYTHON_VERSION regardless of whether a Python virtual env is being used.

First rosdep will try the commands pip2 or pip3 since those are similar to the previous usage of pip. If those don't exist and sys.version[0] is the same value as ROS_PYTHON_VERSION then it will try sys.executable -m pip. If that didn't work, then as a last resort it will try the commands python2 -m pip or python3 -m pip.

Shebangs and reliance on the Python command

Python scripts on UNIX systems typically have shebang lines written as:

#!/usr/bin/env python

PEP 394 recommends distributed Python scripts to use either python2 or python3 [7]. The python command cannot be trusted to a specific Python version. On older ROS distros, scripts can continue to use python since they're known to work on those platforms. These shebangs must be rewritten to the specific version of Python supported. Packages can use the CMake macro catkin_install_python() to install Python scripts with rewritten shebangs. It will create a relay script in the devel space with a rewritten shebang.

The same issue appears in scripts that call the python command directly. If they are Python scripts, they should invoke sys.executable. Otherwise, they should invoke the specific version of Python they require, which means templating the script to invoke the Python interpreter found when the package was built.

Dependencies and package.xml

On platforms where the target version of Python is 2, the package.xml of a ROS package must refer to Python 2 dependencies, and when the target Python version is 3 it must refer to Python 3 dependencies. Packages which release from different branches for each ROS distro can replace rosdep keys that resolve to Python 2 dependencies with ones that resolve to Python 3 equivalents. Packages using the same code base for multiple ROS distros should instead use conditional dependencies as described in REP 149 [5].

<depend condition="$ROS_PYTHON_VERSION == '2'">python-numpy</depend>
<depend condition="$ROS_PYTHON_VERSION == '3'">python3-numpy</depend>

If ROS_PYTHON_VERSION is relied upon at build time, such as when using catkin_install_python() to rewrite shebangs, then the package must declare a <buildtool_depend> on ros_environment. Any ROS package which uses ROS_PYTHON_VERSION in a script intended to be run at runtime should add an <exec_depend> tag for ros_environment.

Making Python fixes available to downstream packages

Transitioning to Python 3 is expected to be a significant effort. Maintainers should release packages with Python 3 fixes to Noetic as soon as possible, even if they intend to make breaking changes later. Maintainers should also add source entries for their Noetic branches to this file to enable downstream users to use rosinstall_generator with the --upstream-development flag to get Python 3 fixes. Instructions to build from source using Python 3 have been made available to the ROS community [9].