Binning

Binning is the process of summing small neighborhoods of pixels on chip into larger "bins." For example, 2x2 binning reduces the resolution of the camera by half both horizontally and vertically. The chief advantage of binning is to increase the signal to noise ratio (SNR). This is especially useful in low-light environments. A second benefit is that decimating the resolution may allow for increased frame rate.

Giving up resolution for increased SNR and/or frame rate is a useful trade-off in many vision applications. The utility of binning, combined with its wide availability in camera hardware, makes this a particularly valuable feature to support in CameraInfo.

The correct geometric interpretation of a binned image requires scaling the focal lengths and principal point calculated during calibration at full resolution. The current CameraInfo message unfortunately has no way to encode the binning parameters.

Alternate Distortion Models

Currently CameraInfo assumes the "Plumb Bob" model of distortion [2], a 5-parameter polynomial approximation of radial and tangential distortion. Recently OpenCV has added support for an 8-parameter rational polynomial distortion model. This new model offers improved stability and accuracy for wide angle and fisheye lenses, which can severely distort the image. Unfortunately the size of the distortion parameter array D in CameraInfo is fixed at 5, so there is currently no way to support more complex models.

This REP makes D variable-length and introduces a string distortion_model to distinguish which distortion model was used in the calibration. These changes also accommodate adding other distortion models in the future, should we find it necessary.

Region of Interest

Region of interest is another common camera feature, allowing the user to instruct the imager to only capture a desired sub-rectangle of its full resolution. This may be used to reduce bandwidth, particularly with high-definition cameras that could overwhelm a network connection at full resolution. Like binning, it may increase the frame rate for some cameras. Finally, it may improve the imagery of the object we are interested in by applying smart camera features such as auto-exposure only to the relevant image patch.

Applications of ROI to vision include performing high-speed tracking of an object, or acquiring a high-resolution close-up of an object with known location.

CameraInfo already supports ROI in that it contains fields for the sub-rectangle captured by the imager. This ROI is given in raw image coordinates, as supplied to the imager, as that is all the camera driver is expected to know and care about. Given the ROI and the camera calibration, we (or rather image_geometry) can rectify the image patch. But what is the ROI of the rectified patch in rectified coordinates; that is, given the full rectified image, what sub-rectangle inside of it corresponds to the rectified patch?

Currently image_geometry (by extension, image_proc [15]) assumes that the ROI in rectified coordinates is the same as the ROI in raw (unrectified) coordinates. This behavior is broken. For small ROIs and/or large amounts of distortion, the raw patch (once rectified) may not even coincide with the same ROI in rectified coordinates. In that case the rectified image patch is set to all black, as the data required to fill it was not even captured [3]. Therefore we must introduce a mapping between the ROI in raw coordinates and the corresponding ROI in rectified coordinates.

CameraInfo Message

The proposed CameraInfo message is listed (stripped of comments) below. Additions and changes are noted on the right.

Header header
  uint32 seq
  time stamp
  string frame_id

uint32 height
uint32 width

# roi used to be located here.
# I've moved it to after the calibration parameters.

string distortion_model            # New field

float64[] D                        # Made variable-length
float64[9] K
float64[9] R
float64[12] P

uint32 binning_x                   # New field
uint32 binning_y                   # New field

sensor_msgs/RegionOfInterest roi   # Moved field
  uint32 x_offset
  uint32 y_offset
  uint32 height
  uint32 width
  bool do_rectify                  # New field

Interpretation

There are three parts to CameraInfo.

• ROS header. The time stamp is the time at which the image was captured. frame_id is the name of the optical coordinate frame with origin at the optical center of the camera.
• Calibration parameters. These are fixed during camera calibration. Their values will be the same in all messages until the camera is recalibrated. Note that self-calibrating systems may "recalibrate" frequently.
• Operational parameters. These define the image region actually captured by the camera driver. Although they affect the geometry of the output image, they may be changed freely without recalibrating the camera.

Raw and Rectified ROI

When working with distorted images, a desired raw ROI can be given directly to the camera driver. More commonly, however, the consumer wants the camera to provide an ROI in the rectified image, and does not particularly care how the rectified image patch is acquired. To bridge the gap between user (who works in rectified coordinates) and camera driver (which understands only raw coordinates), we define a mapping between raw and rectified ROI.

Given a rectified ROI, the corresponding raw ROI is the smallest sub-rectangle such that every pixel in the rectified ROI maps to a point inside the raw ROI. In other words, the raw ROI must contain all the information needed to fill the rectified ROI. Geometrically, if we distort the outline of the rectified ROI into raw coordinates, the raw ROI circumscribes the resulting curve.

Likewise, given a raw ROI, the corresponding rectified ROI is the largest sub-rectangle such that every pixel in the rectified ROI maps to a point inside the raw ROI. If we rectify the outline of the raw ROI, the rectified ROI inscribes the resulting curve.

When a full resolution image is captured, the behavior is different. During the camera calibration process, the user chooses a scaling which trades off between having all valid pixels in the rectified image (but discarding some pixels from the raw image) versus discarding no raw image pixels (but permitting invalid pixels in the rectified image). The assumption that all rectified pixels should be valid does not necessarily hold; that is up to the user to decide during calibration. The raw and rectified images have the same resolution, and hence the same "ROI" (the full image). roi.do_rectify is set to False to indicate that no ROI mapping should be done.

When the raw ROI is only part of the full resolution, roi.do_rectify is set to True. The raw ROI is mapped to the rectified ROI as described above.

It is permitted to set roi.do_rectify = False when the ROI is not actually the same as the full image resolution to suppress the ROI mapping. This feature can be useful in special situations, for example if the camera supports particular video modes with the resolution cropped to a smaller field of view. See use case #3 Cropped Video Mode below.

To complete the set of possibilities, the user could request an ROI of the full image resolution but with roi.do_rectify = True. This ensures that the rectified image contains no invalid pixels, but it will also discard pixels from the raw image.

Polled Camera Interface

The polled_camera/GetPolledImage service is updated as below:

string response_namespace

uint32 binning_x                   # New field
uint32 binning_y                   # New field

sensor_msgs/RegionOfInterest roi
  uint32 x_offset
  uint32 y_offset
  uint32 height
  uint32 width
  bool do_rectify                  # New field
---
bool success                       # New field
string status_message              # New field
time stamp

This revision allows users to specify the binning as well as the ROI. It also allows the camera driver to return failure (and an explanatory message) if the request could not be met; see Guidelines for Camera Drivers below.

Guidelines for Camera Drivers

The main purpose of a camera driver is to expose useful functions of the camera hardware. If a camera does not support binning or ROI in hardware, the driver has no obligation to implement these features in software; indeed, that might mislead users as to the camera's actual capabilities. Such post-processing can always be performed by a separate node.

Raw and Rectified ROI described a range of ways in which the ROI fields may be set for different purposes. In practice most camera drivers only need one or two behaviors. When capturing at full resolution, CameraInfo/roi can be left at the default of all zeros to signify full resolution. Drivers that do not support ROI need not touch CameraInfo/roi at all. When the user has requested some ROI (perhaps through dynamic_reconfigure), the driver should set CameraInfo/roi with the ROI offsets and size used and do_rectify = True.

Drivers are not obligated to implement the polled camera interface [6]. This feature is most useful for cameras that support software triggering or one-shot capture, and especially high-resolution cameras that may be expensive to stream at full resolution.

If a GetPolledImage request asks for binning settings that the camera does not support, the driver shall return failure (success = False). If the request asks for a ROI that is not full resolution and the camera does not support ROI, or if the requested ROI does not match any available camera resolution, the driver shall return failure. There may be cases when the camera supports ROI but cannot achieve precisely the ROI requested, e.g. if the camera requires the ROI offsets to be multiples of the binning sizes. In that situation the driver should expand the requested ROI to one attainable by the camera, return success, and set CameraInfo/roi to the expanded ROI. roi.do_rectify should always be copied as-is from the request to the output CameraInfo.

image_geometry API

Several new methods will be added to image_geometry's PinholeCameraModel class [7]. The signatures below are for C++; the Python API will be equivalent.

class PinholeCameraModel
{
  // Resolution of the camera when it was calibrated
  cv::Size fullResolution() const;

  // Current resolution. May be reduced by binning, or by ROI
  // when roi.do_rectify == False
  cv::Size currentResolution() const;

  // The current binning settings
  int binningX() const;
  int binningY() const;

  // The current raw and rectified ROIs
  cv::Rect rawRoi() const;
  cv::Rect rectRoi() const;

  // Compute the rectified ROI corresponding to a given raw ROI
  cv::Rect rectifyRoi(cv::Rect roi_raw) const;

  // Compute the raw ROI corresponding to a given rectified ROI.
  // The second overload is more convenient for use with the
  // polled_camera interface.
  cv::Rect unrectifyRoi(cv::Rect roi_rect) const;
  void unrectifyRoi(cv::Rect roi_rect,
                    sensor_msgs::RegionOfInterest& roi_raw) const;
};

Internally, PinholeCameraModel will be updated to compute the rectified ROI from CameraInfo/roi, and will use it when (un)rectifying points and images.

#1 Full Resolution

The most basic case - we capture full 752x480 images. The relevant CameraInfo settings are:

height: 480
width:  752
binning_x: 0
binning_y: 0
roi
    offset_x:   0
    offset_y:   0
    height:     0
    width:      0
    do_rectify: False

For this example we have left the binning and ROI fields at their defaults of 0. Drivers that do not support binning and/or ROI do not need to touch these fields at all. The above message has the same meaning as:

height: 480
width:  752
binning_x: 1
binning_y: 1
roi
    offset_x:   0
    offset_y:   0
    height:     480
    width:      752
    do_rectify: False

#2 Region of Interest

Let's capture a 200x300 ROI with top left corner (50, 70):

height: 480
width:  752
binning_x: 1
binning_y: 1
roi
    offset_x:   50
    offset_y:   70
    height:     300
    width:      200
    do_rectify: True

The driver simply needs to fill in the roi field. roi.do_rectify is now set to True, as the best-fitting rectified ROI may overlap poorly with the original raw ROI.

#3 Cropped Video Mode

Now we change the camera to 640x480 mode, cropping 56 columns on each side:

height: 480
width:  752
binning_x: 1
binning_y: 1
roi
    offset_x:   56
    offset_y:   0
    height:     480
    width:      640
    do_rectify: False

The cropping effect of the lower-resolution mode is encoded in CameraInfo as a ROI. roi.do_rectify = False because we wish to pretend that we are in fact running a 640x480 camera at full resolution. The rectified image will also be 640x480.

#4 Cropped Video Mode with ROI

Again capturing a 200x300 ROI with top left corner (50, 70), this time with respect to the 640x480 image:

height: 480
width:  752
binning_x: 1
binning_y: 1
roi
    offset_x:   106
    offset_y:   70
    height:     300
    width:      200
    do_rectify: True

Notice that the ROI in CameraInfo is actually specified in the full 752x480 resolution we calibrated with. roi.offset_x = 106 = 56 + 50.

#5 Binned, Cropped Video Mode

Staying in 640x480 cropped video mode, we enable 2x2 binning:

height: 480
width:  752
binning_x: 2
binning_y: 2
roi
    offset_x:   56
    offset_y:   0
    height:     480
    width:      640
    do_rectify: False

The output resolution is reduced to 320x240. Since ROI is specified in unbinned coordinates, the roi fields are unchanged from #3.

#6 Binned, Cropped Video Mode with ROI

Finally, we capture the same ROI as in #4:

height: 480
width:  752
binning_x: 2
binning_y: 2
roi
    offset_x:   106
    offset_y:   70
    height:     300
    width:      200
    do_rectify: True

Again, the roi fields are unchanged from #4.

Why Include Operational Parameters

In defining CameraInfo, we have included both calibration parameters (fixed until the next calibration) and operational parameters (set by the camera driver, may change freely). The Use Cases section demonstrates how the operational parameters can be used to describe a variety of useful capture modes without recalibrating.

One could ask, however, why include the operational parameters in CameraInfo at all? Binning rescales the focal lengths and principal point. The ROI offset shifts the principal point. A major alternative approach is to modify the projection matrix in the camera driver to describe the geometry of the individual output image. We have three objections to this approach.

First is separation of concerns. In ROS we consider camera drivers to have a very specific function: read pixels off the imager, shove them directly into a sensor_msgs/Image message, and publish. They are not expected to understand the camera model at all, merely regurgitate the calibration parameters provided to them externally. This keeps drivers simple and focused, and avoids requiring external dependencies such as OpenCV.

Second, the operational parameters are useful information in certain vision tasks, particularly ones which actively interact with the camera. The initial motivation for including ROI in CameraInfo was an application which tracked a small object at high speed by updating the ROI of a high-definition camera after each detection [9]. In this case, we needed to know the ROI of the incoming image to calculate the updated ROI.

Third, CameraInfo as specified permits some convenient optimizations. Without getting deeply into implementation details, rectifying a full resolution image for the first time is a fairly heavyweight operation. It involves generating matrices mapping each rectified pixel to a location in the raw image. These maps can be reused, however, on subsequent images, greatly reducing the expense of rectification. Furthermore, when ROI is used you can cheaply extract and use the same ROI from the full resolution map; this is a nice win when the ROI is frequently changing, as in the tracking application mentioned above. With the alternate approach, changing the ROI likely entails generating a new map specific to that ROI.

Unbinned Coordinates for ROI

The original draft of this REP expressed ROI in binned coordinates with the theory that this is slightly more intuitive to the user. One problem is that binned coordinates are less expressive than the full resolution coordinates; they do not allow bins that start at coordinates that are not multiples of the binning factor. Another advantage of using unbinned coordinates is that the ROI represents the same field of view regardless of the binning settings.

The IIDC (DCAM) 1394 specification [10] uses unbinned coordinates and explicitly does allow "off-grid" ROIs in Format_7 (Partial Image Size Format) modes. In contrast, the GenICam Standard Features Naming Convention [11] uses binned coordinates for ROI. Thus there is some disagreement among standards, but in the absence of any strong argument for using binned coordinates we opt for the technical advantages of unbinned coordinates.

Rejected Features

Some requested features are not included in this REP, generally because we do not see enough benefit to justify the added complexity.