Package Summary

Assistive_teleop provides an accessible interface for control of a PR2 by persons with severe motor disabilities. It is designed for use with an assistive HCI device providing control of mouse movement and a single (left) click. The interface is presented in a web browser, requiring no installation by users, and allows low-level control of all degrees of freedom in the PR2, provides visual feedback, and provides semi-autonomous functions for general manipulation and performance of Activities of Daily Living (ADLs), with the intention of allowing users greater control over their environment as well as their own bodies, and providing increased independence. This package is a direct product of the Robots for Humanity effort, a collaboration between Willow Garage and the Healthcare Robotics Lab at Georgia Tech.

• Author: Phillip M. Grice, Advisor: Prof. Charlie Kemp, Lab: The Healthcare Robotoics Lab at Georgia Tech.
• License: BSD
• Repository: gt-ros-pkg
• Source: svn http://gt-ros-pkg.googlecode.com/svn/trunk/hrl/assistive_teleop

Usage

Server (PR2)

roslaunch assistive_teleop assistive_teleop.launch

This will likely throw errors regarding being unable to access the grasp warehouse database. This is ok, as this is a server hosted internally to Willow Garage, and not needed for operation.

Once this has finished launching, launch the Georgia Tech kinect launch file:

roslaunch hrl_pr2_lib openni_kinect_polled.launch

Client (Desktop)

Copy assistive_teleop/website/ directory to local machine. Entire folder (not just assistive_teleop.html) is required, as it provides supporting javascript and image files.

In assistive_teleop/website/assistive_teleop.html at l.16 set robot to your robot's network address. (i.e. pr2c1.gatech.edu, prz.willowgarage.com, 10.68.0.1, etc.)

Using Google Chrome (currently the only fully supported browser), browse to:

file:///.../website/assistive_teleop.html

or drag-and-drop the file into the browser window.

Client (Website)

The assistive_teleop/website/ directory may be served through a standard web server. Once served, browse to http://www.YOUR_WEBSITE.com/.../website/assistive_teleop.html.

This should bring up the interface and establish a connection to the robot, (if the server is launched on the robot.)

Interface

Feedback

Video

The interface provides visual feedback using the mjeg_server from bosch-ros-pkg to stream images from the PR2's right wide_stereo camera, either of the forearm cameras, and an added kinect sensor on the head of the robot. The forearm camera image streams are rotated using the image_rotate package so that the image is always 'right-side-up'. Buttons beneath the image stream allow for user camera selection. Also, when using Normal Approach, the camera stream will be modified to include an arrow indicating the point in the image and normal direction which has been selected, in order to provide confirmation that the robot is attempting the desired behavior.

Text

The head pan and tilt are both reported, in degrees, directly beneath the camera view, in order to make it easier for remote users, who can only see the camera view, to remain conscious of the head position. With the head turned, remote users often have the perception of 'forward' being directly in front of the camera, and making this assumption unconsciously can lead to unexpected behavior when driving. However, to avoid potential confusion, driving is entirely carried out with respect to the /base_frame of the robot, though this may change with further evaluation.

Also, a general log of feedback is provided at the bottom of the interface, which provides an indication of the action currently being taken by the robot. While somewhat trivial for low-level, immediately responsive control, this becomes extremely valuable for longer-term, semi-autonomous behaviors, so that the user is constantly aware of what the robot is doing.

Control

In order to manage the PR2's many degrees of freedom, control of the robot has been subdivided into multiple sub-divisions, in addition to some persistently available control inputs. The selection of a component to control is placed in the lower left of the interface, and alters the actions performed by the control inputs (bottom center). All motions (except base movement) are performed in discrete movements. The relative step size for a given input can be adjusted (+/-100% of the default) with the slider to the left of the command buttons. The scaling value will remain set for each portion separately, and will reset when the page is refreshed.

Control Head

When Control Head is selected, the control inputs provide arrows for looking up(^), down(v), left(<), and right(>). In addition, the center button (o) will bring the head pan/tilt to zero degrees, looking directly ahead, which can be used for centering the view with respect to the robot's body. Lastly, this also exposes controls to direct the head to persistently follow either the left or right gripper.

Control Right/Left Arm

Control Right/Left Arm provides access to low-level control of the arm position of the PR2. The arrow buttons (^,v,<,>) send commands in the xy-plane of /torso_lift_link, with separate buttons for UP and DOWN vertical movements. These commands attempt to move the arm a small distance in the given direction while maintaining the gripper pose. If the pose cannot be maintained, the hand will be moved as far along the direction as possible, and indicative feedback will be provided.

Relatively small motions of the end effector (<20cm) will be performed without motion planning or collision, allowing the gripper to come into contact with the environment, as required for grasping and performing contact-dependent activities of daily living. However, larger movement requests will use the move_arm motion planning package to avoid contact with the environment. Currently, this method will attempt to reach a goal with the same pose as the starting position, but will not necessarily maintain that pose throughout the movement path planned.

Additionally, this sub-division also exposes controls for performing a Normal Approach, which will attempt to place the gripper perpendicular to any arbitrarily indicated point visible in the kinect image. This is performed by selecting Normal Approach for the desired arm, and clicking a desired point on the image. Using pixel_2_3d to determine the associated 3D point in space and a vector normal to the surface at that point, the interface attempts to use move_arm motion planning to place the gripper 20cm away from the point, with the tips of the gripper pointing perpendicular to the surface.

In combination with this function are controls to Advance and Retreat the gripper. These commands will move the gripper directly forward or backward along the current orientation, allowing for easy grasping and manipulation at arbitrary gripper orientations. While designed for use with Normal Approach, these functions are completely general, and will work at any point in time when using the interface to control the robot.

Control Right/Left Hand

Selecting Control Right/Left Hand exposes controls for directly commanding the joints of the PR2's 'wrist': Forearm Roll, Wrist Flex, and Wrist Roll. The position of the arm is determined by the origin of the /wrist_roll_link, and so these commands can extend the reach of the PR2 by extending the wrist flex beyond the current position of the arm.

Control Base

When Control Base is selected, control inputs are exposed for driving the robot forward/backward, strafing left/right, and turning in place. Driving is the only motion which receives velocity inputs, and consistent motion can be achieved with a click-and-hold.

Sliders (Persistent)

The torso height of the robot is controlled using a slider, which both receives commands and reports the current position. Adjust the height of the robot by moving the slider bar to the desired position. Once the goal is sent, the slider will display the motion of the torso, as the current position continues to be updated.

Image (Persistent)

Currently, clicking anywhere on the image defaults to pointing the head to look at the 3D point associated with the image point where the click occurs, as returned by pixel_2_3d.

Similarly, the open/close state of the grippers is controlled by sliders, with the grippers being open at the top of the bar, and closed at the bottom. These sliders also report the current position of the gripper, and will display changes over time if the grippers are sent large commands.

Run-Stop (Persistent)

At the bottom of the interface, a large button labeled STOP ROBOT acts as a user-controlled run-stop, and immediately halts the PR2's motors. This replaces the interface with a warning message, and a Reactivate button to restart the motors. Be careful of the movement of the robot when run-stopped (gravity acting on head and arms, spring tension in arms, etc. ) as well as the possibility of the robot dropping any held objects.

Acknowledgements

First and foremost, the work depends upon the insight, drive, and feedback of Henry Evans, as well as the support of his family.

This package depends heavily upon the rosbridge and mjpeg_server packages from brown-ros-pkg and bosch-ros-pkg respectively. Their work has made this possible.

In addition, from the Healthcare Robotics Lab:

Tiffany Chen has provided background work on many aspects of ADL's, performed extensive needs evaluation with Henry Evans, and coordinated IRB approvals for all related research.

Kelsey Hawkins produced pixel_2_3d, which is used extensively in the interface, and has contributed specialized behaviors for performing ADL's to be used through this interface. Please contact for further details.

Significant support for the project has been received from Willow Garage, and significant collaboration has occurred in the development of the tools in the interface.

All related activities are performed under the guidance of Prof. Charlie Kemp.