Benchmarking Cluttered Robot Pick-and-Place Manipulation with the Box and Blocks Test

Andrew S. Morgan, Kaiyu Hang, Walter G. Bircher, Fadi M. Alladkani, Abhinav Gandhi, Berk Calli, and Aaron M. Dollar
Robotics and Automation Letters, 2019.
Abstract:
In this work, we propose a pick-and-place benchmark to assess the manipulation capabilities of a robotic system. The benchmark is based on the Box and Blocks Test (BBT), a task utilized for decades by the rehabilitation community to assess unilateral gross manual dexterity in humans. We propose three robot benchmarking protocols in this work that hold true to the spirit of the original clinical tests-the Modified-BBT, the Targeted-BBT, and the Standard-BBT. These protocols can be implemented by the greater robotics research community, as the physical BBT setup has been widely distributed with the Yale-CMU-Berkeley (YCB) Object and Model Set. Difficulty of the three protocols increase sequentially, adding a new performance component at each level, and therefore aiming to assess various aspects of the system separately. Clinical task-time norms are summarized for able-bodied human participants. We provide baselines for all three protocols with off-the-shelf planning and perception algorithms on a Barrett WAM and a Franka Emika Panda manipulator, and compare results with human performance.