Open-ended Grasping with QD-Grasp

A key challenge in contemporary robotics is the development of open-ended grasping abilities for robots. The very high diversity of manipulation tasks that can be encountered when deploying robots means robots need to be able to deploy diverse grasping strategies on arbitrary objects and traditional grasping approaches, which rely on strong priors, tend to fall short.

Within PILLAR, we developed a family of grasping learning algorithms called QD-grasp. These rely on evolutionary approaches designed to optimize diversity, improving exploration and enabling robots to broadly explore grasp options for objects.

Simulation, deployment and demonstration

To evaluate candidate grasps, the robot’s end-effector and a 3D model of the object are injected in a simulation, which is used to evaluate whether the end-effector stays in contact with the object (that is, successfully grasps it), even when subjected to shaking motions. To further enhance the robustness of the learning process, the physical parameters of the object (mass, surface friction coefficients) are randomly perturbed. This avoids learning grasps that rely on physically unrealistic behaviors.

When physically deploying the grasps, the robot must know the object’s position. We use off-the-shelf models that rely on the same object models that are used for the grasp learning process. This model allows us to transfer grasps to any object configuration.

In the work package 9 demonstration, we showcase the usefulness of learning diverse grasps: the objects are placed on a work desk, within a cluttered environment. Having learned diverse grasp archives, the TiAGo is able to select a grasp that is compatible with the environment and pick the object. The demonstration also showcases the utility of simulating the robot’s end-effector when learning the grasps, as this makes the process able to adapt to non-standard hardware, in this case a Robotiq gripper.

This learning technique presents exciting perspectives. For instance, we found that foundation models can be used to leverage grasp diversity by specifying which part of an object should be grasped for a given manipulation task, which is a prerequisite for higher skills such as tool use