In PILLAR-robots, the industrial use case is designed to demonstrate how purposeful, lifelong-learning robots can help workers in professional settings. Industrial environments present unique challenges, requiring accurate decision-making in open worlds, interaction with non-expert users and the ability to handle uncommon, challenging objects.

In the scenario we envision, the PILLAR robot helps workers in the PAL workshop. The robot is given tasks such as fetching objects for the factory workers and clearing their workspace of discarded tools, freeing up their time for more complex work. This seemingly simple mission presents several challenges.

First, the robot must be able to interact with humans. For rapid, open-ended communication with untrained users, we choose to use natural language. This poses a challenge, as language is less structured than other modes of communication such as machine code. To address this challenge, we use Large Language Models, able to understand user commands. These models are used to manage user requests, asking for clarifications if instructions are unclear, and to plan the robot’s actions to solve these instructions. The purposeful intrinsically-motivated cognitive architecture is used to memorize which motion primitive sequences were used to tackle each instruction and the associated environmental states, enabling grounded lifelong learning.

A second requirement for this use-case is the ability to perceive the environment and extract pertinent, task-relevant information. Semantic perception is required to recognize and classify objects, allowing the robot to understand how to interact with them and increasing its autonomy. On the other hand, grounded, low-level perceptual pipelines such as estimation of each object’s position are also critical for manipulation. Perception modules fulfilling both of these needs are developed by ARC.

Finally, manipulation skills allow the robot to interact with the objects in the environment. These skills, such as opening doors and drawers or grasping objects, are challenging to design robustly due to the complexity of the physical interactions involved. Inaccuracies in simulations can lead to difficulties in applying skills to the real world. Algorithms developed at ISIR, leveraging domain-randomized simulations and evolutionary methods, address the need for robust manipulation skills. 

While industrial environments are challenging for autonomous agents due to their complexity, open-ended learning can help robots acquire the perceptions, skills and knowledge required to safely, efficiently supplement and assist human workers.