To Örebro University

In the domain of cognitive robotics, many tasks that involve the interaction of robots and humans rely on the successful grounding of linguistic expressions in the physical environment, referred to as language grounding. The important factors that affect language grounding are the physical environment and the user, and a robot stores the perceived information about those in its context model. How a language-grounding robot creates and uses its model of context can be divided into two categories: symbolic and subsymbolic. In either case, anguage-grounding robots have to make assumptions about the environment or users that in real-world scenarios are not always guaranteed. The main question left to be answered is how to develop a language grounding method enabling robots to perform language grounding in real-world scenarios, where such assumptions are not guaranteed. We tackle this challenge by revisiting how a robot’s model of context should be defined and used in order to take advantage of both symbolic and subsymbolic systems together.

The context model contains all necessary information that is acquired at runtime for performing language grounding. A robot can store different features of the environment and user in its context model. In this thesis, we focus on information related to objects in the robot’s surroundings and a user who may verbally communicate with the robot about the objects, to give information about them, or to refer to them. Symbolic and subsymbolic techniques, when used separately, inherently limit language-grounding robots in their ability to process and ground information. When symbolic techniques are deployed in language grounding, common assumptions have to be made regarding the user, such as the user can refer to a limited and fixed set of objects, using a fixed set of grammar and vocabulary. When subsymbolic techniques are used, common assumptions have to be made regarding the environment, like that the environment is fixed, or that nothing hinders the robot’s full observability over objects.

To ensure the robot’s language grounding performance in real-world scenarios where the user and environment have variabilities, we propose an approach that integrates symbolic and subsymbolic techniques by revisiting the theory and practice of context. The initial step in our approach is defining the context, which serves as the cornerstone for integrating the robot’s grounding, perceptual, and interactive skills, tailored for real-world scenarios. Based on the defined model of context, we develop methods that enable the robot to perceive and store information from the environment and user in the context model, alongside a grounding methodology within the acquired information. The robot’s visual and verbal perception modules utilize neural techniques that guarantee the robot’s robustness in responding to the user’s variabilities in real-world scenarios. To guarantee the robot’s language grounding performance amidst the challenges posed by real-world variabilities of the environments, we focus on the robot’s skills in handling the challenges. Specifically, when a robot’s perception is hindered or an object new to the robot is introduced, we develop techniques based on planning and reasoning to enable the robot to perceive its required information.

The outcome of this dissertation advances cognitive robots that have to execute real-world tasks using language grounding. Our proposed methods relax the assumptions inherent in the current systems, enabling robots to perform language grounding in challenging real-world scenarios. We demonstrate how symbolic and subsymbolic techniques can be integrated using a model of context, showcasing how this integration enables robots to perform language grounding in real-world scenarios, even in complex environments. We evaluate and validate our methods experimentally in real-world scenarios, underscoring the effectiveness and practical applicability of our approaches.