Authors
- Jesse Thomason*
- Aishwarya Padmakumar*
- Jivko Sinapov*
- Nick Walker*
- Yuqian Jiang*
- Harel Yedidsion*
- Justin Hart*
- Peter Stone
- Raymond J. Mooney*
* External authors
Venue
- IJCAI-2021
- The Journal of Artificial Intelligence Research
Date
- 2021
Share
Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog
Jesse Thomason*
Aishwarya Padmakumar*
Jivko Sinapov*
Nick Walker*
Yuqian Jiang*
Harel Yedidsion*
Justin Hart*
Raymond J. Mooney*
* External authors
IJCAI-2021
The Journal of Artificial Intelligence Research
2021
Abstract
In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clarification questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the fly while completing a real-world task.
Related Publications
Goal-conditioned reinforcement learning (GCRL) has a wide range of potential real-world applications, including manipulation and navigation problems in robotics. Especially in such robotics tasks, sample efficiency is of the utmost importance for GCRL since, by default, the …
In reinforcement learning (RL), a reward function that aligns exactly with a task's true performance metric is often sparse. For example, a true task metric might encode a reward of 1 upon success and 0 otherwise. These sparse task metrics can be hard to learn from, so in pr…
Current approaches to multi-agent cooperation rely heavily on centralized mechanisms or explicit communication protocols to ensure convergence. This paper studies the problem of distributed multi-agent learning without resorting to centralized components or explicit communic…
JOIN US
Shape the Future of AI with Sony AI
We want to hear from those of you who have a strong desire
to shape the future of AI.