Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention

Authors: Khanh Nguyen, Debadeepta Dey, Chris Brockett, Bill Dolan.

This repo contains code and data-downloading scripts for the paper Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention (CVPR 2019). We present Vision-based Navigation with Language-based Assistance (VNLA, pronounced as "Vanilla"), a grounded vision-language task where an agent with visual perception is guided via language to find objects in photorealistic indoor environments.

Development system

Our instructions assume the followings are installed:

• Ubuntu 18.04 LTS
• Anaconda Python 3.6
• PyTorch 1.0.1 with CUDA 10.0

See setup simulator for packages required to install the Matterport3D simulator.

The Ubuntu requirement is not mandatory. As long as you can sucessfully Anaconda, PyTorch and other required packages, you are good!

Let's play with the code!

0. Clone this repo git clone --recursive https://github.com/debadeepta/vnla.git (don't forget the recursive flag!)
1. Download data.
2. Setup simulator.
3. Run experiments.
4. Extend this project.

Please create a Github issue or email kxnguyen@cs.umd.edu, dedey@microsoft.com for any question or feedback.

FAQ

Q: What's the difference between this task and the Room-to-Room task?

A: In R2R, the agent's task is given by a detailed language instruction (e.g., "Go the table, turn left, walk to the stairs, wait there"). The agent has to execute the instruction without additional assistance.

In VNLA (our task), the task is described as a high-level end-goal (the steps for accomplishing the task are not described) (e.g., "Find a cup in the kitchen"). The agent is capable of actively requesting additional assistance (in the form of language subgoals) while trying to fulfill the task.

Citation

If you want to cite this work, please use the following bibtex code

@InProceedings{nguyen2019vnla,
author = {Nguyen, Khanh and Dey, Debadeepta and Brockett, Chris and Dolan, Bill},
title = {Vision-Based Navigation With Language-Based Assistance via Imitation Learning With Indirect Intervention},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}