This is the code base for the paper "Verifiably Following Complex Robot Instructions with Foundation Models". We present a novel approach that leverages pre-trained foundation models and temporal logics to enable robots verifiably follow expressive and complex open ended instructions in real world environments without prebuilt semantic maps.
- Create conda environment and install relevant packages:
conda env create -f environment.yml - Activate your conda environment:
conda activate limp - Setup the Open Spatial Grounding Library (OSG) submodule
- Update submodule
git submodule update --init --recursivethengit submodule foreach git pull origin main - Run
pip install -e open-spatial-grounding - In the
limpconda environment install Mobile SAM or Segment Anything as stated in OSG's installation instructions. - Copy the osg folder from the Open Spatial Grounding library and place it in this root directory.
- Update submodule
- Obtain an Openai api key and add it to your system variables.
- This codebase supports directly using RGBD data from robot exploration however the easiest way to get started is using an Iphone Pro equiped with lidar sensors.
- Download the Record3D app, scan the scene of your choice and export the *.r3d file to your computer.
- Make sure to capture all test objects in your recording.
- You can download a sample r3d file from Ok-Robot a cool project on open-vocabulary navigation.
- Walkthrough the demo notebook
The methods implemented in this codebase were proposed in the paper "Verifiably Following Complex Robot Instructions with Foundation Models". If you find any part of this code useful, please consider citing:
@article{quartey2024verifiably,
title={Verifiably Following Complex Robot Instructions with Foundation Models},
author={Quartey, Benedict and Rosen, Eric and Tellex, Stefanie and Konidaris, George},
journal={arXiv preprint arXiv:2402.11498},
year={2024}
}