This directory is supplementary material for our work presented at The 39th International Conference on Machine Learning (ICML 2022):
"Imitation Learning by Estimating Expertise of Demonstrators" Mark Beliaev*, Andy Shih*, Stefano Ermon, Dorsa Sadigh, Ramtin Pedarsani.
All relevant citations for methods used are found in the paper's list of references
We recommend using pacakge manager pip as well as cuda to install the relative packages:
NOTE: This repository contains an indepedent package:
-- ileed our implementation of ILEED and behavior cloning (BC)
We place the package in its own directory /ileed/. The corresponding data generated by this packages is stored in /data/ileed/.
First create a new environment that has pytorch. We are using the 1.10.0 pytorch build and python 3.6.13 Following this we install all required packages for ileed using pip and conda.
conda create -n ileed python==3.6.13
conda activate ileed
conda install pytorch==1.10 torchvision torchaudio cudatoolkit=11.3 jupyter tensorboard==2.5 numpy -c
pip install gym==0.18.0 stable-baselines3==1.2.0 gym_minigrid tqdm
cd .../ileed/ileed
pip install .
export PYTHONPATH=.../ileed/:.../ileed/ileed/Now we can test the installation.
python test/test_ileed.py This will run the full framework briefly, with irrelevant performance. The results will be saved in /test/data/.
To utilize the repository one must first pre-train policies to generate the noisy trajectories. We do this utilizing sb3's PPO implementation, with the code provided in /scripts/train.py
For example, to save policies for all 4 MiniGrid environments we utilized in our experiments run:
python scripts/train.py --seed 0 All of the main code is provided in /ileed/, but we have abstracted the package out by providing high level algorithm blocks in /functions/algorithms.py, which contain implementations for ILEED and BC.
To use this implementations, one can run the code provided in /scripts/learn.py to replicate our MiniGrid experiments:
python scripts/learn.py --seed 0