Intention-Aware-Video-Prediction

The code for the paper: "Intention-Aware Frequency Domain Transformer Networks for Video Prediction"

If you use the code for your research paper, please cite the following paper:

Hafez Farazi and Sven Behnke:
Intention-Aware Frequency Domain Transformer Networks for Video Prediction [PDF]
Accepted for 31st International Conference on Artificial Neural Networks (ICANN), Bristol, UK, September 2022.

BIB:

@Conference{Farazi2022_ICANN,
  Title                    = {Intention-Aware Frequency Domain Transformer Networks for Video Prediction},
  Author                   = {Farazi, Hafez and Behnke, Sven},
  Booktitle                = {International Conference on Artificial Neural Networks (ICANN)},
  Year                     = {2022},
  Address                  = {Bristol, UK}
}

Dependencies

The code was tested with Ubuntu 20.04 and PyTorch 1.10

Sample Result

GT | z=0 | Diff | z=1 | Diff | z=2 | Diff | z=3

PropMNIST_4D:

GT | z=-1 | Diff | z=-0.8 | Diff | z=-0.6 | Diff | z=-0.4 | Diff | z=-0.2 | Diff | z=0 | Diff | z=0.2 | Diff | z=0.4 | Diff | z=0.6 | Diff | z=0.8 | Diff | z=1

PropMNIST_1C:

GT | z=-1 | Diff | z=-0.8 | Diff | z=-0.6 | Diff | z=-0.4 | Diff | z=-0.2 | Diff | z=0 | Diff | z=0.2 | Diff | z=0.4 | Diff | z=0.6 | Diff | z=0.8 | Diff | z=1

Skeleton_1C:

Run

python app.py --data_key=PropMMNIST --batch_size=500 --useGlobalLFT=True --res_x=64 --res_y=64 --inference=False --ArrowScale=2 --futureAwareMPFNetwrokTestTime=Same --futureAwareMPFDropout=0 --futureAwareMPFHistory_len=5 --futureAwareMPFChannelNumber=1 --futureAwareMPF=Network --sequence_length=6 --sequence_seed=3 --history_len=4 --digitCount=1 --futureAwareMPFContinuous=True --refine_output=True --M_transform_lr=0.007 --epochs=30 --refine_lr=0.0011