Would it be faster if using RGBH ?

[hooke007]

edit: outdated, see #19

From this benchmark I saw it's faster. HolyWu/vs-rife#19 (comment)

[WolframRhodium]

Obviously it will, but maybe not significantly.

Let me explain the benchmark first. It's hard to benchmark different RIFE implementations in a fair way. RIFE uses an operation called GridSample whose performance depends on input video，so I use the following scripts to benchmark vs-rife v3.0.0 (pytorch 1.13.0+cuda117) and vsmlrt v12 on AWS's g5.2xlarge instance (NVIDIA A10G, driver 522.33, win server 2022 (20348.1249)) for reproducibility:

import vapoursynth as vs
core = vs.core

src = core.std.BlankClip(width=1920, height=1088, format=vs.RGBS, color=[0.5]*3, length=3000)

import vsmlrt
# without lexpr installation
flt = vsmlrt.RIFE(src, model=46, backend=vsmlrt.Backend.TRT(fp16=True, workspace=1024, num_streams=3))

flt.set_output()

205.90 fps

import vapoursynth as vs
core = vs.core

src = core.std.BlankClip(width=1920, height=1088, format=vs.RGBH, color=[0.5]*3, length=3000)

import vsrife
flt = vsrife.RIFE(src, num_streams=3, trt=True, trt_max_workspace_size=1<<30, model='4.6', sc=False)

flt.set_output()

133.54 fps

It has been shown that vsmlrt's trt uses less vram than its counterpart, and is not reported here.

What's more, vsrife's trt implementation produces different results than any other implementation available. I have no idea.

Back to the topic, using RGBH will certainly reduce the amount of data required to be transferred between memory and vram. This means vs-mlrt's performance could be further improved, but I dropped it here for simplicity.

[AkarinVS]

We did have discussed in length whether to support RGBH and whether using it to trigger the fp16=True, but ultimately decided against this.

The main reason is that, almost none of the common VS filters support 16-bit floating point format (iirc, only zimg, aka resize supports it), so even if we were to support it, the user will most likely have to convert to RGBH just for mlrt and then immediately convert the resulting RGBH clip back into another format, so even though using RGBH saves some PCIe bandwidth, it just pushes the conversion overhead elsewhere.

And note that unless your GPU bandwidth is severely limited, the computation and transfer are fully overlapped when num_streams>1, so in practice it also matters much less than you'd expect.

[WolframRhodium]

(these scripts are also unfair, to be honest)

In this example, vs-rife will pad the input image on gpu to 1920x1152, while vs-mlrt does not requires padding since it has been done. However, vs-mlrt is still faster on BlankClip(width=1920, height=1152).

In another interpretation, given a 1920x1080 input, vs-mlrt requires users to manually pad the input and crop the output. On machine with a poor cpu, these operations may reduce overall performance. This is not revealed in this benchmark.