Does ring-attn not support dropout?

[chinapanda]

In the backward function of ring-attn, rng_state does not use the value from forward function, but directly passes in None.
Does this indicate that ring-attn does not support dropout?

[zhuzilin]

Good catch! yeah, I think it's hard to support dropout with the current implementation...

[chinapanda]

咱们有没有研究过te中的cp并行呢？这块是支持dropout的。这块逻辑是否可借鉴呢？
https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py#L1018-L1025

[zhuzilin]

嗯... 可以确实是可以... 就是要单独存 ring_size 份 rng_state，我有点不太确定 rng_state 的显存占用多不多...