You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the backward function of ring-attn, rng_state does not use the value from forward function, but directly passes in None.
Does this indicate that ring-attn does not support dropout?
The text was updated successfully, but these errors were encountered:
In the backward function of ring-attn, rng_state does not use the value from forward function, but directly passes in None.
Does this indicate that ring-attn does not support dropout?
The text was updated successfully, but these errors were encountered: