Are gradients of persistent_parameters synced in backward with reduce-scatter?

[zarzen]

Hi there,
I noticed there is a all_gather step in _post_step function of stage3. The all-gather is used, instead of all-reduce, is it because the gradients of the persistent parameter is synchronized via reduce-scatter in the backward pass?
https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/zero/stage3.py#L1707