Is communication in DeepSpeed pipeline parallelism async?

[jiashu-z]

Hi DeepSpeed community,

DeepSpeed 0.12.2

I used DeepSpeed pipeline parallelism to train a model, one time with communication log, and another time without it. I plotted the pipeline steps of two training, whick looked very different.

When communication log is on, the figure is like this.

Forward and backward passes are synchronized.

However, when the communication log is off, the pipeline steps are strange.

I wonder if this is because DeepSpeed pipeline parallelism uses async communication. On the webpage of communication log, it says that

All logging communication calls are synchronized in order to provide accurate timing information. This may hamper performance if your model heavily uses asynchronous communication operations.

But I also notice that in DeepSpeed/deepspeed/runtime/pipe/p2p.py, async_op=False is always the case for send and recv, which confused me.

Thanks!