Any explaination about the training speed changes?

[RyougiKukoc]

Basically I built a QAT ViT (with some input_quant=Int8ActPerTensorFloat and output_quant=Int8ActPerTensorFloat), but I found the training progress is kind of 8x slower at the first few epochs (around 5 or 6 epochs) and then speed up to normal (that is a little little slower than fp model). What is the reason behind it? Is it influenced by the statistics collecting scheme?

[Giuseppe5]

During QAT, for the first training steps, the activation scale factors are computed through statics and this causes the initial slowdown. After that the scales are converted to a parameters, so it is faster since it is not necessary anymore to compute statics.

[RyougiKukoc]

Got it, thank you!