Unit 9 Exercise 1: Mixed-Precision Results

[rasbt]

This is a discussion for the upcoming Unit 9 exercise.

[spayot]

results on Apple M1 Pro are not super conclusive:

precision	its / second	test accuracy
32-true	1.07	0.9261
16-mixed	1.15	0.9194
16-true	NA	NA
64-true	Not Supported for MPS	Not Supported for MPS
bf16-mixed	Not Supported for MPS	Not Supported for MPS

• hardware: MPS (Apple M1 Pro)
• batch size: 12
• test accuracy measured after 1 epoch of finetuning
• all params trainable

Couple of questions:

• is "16-true" really a valid precision value? couldn't find any reference to it when looking at the source code for lightning.fabric.plugins.precision.precision.py (watermark says i'm on lightning v2.0.2), and the script return a ValueError when i try it with this value.
• switching to 16-mixed precision seems to yield much lower training speed gains than in the video when done on the M1 Pro. is that a documented limitation of M1 Pros? or did i do something wrong?

Thanks!

[spayot]

a quick googling surfaced this open Issue on the pytorch project. probably answers my second question.

[rasbt]

Thanks for sharing @spayot !

is "16-true" really a valid precision value?

Yes, it should be. I always mix up "16-true" and "16-full" because I think of it as "full" (as opposed to "mixed") 16-bit precision. But I just double-checked and it should be 16-true. Maybe it was just added a tad later after v2.0.2 (I don't recall off the top of my head).

switching to 16-mixed precision seems to yield much lower training speed gains than in the video

I think this is likely because the A100 GPUs I used have tensor cores that are better optimized for 16-bit

[mingfei-li]

Here are the results I got on Google Cloud Virtual Machine with the following setup: n1-standard-4 (4 vCPU, 15GB memory) + 1 Nvidia V100 GPU. It's interesting to see that 16-mixed achieved higher test accuracy than 32-true (probably just random noise).

Precision option	Elapsed time for 3 epochs	Test accuracy
64-true	54.24 min	0.9265000224113464
32-true	26.86 min	0.9225000143051147
16-true	not supported	not supported
16-mixed	11.29 min	0.9264000058174133
bf16-mixed	not supported	not supported

[rasbt]

Very interesting @mflipro . I occasionally also see mixed or lower precision resulting in slightly higher accuracy. Maybe the "imprecision" has some sort of regularizing effect that slightly reduces overfitting.

[pardeep-singh]

I am sharing the results I got on the P100 GPU on Kaggle. bf16-mixed precision was unavailable on this GPU.

Config

• Epochs: 3
• Batch Size: 16

Results

Precision option	Elapsed time	Test Accuracy
64-true	1 hr 44 min	0.9260
32-true	49 min	0.9241
16-true	45 min	0.5006
16-mixed	49 min	0.9279
bf16-mixed	not supported	not supported

Link to the notebook

@rasbt Is there any specific reason behind the low accuracy for 16-true?

[rasbt]

Thanks for sharing those! First, I must say that it's interesting that there are so little time savings with the P100. I think it doesn't have tensor cores yet, which are optimized for lower precision computations.

Regarding 16-true and the bad accuracy. That's likely because it's too much of a reduction in precision during training. In mixed-precision, you compute the gradients in 16 bit precision but convert them back to 32bit when updating the weights, so maybe the loss in precision, if everything is in 16 bit, is too harsh. Btw sometimes I get 16-true to work well, mostly when working with LLMs or vision transformers.

[pardeep-singh]

@rasbt Thanks for your response and clarification on 16-true