fp8 dtype support

[cyatarow]

In AUTOMATIC1111's WebUI repo, the FP8 dtype support is proposed.
It reduces VRAM usage by almost HALF compared to FP16 with a speed decrease of only 5% or less.
It needs PyTorch 2.1.0 or newer, and so far AUTOMATIC1111's WebUI doesn't have the 2.1.0 support, but SD.Next does.

What do you think of FP8? And can it be implemented in SD.Next?

[vladmandic]

i'm following the progress, but i don't see it as production ready just yet as torch/cuda does not have compute fp8 capabilities, so this is applies to storage only and relies on autocast to fp16 during runtime for processing - and exact gpus which are likely to be memory starved are same ones that are likely to have either autocast (e.g. directml) or fp16 precision issues (e.g. nvidia series 1xxx).

[drax-xard]

Honestly, for memory starved setups the best answer may come when stable-difussion.cpp matures to a usable point, since they use the GGML framework from llamacpp that allows really low-bit quantizations (among other cool things).
Even with fp16 they are reporting sub-2gb usage when used with flash attention.

[cyatarow]

[Edit] changed the title: FP8 Precision -> fp8 dtype support

The above proposal was merged into their dev branch.
Frankly, if VRAM usage is drastically reduced as their proposal says, I'm OK even if it is only applied to the storage side.
And if IPEX also can use it, that's even better.