[Feature] Is there a plan to support BF16 inference?

[qingchanghan]

Motivation

It appears that LMDeploy does not support native BF16 inference, but instead converts it to FP16.

But this may lead to precision overflow (#385), is there a plan to support BF16 inference?

Related resources

No response

Additional context

No response

[lzhangzz]

Yes, we will bring back BF16 support after a few major refactoring to lower the maintainance cost.

BTW, which model are you testing has overflow issue using f16 type?

[qingchanghan]

Thank you! Look forward to the follow-up progress.

And we encountered this problem with an internal inference framework and internal models. These models are trained with BF16 precision.

[lzhangzz]

I see.

Have you checked in which layer or operator the overflow arises?

[qingchanghan]

Sorry, I didn't check it carefully.

But when I changed the precision to BF16, I can get the correct result.

[lvhan028]

PR #803