Check bf16 model in torch engine #1270
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lvhan028
reviewed
Mar 11, 2024
| model_config = ModelConfig.from_hf_config(config, | ||
| model_path=model_path) | ||
| if model_config.dtype == torch.bfloat16: | ||
| assert not torch.cuda.is_bf16_supported(), ( |
There was a problem hiding this comment.
这种情况下，告警，并 fallback到 fp16，如何？
There was a problem hiding this comment.
warning 用户不一定会注意，这样更醒目，也能强调 fallback 会造成精度问题。
There was a problem hiding this comment.
有没有可以在 engine 中 fallback 到 fp16的办法呢？
There was a problem hiding this comment.
lmdeploy/lmdeploy/pytorch/engine/model_agent.py
Lines 493 to 495 in 9c3069f
lmdeploy/lmdeploy/pytorch/engine/model_agent.py
Lines 710 to 712 in 9c3069f
可以，但是没有办法保证这样的计算结果的正确性。用户也很容易无视 warning。
lvhan028
approved these changes
Mar 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
check bf16 support on v100.