Add ONNX and ORT support for Falcon #1391
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Fixes #1172 |
fxmarty
requested review from
michaelbenayoun,
JingyaHuang,
echarlaix and
mht-sharma
Comment on lines
+322
to
+323
| # we need to set output_attentions=True in the model input to avoid calling | ||
| # torch.nn.functional.scaled_dot_product_attention that is not supported by the ONNX export |
There was a problem hiding this comment.
nit: I would move this comment inside the method.
| generation_config=generation_config, | ||
| **kwargs, | ||
| ) | ||
| # self.num_kv_heads = config.num_kv_heads if (config.new_decoder_architecture or not config.multi_query) else 1 |
There was a problem hiding this comment.
Let's keep it for now
| @@ -211,7 +211,7 @@ class NormalizedConfigManager: | |||
| "blenderbot": BartLikeNormalizedTextConfig, | |||
| "blenderbot_small": BartLikeNormalizedTextConfig, | |||
| "bloom": NormalizedTextConfig.with_args(num_layers="n_layer"), | |||
| "falcon": NormalizedTextConfig.with_args(num_layers="num_hidden_layers", num_attention_heads="num_kv_heads"), | |||
| "falcon": NormalizedTextConfig, | |||
There was a problem hiding this comment.
Question: does NormalizedConfig have a NUM_KV_HEADS attribute to normalize it or not?
This was referenced Feb 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This one was more painful than it should have been because:
position_idsinput and then usinggenerateis bugged because the generated position_ids https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/models/falcon/modeling_falcon.py#L932 have a different shape than theposition_idsgenerated ingenerate. I believe this is a bug in many Transformers models.normalized_config- this should be refactored with inheritance to avoid any controlflow at all.Triluop onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for Trilu(14) node with name '/decoder/Trilu' microsoft/onnxruntime#16189Remaining issue: I think
repeat_interleaveONNX export insertsLoopin the ONNX, which we may want to avoid. EDIT: fixed in pytorch 2.1