[ARC-WSL-Ubuntu24.04] 15 Timm models got fail_accuracy

[libohao1201]

🐛 Describe the bug

14 Timm models got error message fail_accuracy on ARC-WSL.

cp torch-xpu-ops\.github\scripts\inductor_xpu_test.sh pytorch/
cd pytorch
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 botnet26t_256 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 convmixer_768_32 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 cspdarknet53 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 eca_botnext26ts_256
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 eca_halonext26ts 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 fbnetv3_b 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 gluon_inception_v3 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 lcnet_050 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 levit_128
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 mixer_b16_224 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 mobilenetv2_100 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 resnest101e 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 rexnet_100 
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 sebotnet33ts_256
bash inductor_xpu_test.sh timm_models bfloat16 training accuracy xpu 0 static 1 0 tinynet_a

Versions

Machine	ARC - Win11 shwde-mlp-arc02.ccr.corp.intel.com
stock pytorch	pip install torch --index-url https://download.pytorch.org/whl/test/xpu git clone https://github.com/pytorch/pytorch.git git checkout b1940b5867e40e40ebdce4db76f76d3d0b71d3f4
torch-xpu-ops	Commit (pin): 026b2c8
Driver	32.0.101.6647 WSL: Driver UUID 32342e33-392e-3331-3239-340000000000 Driver Version 24.39.31294
Conda	python 3.10

[libohao1201]

3 HF models also got fail_accuracy.

• DebertaV2ForQuestionAnswering bf16/fp16/fp32 training
• AlbertForQuestionAnswering amp_bf16/amp_fp16/bf16/fp16 training
• AlbertForMaskedLM amp_fp16/fp16 training

[chuanqi129]

@libohao1201 please check the latest status based on 2.9 test results

[libohao1201]

Verified on ARC-WSL-Ubuntu24.04 with pt2.9 and windows driver 32.0.101.8136 (WSL driver: 24.52.32224), 10 cases still failed with fail_accuracy
TM:
bf16_accuracy_training: gluon_inception_v3 / mixer_b16_224 / mobilenetv2_100 sebotnet33ts_256

HF：
AlbertForQuestionAnswering amp_bf16/amp_fp16/bf16/fp16 training
AlbertForMaskedLM amp_fp16/fp16 training

========================timm==========================
Testing model gluon_inception_v3
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:09, ?it/s]
xpu  train gluon_inception_v3                 
WARNING:common:fp64 golden ref were not generated for gluon_inception_v3. Setting accuracy check to cosine
W1104 16:00:17.774000 19117 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.957487940788269
E1104 16:00:17.776000 19117 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name Conv2d_1a_3x3.conv.weight.grad
fail_accuracy


Testing model mixer_b16_224
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:04, ?it/s]
xpu  train mixer_b16_224                      
WARNING:common:fp64 golden ref were not generated for mixer_b16_224. Setting accuracy check to cosine
W1104 16:04:04.116000 26287 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.9897714257240295
E1104 16:04:04.119000 26287 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name stem.proj.weight.grad
fail_accuracy


Testing model mobilenetv2_100
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:05, ?it/s]
xpu  train mobilenetv2_100                    
WARNING:common:fp64 golden ref were not generated for mobilenetv2_100. Setting accuracy check to cosine
W1104 16:05:17.572000 27018 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.9745811223983765
E1104 16:05:17.576000 27018 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name blocks.0.0.conv_pw.weight.grad
fail_accuracy


Testing model sebotnet33ts_256
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:06, ?it/s]
xpu  train sebotnet33ts_256                   
WARNING:common:fp64 golden ref were not generated for sebotnet33ts_256. Setting accuracy check to cosine
W1104 16:11:45.013000 37759 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.2610626816749573
E1104 16:11:45.015000 37759 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name stages.0.0.conv1_1x1.conv.weight.grad
fail_accuracy

========================hf==========================
Testing model AlbertForQuestionAnswering
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:04, ?it/s]
xpu  train AlbertForQuestionAnswering         
WARNING:common:fp64 golden ref were not generated for AlbertForQuestionAnswering. Setting accuracy check to cosine

ShaderDumpEnable Warning! BufferVec[] has 2928 elements. Including first 1000 items in ShaderDumps. To print all elements set IGC_ShowFullVectorsInShaderDumps register flag to True. ShaderOverride flag may not work properly without IGC_ShowFullVectorsInShaderDumps enabled.

W1104 16:18:54.871000 45823 site-packages/torch/_dynamo/utils.py:3025] Similarity score=-0.13060349225997925
E1104 16:18:54.872000 45823 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name albert.embeddings.token_type_embeddings.weight.grad
fail_accuracy

Testing model AlbertForQuestionAnswering
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:03, ?it/s]
xpu  train AlbertForQuestionAnswering         
WARNING:common:fp64 golden ref were not generated for AlbertForQuestionAnswering. Setting accuracy check to cosine

ShaderDumpEnable Warning! BufferVec[] has 2928 elements. Including first 1000 items in ShaderDumps. To print all elements set IGC_ShowFullVectorsInShaderDumps register flag to True. ShaderOverride flag may not work properly without IGC_ShowFullVectorsInShaderDumps enabled.

W1104 16:20:03.866000 46520 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.25127941370010376
E1104 16:20:03.868000 46520 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name albert.embeddings.token_type_embeddings.weight.grad
fail_accuracy


Testing model AlbertForQuestionAnswering
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:03, ?it/s]
xpu  train AlbertForQuestionAnswering         
WARNING:common:fp64 golden ref were not generated for AlbertForQuestionAnswering. Setting accuracy check to cosine

ShaderDumpEnable Warning! BufferVec[] has 2928 elements. Including first 1000 items in ShaderDumps. To print all elements set IGC_ShowFullVectorsInShaderDumps register flag to True. ShaderOverride flag may not work properly without IGC_ShowFullVectorsInShaderDumps enabled.

W1104 16:21:17.052000 47250 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.219701886177063
E1104 16:21:17.058000 47250 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name albert.embeddings.token_type_embeddings.weight.grad
fail_accuracy


Testing model AlbertForQuestionAnswering
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:04, ?it/s]
xpu  train AlbertForQuestionAnswering         
WARNING:common:fp64 golden ref were not generated for AlbertForQuestionAnswering. Setting accuracy check to cosine

ShaderDumpEnable Warning! BufferVec[] has 2928 elements. Including first 1000 items in ShaderDumps. To print all elements set IGC_ShowFullVectorsInShaderDumps register flag to True. ShaderOverride flag may not work properly without IGC_ShowFullVectorsInShaderDumps enabled.

W1104 16:22:32.057000 48081 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.2998909056186676
E1104 16:22:32.061000 48081 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name albert.embeddings.token_type_embeddings.weight.grad
fail_accuracy


Testing model AlbertForMaskedLM
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:04, ?it/s]
xpu  train AlbertForMaskedLM                  
WARNING:common:fp64 golden ref were not generated for AlbertForMaskedLM. Setting accuracy check to cosine

ShaderDumpEnable Warning! BufferVec[] has 2928 elements. Including first 1000 items in ShaderDumps. To print all elements set IGC_ShowFullVectorsInShaderDumps register flag to True. ShaderOverride flag may not work properly without IGC_ShowFullVectorsInShaderDumps enabled.

W1104 16:23:38.228000 48937 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.9289129972457886
E1104 16:23:38.231000 48937 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name albert.embeddings.token_type_embeddings.weight.grad
fail_accuracy


Testing model AlbertForMaskedLM
Testing with training mode.

loading model: 0it [00:00, ?it/s]
loading model: 0it [00:04, ?it/s]
xpu  train AlbertForMaskedLM                  
WARNING:common:fp64 golden ref were not generated for AlbertForMaskedLM. Setting accuracy check to cosine

ShaderDumpEnable Warning! BufferVec[] has 2928 elements. Including first 1000 items in ShaderDumps. To print all elements set IGC_ShowFullVectorsInShaderDumps register flag to True. ShaderOverride flag may not work properly without IGC_ShowFullVectorsInShaderDumps enabled.

W1104 16:24:49.930000 49626 site-packages/torch/_dynamo/utils.py:3025] Similarity score=0.956402599811554
E1104 16:24:49.931000 49626 site-packages/torch/_dynamo/utils.py:2976] Accuracy failed for key name albert.embeddings.token_type_embeddings.weight.grad
fail_accuracy