Failing Torchbench Models: tracking issue

[ysiraichi]

Summary of Contributions (9th Feb)

1. Improve the number of models in TorchBench that work with Dynamo as a tracer: These passing rates are now comparable to those from torch.compile using Inductor. Some of the fixes also improved the previous tracer that PyTorch/XLA used to use.

   	Inference	Training
   Inductor	87	63
   Dynamo	60 to 82	41 to 53
   Non-Dynamo	79 to 82	54 to 56

2. Improve the benchmarking tools used by Google: The initial Google runs benchmarking these models showed a discrepancy of about 15 models with the results reported. We identified and fixed 10+ issues that helped reconcile Google's benchmarks with those reported and, in turn, with the PyTorch HUD.

Current State

This post has two lists:

• Failing inference models
• Failing training models

Each of them shows the failing models:

• Tracing without Dynamo (Eager-mode)
• Tracing with Dynamo into openxla (Dynamo+openxla)

These lists were created using the benchmarking scripts that currently live in the upstream. The following command was executed:

python xla/benchmarks/experiment_runner.py \
       --suite-name torchbench \
       --accelerator cuda \
       --xla PJRT --xla None \
       --dynamo openxla --dynamo inductor --dynamo None \
       --test eval --test train \
       --repeat 30 --iterations-per-run 5 \
       --print-subprocess \
       --no-resume

Environment

• GPU: A100 40GB

Inference

Non-Dynamo. Pass rate: 78/81 - 96% (against inductor)

• [x] DALLE2_pytorch
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Moved to canary models: Upgrade numpy to 2.0.0rc benchmark#2311
• cm3leon_generate
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• hf_Longformer
  • Issue: No support for overlapped tensors. #5835
    • PyTorch/XLA PR: Make as_strided_copy materialize a new tensor with index. #6624
• hf_T5_generate
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• moco
  • Issue: [torchbench] moco fails to run. #6083
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Issue: [torchbench] moco inference fails to run on dynamo. #7636
  • Issue: [torchbench] moco fails to run with CUDA OpenXLA fallback. #7647
• nvidia_deeprecommender
  • Issue: [torchbench] nvidia_deeprecommender fails to run. #6006
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• pytorch_CycleGAN_and_pix2pix
  • Issue: [torchbench] pytorch_CycleGAN_and_pix2pix fails to run. #6007
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• [ ] simple_gpt
  • RTX 2060 doesn't support BF16
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (only work with multiprocess enabled -- torchbench.yaml)
• [ ] simple_gpt_tp_manual
  • RTX 2060 doesn't support BF16
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (inside skip list -- torchbench.yaml)
• [ ] tacotron2
  • Issue: [torchbench] tacotron2 fails to run in eager-mode. #6112
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (inside skip list -- torchbench.yaml)
• timm_efficientdet
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
• vision_maskrcnn
  • PyTorch/XLA PR: Fix XLA tensor storage device by using XlaDeviceToAtenDevice. #5743
  • PyTorch PR: Skip aliasing correction for lift_fresh. pytorch#112202
  • Issue: [torchbench] vision_maskrcnn failing on inference with dynamo after bfloat16 conversion. #6557
    • PyTorch/XLA PR: index: fix index of 0-element tensor by 0-element tensor. #7113

Dynamo+openxla. 78/81 - 96% (against inductor)

• [x] DALLE2_pytorch
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Moved to canary models: Upgrade numpy to 2.0.0rc benchmark#2311
• Super_SloMo
  • PyTorch/XLA PR: Add support for _unsafe_index. #5707
  • PyTorch/benchmark PR: Register grid buffers for Super_SloMo. benchmark#2038
• cm3leon_generate
  • Issue: [torchbench] Check failed: cachedComputation #5967
• detectron2_fasterrcnn_r_101_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_101_dc5
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_101_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_50_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_50_dc5
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_50_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fcos_r_50_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_101_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_101_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_50_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_50_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• dlrm
  • PyTorch/XLA PR: Fix XLA tensor storage device by using XlaDeviceToAtenDevice. #5743
  • PyTorch PR: Skip aliasing correction for lift_fresh. pytorch#112202
• hf_BigBird
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• hf_GPT2
  • PyTorch/XLA PR: [dynamo] Move CPU tensor constructor nodes to XLA. #5922
• hf_GPT2_large
  • PyTorch/XLA PR: [dynamo] Move CPU tensor constructor nodes to XLA. #5922
• hf_Longformer
  • Issue: No support for overlapped tensors. #5835
    • PyTorch/XLA PR: Make as_strided_copy materialize a new tensor with index. #6624
• hf_Reformer
  • Issue: Dynamo (openxla) fails when returning tensor.expand. #5837
    • PyTorch PR: [AOTAutograd] Replay views on output using FunctionalTensor metas. pytorch#121007
• hf_T5_generate
  • Issue: [torchbench] Check failed: cachedComputation #5967
• moco
  • Issue: [torchbench] moco fails to run. #6083
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Issue: [torchbench] moco inference fails to run on dynamo. #7636
  • Issue: [torchbench] moco fails to run with CUDA OpenXLA fallback. #7647
• nvidia_deeprecommender
  • Issue: [torchbench] nvidia_deeprecommender fails to run. #6006
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• pyhpc_isoneutral_mixing
  • PyTorch/XLA PR: Fix XLA tensor storage device by using XlaDeviceToAtenDevice. #5743
  • PyTorch PR: Skip aliasing correction for lift_fresh. pytorch#112202
• pyhpc_turbulent_kinetic_energy
  • PyTorch/XLA PR: Fix XLA tensor storage device by using XlaDeviceToAtenDevice. #5743
  • PyTorch PR: Skip aliasing correction for lift_fresh. pytorch#112202
• pytorch_CycleGAN_and_pix2pix
  • Issue: [torchbench] pytorch_CycleGAN_and_pix2pix fails to run. #6007
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• speech_transformer
  • PyTorch/XLA PR: Sync xla_args before computation. #5823
• timm_efficientdet
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071

Models also Failing on Inductor

Inference Failing on Inductor CUDA with the Same Error

Benchmarks that raise the same error on inductor:

• hf_clip
  • 'str' object has no attribute 'shape'
• mobilenet_v2_quantized_qat
• resnet50_quantized_qat

Inference Failing on Inductor CUDA with Different Errors

• simple_gpt
  • RTX 2060 doesn't support BF16
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (only work with multiprocess enabled -- torchbench.yaml)
• simple_gpt_tp_manual
  • RTX 2060 doesn't support BF16
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (inside skip list -- torchbench.yaml)
• tacotron2
  • Issue: [torchbench] Check failed: xtensor #6005
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (inside skip list -- torchbench.yaml)

Training

Non-Dynamo. Pass rate: 64/66 - 96% (against inductor)

• [ ] DALLE2_pytorch
  • Issue: [torchbench] Training benchmarks failing with: tensor does not require grad #6084
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Moved to canary models: Upgrade numpy to 2.0.0rc benchmark#2311
• demucs
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• densenet121
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• detectron2_fasterrcnn_r_101_c4
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_fasterrcnn_r_101_dc5
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_fasterrcnn_r_101_fpn
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_fasterrcnn_r_50_c4
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_fasterrcnn_r_50_dc5
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_fasterrcnn_r_50_fpn
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_fcos_r_50_fpn
  • Skipped by the benchmarking script
• detectron2_maskrcnn_r_101_c4
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_maskrcnn_r_101_fpn
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_maskrcnn_r_50_c4
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• detectron2_maskrcnn_r_50_fpn
  • Issue: [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• dlrm
  • Issue: [torchbench] dlrm fails to run on training. #6008
    • PyTorch/XLA PR: Fallback _embedding_bag_backward and force sparse=false. #7584
• hf_GPT2_large
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• hf_Longformer
  • Issue: No support for overlapped tensors. #5835
    • PyTorch/XLA PR: Make as_strided_copy materialize a new tensor with index. #6624
• hf_T5_base
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• [ ] llama_v2_7b_16h
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
  • SKIP (training not supported -- torchbench.yaml)
• moco
  • Issue: [torchbench] moco fails to run. #6083
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Issue: [torchbench] moco inference fails to run on dynamo. #7636
  • Issue: [torchbench] moco fails to run with CUDA OpenXLA fallback. #7647
• nvidia_deeprecommender
  • RTX 2060 OOM
  • Issue: [torchbench] nvidia_deeprecommender fails to run. #6006
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• pytorch_CycleGAN_and_pix2pix
  • Issue: [torchbench] pytorch_CycleGAN_and_pix2pix fails to run. #6007
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• stable_diffusion_unet
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• [ ] tacotron2
  • Issue: [torchbench] tacotron2 fails to run in eager-mode. #6112
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • SKIP (inside skip list -- torchbench.yaml)
• timm_efficientdet
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
• timm_nfnet
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• timm_vision_transformer_large
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• yolov3
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071

Dynamo+openxla. Pass rate: 55/66 - 83% (against inductor)

• demucs
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• densenet121
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• dlrm
  • Issue: [torchbench] dlrm fails to run on training. #6008
    • PyTorch/XLA PR: Fallback _embedding_bag_backward and force sparse=false. #7584
• hf_BigBird
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• hf_GPT2
  • PyTorch/XLA PR: [dynamo] Move CPU tensor constructor nodes to XLA. #5922
• hf_GPT2_large
  • PyTorch/XLA PR: [dynamo] Move CPU tensor constructor nodes to XLA. #5922
• hf_Longformer
  • Issue: No support for overlapped tensors. #5835
    • PyTorch/XLA PR: Make as_strided_copy materialize a new tensor with index. #6624
• hf_Reformer
  • Issue: [torchbench] hf_Reformer fails to run on dynamo+openxla training. #6009
    • PyTorch PR: [AOTAutograd] Replay views on output using FunctionalTensor metas. pytorch#121007
• moco
  • Issue: [torchbench] moco fails to run. #6083
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Issue: [torchbench] moco inference fails to run on dynamo. #7636
  • Issue: [torchbench] moco fails to run with CUDA OpenXLA fallback. #7647
• nvidia_deeprecommender
  • Issue: [torchbench] Training benchmarks failing with: tensor does not require grad #6084
  • Issue: [torchbench] nvidia_deeprecommender fails to run. #6006
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• pytorch_CycleGAN_and_pix2pix
  • Issue: [torchbench] pytorch_CycleGAN_and_pix2pix fails to run. #6007
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
• stable_diffusion_unet
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• timm_efficientdet
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
  • Issue: [torchbench] Models require initialization on CUDA device. #6011
    • PyTorch/XLA PR: Re-land: Fix model initialization. #6296
    • PyTorch/XLA PR: Fix model initialization. #6076
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
• timm_vision_transformer
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• timm_vision_transformer_large
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
• torch_multimodal_clip
  • Issue: [torchbench] Check failed: xtensor #6005
• yolov3
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071

Models also Failing on Inductor

No Training Support on Inductor CUDA

Benchmarks that raise the error: Model's DEFAULT_TRAIN_BSIZE is not implemented.

• cm3leon_generate
• detectron2_fcos_r_50_fpn
• doctr_det_predictor
• doctr_reco_predictor
• hf_T5_generate
• llama
• phi_1_5
• pyhpc_equation_of_state
• pyhpc_isoneutral_mixing
• pyhpc_turbulent_kinetic_energy
• sam
• simple_gpt
• simple_gpt_tp_manual

Training Failing on Inductor CUDA with the Same Error

Benchmarks that raise the same error on inductor:

• DALLE2_pytorch
  • Issue: [torchbench] Training benchmarks failing with: tensor does not require grad #6084
  • Issue: [torchbench] Moving models from CUDA to XLA raise segmentation fault. #6010
    • PyTorch/XLA PR: Move CUDA tensors to CPU before moving to XLA. #6060
    • PyTorch/XLA PR: Move 0-dimensional tensors to CPU before copying to XLA. #6071
  • Moved to canary models: Upgrade numpy to 2.0.0rc benchmark#2311
• llama_v2_7b_16h
  • Issue: [torchbench] Training benchmarks failing with: OOM #6003
  • SKIP (training not supported -- torchbench.yaml)
• maml
  • Issue: [torchbench] Training benchmarks failing with: tensor does not require grad #6084
  • SKIP (training not supported -- torchbench.yaml)
• vision_maskrcnn
  • targets should not be none when in training mode
  • Fix Decomposition for upsample_linear{1d, 3d} pytorch#114774

Training Failing on Inductor CUDA with Different Errors

• detectron2_fasterrcnn_r_101_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_101_dc5
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_101_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_50_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_50_dc5
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_fasterrcnn_r_50_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_101_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_101_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_50_c4
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• detectron2_maskrcnn_r_50_fpn
  • Issue: [torchbench] AssertionError: All tensors should be on xla #5966
    • PyTorch/XLA PR: Ignore non-XLA nodes and their direct dependents. #6170
• opacus_cifar10
  • Issue: [torchbench] Check failed: cachedComputation #5967

cc @JackCaoG @miladm

[lezcano]

State after 7 weeks of work:

Models fixed so far:

• pyhpc_isoneutral_mixing
• pyhpc_turbulent_kinetic_energy
• dlrm
• Super_SloMo
• speech_transformer

PRs to fix the models. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• Fix masked_fill broadcasting. #5688
• Filter tensor arguments from traced model. #5689
• Add support for _unsafe_index. #5707
• Fix XLA tensor storage device by using XlaDeviceToAtenDevice. #5743
• Remove _unsafe_index implementation. #5769
• Sync xla_args before computation. #5823
• Fix as_strided for inputs smaller than the arguments specification. #5914
• Skip aliasing correction for lift_fresh. pytorch#112202
• Refactor move_constructor_to_cuda. pytorch#114626
• Refactor move_constructor_to_cuda. pytorch#114626
• Register grid buffers for Super_SloMo. benchmark#2038

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Fix hard-coded cuda device in ConstructorMoverPass. pytorch#114932
• [dynamo] Move CPU tensor constructor nodes to XLA. #5922
• Use subprocess for checking XLA supported devices. #5960
• Reduce benchmark experiment filename string. #5963
• Allow benchmark runner config: openxla for inference runs. #5939
• Install diffusers before checking for token. benchmark#2072

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• No support for overlapped tensors. #5835
• Dynamo (openxla) fails when returning tensor.expand. #5837
• Mutation after tensor.expand returns wrong result. #5839
• Failing Torchbench Models: tracking issue #5932
• Modifying PjRt device at runtime doesn't work. #5942
• CapabilityBasedPartitioner returns invalid partitions. pytorch#111033
• Mutation after tensor.expand returns wrong result. pytorch#114302

[ysiraichi]

Weekly update (Dec 1~Dec 10):

Models fixed:

• DALLE2_pytorch
  • training is now failing with the same error as inductor
• stable_diffusion_unet
  • training is still failing with OOM
• stable_diffusion_text_encoder
• hf_GPT2
• hf_GPT2_large
  • training without dynamo is still failing
• yolov3
  • Failing possibly due to a cuNND error, which is likely an OOM, on a RTX 2060. Haven't tested it yet on a A100, though

PRs merged. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• [dynamo] Move CPU tensor constructor nodes to XLA. #5922
• Allow benchmark runner config: openxla for inference runs. #5939
• Move CUDA tensors to CPU before moving to XLA. #6060
• Use subprocess for checking XLA supported devices. #6068
• Reduce benchmark experiment filename string. #6069
• Move 0-dimensional tensors to CPU before copying to XLA. #6071
• Install diffusers before checking for token. benchmark#2072
• Fix hard-coded cuda device in ConstructorMoverPass. pytorch#114932

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Fix model initialization. #6076
• Gather more summary data when running result_analyzer.py. #6067
• Enable CUDA support for PyTorch on CI. #6070
• Remove branch on undefined variable. #6072

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• [torchbench] AssertionError: All tensors should be on xla #5966
• [torchbench] Check failed: cachedComputation #5967
• [torchbench] Training benchmarks failing with: OOM #6003
• [torchbench] Benchmarks failing on eager-mode with: TIMEOUT (30 minutes) #6004
• [torchbench] Check failed: xtensor #6005
• [torchbench] dlrm fails to run on training. #6008
• [torchbench] hf_Reformer fails to run on dynamo+openxla training. #6009
• [torchbench] moco fails to run. #6083
• test_autocast_torch_bf16 fails if PyTorch is compiled with CUDA support. #6085
• test_grad_checkpoint.py fails if PyTorch is compiled with CUDA support. #6086

[ysiraichi]

Weekly update (Dec 11~Dec 15):

Models fixed:

• pytorch_CycleGAN_and_pix2pix
• nvidia_deeprecommender
  • dynamo+openxla training is still failling
• simple_gpt and simple_gpt_tp_manual
  • failing due to the same reasons as inductor
• moco
  • failing due to distributed backend
• timm_efficientdet
  • dynamo+openxla training is still failing

PRs merged. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• Remove branch on undefined variable. #6072
• Fix model initialization. #6076
• Move test_result_analyzer.py to the test directory. #6130
• [benchmarks] Collect XLA compilation and execution metrics. #6153
• Re-land: Fix model initialization. #6182

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Enable CUDA support for PyTorch on CI. #6070
• Move tests that need CUDA to test_operations.py. #6160
• Ignore non-XLA nodes and their direct dependents. #6170
• Adapt a few torch.utils.checkpoint functions for PyTorch/XLA. #6178
• [benchmarks] Collect dynamo counters. #6180
• Try creating a bf16 tensor as a last resort of is_bf16_supported(). pytorch#115924

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• [torchbench] Training benchmarks failing with: tensor does not require grad #6084
• [torchbench] tacotron2 fails to run in eager-mode. #6112
• Can't run autocast on XLA, if PyTorch was compiled with CUDA support. pytorch#115900

[miladm]

Can we please add a pass rate table in the weekly report that includes:

Inference

• Inductor, Dynamo+PyTocrh/XLA:GPU, Non-Dynamo+PyTocrh/XLA:GPU

Training

• Inductor, Dynamo+PyTocrh/XLA:GPU, Non-Dynamo+PyTocrh/XLA:GPU

[ysiraichi]

Weekly update (Jan 8 ~ Jan 12):

Pass rate (out of 99 benchmarks):

	Inference	Training
Inductor	91	64
Non-Dynamo	87	67
Dynamo	86	57

Models fixed:

• detectron2 models (inference with dynamo)
• hf_BigBird (inference and training with dynamo)
• torch_multimodal_clip (training with dynamo)
• timm_vision_transformer (training with dynamo)
• Likely not due to the merged PRs below:
  • detectron2 models: all but detectron2_fcos_r_50_fpn (training without dynamo)

PRs merged. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• Try creating a bf16 tensor as a last resort of is_bf16_supported(). pytorch#115924

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Use spawn as the fork method for the profiler test. #6302
• Re-land: Fix model initialization. #6296
• Move tests that need CUDA to test_operations.py. #6160
• Enable CUDA support for PyTorch on CI. #6070

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• test_trace_and_metrics fail if PyTorch has CUDA support. #6292

[ysiraichi]

Weekly update (Jan 15 ~ Jan 19):

Pass rate (out of 99 benchmarks):

	Inference	Training
Inductor	85	62
Non-Dynamo	70	57
Dynamo	71	55

Models that started failing:

• After Re-land: Fix model initialization. #6296:
  • detectron2_fasterrcnn_r_101_c4
  • detectron2_fasterrcnn_r_101_dc5
  • detectron2_fasterrcnn_r_101_fpn
  • detectron2_fasterrcnn_r_50_c4
  • detectron2_fasterrcnn_r_50_dc5
  • detectron2_fasterrcnn_r_50_fpn
  • detectron2_fcos_r_50_fpn
  • detectron2_maskrcnn_r_101_c4
  • detectron2_maskrcnn_r_101_fpn
  • detectron2_maskrcnn_r_50_c4
  • detectron2_maskrcnn_r_50_fpn
  • mobilenet_v3_large
  • timm_regnet
  • hf_Bart
• Started being skipped:
  • pytorch_CycleGAN_and_pix2pix
  • pytorch_unet
• Unsupported precision:
  • pytorch_unet
  • yolov3
• cuDNN error:
  • Super_SloMo (inductor)

PRs merged. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• Use spawn as the fork method for the profiler test. #6302
• Re-land: Fix model initialization. #6296
• Clean-up CUDA cache. #6325

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Move tests that need CUDA to test_operations.py. #6160
• Enable CUDA support for PyTorch on CI. #6070

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• [torchbench] Detectron2 benchmarks failing to run. #6336

[miladm]

Can we track separate passrate tables for L4 and A100 GPUs going forward @ysiraichi?

cc @frgossen @golechwierowicz @cota

[ysiraichi]

Weekly update (Jan 22 ~ Jan 26):

Pass rate (out of 99 benchmarks):

	Inference	Training
Inductor	88	63
Non-Dynamo	69	57
Dynamo	72	55

Models fixed:

• (inductor) moco
• (inductor) Super_SloMo
  • Failed when executed with all other benchmarks
  • Passed when executed alone (by specifying --filter argument)
• (inference) llama_v2_7b_16h

Models that started failing:

• (inference + non-dynamo) timm_efficientnet (to be fixed by: #6389)
• (inference + non-dynamo) timm_nfnet (to be fixed by: #6389)

PRs merged. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• [torchbench] Force llama_v2_7b_16h to use SGD. #6350
• Fix walrus operator precedence. #6374
• Use benchmark_cls for checking precision. #6375
• Do not skip llama_v2_7b_16h on NVIDIA A100 40G. benchmark#2124
• Move skip sets into a new file. pytorch#118032

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Force to fp16 models to fp32 if XLA_USE_FP16 is already set. #6389
• Move tests that need CUDA to test_operations.py. #6160
• Enable CUDA support for PyTorch on CI. #6070

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• [torchbench] Regression after changing load_benchmark method. #6348
• [torchbench] Regression: detectron2_maskrcnn training fails on non-dynamo.  #6353
• [torchbench] Super_SloMo inference fails on inductor. #6366
• [torchbench] moco inference and training fail on inductor. #6367
• [torchbench] opacus_cifar10 memory not freed after each run. #6380
• [torchbench] opacus_cifar10 training runs unexpectedly without errors. #6391

[ysiraichi]

Weekly update (Jan 29 ~ Feb 2):

Pass rate (out of 99 benchmarks):

A100

	Inference	Training
Inductor	87 (last: 88)	63
Non-Dynamo	82 (last: 69)	56 (last: 57)
Dynamo	82 (last: 72)	53 (last: 55)

L4

	Inference	Training
Inductor	86	60
Non-Dynamo	81	53
Dynamo	82	49

Models Summary (for A100)

• Inductor: Inference (-4, +3)
  • (fail) New skips by PyTorch's torchbench skip list:
    • detectron2_maskrcnn
    • hf_Bert
    • hf_Bert_large
    • maml
  • (pass) Remove outdated skip:
    • vision_maskrcnn
  • (pass) AMP supported:
    • pytorch_unet
    • yolov3
• Inductor: Training (-3, +3)
  • (fail) New skips by PyTorch's torchbench skip list:
    • hf_Bert
    • hf_Bert_large
  • (fail) Failing due to sparse error:
    • dlrm
  • (pass) AMP supported:
    • pytorch_unet
  • (pass) No OOM:
    • demucs
    • opacus_cifar10
• XLA:GPU (non-dynamo): Inference (-3, +16)
  • (fail) New skips by PyTorch's torchbench skip list:
    • detectron2_maskrcnn
    • hf_Bert
    • hf_Bert_large
  • (pass) Forcing fp32 precision (while setting XLA_USE_FP16):
    • detectron2 benchmarks (11)
    • mobilenet_v3_large
    • timm_efficientnet
    • timm_nfnet
    • timm_regnet
  • (pass) AMP supported:
    • yolov3
• XLA:GPU (non-dynamo): Training (-2, +1)
  • (fail) New skips by PyTorch's torchbench skip list:
    • hf_Bert
    • hf_Bert_large
  • (pass) No OOM:
    • hf_GPT2_large
• XLA:GPU (dynamo): Inference (-4, +14)
  • (fail) New skips by PyTorch's torchbench skip list:
    • detectron2_maskrcnn
    • hf_Bert
    • hf_Bert_large
    • maml
  • (pass) Remove outdated skip:
    • vision_maskrcnn
  • (pass) Forcing fp32 precision (while setting XLA_USE_FP16):
    • detectron2 benchmarks (11)
    • hf_Bart
  • (pass) AMP supported:
    • yolov3
• XLA:GPU (dynamo): Training (-2, +0)
  • (fail) New skips by PyTorch's torchbench skip list:
    • hf_Bert
    • hf_Bert_large

---

PRs merged. For an updated list see [XLA, pytorch/benchmarks, pytorch/pytorch]

• Enable CUDA support for PyTorch on CI. #6070
• Move tests that need CUDA to test_operations.py. #6160
• Force to fp16 models to fp32 if XLA_USE_FP16 is already set. #6389
• [torchbench] Re-land: Force llama_v2_7b_16h to use SGD. #6402
• Check precision only if on CUDA and not inductor. #6407
• Use PyTorch's dynamo benchmark skip-list. #6416
• [benchmarks] Fix CUDA device check. #6419
• Fix distributed checkpoint errors when PyTorch CUDA is enabled. #6421
• Add benchmarks/dynamo search option for finding PyTorch skip file. #6446
• [benchmarks] Add support for AMP execution. #6447

PRs in flight. For an updated list see [XLA, pytorch/pytorch, pytorch/benchmarks]

• Implement shallow copy functions for FunctionalTensorWrapper. pytorch#118783

Issues identified that the PRs in flight do not fix. For an updated list see [XLA, pytorch/pytorch]

• Unexpected upcasted output when XLA_USE_FP16 and XLA_USE_BF16 are set. #6403
• Tracking issue: PyTorch precision upcast issue. #6404