implement torch.linalg.svd xpu backend

[yao-matrix]

🚀 The feature, motivation and pitch

torch.linalg.svd is need by PEFT LOFTQ and PISSA, but on XPU it shows the log as below:

usr/local/lib/python3.11/dist-packages/peft/utils/loftq_utils.py:182: UserWarning: Aten Op fallback from XPU to CPU happends. This may have performance implications. If need debug the fallback ops please set environment variable PYTORCH_DEBUG_XPU_FALLBACK=1 (Triggered internally at /pytorch/build/xpu/ATen/RegisterXPU_0.cpp:54525.)
U, S, Vh = torch.linalg.svd

pls plan torch.linalg.svd xpu backend rather than falling back to cpu.

Alternatives

No response

Additional context

No response

[CuiYifeng]

Need to discuss performance issue with oneMKL.

[CuiYifeng]

@yao-matrix Could you provide the proportion of torch.linalg.svd in E2E time so that we can have data to discuss with oneMKL?

[yao-matrix]

For this op, w/o xpu backend, we cannot finish the finetuning, just waiting forever, so I cannot give you a E2E percentage, since i cannot even run the e2e workload w/o it, it's kind of a showstopper.

[CuiYifeng]

For this op, w/o xpu backend, we cannot finish the finetuning, just waiting forever, so I cannot give you a E2E percentage, since i cannot even run the e2e workload w/o it, it's kind of a showstopper.

Cannot complete even 1 iteration in the finetuning? A percentage of SVD in 1 iteration is also worth referencing. Just a reminder: could you check if "waiting forever" is due to a hang?

Some comments about SVD from MKL: "Note that even for a GPU-optimized SVD, some of the computations may be performed on the CPU".