[LoRA] allow big CUDA tests to run properly for LoRA (and others) (#9… #4257

Job	Run time
Flax TPU Tests	3m 3s
ONNX CUDA Tests	4m 36s
Setup Torch Pipelines CUDA Slow Tests Matrix	2m 26s
Torch CUDA Tests (models)	6m 54s
Torch CUDA Tests (schedulers)	5m 12s
Torch CUDA Tests (lora)	5m 58s
Torch CUDA Tests (others)	41s
Examples PyTorch CUDA tests on Ubuntu	30m 49s
PyTorch Compile CUDA tests	3m 2s
Torch CUDA Tests (single_file)	1m 27s
PyTorch xformers CUDA tests	5m 29s
Torch Pipelines CUDA Tests (cogvideo)	3m 45s
Torch Pipelines CUDA Tests (controlnet)	12m 38s
Torch Pipelines CUDA Tests (ddpm)	39s
Torch Pipelines CUDA Tests (flux)	3m 37s
Torch Pipelines CUDA Tests (ip_adapters)	2m 23s
Torch Pipelines CUDA Tests (kandinsky2_2)	5m 3s
Torch Pipelines CUDA Tests (latent_consistency_models)	3m 1s
Torch Pipelines CUDA Tests (ltx)	1m 6s
Torch Pipelines CUDA Tests (pixart_alpha)	2m 36s
Torch Pipelines CUDA Tests (stable_diffusion)	5m 32s
Torch Pipelines CUDA Tests (stable_diffusion_2)	6m 26s
Torch Pipelines CUDA Tests (stable_diffusion_3)	1m 27s
Torch Pipelines CUDA Tests (stable_diffusion_adapter)	3m 28s
Torch Pipelines CUDA Tests (stable_diffusion_xl)	6m 37s
Torch Pipelines CUDA Tests (stable_video_diffusion)	51s
	2h 8m 46s

Provide feedback