CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

[Xuhaocheng2023]

When I ran the Pre-trained model
python ase/run.py --test --task HumanoidReach --num_envs 16 --cfg_env ase/data/cfg/humanoid_sword_shield_reach.yaml --cfg_train ase/data/cfg/train/rlg/hrl_humanoid.yaml --motion_file ase/data/motions/reallusion_sword_shield/RL_Avatar_Idle_Ready_Motion.npy --llc_checkpoint ase/data/models/ase_llc_reallusion_sword_shield.pth --checkpoint ase/data/models/ase_hlc_reach_reallusion_sword_shield.pth
I got "RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)"

the details:
Traceback of TorchScript (most recent call last):
File "/home/xhc/ASE/ASE-main/ase/env/tasks/humanoid.py", line 620, in compute_humanoid_observations_max

root_h = root_pos[:, 2:3]

heading_rot = torch_utils.calc_heading_quat_inv(root_rot)
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
if (not root_height_obs):

File "/home/xhc/ASE/ASE-main/ase/utils/torch_utils.py", line 177, in calc_heading_quat_inv

# the heading is the direction on the xy plane
# q must be normalized
heading = calc_heading(q)
          ~~~~~~~~~~~~ <--- HERE
axis = torch.zeros_like(q[..., 0:3])
axis[..., 2] = 1

File "/home/xhc/ASE/ASE-main/ase/utils/torch_utils.py", line 153, in calc_heading

ref_dir = torch.zeros_like(q[..., 0:3])
ref_dir[..., 0] = 1
rot_dir = quat_rotate(q, ref_dir)
          ~~~~~~~~~~~ <--- HERE
heading = torch.atan2(rot_dir[..., 1], rot_dir[..., 0])

File "/home/xhc/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/torch_utils.py", line 68, in quat_rotate
shape[0], 3, 1))

c = q_vec * \
    torch.bmm(q_vec.view(shape[0], 1, 3), v.view(
    ~~~~~~~~~ <--- HERE
        shape[0], 3, 1)).squeeze(-1) * 2.0
return a + b + c

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling cublasCreate(handle)

[xbpeng]

It looks like a CUDA specific error. Maybe CUDA is not set up correctly. You can probably google that to see if there are any suggested solutions online.