cudaFree(rawptr) failed with error an illegal memory access was encountered

[yoker5262]

ubuntu
cuda 11.8
instant-ngp can use snapshot, but nerfshop will be wrong when using snapshot .
How to solve this problem please?

(base) yoker@yoker-OMEN:~/github_clone/nerfshop$ ./build/nerfshop --scene /opt/image-synthesis/output_data/WeChat_20230828181639/transforms.json --snapshot /home/yoker/instant-ngp_test_base.ingp --no-train 16:50:19 INFO Loading NeRF dataset from 16:50:19 INFO /opt/image-synthesis/output_data/WeChat_20230828181639/transforms.json 16:50:20 SUCCESS Loaded 222 images of size 1920x1080 after 0s 16:50:20 INFO cam_aabb=[min=[-0.761813,-0.919145,0.91396], max=[1.5125,1.63957,1.81485]] 16:50:20 INFO Loading network config from: /home/yoker/instant-ngp_test_base.ingp 16:50:20 INFO GridEncoding: Nmin=16 b=2.97199 F=4 T=2^19 L=8 Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered Could not free memory: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/gpu_memory.h:129 cudaFree(rawptr) failed with error an illegal memory access was encountered 16:50:20 ERROR Uncaught exception: /home/yoker/github_clone/nerfshop/dependencies/tiny-cuda-nn/include/tiny-cuda-nn/encodings/grid.h:924 cudaMemcpy(m_hashmap_offsets_table.data(), m_hashmap_offsets_table_cpu.data(), (m_n_levels+1) * sizeof(uint32_t), cudaMemcpyHostToDevice) failed with error an illegal memory access was encountered