How to release the momory used to load scenes?

[Galaxeaaa]

I am trying to use Mitsuba 3 to render a lot of scenes in one single execution of a python file, which involves many invokes of mi.load_dict(). After some iterations I will get error Critical Dr.Jit compiler failure: cuda_check(): API error 0700 (CUDA_ERROR_ILLEGAL_ADDRESS): "an illegal memory access was encountered" in D:\a\drjit\drjit\ext\drjit-core\src\init.cpp:454. I checked my GPU memory usage and it does increase linearly as the loop keeps going. Therefore I suppose this error results from unreleased scene memory. Is there a way to release the memory for a loaded scene? And do you think there are other reasons for this error?

[merlinND]

Hello @Galaxeaaa,

Have a look at the clean_up() function that runs between unit tests:

mitsuba3/src/conftest.py

Lines 45 to 63 in c9e881b

	def clean_up():
	'''
	Ensure no leftover instances from other tests are still in registry, wait
	for all running kernels to finish and reset the JitFlag to default. Also
	periodically frees the malloc cache to prevent the testcases from hogging
	all system memory.
	'''
	gc.collect()
	gc.collect()
	dr.kernel_history_clear()
	dr.flush_malloc_cache()
	dr.malloc_clear_statistics()
	dr.flush_kernel_cache()
	if hasattr(dr, 'sync_thread'):
	dr.sync_thread()
	dr.registry_clear()
	dr.set_flags(dr.JitFlag.Default)

You should be able to use it, or a subset, to load a large number of scenes in sequence.

[Galaxeaaa]

Hi @merlinND,

Thank you for your reply! I copied the clean_up() function and calls it after one scene is rendered. However I got this in my output:

Unhandled exception caught in c10/util/AbortHandler.h
00007FFAA8477FF400007FFAA845C130 torch_python.dll!THPGenerator_initDefaultGenerator [<unknown file> @ <unknown line number>]
00007FFBA117EE1200007FFBA117EDF0 ucrtbase.dll!terminate [<unknown file> @ <unknown line number>]
00007FFB98671ABF00007FFB98671150 VCRUNTIME140_1.dll!_NLG_Return2 [<unknown file> @ <unknown line number>]
00007FFB9867232B00007FFB98671150 VCRUNTIME140_1.dll!_NLG_Return2 [<unknown file> @ <unknown line number>]
00007FFB986740E900007FFB98674040 VCRUNTIME140_1.dll!_CxxFrameHandler4 [<unknown file> @ <unknown line number>]
00007FFBA36349FF00007FFBA36348D0 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FFBA35AE46600007FFBA35AD9D0 ntdll.dll!RtlFindCharInUnicodeString [<unknown file> @ <unknown line number>]
00007FFBA35E446500007FFBA35E42D0 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FFBA0A7543C00007FFBA0A753D0 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FFB27A0622000007FFB27A06190 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FFB72DD15CD <unknown symbol address> drjit-core.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFB72DEE95E <unknown symbol address> drjit-core.dll!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFB72E28A6800007FFB72E28A10 drjit-core.dll!jit_registry_remove [<unknown file> @ <unknown line number>]
00007FFACEB3BF5700007FFACEB3BF40 mitsuba.dll!mitsuba::Sensor<drjit::DiffArray<drjit::CUDAArray<float> >,mitsuba::Color<drjit::DiffArray<drjit::CUDAArray<float> >,3> >::operator delete [<unknown file> @ <unknown line number>]
00007FFB7376B8D900007FFB7376B680 obj.dll!mitsuba::Struct::Field::operator!= [<unknown file> @ <unknown line number>]
00007FFACEB0301B00007FFACEB02E50 mitsuba.dll!mitsuba::Logger::class_ [<unknown file> @ <unknown line number>]
00007FFACEC3336600007FFACEC332A0 mitsuba.dll!mitsuba::Scene<drjit::DiffArray<drjit::CUDAArray<float> >,mitsuba::Color<drjit::DiffArray<drjit::CUDAArray<float> >,3> >::~Scene<drjit::DiffArray<drjit::CUDAArray<float> >,mitsuba::Color<drjit::DiffArray<drjit::CUDAArray<float> >,3> > [<unknown file> @ <unknown line number>]
00007FFACEC245A700007FFACEC22470 mitsuba.dll!mitsuba::PCG32Sampler<drjit::DiffArray<drjit::LLVMArray<float> >,mitsuba::Color<drjit::DiffArray<drjit::LLVMArray<float> >,3> >::PCG32Sampler<drjit::DiffArray<drjit::LLVMArray<float> >,mitsuba::Color<drjit::DiffArray<drjit::LLVMArray<float> >,3> > [<unknown file> @ <unknown line number>]
00007FFAA7BF181800007FFAA7A9E8E8 mitsuba_cuda_ad_rgb_ext.cp310-win_amd64.pyd!ScopedSignalHandler::operator= [<unknown file> @ <unknown line number>]
00007FFACEEA646E <unknown symbol address> drjit_ext.cp310-win_amd64.pyd!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFAFBE934FF00007FFAFBE92EF0 python310.dll!PyDict_MaybeUntrack [<unknown file> @ <unknown line number>]
00007FFAFBF7917D00007FFAFBF77A70 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFAFBF7CE7B00007FFAFBF77A70 python310.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFAFBF778C200007FFAFBF77840 python310.dll!PyEval_EvalCode [<unknown file> @ <unknown line number>]
00007FFAFBFEE08E00007FFAFBFEDD50 python310.dll!PyRun_FileExFlags [<unknown file> @ <unknown line number>]
00007FFAFBFEE16800007FFAFBFEDD50 python310.dll!PyRun_FileExFlags [<unknown file> @ <unknown line number>]
00007FFAFBFEDD1800007FFAFBFEDBB0 python310.dll!PyRun_StringFlags [<unknown file> @ <unknown line number>]
00007FFAFBFEBF7500007FFAFBFEBCD0 python310.dll!PyRun_SimpleFileObject [<unknown file> @ <unknown line number>]
00007FFAFBFEB12400007FFAFBFEB060 python310.dll!PyRun_AnyFileObject [<unknown file> @ <unknown line number>]
00007FFAFBE085DC00007FFAFBE00390 python310.dll!PyObject_GC_IsFinalized [<unknown file> @ <unknown line number>]
00007FFAFBE08FED00007FFAFBE00390 python310.dll!PyObject_GC_IsFinalized [<unknown file> @ <unknown line number>]
00007FFAFBE09E9300007FFAFBE09310 python310.dll!Py_RunMain [<unknown file> @ <unknown line number>]
00007FFAFBE09F0600007FFAFBE09EE0 python310.dll!Py_Main [<unknown file> @ <unknown line number>]
00007FF73D71149400007FF73D711110 python.exe!OPENSSL_Applink [<unknown file> @ <unknown line number>]
00007FFBA1A1257D00007FFBA1A12560 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FFBA35EAA4800007FFBA35EAA20 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]

Do you have any idea on how to solve this?

[merlinND]

I'm not sure, it's difficult to tell from the stack trace. The clean_up() function is very aggressive and will lead to segfaults if anything from the previously loaded scene is still referenced or used in any way.
So I would try to make sure that everything is out of scope (including no "hidden" references via an AD graph) before calling it.

You can also try commenting out different parts of the clean_up() function to see if it's enough to reclaim memory while being less aggressive.

[njroussel]

One small addition, I just wanted to be clear about the following: any object, including scenes, should be cleaned up automatically as soon as they are no longer referenceable. As Merlin mentionned, this can be quite sneaky sometimes, variable dependencies aren't always obvious.

[Galaxeaaa]

FYI, I still didn't make clean_up() or its subset work, but I eventually bypassed this problem by writing a script to run a python file multiple times with different arguments.