Scene environment and scene background are leaking after deep clean

[catalin-enache]

Description

After deep cleaning the scene, the render info still reports geometries and textures depending on the scenario.
It looks like while traversing the scene we cannot reach certain geometries and textures so that we can dispose them.

Reproduction steps

When no scene.background and no scene.environment, all geometries and textures from scene meshes are disposed properly.
E.g. when using a custom mesh or GLTF without scene.background and scene.environments the scene is cleaned up properly.

When FBX + scene.environment all geometries and textures from scene meshes are disposed properly.

Background issue:
When using only the scene.background (not any meshes in the scene) it is not disposed after scene.background.dispose(); scene.background = null;

Environment issue:
When scene.environment is added in conjunction with custom mesh or GLTF, geometries and textures are still reported after cleaning up the scene.
Strange fact: When scene.environment is added in conjunction with FBX, geometries and textures are zeroed out after cleaning up the scene, which is expected behaviour whatever the scenario would be.

Note:
even if not setting the loaded texture on scene.environment but setting it directly to the mesh.texture.envMap
it triggers the same leaking behaviour after deepClean.

Tested with UltraHDRLoader, RGBELoader, and locally with CubeTexture.
The behaviour is the same, which looks like not being a loader issue.

Code

This is the function used to deep clean the scene.

const deepCleanScene = (scene) => {
  const geometriesSet = new Set();
  const materialsSet = new Set();
  const texturesSet = new Set();
  const skeletonsSet = new Set();
  const disposableSet = new Set();
  const renderTargetsSet = new Set();

  scene.traverse((descendant) => {
    if (descendant instanceof THREE.Scene) {
      if (descendant.background instanceof THREE.Texture) {
      	console.log('disposing of scene.background');
        descendant.background.dispose();
        descendant.background = null;
      }
      if (descendant.environment instanceof THREE.Texture) {
        console.log('disposing of scene.environment');
        descendant.environment.dispose();
        descendant.environment = null;
      }
    }

    if ('dispose' in descendant) {
      disposableSet.add(descendant);
    }

    if (descendant.isLight && descendant.shadow && descendant.shadow.map) {
      renderTargetsSet.add(descendant.shadow.map);
    }

    if (descendant.renderTarget) {
      renderTargetsSet.add(descendant.renderTarget);
      const _texture = descendant.renderTarget.texture;
      const _textures = descendant.renderTarget.textures;
      const texture = !_texture ? [] : Array.isArray(_texture) ? _texture : [_texture];
      const textures = !_textures ? [] : Array.isArray(_textures) ? _textures : [_textures];
      const allTextures = [...texture, ...textures];
      allTextures.forEach((tex) => {
        texturesSet.add(tex);
      });
    }

    if (descendant.material) {
      const materials = Array.isArray(descendant.material)
        ? descendant.material
        : [descendant.material];
      materials.forEach((mat) => {
        materialsSet.add(mat);
        
        Object.keys(mat).forEach((key) => {
          if (mat[key] instanceof THREE.Texture) {
            texturesSet.add(mat[key]);
            mat[key] = null;
          }
        });

        if (mat.uniforms) {
          console.log('Cleaning up material found uniforms', { uniforms: mat.uniforms, obj: descendant });
          for (const value of Object.values(mat.uniforms)) {
            if (value) {
              const uniformValues = Array.isArray(value.value) ? value.value : [value.value];
              uniformValues.forEach((uniformValue) => {
                if (uniformValue instanceof THREE.Texture) {
                  console.log('Cleaning up material disposing uniformValue as Texture', {
                    obj: descendant,
                    uniformValue
                  });
                  texturesSet.add(uniformValue);
                }
              });
            }
          }
        }
        // mat.needsUpdate = true;
      });
    }

    if (descendant.geometry) {
      geometriesSet.add(descendant.geometry);
    }

    if (descendant.skeleton) {
      skeletonsSet.add(descendant.skeleton);
    }
    
    if (descendant.skeleton && descendant.skeleton.boneTexture) {
      texturesSet.add(descendant.skeleton.boneTexture);
    }

  });

  console.log('Disposing of', {
    geometries: geometriesSet.size,
    materials: materialsSet.size,
    textures: texturesSet.size,
    skeletons: skeletonsSet.size,
    disposables: disposableSet.size,
    renderTargets: renderTargetsSet.size
  });

  const childrenToRemove = [...scene.children];

  childrenToRemove.forEach((child) => {
    child.removeFromParent();
  });

  geometriesSet.forEach((geometry) => {
    geometry.dispose();
  });
  geometriesSet.clear();

  texturesSet.forEach((texture) => {
    if (texture instanceof ImageBitmap) {
      texture.close();
    }
    texture.dispose();
  });
  texturesSet.clear();

  materialsSet.forEach((material) => {
    material.dispose();
  });
  materialsSet.clear();

  disposableSet.forEach((disposable) => {
    disposable.dispose();
  });
  disposableSet.clear();

  skeletonsSet.forEach((skeleton) => {
    skeleton.dispose();
  });
  skeletonsSet.clear();

  renderTargetsSet.forEach((renderTarget) => {
    renderTarget.dispose();
  });
  renderTargetsSet.clear();
};

Live example

https://jsfiddle.net/catalin_enache/Lc319amo/433/

There are some flags in the fiddle to play with in order to trigger some scenario from the described scenarios

Screenshots

No response

Version

0.173.0

Device

Desktop

Browser

Chrome

OS

MacOS

[Mugen87]

For performance reasons the renderer keeps a few references to internal materials and geometries like the skybox mesh. As long as the memory is not keep growing we do not consider this as a bug or misbehavior.

Maybe we should document this in the FAQ section for a better understanding. Meaning even if you perform a deep clean, you potentially see a few entries in renderer.info.

[catalin-enache]

I see your point.

For the scenario where the scene is cleared and repopulated multiple times, the texture related to the usage of scene.environment or material.envMap keeps adding up, one texture for each cycle.

One would end up with as many textures hanging around as cleaning/repopulating cycles.

For this issue I managed to make a hack by overriding PREMGenerator._compileMaterial private method.
To collect the PMREMGenerator instances used by ThreeJs under the hood into the WebGLRenderer instance.

THREE.PMREMGenerator.prototype._compileMaterial = (function () {
  const oldCompileMaterial = THREE.PMREMGenerator.prototype._compileMaterial;
  return function (this: THREE.PMREMGenerator, material: THREE.Material): void {
    oldCompileMaterial.call(this, material);
    this._renderer._pmremGenerators = this._renderer._pmremGenerators || new Set();
    this._renderer._pmremGenerators.add(this);
  };
})();

then in the deepCleanScene helper

(renderer._pmremGenerators as Set<THREE.PMREMGenerator>)?.forEach((pmremGenerator) => {
    pmremGenerator.dispose();
  });
// not clearing the set cos PMREMGenerator is instantiated only once by WebGLCubeUvMaps
// renderer._pmremGenerators.clear(); 

With this hack, the hanging material and almost all hanging geometries (except one) are disposed.

But I'm relying on private method to collect PMREMGenerator instances that are out of my control, which is not future proof.

There remain one geometry hanging around related to scene.background usage.
Fortunately this one geometry does not add up during consecutive scene cleanings and repopulating cycles.

But is annoying to see it remaining around.

With this one there cannot be a hack from what I've researched.
That geometry is related to WebGLBackground which is not exported from ThreeJs, thus cannot be approached.
Neither through WebGLRenderer cannot be approached, cos WebGLRenderer is keeping background instance around as a closure.

The only place where WebGLBackground is disposed is in WebGLRenderer.dispose() but that means you cannot continue using that WebGLRenderer anymore due to other side effects.

It would be nice to have an official ThreeJs helper deepCleanScene that should destroy everything, while leaving the scene and renderer alive.

It would be useful for cases where multiple scene cleanings and repopulating cycles are needed.

If not, at least could the background instance be exposed from WebGLRenderer ? So that we can it can be disposed manually ? and even better to also expose PMREMGenerator instance used under the hood so that it also can be disposed manually.
On the same model as renderLists, properties, info are exposed.

I can make PR if agreed to expose these.
That would be exposing background, cubeuvmaps on WebGLRenderer instance and pmremGenerator on WebGLCubeUVMaps instance.

Then when cleaning up:

renderer.background.dispose()
renderer.cubeuvmaps.pmremGenerator.dispose()

Most likely exposing cubemaps on WebGLRenderer instance would also help (I'm not sure about them yet)

Probably it can be suggested to throw away the scene and the renderer and start with new instances.
But I think that throwing the renderer away with textures or geometries still reported as being in use will leave them alive in GPU.

Note: renderer.dispose() does not properly cleanup the hanging texture related to the usage of scene.environment or material.envMap which are related to the internal use of PMREMGenerator.
Only with that hack mentioned before can that texture be properly disposed, and which if not disposed keeps adding up
on multiple clean/repopulating cycles.
Now, starting with a new renderer after calling oldRenderer.dispose() which currently still leaves that texture around ,
wont let the texture leave in GPU as an uncollected garbage ?

[Mugen87]

I can make PR if agreed to expose these.

I do not vote to expose more components of the renderer.

If you call dispose() on an envmap, the internal PMREM should be disposed as well. There is an explicit dispose() event listener for that purpose in WebGLCubeUVMaps.

three.js/src/renderers/webgl/WebGLCubeUVMaps.js

Lines 97 to 112 in 9c01215

	function onTextureDispose( event ) {
	const texture = event.target;
	texture.removeEventListener( 'dispose', onTextureDispose );
	const cubemapUV = cubeUVmaps.get( texture );
	if ( cubemapUV !== undefined ) {
	cubeUVmaps.delete( texture );
	cubemapUV.dispose();
	}
	}

Are you saying this listener does not work correctly?

[catalin-enache]

onTextureDispose => cubemapUV.dispose(); is disposing of the WebGLRenderTarget instance returned from PMREMGenerator.
But the PMREMGenerator instance itself is not disposed.
Inside that instance are the geometries that hang and the texture that keeps adding up on consecutive cleanUp/rePopulate cycles.
Specifially _pingPongRenderTarget inside PMREMGenerator contains that texture and it is not disposed until PMREMGenerator is disposed.

If it would be like that:

if ( cubemapUV !== undefined ) {
		cubeUVmaps.delete( texture );
		cubemapUV.dispose();
		pmremGenerator.dispose();
}

it should clean up properly. Just tested that. Nice idea to focus on that area.
With this as a possible patch the only thing remaining around is one geometry related to WebGLBackground
which if would be exposed on the WebGLRenderer instance could be disposed manually.