[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly #348

nauman-tintash · 2020-12-04T13:24:24Z

Summary

When I keep vertex positions buffer as optimization parameters, the loss is apparently being backpropogated incorrectly. Even if I keep the reference image and new image exactly the same, over a few iterations, the vertex positions keep getting updated and the loss keeps increasing.

System configuration

Platform: Ubuntu 20.4
Compiler: CMAKE 3.16.3
Python version: 3.8.5
Mitsuba 2 version: mitsuba-2
Branch : pathreparam-optix7
Compiled variants:
- scalar_rgb
- gpu_autodiff_rgb

Description

Even if my source and target scene is exactly the same, if I keep the vertex positions buffer as differentiable parameter, the loss keep getting increased. Following is a sample of losses and corresponding images for the first 4 iterations from one example.

Reference Render:

Iteration 1 : (Loss = 4.21295e-16)

Iteration 2 : (Loss = 0.000416789)

Iteration 3 : (Loss = 0.0170517)

Following is my relevant code snippet :

scene = load_file('bagel/scene.xml')
image_ref = render(scene, spp=2)

params = traverse(scene)
params.keep(['object.vertex_positions_buf'])
opt = Adam(params, lr=.02)

for it in range(iterations):
      image = render(scene, spp=2)
      ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
      ek.backward(ob_val)
      opt.step()

The text was updated successfully, but these errors were encountered:

Speierers · 2020-12-07T08:25:32Z

Hi @nauman-tintash ,

This is strange indeed. At iteration 1, the loss you are getting is very small and I am surprised the first gradient step causes such a difference on iteration 2. Could it be that your learning rate is too high?

Also it would be useful to verify that the two renders are indeed identical (e.g. use the same random numbers), otherwise a noise difference in the primal images would cause non-zero gradients. You could for instance run the following:

scene = load_file('bagel/scene.xml')
image_ref = render(scene, spp=2)
ob_val = ek.hsum(ek.sqr(image_ref - image_ref)) / len(image_ref )

to check whether ob_val ==0.0 or if this is simply due to numerical imprecisions.

Another thing that comes to my mind is that the pathreparam integrator produces biased gradients (see publication for more info on that). This bias might affect such a simple case.

One last thing: in the provided optimization examples using the pathreparam integrator, we usually constraint the optimization problem (e.g. optimize translation, displacement along normals,...) to avoid creating invalid meshes with overlapping triangles. It looks like you 3rd iteration is already having this issue.

HsiangYangChu · 2021-10-10T12:47:13Z

I have the same system configuration with @nauman-tintash, but I meet this error:

Traceback (most recent call last):
  File "sphere2cow.py", line 101, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

and my code is:

import torch
import enoki as ek
import mitsuba 
mitsuba.set_variant("gpu_autodiff_rgb")

from mitsuba.core import Thread, Bitmap, Struct, ScalarTransform4f
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, render_torch, Adam

sphere = """
<scene version='2.0.0'>
    <integrator type="path">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./sphere.obj"/>
        <bsdf type="conductor"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

cow = """
<scene version='2.0.0'>
    <integrator type="path">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./cow_mesh/cow.obj"/>
        <bsdf type="conductor"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

scene = load_string(cow)
image_ref = render(scene, spp=16)


scene = load_string(sphere)
camera = scene.sensors()[0]
crop_size = camera.film().crop_size()

params = traverse(scene)
print(params)
# varToOptimize = 'OBJMesh.vertex_positions_buf'
# ek.requires_gradient(params['OBJMesh.vertex_positions_buf'])
# params.set_dirty(varToOptimize)
params.keep(['OBJMesh.vertex_positions_buf'])
# params.update()
opt = Adam(params, lr=.02)

for it in range(100):
    # image = render(scene, optimizer=opt, unbiased=True, spp=8)
    image = render(scene, spp=2)
    write_bitmap('./output/out_%03i.png' % it, image, crop_size)
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
    ek.backward(ob_val)
    opt.step()

Speierers · 2021-10-11T06:42:54Z

You will most likely need to use the pathreparam integrator (here) in order to optimize geometry in Mitsuba 2. This integrator will account for visibility discontinuities when differentiating the light transport simulation. Maybe this will solve you problem.

HsiangYangChu · 2021-10-11T07:29:05Z

You will most likely need to use the pathreparam integrator (here) in order to optimize geometry in Mitsuba 2. This integrator will account for visibility discontinuities when differentiating the light transport simulation. Maybe this will solve you problem.
I've done what you said, but it's still:

Traceback (most recent call last):
  File "sphere2cow.py", line 104, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

This is my new code:

import torch
import enoki as ek
import mitsuba 
mitsuba.set_variant("gpu_autodiff_rgb")

from mitsuba.core import Thread, Bitmap, Struct, ScalarTransform4f
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, render_torch, Adam

sphere = """
<scene version='2.2.1'>
    <integrator type="pathreparam">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./sphere.obj"/>
        <bsdf type="diffuse"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

cow = """
<scene version='2.2.1'>
    <integrator type="path">
        <integer name="max_depth" value="2"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./cow_mesh/cow.obj"/>
        <bsdf type="conductor"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

scene = load_string(cow)
image_ref = render(scene, spp=2)


scene = load_string(sphere)
camera = scene.sensors()[0]
crop_size = camera.film().crop_size()

params = traverse(scene)
print(params)
# varToOptimize = 'OBJMesh.bsdf.reflectance.value'
# varToOptimize = ['OBJMesh.bsdf.specular_reflectance.value', 'OBJMesh.bsdf.eta.value', 'OBJMesh.bsdf.k.value']
varToOptimize = 'OBJMesh.vertex_positions_buf'
# ek.requires_gradient(varToOptimize)
# params.set_dirty(varToOptimize)
params.keep(varToOptimize)
# params[varToOptimize] = [.9, .9, .9]
# params.update()
opt = Adam(params, lr=.02)

for it in range(100):
    # image = render(scene, optimizer=opt, unbiased=True, spp=8)
    image = render(scene, spp=2)
    write_bitmap('./output/out_%03i.png' % it, image, crop_size)
    ob_val = ek.hsum(ek.sqr(image - image_ref)) / len(image)
    ek.backward(ob_val)
    opt.step()

Speierers · 2021-10-11T07:46:07Z

Does it works with this new code?

HsiangYangChu · 2021-10-11T07:47:40Z

Does it works with this new code?

No! :(

Traceback (most recent call last):
  File "sphere2cow.py", line 104, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

The log:

(mt) hsiangyangchu@zhuxiangyang-System-Product-Name:~/work/mitsuba-test/testgeo$ python sphere2cow_copy.py 
2021-10-11 15:53:43 INFO  main  [optix_api.cpp:56] Dynamic loading of the Optix library ..
2021-10-11 15:53:43 INFO  main  [xml.cpp:353] "<string>": in-memory version upgrade (v2.2.1 -> v2.1.0) ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/path.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/envmap.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/uniform.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/conductor.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/obj.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/hdrfilm.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/gaussian.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/independent.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/perspective.so" ..
2021-10-11 15:53:43 INFO  main  [Scene] Building scene in OptiX ..
2021-10-11 15:53:43 INFO  main  [xml.cpp:353] "<string>": in-memory version upgrade (v2.2.1 -> v2.1.0) ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/pathreparam.so" ..
2021-10-11 15:53:43 INFO  main  [PluginManager] Loading plugin "plugins/diffuse.so" ..
2021-10-11 15:53:43 INFO  main  [Scene] Building scene in OptiX ..
Traceback (most recent call last):
  File "sphere2cow_copy.py", line 97, in <module>
    ek.backward(ob_val)
RuntimeError: set_gradient(): no gradients are associated with this variable (a prior call to requires_gradient() is required.)

Speierers · 2021-10-11T07:59:08Z

It looks like the rendered image doesn't depend on the parameters that you are trying to optimize at all. Please check that the geometry OBJMesh is in sight.

Also you could use ek.foward() to produce a gradient image, this one shouldn't be black (see the tutorial in the doc).

HsiangYangChu · 2021-10-11T08:10:14Z

It looks like the rendered image doesn't depend on the parameters that you are trying to optimize at all. Please check that the geometry OBJMesh is in sight.

Also you could use ek.foward() to produce a gradient image, this one shouldn't be black (see the tutorial in the doc).

You mean the forward_diff.py in the doc?

Speierers · 2021-10-11T08:17:38Z

yes

HsiangYangChu · 2021-10-11T08:47:44Z

yes

I have a try, the result is:

(mt) hsiangyangchu@zhuxiangyang-System-Product-Name:~/work/mitsuba-test/testgeo$ python sphere2cow_forward.py 
2021-10-11 16:38:14 INFO  main  [optix_api.cpp:56] Dynamic loading of the Optix library ..
2021-10-11 16:38:14 INFO  main  [xml.cpp:353] "<string>": in-memory version upgrade (v2.2.1 -> v2.1.0) ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/pathreparam.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/envmap.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/uniform.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/diffuse.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/obj.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/hdrfilm.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/gaussian.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/independent.so" ..
2021-10-11 16:38:14 INFO  main  [PluginManager] Loading plugin "plugins/perspective.so" ..
2021-10-11 16:38:14 INFO  main  [Scene] Building scene in OptiX ..
Traceback (most recent call last):
  File "sphere2cow_forward.py", line 52, in <module>
    image = render(scene, spp=8)
  File "/home/hsiangyangchu/mitsuba2/build/dist/python/mitsuba/python/autodiff.py", line 203, in render
    image = _render_helper(scene, spp=spp, sensor_index=sensor_index)
  File "/home/hsiangyangchu/mitsuba2/build/dist/python/mitsuba/python/autodiff.py", line 43, in _render_helper
    spec, mask, aovs = scene.integrator().sample(scene, sampler, rays)
RuntimeError: cuda_malloc(): out of memory!

My code is:

import torch
import enoki as ek
import mitsuba 
mitsuba.set_variant("gpu_autodiff_rgb")

from mitsuba.core import Float
from mitsuba.core.xml import load_string
from mitsuba.python.util import traverse
from mitsuba.python.autodiff import render, write_bitmap, render_torch, Adam

sphere = """
<scene version='2.2.1'>
    <integrator type="pathreparam">
        <integer name="max_depth" value="3"/>
    </integrator>
    <emitter type="envmap" id="my_envmap">
        <string name="filename" value="museum.exr"/>
    </emitter>
    <shape type="obj">
        <string name="filename" value="./sphere.obj"/>
        <bsdf type="diffuse"/>
    </shape>
    <sensor type="perspective">
        <transform name="to_world">
            <lookat origin="5, 5, 5"
                    target="0, 0, 0"
                    up="0, 1, 0"/>
        </transform>

        <float name="fov" value="40"/>

        <film type="hdrfilm">
            <string name="pixel_format" value="rgb"/>
            <integer name="width" value="512"/>
            <integer name="height" value="512"/>
        </film>

        <sampler type="independent">
            <integer name="sample_count" value="10"/>
        </sampler>
    </sensor>
</scene>
"""

scene = load_string(sphere)

params = traverse(scene)

param_0 = params['OBJMesh.vertex_positions_buf']
ek.set_requires_gradient(param_0)

image = render(scene, spp=8)

ek.set_gradient(param_0, [1, 1, 1], backward=False)

Float.forward()

image_grad = ek.gradient(image)

crop_size = scene.sensors()[0].film().crop_size()
fname = 'out.png'
write_bitmap(fname, image_grad, crop_size)
print('Wrote forward differentiation image to: {}'.format(fname))

HsiangYangChu · 2021-10-11T10:03:12Z

yes

Could you provide me with an example of geometry optimization?

Speierers · 2021-10-11T10:58:01Z

Please take a look at this one

HsiangYangChu · 2021-10-11T12:42:20Z

Please take a look at this one

It seems that the folder scene_folder is lost.

binorchen · 2022-04-14T21:26:57Z

Hello @HsiangYangChu @Speierers ,

Have you solved this problem? I am suffering exactly the same problem.

Thank you!

Speierers closed this as completed Oct 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly #348

[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly #348

nauman-tintash commented Dec 4, 2020

Speierers commented Dec 7, 2020 •

edited by merlinND

Loading

HsiangYangChu commented Oct 10, 2021

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021 •

edited

Loading

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021 •

edited

Loading

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021 •

edited

Loading

binorchen commented Apr 14, 2022

[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly #348

[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly #348

Comments

nauman-tintash commented Dec 4, 2020

Summary

System configuration

Description

Reference Render:

Speierers commented Dec 7, 2020 • edited by merlinND Loading

HsiangYangChu commented Oct 10, 2021

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021 • edited Loading

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021 • edited Loading

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021

Speierers commented Oct 11, 2021

HsiangYangChu commented Oct 11, 2021 • edited Loading

binorchen commented Apr 14, 2022

Speierers commented Dec 7, 2020 •

edited by merlinND

Loading

HsiangYangChu commented Oct 11, 2021 •

edited

Loading

HsiangYangChu commented Oct 11, 2021 •

edited

Loading

HsiangYangChu commented Oct 11, 2021 •

edited

Loading