How to inverse some scene params using pytorch?

[owaranainatsu]

Based on the tutorial, I want to use pytorch to implement it but I am confused about how to expose a scene parameter to the PyTorch optimizer in mitsuba3, like the codes shown here.

I tried that using the following codes .

scene = mi.load_file('./scenes/cbox.xml', res=256, integrator='prb')
image_ref = mi.render(scene, spp=512)

mi.util.convert_to_bitmap(image_ref)
@dr.wrap_ad(source='torch', target='drjit')
def render_torch(r, spp=256, seed=1):
    params[key] = mi.Color3f(r)
    params.update()
    return mi.render(scene, params, spp=spp, seed=seed, seed_grad=seed+1)

params = mi.traverse(scene)
key = 'red.reflectance.value'
param_ref = mi.Color3f(params[key])

inital_r = torch.tensor([[0.01, 0.2, 0.9]], dtype=torch.float32, requires_grad=True).to(device)
image_init = render_torch(inital_r)
mi.util.convert_to_bitmap(image_init)

And I got the error message:

Exception                                 Traceback (most recent call last)
File ~/anaconda3/envs/mitsuba3/lib/python3.9/site-packages/drjit/detail.py:189, in array_init(self, args)
    187 if self.IsJIT and getattr(t, "IsJIT", False) and \
    188    self.Depth == 1 and t.Depth == 1:
--> 189     raise _dr.Exception(
    190         "Refusing to do an extremely inefficient "
    191         "element-by-element array conversion from type %s "
    192         "to %s. Did you forget a cast or detach operation?"
    193         % (str(type(o)), str(type(self))))
    195 if isinstance(o[0], value_type) or self.IsMatrix:

Exception: Refusing to do an extremely inefficient element-by-element array conversion from type <class 'drjit.cuda.ad.TensorXf'> to <class 'drjit.cuda.ad.Float'>. Did you forget a cast or detach operation?

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
File ~/anaconda3/envs/mitsuba3/lib/python3.9/site-packages/drjit/detail.py:200, in array_init(self, args)
    199             for i in range(size):
--> 200                 self.set_entry_(i, value_type(o[i]))
    201 elif issubclass(t, (int, float)) and (not self.IsJIT or self.Depth > 1):

File ~/anaconda3/envs/mitsuba3/lib/python3.9/site-packages/drjit/detail.py:284, in array_init(self, args)
    283 if dynamic:
--> 284     raise TypeError("%s constructor expects: arbitrarily many values "
...
    291                     "PyTorch/TF/Jax array." % (type(self).__name__, "" if
    292                                                size == 1 else "1 or ", size,
    293                                                value_type.__name__)) from err

TypeError: Color3f constructor expects: 1 or 3 values of type "Float", a matching list/tuple, or a NumPy/PyTorch/TF/Jax array.

[Speierers]

In your case, you need to be more careful when converting the tensor argument of render_torch into a Color3f object. See this issue for more detailed information.

You will likely need to use something like this to make it work:

params[key] = dr.unravel(mi.Color3f, r.array)

[owaranainatsu]

Thank you very much!