Implementation Tutorial for Neural Radiosity [Hadadan et al. 2021] in Mitsuba 3

[ciy405x]

Hello folks,

For a recent project, I needed to implement a novel rendering technique called Neural Radiosity (SIGGRAPH 2021) using Mitsuba 3.

https://github.com/krafton-ai/neural-radiosity-tutorial-mitsuba3/blob/main/neural_radiosity.ipynb

In the Python notebook above, I provide a step-by-step guide to implementing NR, including sampling surface interactions uniformly on a mesh, integrating neural nets and Mitsuba 3, training with gradient descent, and writing code for various visualizations. These tasks are relevant not only to NR but also to various Mitsuba-PyTorch fusion projects. I believe other users will find it helpful as I utilized various methods from Mitsuba and Dr. Jit.

While Mitsuba is a fantastic rendering library, there are few open-source codes or tutorials available, which made implementation challenging. Fortunately, with the help of the fantastic Discussions section of this repo, as well as the dedication of programmers and researchers at EPFL, I was able to complete the code.

Recently, some Show-and-tell posts have been published by its users, and I would like to join in and help others. :)
I hope to see more Mitsuba 3 open-source projects in the future as well.

[DoeringChristian]

Hey, thank you so much for the tutorial! It already helped me a lot.
One note, you could set the shape pointer of the SurfaceInteraction i.e. si.shape = shape in the sample_si function.
This would allow you to query the BSDF later using the si.bsdf() function.

[ciy405x]

Bug fixed. Thanks very much for your bug finding ;) I will update the main repository in minutes.

TL;DR:
Change a part of render_rhs function:

si, β2, null_face = first_non_specular_or_null_si(scene, si, sampler)
ds = mi.DirectionSample3f(scene, si=si, ref=prev_si)
mis = mis_weight(
    prev_bsdf_pdf,
    scene.pdf_emitter_direction(prev_si, ds, ~prev_bsdf_delta),
)

as follow:

ds = mi.DirectionSample3f(scene, si=si, ref=prev_si)
mis = mis_weight(
    prev_bsdf_pdf,
    scene.pdf_emitter_direction(prev_si, ds, ~prev_bsdf_delta),
)
si, β2, null_face = first_non_specular_or_null_si(scene, si, sampler)
```.

[DoeringChristian]

Oh, that makes sense thanks for the quick fix.

[DoeringChristian]

Hi,
I tested your implementation further and everything seems to work. However, as far as I can tell you are only constructing M/2 RHS samples per LHS sample i.e. _r_sampler.seed(step, batch_size * M // 2). Is there a reason for this?

[ciy405x]

@DoeringChristian
It depends on the interpretation of the Neural Radiosity paper. In Section 4.7 of the paper, it states that BSDF and emitter sampling are used to sample the direction of the RHS (Right-Hand Side), and a total of M samples are taken. Therefore, BSDF and emitter sampling are performed M/2 times each to construct M direction vectors. However, even when using the next event estimation in MC (Monte Carlo) path tracing, it is common to denote it as 1 spp (sample per pixel). Similarly, it may still be referred to as M spp if BSDF and emitter sampling are performed M times each. If you prefer the latter interpretation, you can double the value of M. Of course, it would be even better if the author @saeedhd96 could provide a direct explanation.

[saeedhd96]

Hi, we have released the neural radiosity code: https://github.com/saeedhd96/neural-radiosity