Implementation of the `pathreparam` integrator with OptiX 7 #157

Speierers · 2020-05-28T14:42:47Z

Continuation of PR #44.

abhinavvs · 2020-05-29T14:24:14Z

Does this mean that "pathreparam" plug-in still is not fully released yet (not even on OptiX 6)?

Any rough idea on when this plug-in will be available (and fully integrated with Mitsuba2)? I am really looking forward to this plug-in as I need this feature for my research project.

Thanks!

Speierers · 2020-06-02T07:54:45Z

The previous implementation based on "Optix 6" wasn't release, although it is publicly available here:
https://github.com/loubetg/mitsuba2

Co-authored-by: Sebastien Speierer <[email protected]>

realWDC · 2020-06-21T15:17:57Z

Great works!

Quick question:
How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?

I have seen following comments in this PR; Any hints on how to proceed?

 // TODO: make si differentiable w.r.t. shape parameters if necessary

Thanks!

Speierers · 2020-06-22T05:36:08Z

Great works!

Quick question:
How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?

I have seen following comments in this PR; Any hints on how to proceed?
 // TODO: make si differentiable w.r.t. shape parameters if necessary
Thanks!

This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins.
This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.

realWDC · 2020-06-28T22:27:47Z

src/shapes/disk.cpp

-        si_out[active] = si;
+    std::pair<Point3f, Normal3f> differentiable_position(const SurfaceInteraction3f &si,
+                                                         Mask /*active*/) const override {
+        auto [c, s] = sincos(si.uv.y() * math::TwoPi<Float>);


Should it be [s, c] instead of [c, s]?

auto [c, s] = sincos(si.uv.y() * math::TwoPi<Float>); // wrong? auto [s, c] = sincos(si.uv.y() * math::TwoPi<Float>); // correct?

realWDC · 2020-07-03T23:22:03Z

src/librender/mesh.cpp

+    MTS_MASK_ARGUMENT(active);
+
+    // Check whether the SurfaceInteraction need to be differentiable w.t.r. m_vertex_positions_buf
+    bool differentiable_pos = is_diff_array_v<Float> && requires_gradient(m_vertex_positions_buf);


To enable scalar_* variants compilation, how about changing from

bool differentiable_pos = is_diff_array_v<Float> && requires_gradient(m_vertex_positions_buf);

to

bool differentiable_pos = false; if constexpr (is_diff_array_v<Float>) { differentiable_pos = requires_gradient(m_vertex_positions_buf); }

wjakob

This is a a big PR, which makes this very difficult to review. Can we please split it up into smaller parts with the key parts?

Specifically:

differentiable intersections
other things needed by this plugin (smootharea, etc.)
the plugin itself

Since these depend on each other, we could do them one after the other instead of having 3 parallel PRs that would be difficult to keep in sync.

For the differentiable intersections, it will be essential to have testcases to make sure that the differentiation is working as expected. (Imagine how difficult it would be to investigate a poorly converging gradient descent, only to find that derivatives are not correctly propagated between mesh positions and derived shading normals.)

For the first part, it would also be good to discuss some of the design decisions that were made. For instance, how fill_intersection gets an intersection record as input and fills it with further information? I understand why this was done due to difficulties in modifying parameters via references in the GPU backends. But there are also downsides to this approach: the masking that this is now needed in all implementations of this method (a source of errors when a mask is forgotten somewhere?), along with the general costs of passing around this data structure (it’s the largest one we have in Mitsuba2). What I would suggest is that we develop an interface where the input SurfaceInteraction3f is not needed.

wjakob · 2020-07-04T09:16:56Z

docs/src/inverse_rendering/advanced.rst

+
+First, we define two helper functions that we will use to transform the mesh
+parameter buffers (flatten arrays) into ``VectorXf`` type (and the other way around).
+Note that those functions will be natively supported by ``enoki`` in a futur release.


wjakob · 2020-07-04T09:20:42Z

include/mitsuba/core/warp.h

@@ -554,9 +554,9 @@ MTS_INLINE Value square_to_beckmann_pdf(const Vector<Value, 3> &m,
 // =======================================================================

 /// Warp a uniformly distributed square sample to a von Mises Fisher distribution
-template <typename Value>
+template <typename Value, typename KappaValue = scalar_t<Value>>


Important: when you declare a template argument that is used as a function parameter, it is meaningless to specify a default type because it will always be overwritten by what the function is called with. In this change and the next few, can we just change KappaValue to Value directly?

wjakob · 2020-07-04T09:21:24Z

include/mitsuba/render/interaction.h

@@ -244,6 +244,37 @@ struct SurfaceInteraction : Interaction<Float_, Spectrum_> {
                          fmsub(a00, b1y, a01 * b0y) * inv_det);
    }

+    /// Fill with detailed information describing the intersection
+    void fill_surface_interaction(const Ray3f &ray,
+                                  const void *cache, // TODO should be Float* but can't because of ENOKI_STRUCT_SUPPORT (pointer to reference issue)


The issue mentioned here doesn't ring a bell, can we discuss it?

We can of course ignore if it is this something that won't play any role in the upcoming Enoki that doesn't allow Array<Float&, ... anymore.

wjakob · 2020-07-04T09:23:02Z

include/mitsuba/render/interaction.h

@@ -244,6 +244,37 @@ struct SurfaceInteraction : Interaction<Float_, Spectrum_> {
                          fmsub(a00, b1y, a01 * b0y) * inv_det);
    }

+    /// Fill with detailed information describing the intersection


It's not immediately clear to me why this interface is needed, so some more documentation would be good.

This appears to just be copying a SurfaceInteraction, why not do an assignment at the caller?

wjakob · 2020-07-04T09:24:51Z

include/mitsuba/render/scene.h

+ * \brief Specifies the surface interaction computation mode when tracing rays
+ */
+enum class HitComputeMode : uint32_t {
+    /// Let the tracer engine (Embree, Optix) compute the surface interaction


tracer engine -> ray tracing engine

wjakob · 2020-07-04T09:25:40Z

include/mitsuba/render/scene.h

+    Default = 0,
+
+    /// Compute a differential surface interaction if shape parameters require gradients. This
+    /// mode will disable the computation by the tracer engine.


tracer engine -> ray tracing engine. Is this comment accurate? IIRC we are still doing most of the ray tracing work with OptiX/Embree, it's just the final intersection record that is computed differently.

wjakob · 2020-07-04T09:26:24Z

include/mitsuba/render/scene.h

+    Differentiable = 1,
+
+    /// Only compute si.t, si.p, si.uv (barycentric), si.shape, and si.prim_index
+    Least = 2


Is this also differentiable? Perhaps Minimal instead of Least (which would be an appropriate naming scheme if there was also Most or something like that.)?

wjakob · 2020-07-04T09:27:16Z

src/emitters/smootharea.cpp

+     at the borders of the area light, in uv space. (Default: 0.1)
+
+This plugin implements an area light with a smooth transition from full emission
+to zero (black) at its borders. This type of light is usefull for differentiable


usefull->useful

wjakob · 2020-07-04T09:28:30Z

src/emitters/smootharea.cpp

+
+    Float smooth_profile(Float x) const {
+        Float res(0);
+        res = select(x >= m_blur_size && x <= Float(1) - m_blur_size, Float(1), res);


Could this be expressed with 1-2 max()/min calls?

priyamvad · 2020-09-21T21:13:52Z

Hello, any movement on this PR? I'd love to test out the ability to derive a displacement map to match the reference render, something it sounds like this feature should help with?

Speierers · 2020-09-22T06:43:36Z

Hello, any movement on this PR? I'd love to test out the ability to derive a displacement map to match the reference render, something it sounds like this feature should help with?

Hi @priyamvad,
We are currently busy working on a major refactoring of the Mitsuba 2 + Enoki codebase.

You should be able to use this branch as it is for you experiments. Have a look at the heightfield optimization tutorial to get started.

RiverIntheSky · 2021-01-13T13:44:51Z

Great works!
Quick question:
How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?
I have seen following comments in this PR; Any hints on how to proceed?
 // TODO: make si differentiable w.r.t. shape parameters if necessary
Thanks!
This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins.
This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.

Excuse me, is there any update on differentiable to_world? Is this pathreparam branch necessary for this goal compared to master branch? If I'm going to proceed in this direction myself, where should I make changes? Currently I can't pass Transform4f to optix::Transform4f

Speierers · 2021-01-13T14:03:42Z

Great works!
Quick question:
How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?
I have seen following comments in this PR; Any hints on how to proceed?
 // TODO: make si differentiable w.r.t. shape parameters if necessary
Thanks!
This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins.
This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.
Excuse me, is there any update on differentiable to_world? Is this pathreparam branch necessary for this goal compared to master branch? If I'm going to proceed in this direction myself, where should I make changes? Currently I can't pass Transform4f to optix::Transform4f

We are busy working on a major refactoring and haven't made any progress on this unfortunately.

Differentiable to_world will for sure introduce discontinuities in the rendering equation, therefore using pathreparam will be necessary to account for those.

You will need to make changes to the shape class and the shape plugins so that there traverse() and parameters_changed() method expose and handle changes in m_to_world.

To pass the transform to optix, you will need to evaluate it and reduce it to a scalar value (e.g. using enoki::hsum which won't do much as every entry of that matrix is a single value).

Good luck!

RiverIntheSky · 2021-01-18T16:32:28Z

Great works!
Quick question:
How to auto-diff w.r.t. shape parameters, for example m_radius in src/shapes/sphere.cpp?
I have seen following comments in this PR; Any hints on how to proceed?
 // TODO: make si differentiable w.r.t. shape parameters if necessary
Thanks!
This is not supported at the moment. For this to work, Shape::m_to_world would need to become a differentiable type (e.g. Transform<Point<CUDAArray<float>, 4>> which involves a lot of changes accross the different shape plugins.
This will definitively be part of a future PR, but I won't be able to give any ETA at the moment.
Excuse me, is there any update on differentiable to_world? Is this pathreparam branch necessary for this goal compared to master branch? If I'm going to proceed in this direction myself, where should I make changes? Currently I can't pass Transform4f to optix::Transform4f
We are busy working on a major refactoring and haven't made any progress on this unfortunately.

Differentiable to_world will for sure introduce discontinuities in the rendering equation, therefore using pathreparam will be necessary to account for those.

You will need to make changes to the shape class and the shape plugins so that there traverse() and parameters_changed() method expose and handle changes in m_to_world.

To pass the transform to optix, you will need to evaluate it and reduce it to a scalar value (e.g. using enoki::hsum which won't do much as every entry of that matrix is a single value).

Good luck!

I am not sure if I understand enoki::hsum correctly. As I read in Enoki Reference, value_t<Array> hsum(Array value), so hsum(Float m_radius) should return value_t<Float>, which I tested is float (my compiler says 'value_t<enoki::DiffArray<CUDAArray<float>>>' (aka 'float'). But as I tried to assign it to float it failed:

error: no viable conversion from 'enoki::DiffArray<CUDAArray<float>>' to 'float'
        float radius = hsum(m_radius);
              ^        ~~~~~~~~~~~~~~

Why is this? It looks like hsum(Float m_radius) still returns Float.

Speierers · 2021-01-20T06:45:31Z

Hi @RiverIntheSky ,

Could you please open a seperate issue for this? Could you try something like hsum(detach(m_radius)) maybe?

lo-zed · 2021-02-03T14:33:27Z

Dear developers,
First of all thank you for your great tool and all the work done.
If I understand correctly differentiation of weights for blended BSDFs as well as mesh positions depend on this PR? Or was it already included as part of another refactoring mentioned above?
Do you have an idea if work on this PR will start again in the near future?
Thank you very much for your time.

Speierers · 2021-02-03T15:09:10Z

Hi @Idiot-z ,

This PR will stay as it is for now. The refactored codebase will be very different, requiring a complete re-write of this plugin to fully leverage the various new features of the system. I can't say for now whether it will included or not "natively" in the upcoming release.

lo-zed · 2021-02-03T15:21:11Z

Hi @Speierers
Thank you for your prompt response. Just to be sure, is differentiation of weights for blended bsdf supposed to be functional in the current release (fe2a1bd)? Because there is a star next to the corresponding parameter, however I get very strange results.

Speierers · 2021-02-04T08:11:17Z

Hi @Idiot-z ,
Please open a seperate issue for this as it is completely unrelated to this PR ;)

mehrab2603 · 2021-03-15T14:54:40Z

Is there any plan of merging this with master? I need some features from master (projector emitter) that were implemented after this. Tried merging myself but got so many conflicts that I don't know how to fix.

Speierers · 2021-03-16T09:27:46Z

@mehrab2603 No we are not going to merge this with master as we are about to release our refactored/redesigned version of the codebase which will render this PR completely out of sync.

In the future, we are planning on implementing integrators that can deal with discontinuities. But this won't be part of the upcoming release unfortunately.

Speierers mentioned this pull request May 28, 2020

Implementation of the pathreparam integrator #44

Closed

8 tasks

Speierers force-pushed the pathreparam-optix7 branch from 2bbb590 to 043b0b6 Compare June 2, 2020 08:02

loubetg and others added 3 commits June 2, 2020 10:06

Add pathreparam and smootharea emitter

2b8655e

Co-authored-by: Sebastien Speierer <[email protected]>

Add invert pose tutorial

4df3343

Update data submodule

175a8f4

Speierers force-pushed the pathreparam-optix7 branch from 043b0b6 to 175a8f4 Compare June 2, 2020 14:39

Speierers added 6 commits June 3, 2020 09:10

Fix MSVC warnings

6395ab4

Check for sufficient GPU memory in tests

21a5bcc

Add cuda sync and params.set_dirty()

67b2adb

Update test_pathreparam using set_dirty

3e85a0c

Mesh::fill_surface_interaction checks whether gradients are needed

2d3b2a8

Fix pathreparams for MSVC

70af33f

Speierers requested a review from wjakob June 12, 2020 10:01

arpit15 mentioned this pull request Jun 19, 2020

Geometric differentiation cannot optimize vertex positions at all. #182

Closed

realWDC reviewed Jun 28, 2020

View reviewed changes

realWDC reviewed Jul 3, 2020

View reviewed changes

wjakob reviewed Jul 4, 2020

View reviewed changes

Speierers marked this pull request as draft July 10, 2020 07:29

Speierers mentioned this pull request Jul 13, 2020

Differentiable Surface Interaction #209

Merged

6 tasks

aluo-x mentioned this pull request Sep 2, 2020

Will Pytorch3D support the ray-tracing method? facebookresearch/pytorch3d#340

Open

lo-zed mentioned this pull request Feb 4, 2021

[🐛 bug report] weight differentiation for blended BSDFs dysfunctional #381

Open

Speierers mentioned this pull request Jul 7, 2021

[❔ other question] Are the examples from reparameterizing paper available? #464

Closed

Speierers mentioned this pull request Oct 11, 2021

[🐛 bug report] Vertex Positions Loss Backpropagating Incorrectly #348

Closed

Speierers mentioned this pull request Jan 17, 2022

[❔ other question] How to split "aov" rendering while remaining differentiable? #559

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of the `pathreparam` integrator with OptiX 7 #157

Implementation of the `pathreparam` integrator with OptiX 7 #157

Speierers commented May 28, 2020

abhinavvs commented May 29, 2020

Speierers commented Jun 2, 2020

realWDC commented Jun 21, 2020

Speierers commented Jun 22, 2020

realWDC Jun 28, 2020

realWDC Jul 3, 2020

wjakob left a comment •

edited

Loading

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

wjakob Jul 4, 2020

priyamvad commented Sep 21, 2020

Speierers commented Sep 22, 2020

RiverIntheSky commented Jan 13, 2021

Speierers commented Jan 13, 2021

RiverIntheSky commented Jan 18, 2021

Speierers commented Jan 20, 2021

lo-zed commented Feb 3, 2021

Speierers commented Feb 3, 2021

lo-zed commented Feb 3, 2021

Speierers commented Feb 4, 2021

mehrab2603 commented Mar 15, 2021

Speierers commented Mar 16, 2021

Implementation of the pathreparam integrator with OptiX 7 #157

Are you sure you want to change the base?

Implementation of the pathreparam integrator with OptiX 7 #157

Conversation

Speierers commented May 28, 2020

abhinavvs commented May 29, 2020

Speierers commented Jun 2, 2020

realWDC commented Jun 21, 2020

Speierers commented Jun 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjakob left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priyamvad commented Sep 21, 2020

Speierers commented Sep 22, 2020

RiverIntheSky commented Jan 13, 2021

Speierers commented Jan 13, 2021

RiverIntheSky commented Jan 18, 2021

Speierers commented Jan 20, 2021

lo-zed commented Feb 3, 2021

Speierers commented Feb 3, 2021

lo-zed commented Feb 3, 2021

Speierers commented Feb 4, 2021

mehrab2603 commented Mar 15, 2021

Speierers commented Mar 16, 2021

Implementation of the `pathreparam` integrator with OptiX 7 #157

Implementation of the `pathreparam` integrator with OptiX 7 #157

wjakob left a comment •

edited

Loading