Tips for the optimization of envmap.data

[maxfrei750]

I'd like to optimize an envmap and have a few questions about best practices:

1. Did I overlook a tutorial example, where this is all already explained?
2. I understood that a random initialization of the envmap.data is preferred over a constant initialization (see here). Is that right?
3. The format of envmap.data is RGBE, correct?
4. If the format is RGBE, then what about clipping? Often it makes sense to clip variables to a range of [0,1]. IMHO, this still makes sense for the RGB part, but not for the E (exponent) part, correct? If so, then how would you go about clipping just part of the tensor?
5. What about envmap.scale, does it make sense to optimize it along with envmap.data or would that be redundant?
6. What about *_mono variants? If I checked it correctly, they still use RGBE for envmap.data. Is that right, and would it make sense to maneuver around this, to reduce the dimensionality of the optimization problem? If so, then how?
7. Are there any other considerations?

Quite a lot of questions, but I hope that the answers will also benefit others. Thanks!

[Speierers]

Hi @maxfrei750,

Those are good questions, some of them even deserve their own research project so I will do my best to give you some pointers:

1. There used to be a tutorial about this in Mitsuba 2. It shouldn't be too hard to reproduce this in Mitsuba 3 starting from there.
2. I would say that this will depend on the target illumination. But in theory, both initialization should eventually converge to the right solution.
3. The data format is RGBA
4. Environment lights often have regions with intensity over 1.0 so clipping might be dangerous in this case. Think about HDR vs LDR illumination. When evaluated, the alpha value in the data is discarded so its value shouldn't even matter here.
5. This question is related to the parameterization of your optimization problem. In some cases, it might help to give more handles to your optimizer so that it can better navigate the optimization landscape. Although my intuition would be that it isn't necessary to optimize the scale and data parameters jointly.
6. Yes, in *_mono variant the data format is still RGBA, but once again, the alpha channel will be dropped in the luminance evaluation.
7. I would recommend you try a coarse-to-fine approach, first optimizing a low resolution envmap, and increase its resolution progressively using the dr.upsample() function after a few iterations.

Good luck!

[maxfrei750]

That's some great input! Thanks for taking the time.

[maxfrei750]

Follow-up question: If the alpha channel is being discarded during evaluation, why is it there in the first place? For the optimization scenario, this would significantly reduce the number of parameters, or wouldn't it?

[Speierers]

Not quite sure why we keep the alpha channel here. We could try to get rid of it in a future release.

But note that if those aren't use in the envmap, then they will have a gradient of zero, hence it shouldn't impact the optimization process.

[maxfrei750]

Note: Apparently, for the "*_mono" variants, only the R channel is being optimized, which makes sense, but leads to funky looking textures, when plotted as RGB:

Optimization started with a noise, red pixels were increased, and blue pixels were reduced. Grayscale pixels remained unchanged.