Stokes vector output on perspective cameras

[fbergama]

I understand that the "Stokes integrator has to perform a last coordinate frame rotation to make sure the Stokes vector is saved in a consistent format, where its reference basis is aligned with the horizontal axis of the camera frame", as described in the documentation.

However, in this respect, I have 2 questions:

1. In a perspective camera, the sensor's reference frame is not orthogonal to the light ray (unless you consider the central pixel). So, is it implemented correctly in Mitsuba even for perspective cameras?
2. If so, why is such transformation implemented in that way? Can you cite a textbook where this is explained? Because Mueller calculus typically involves aligning reference frames parallel to each other or related by Fresnel reflection/refraction.

Thank you so much for your answers!

[tizian]

Here is the code that computes this final reference frame used for the rotation at the sensor:

mitsuba3/src/integrators/stokes.cpp

Line 99 in f47430d

/* The Stokes vector that comes from the integrator is still aligned

The motivation behind it is:

• As you mention, it needs to be orthogonal to the incident light ray. Anything else would be incorrect.
• It is also natural to rotate the frame s.t. its "main" horizontal axis is aligned with the image X-axis.
• This is also consistent with how ART saves its polarization output.

[fbergama]

I understand that it is "natural" to provide a new frame orthogonal to the light ray but with its main horizontal axis aligned with the image X axis.

But still, the general question is:

I have a Stokes vector S defined in a certain reference frame X,Y,Z. I want to represent it in a new frame X',Y',Z' where Z' is different from Z. Is it just a matter of computing B = Z x Y' and then rotate_stokes_basis(Z,X,B)*S ? Looks somehow arbitrary to me because it does not take into account Z'...
I would instead project Ex and Ey (defined in the ray reference frame) onto the camera reference frame, resulting in new vectors that are not orthogonal anymore because the camera is tilted with respect to the ray. The new Ex' and Ey' should then be expressed in the camera basis. In other words, the transformation should be a projection of Ex, Ey from X, Y , Z into X', Y', Z'

Please tell me if I'm completely wrong, I just feel that this step is simply not well discussed in any textbook...

Thank you so much for your time

[tizian]

Yeah I don't know of any textbook that talks about what exactly should happen at the sensor level.

In your example, is the Z-axis along the light ray direction? I don't think it is valid to construct a new frame with Z' != Z then. The Stokes vector really only has a meaning for one specific Z-axis, with the only degree of freedom being a rotation of the remaining X/Y axes around Z.

[fbergama]

In your example, is the Z-axis along the light ray direction?

yes, Z is the ray traveling direction, Z' is the camera optical axis.

I agree that theoretically speaking, it is not valid to construct a new frame with Z' != Z. Indeed, the camera cannot magically sense the Stokes vector, but it has to use linear polarizers for that. So, since the camera internally uses multiple polarizers at different angles (typically 0, 45°, 90°, 135°), would it be more appropriate to model what happens by means of tilted polarizers?

I'm not suggesting modifying the Mitsuba implementation, I just want to understand what's the best (physically reasonable) way to describe what Mitsuba does (which intuitively I think is correct).

[tizian]

FWIW, my interpretation of our Mitsuba implementation is simply that each pixel measures the incident Stokes vector in its own reference frame.

You're right though that a real camera would perform these measurements using a set of filters, e.g. a combination of linear polarizers and quarter-wave plates. Mitsuba can in principle also do this. You'd simply add these elements in front of the camera and then reconstruct Stokes vectors by only measuring the intensity "S0" that is independent of any reference frames.

I think what is stored in the image at the end is only a matter of convention. What you ultimately care about is the polarization state along each incident ray on the sensor, and this you should be able to recover from both of these approaches.

[fbergama]

As you suggested, we did a simple experiment with Mitsuba to simulate a "real" polarimetric camera. The same scene is rendered 4 times with different virtual polarizers in front of the lenses (with angles of 0, 45, 90, 135 degrees) to recover the Stokes vector with the classical formulas (eq. 1,2,3). Then, we compared the result against the "Stokes integrator" we mentioned in this discussion (see the attached code).

Differences are up to 6 degrees in the computation of the Angle of Linear Polarization (AOLP). This is expected, because the tilted polarizer model is only accounted if we explicitly simulate a rotating polarizer in front of the camera.

I'm surprised that nobody seems to care what happens on a polarimetric camera at a sensor level. For focal lengths that are not extremely high, the relative angle between rays and the image plane (on which the polarizers lie) is not negligible at all and has a macroscopic effect on what you measure. Since Mitsuba is intended also for inverse rendering, I think that it should be interesting to understand which model better describes what we might get with a real camera.

The problem is that I don't have yet experimental evidence on that. We want to try an LCD screen as a calibration target, but since LCD screen uses polarizer itself, is also affected by the tilted polarizer model...

mitsuba_tilted_polarizer_tests.zip

[GCChen97]

Hi,

Is there any progress? I notice this problem recently. To implement a virtual DoFP sensor, we need to know in which directions the light rays hit the sensor. But the directions are unknown.

[fbergama]

Hi,
Yes, I think that we've found a model describing what happens on real polarimetric cameras. Preliminary experiments seem to validate our idea. We plan to publish it in a near future, I'll keep you informed!

[GCChen97]

Wow, that will be interesting! I also discuss the problem in my paper https://arxiv.org/abs/2207.09629. Looking forward to your model.

[fbergama]

Super cool! Well... oh no now we have to compare against your method for our CVPR ;D

Actually, we are doing a very similar thing, testing with a glossy plastic board with some markers. I'll read your paper carefully. Thanks for sharing