Bug in reading / writing multichannel EXR in python?

[tatue64]

I rendered a multichannel image aov. For this image aov.split() works as expected. After saving the image with aov.write("test.exr") and afterwards reading it with aovx = mi.Bitmap("test.exr") the image is loaded without problems, but aovx.split() does no longer work but returns the error `RuntimeError: ​[StructConverter] Internal error: source and target alpha have mismatched names!'. Apparently, the image structure is changed in the process of writing and reading the image (see below). The different channels of the image can be read and displayed with the Tev app, and this suggests that the problem occurs during reading.

The original image has the following structure

Bitmap[
pixel_format = multichannel,
component_format = float32,
size = [768, 768],
srgb_gamma = 0,
struct = Struct<116>[
float32 X; // @0, premultiplied alpha, blend = <R * 0.412453 + G * 0.35758 + B * 0.180423>
float32 Y; // @4, premultiplied alpha, blend = <R * 0.212671 + G * 0.71516 + B * 0.072169>
float32 Z; // @8, premultiplied alpha, blend = <R * 0.019334 + G * 0.119193 + B * 0.950227>
float32 A; // @12, premultiplied alpha
float32 integrator.R; // @16, premultiplied alpha
float32 integrator.G; // @20, premultiplied alpha
float32 integrator.B; // @24, premultiplied alpha
float32 integrator.A; // @28, premultiplied alpha
float32 albedo.R; // @32, premultiplied alpha
float32 albedo.G; // @36, premultiplied alpha
float32 albedo.B; // @40, premultiplied alpha
float32 normals.X; // @44, premultiplied alpha
float32 normals.Y; // @48, premultiplied alpha
float32 normals.Z; // @52, premultiplied alpha
float32 uv.U; // @56, premultiplied alpha
float32 uv.V; // @60, premultiplied alpha
float32 shape.I; // @64, premultiplied alpha
float32 prim.I; // @68, premultiplied alpha
float32 dpu.X; // @72, premultiplied alpha
float32 dpu.Y; // @76, premultiplied alpha
float32 dpu.Z; // @80, premultiplied alpha
float32 dpv.X; // @84, premultiplied alpha
float32 dpv.Y; // @88, premultiplied alpha
float32 dpv.Z; // @92, premultiplied alpha
float32 dx.U; // @96, premultiplied alpha
float32 dx.V; // @100, premultiplied alpha
float32 dy.U; // @104, premultiplied alpha
float32 dy.V; // @108, premultiplied alpha
float32 depth.T; // @112, premultiplied alpha
],
data = [ 65.2 MiB of image data ]
]

whereas the structure after reading the saved file is

Bitmap[
pixel_format = multichannel,
component_format = float32,
size = [768, 768],
srgb_gamma = 0,
struct = Struct<116>[
float32 X; // @0, premultiplied alpha
float32 Y; // @4, premultiplied alpha
float32 Z; // @8, premultiplied alpha
float32 A; // @12, alpha
float32 albedo.R; // @16, premultiplied alpha
float32 albedo.G; // @20, premultiplied alpha
float32 albedo.B; // @24, premultiplied alpha
float32 depth.T; // @28, premultiplied alpha
float32 dpu.X; // @32, premultiplied alpha
float32 dpu.Y; // @36, premultiplied alpha
float32 dpu.Z; // @40, premultiplied alpha
float32 dpv.X; // @44, premultiplied alpha
float32 dpv.Y; // @48, premultiplied alpha
float32 dpv.Z; // @52, premultiplied alpha
float32 dx.U; // @56, premultiplied alpha
float32 dx.V; // @60, premultiplied alpha
float32 dy.U; // @64, premultiplied alpha
float32 dy.V; // @68, premultiplied alpha
float32 integrator.R; // @72, premultiplied alpha
float32 integrator.G; // @76, premultiplied alpha
float32 integrator.B; // @80, premultiplied alpha
float32 integrator.A; // @84, alpha
float32 normals.X; // @88, premultiplied alpha
float32 normals.Y; // @92, premultiplied alpha
float32 normals.Z; // @96, premultiplied alpha
float32 prim.I; // @100, premultiplied alpha
float32 shape.I; // @104, premultiplied alpha
float32 uv.U; // @108, premultiplied alpha
float32 uv.V; // @112, premultiplied alpha
],
metadata = {
generatedBy => "Mitsuba version 3.6.2",
pixelAspectRatio => 1,
screenWindowWidth => 1
},
data = [ 65.2 MiB of image data ]
]

Obviously, the first four channels are different and also Integrator A. Presumably, the change from 'premultiplied alpha' to 'alpha' leads to the error.

[njroussel]

Hi @tatue64

Could you provide us with a reproducer??
I've tried the following and it seems to work just fine:

import mitsuba as mi
mi.set_variant('cuda_ad_rgb')

scene = mi.load_dict(mi.cornell_box())
path_integrator = mi.load_dict({
    'type': 'path',
    'max_depth': 2
})
integrator = mi.load_dict({
    'type': 'aov',
    'aovs': 'p:position',
    'my_image': path_integrator

})

mi.render(scene, integrator=integrator)

bmp = scene.sensors()[0].film().bitmap()
out = bmp.split()
bmp.write("test.exr")

reloaded = mi.Bitmap('test.exr')
out = reloaded.split()

[tatue64]

yes. The problem occurs, when the pixel_format is set to xyza or rgba.

import drjit as dr
import mitsuba as mi

mi.set_variant('cuda_rgb')

cbox = mi.cornell_box()

film = {
    'type': 'hdrfilm',
    'pixel_format': 'rgba',
    'component_format': 'float32',
    'file_format': 'openexr',
    'filter': {
        'type': 'gaussian',
        'stddev': 0.5
    },
    'width': 256,
    'height': 256,
    'sample_border': True,
    'compensate': True
}

cbox['sensor']['film'] = film


scene = mi.load_dict(cbox)
path_integrator = mi.load_dict({
    'type': 'path',
    'max_depth': 2
})
integrator = mi.load_dict({
    'type': 'aov',
    'aovs': 'albedo:albedo,normals:sh_normal,uv:uv,shape:shape_index,prim:prim_index,dpu:dp_du,dpv:dp_dv,dx:duv_dx,dy:duv_dy,depth:depth',
    'my_image': path_integrator

})

mi.render(scene, integrator=integrator)

bmp = scene.sensors()[0].film().bitmap()
out = bmp.split()
bmp.write("test.exr")

reloaded = mi.Bitmap('test.exr')
out = reloaded.split()'

[ziyi-zhang]

I can reproduce this bug.

Here is the direct cause of the bug. The following reported result only uses depth aov for simplicity.
The original rendered multi-channel bitmap looks like:

Bitmap[
  pixel_format = multichannel,
  component_format = float32,
  size = [256, 256],
  srgb_gamma = 0,
  struct = Struct<36>[
    float32 R; // @0, premultiplied alpha
    float32 G; // @4, premultiplied alpha
    float32 B; // @8, premultiplied alpha
    float32 A; // @12, premultiplied alpha
    float32 my_image.R; // @16, premultiplied alpha
    float32 my_image.G; // @20, premultiplied alpha
    float32 my_image.B; // @24, premultiplied alpha
    float32 my_image.A; // @28, premultiplied alpha
    float32 depth.T; // @32, premultiplied alpha
  ],
  data = [ 2.25 MiB of image data ]
]

After write to exr and reloading, it becomes:

Bitmap[
  pixel_format = multichannel,
  component_format = float32,
  size = [256, 256],
  srgb_gamma = 0,
  struct = Struct<36>[
    float32 R; // @0, premultiplied alpha
    float32 G; // @4, premultiplied alpha
    float32 B; // @8, premultiplied alpha
    float32 A; // @12, alpha
    float32 depth.T; // @16, premultiplied alpha
    float32 my_image.R; // @20, premultiplied alpha
    float32 my_image.G; // @24, premultiplied alpha
    float32 my_image.B; // @28, premultiplied alpha
    float32 my_image.A; // @32, alpha
  ],
  metadata = {
    generatedBy => "Mitsuba version 3.6.3",
    pixelAspectRatio => 1,
    screenWindowWidth => 1
  },
  data = [ 2.25 MiB of image data ]
]

Later in the split() function, when splitting the channel group my_image, the code handling alpha channels does not support multiple alphas, which triggered the error.
I suppose that code is responsible for inconsistent alpha precomputation: e.g., convert source that has premultiplied alpha to target without premultiplied alpha. This is not needed in this example.
In which case we can defer that name assertion in L1448, when we are certain the above conversion is needed.

Aside: It's not clear to me why the multi-channel RGBA are all marked as premultiplied alpha. If we don't use AOV integrator, the alpha channel is labeled as "alpha".