assigning materials to faces from their index

[AndreaMaestri18]

Summary

I am trying to assign materials to the faces of my mesh, after exporting from Blender.
I can't find a way of assigning the materials directly in Mitsuba, or alternatively without having to find all the centers and then find the corresponding face in Blender again.

Description

As mentioned, from an algorithm that I created, I have a list of indexes of the faces and the corresponding materials:

{"wood": [1,2,3,4,6],
"concrete": [7,8,9,10,11],
...
}

Now what I would like to do here, is to assign all the materials to the faces. The materials are already present, as I created them in blender and exported them. The resulting files are 4 meshes (because I have 4 materials), and all of them consists of all the faces.

The problem that I am facing is that I couldn't find a way to assign materials directly in Mitsuba, or alternatively to find a map between the indexes of the faces in Mitsuba and in Blender. Therefore the only thing that I had in mind to do was to find in with Mitsuba all the centers, and then iterate over all the faces in blender and find the center that is closest to the value that mitsuba gives me.

This however gives me a big problem. If I have N faces, I would have to do $O(N^2)$ operations, which I would like to avoid since my mesh can be very big.

Do you have any clue on how I could approach this problem in Mitsuba directly?

[njroussel]

Hi @AndreaMaestri18

I'm confused by what "centers" are you talking about?

Also, wouldn't it be easier to just extract four partial meshes with your algorithm? You know the face indices, it should be fairly straightforward to create meshes that are the subset of triangle faces - you can most likely even just export them as disjoint triangles. At that point, you would no longer care about how the indices map from Blender/your algorithm to Mitsuba.

[AndreaMaestri18]

Hi @njroussel, thanks for the reply.

By centers I mean the geometrical centers of the faces.

"You know the face indices, it should be fairly straightforward to create meshes that are the subset of triangle faces - you can most likely even just export them as disjoint triangles." I find the face indices in Mitsuba, can I create separate meshes from those indices that I find there?

I couldn't find anything in the documentation.

[njroussel]

Yeah, that's possible. With mi.traverse() you can read the vertex-index triplet of a given face, and you can the read the vertex positions for that triplet too. You'll find some guidance in this guide: https://mitsuba.readthedocs.io/en/latest/src/how_to_guides/mesh_io_and_manipulation.html#Writing-a-mesh-to-disk

[AndreaMaestri18]

this is what i get if I do mi.traverse()

SceneParameters[
  ---------------------------------------------------------------------------------
  Name                          Flags    Type  Parent
  ---------------------------------------------------------------------------------
  bsdf.reflectance.value        ∂        Float UniformSpectrum
  silhouette_sampling_weight             float PLYMesh
  faces                                  UInt  PLYMesh
  vertex_positions              ∂, D     Float PLYMesh
  vertex_normals                ∂, D     Float PLYMesh
  vertex_texcoords              ∂        Float PLYMesh
  vertex_Attribute              ∂        Float PLYMesh
]

If I do mi.traverse(mesh)["faces"] I get an array of >2m elements, where some elements are repeated since my mesh has 600k faces in total. And if I check if one index face is inside (previously found with: idx = scene.ray_intersect(rays).prim_index) it says it is not there.

This also takes me to another problem:
the algorithm I wrote maps a mask to the 3D model, or at least it tries to do that. The problem that I am seeing is that is not exact 100% (attached picture of the 3D model with different materials)

Here you can see that the model is spotted, and where it is supposed to be completely white for example, some faces are coloured with another material even though the masks are not like this.

To do the mapping, I spawn a ray from the camera to the 3D model (like the snipped of code I wrote before), after adding all the camera parameters in my algorithm. It looks like it's working well for the majority of the points, but not for all of them. Do you have any ideas why? Is the result of scene.ray_intersect(rays) only the first hit or is the ray bouncing around? That's the only thing I could have thought of.

[AndreaMaestri18]

Update, the mapping works well, I was doing something wrong when creating the materials in blender... therefore from the camera poses I can identify successfully a point in the mesh from the picture.

Therefore now the problem is only on how to do this in an efficient way. @njroussel I tried like you said but the above snipped of code is what I get from doing like you said. Do you know how to proceed from there?

[AndreaMaestri18]

I'm sorry for all the comments and questions, but I am having another issue. I am trying to save the centers of the faces to a JSON file, starting from the index of a face. This is what I do:

import mitsuba as mi
import numpy as np
import json
mi.set_variant("llvm_ad_rgb")
mesh = mi.load_dict({
    "type": "ply",
    "filename": "../scenes/new_dji/meshes/mesh.ply",
})
# Read JSON file
with open("/home/ubuntu/geoloc-rf/2D_3D_mapping/mat_idx.json", 'r') as json_file:
    data = json.load(json_file)

breakpoint()
dic = {}
for material, set_of_indexes in data.items():
    if material == "car":
        centers = np.zeros(shape=(len(set_of_indexes), 3))
        vertices = mesh.face_indices(list(set_of_indexes)).numpy()
        for i, vert in enumerate(vertices):
            centers[i] = np.mean(mesh.vertex_position(vert).numpy(), axis=0)

        dic[material] = centers

    # Convert NumPy arrays to lists
dic_converted = {material: centers.tolist() for material, centers in dic.items()}

# Save the converted dictionary to a JSON file
with open('car.json', 'w') as json_file:
    json.dump(dic_converted, json_file, indent=4)

print("Data saved to material_centers.json")

But I get the error:
Segmentation fault (core dumped)

Is this the correct way of doing this operation?

[AndreaMaestri18]

Done, I'm still getting the same trace

[merlinND]

Have you checked that all of the values inside of set_of_indexes are legal face indices? I think the segfault is likely an out-of-bounds read.

[AndreaMaestri18]

I was just thinking that yesterday, but the indexes are coming from scene.ray_intersect(rays).prim_index. Can those be illegal indices? Also, are these only about the first hit on the mesh?

Processing in batch seems to work for example, so I'm tempted to say that it's not an out of bound problem

[merlinND]

I don't know what "processing in batch" means in this context.

Valid face indices have values 0 <= idx < mesh.face_count().
Valid vertex indices have values 0 <= idx < mesh.vertex_count().
You could add an assert to make sure that this holds. Also, step through the script line by line (or add prints and run the script with python -u script.py) to find out which line triggers the segfault.

[AndreaMaestri18]

It turned out you were right about the out of bound! Out of all the indices there were 5 that were out of bounds. Removing them fixed the problem. Sorry for the silly mistake I didn't think of checking :)