Scene Semantic API is not correctly working when I hold the smartphone horizontally.

[ForJobOk]

I tried using Scene Semantic API when I hold the smartphone horizontally, but it didn't correctly work.

I changed Orientation at player settings, it didn't relevant to the issue.
I had my doubts about my implementation, but the same problem occurred with the samples you guys have published.

Sample : https://github.com/google-ar/demo-megagolf
Your sample movie : https://github.com/google-ar/arcore-unity-extensions/assets/41860455/1ee42134-7abe-40c3-8213-70599593e445

Of cource, I know Scene Semantic API don't support holding the smartphone horizontally.
However, I don't believe the behavior shown in the video is correct.

Version

Unity 2022.3.24f1
ARCore Extensions 1.42.0
AR Foundation 5.1.3

[15kingben]

This is the expected behavior, the semantics model dimensions are fixed to match the horizontal camera image aspect ratio (4:3 portrait). When the phone is held landscape, the semantics model is run on a portrait crop of the landscape camera image.

[ForJobOk]

@15kingben
Thank you for your reply.

Are you saying there is no way to solve this problem?

[15kingben]

Yes, currently using semantics horizontally requires the empty spaces on the sides.

[ForJobOk]

@15kingben
OK. Do you have a plan updating to use horizontal?

Or we need API to be notified when the empty space occurs on the sides.

One might assume that implementers should control that when the device is rotated, but that confuses us, because the empty space on the sides occurs even in portrait mode in Unity settings.

[ForJobOk]

@15kingben
What's your opinion?

[15kingben]

Hi Kento, I suppose an easy way to check if the semantic image was horizontal or vertical would be to check for unlabeled pixels, e.g. via getSemanticLabelFraction), the black bars on the side will be the unlabeled class. An update to better support horizontal is not currently being worked on

[ForJobOk]

Thank you.

But GetSemanticLabelFraction returns the percentage of unlabeled pixels, so it will also return for non-horizontal holdings. For example, pixels that could not be successfully segmented from the camera image would also return unlabeled, right?

If the library prohibits horizontal holding, it would be better if a mechanism exists to alert against horizontal holding.

[15kingben]

The only other instance where this occurs is due to the forward projection (#197 (comment)). However, after looking into it further, it seems like GetSemanticLabelFraction won't actually represent the number of unlabeled pixels correctly, so it would be required to inspect the pixels separately.

[ForJobOk]

It all makes sense now.

Then, we have to implement a function like GetSemanticLabelFraction by ourself.
I think that a mechanism is required to alert against horizontal holding if you don't allow it.