-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scene Semantic API is not correctly working when I hold the smartphone horizontally. #204
Comments
This is the expected behavior, the semantics model dimensions are fixed to match the horizontal camera image aspect ratio (4:3 portrait). When the phone is held landscape, the semantics model is run on a portrait crop of the landscape camera image. |
@15kingben Are you saying there is no way to solve this problem? |
Yes, currently using semantics horizontally requires the empty spaces on the sides. |
@15kingben Or we need API to be notified when the empty space occurs on the sides. One might assume that implementers should control that when the device is rotated, but that confuses us, because the empty space on the sides occurs even in portrait mode in Unity settings. |
@15kingben |
Hi Kento, I suppose an easy way to check if the semantic image was horizontal or vertical would be to check for unlabeled pixels, e.g. via getSemanticLabelFraction), the black bars on the side will be the unlabeled class. An update to better support horizontal is not currently being worked on |
Thank you. But GetSemanticLabelFraction returns the percentage of unlabeled pixels, so it will also return for non-horizontal holdings. For example, pixels that could not be successfully segmented from the camera image would also return unlabeled, right? If the library prohibits horizontal holding, it would be better if a mechanism exists to alert against horizontal holding. |
The only other instance where this occurs is due to the forward projection (#197 (comment)). However, after looking into it further, it seems like GetSemanticLabelFraction won't actually represent the number of unlabeled pixels correctly, so it would be required to inspect the pixels separately. |
It all makes sense now. Then, we have to implement a function like GetSemanticLabelFraction by ourself. |
I tried using Scene Semantic API when I hold the smartphone horizontally, but it didn't correctly work.
I changed Orientation at player settings, it didn't relevant to the issue.
I had my doubts about my implementation, but the same problem occurred with the samples you guys have published.
Sample : https://github.com/google-ar/demo-megagolf
Your sample movie : https://github.com/google-ar/arcore-unity-extensions/assets/41860455/1ee42134-7abe-40c3-8213-70599593e445
Of cource, I know Scene Semantic API don't support holding the smartphone horizontally.
However, I don't believe the behavior shown in the video is correct.
Version
Unity 2022.3.24f1
ARCore Extensions 1.42.0
AR Foundation 5.1.3
The text was updated successfully, but these errors were encountered: