Prediction on still images has worse performance on iOS than on Lobe desktop

[technoplato]

Hello, love the app, first of all. This is how training should be!

I did run into a snag: I've successfully trained the model to recognize screenshots from audible, youtube, and apple podcasts and that model works very well on desktop using the "Play" section.

However, when I export the model and use it in the iOS Bootstrap, it does very bad at recognizing any screenshot accurately.

All I've done with code is replaced LobeModel with SavedModel directly from Lobe.

Any suggestions?

Video demo link: https://www.youtube.com/watch?v=_rtKnlmsxzM

[technoplato]

Disregard.

There is a popup when you go to export your model in the desktop application that asks if you'd like to optimize your model.

I clicked optimize the second go round and everything works flawlessly now. Thanks!

[technoplato]

Sorry Lobe people for the back and forth, I’m going to reopen this as the performance on device, even with the optimized model is much less accurate than that on the desktop application.

Any tips would be appreciated.

[ellbosch]

Thanks @technoplato—we will be deploying changes as soon as this week to fix these issues. You are correct to note that optimizing your model will not resolve this issue :)

[technoplato]

Well that's great to hear! Glad I'm not crazy.

As a side note, is Object Detection getting close or is that still pretty far out in the timeline?

Thanks!

[ellbosch]

Object detection is a rather large release and a top priority, but we can't say more than that. We are starting an insiders ring to test early features with object detection, if you are interested in joining email lobeai@microsoft.com.

[technoplato]

Totally understand. Quick question: are all models expected to have poor performance or just the CreateML?

…

On Tue, Feb 9, 2021 at 7:03 PM Elliot Boschwitz ***@***.***> wrote: Object detection is a rather large release and a top priority, but we can't say more than that. We are starting an insiders ring to test early features with object detection, if you are interested in joining email ***@***.*** — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU2FGEXYFL5OVTILXYPW3LS6HELNANCNFSM4XJ4TGUQ> .

[ellbosch]

For the particular bug you posted, it's a bug specific to image processing. There's actually nothing wrong with the Core ML model—it should behave as you would expect. We're just plugging in a bad input to the model which creates the weird results you're seeing.

[technoplato]

Now I’m a little more confused. Is the bug with the starter code then? Could you elaborate on what you mean about image processing?

…

On Wed, Feb 10, 2021 at 12:29 AM Elliot Boschwitz ***@***.***> wrote: For the particular bug you posted, it's a bug specific to image processing. There's actually nothing wrong with the Core ML model—it should behave as you would expect. We're just plugging in a bad input to the model which creates the weird results you're seeing. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU2FGE36AJEG4JKEES3E7TS6IKTRANCNFSM4XJ4TGUQ> .

[ellbosch]

You are correct that it's a bug with the starter code. Here's a brief explanation on how iOS-bootstrap works:

1. Video capture: first, the iOS device's camera transmits video feed.
2. Image processing: a frame of the video feed needs formatting (with cropping and orientation adjustments) before we can predict.
3. Prediction: the processed image is assigned a prediction from the Core ML model's classifier.

Step 2 above is what's broken, and should be fixed this week.

I highly recommend our docs if you'd like to learn more! Please let me know if you have any other questions.

[technoplato]

Haven’t had a chance to dig too deep into the code yet but just wanted to make sure that (2) is expected even when an image is selected from the image library.

…

On Wed, Feb 10, 2021 at 11:22 AM Elliot Boschwitz ***@***.***> wrote: You are correct that it's a bug with the starter code. Here's a brief explanation on how iOS-bootstrap works: 1. Video capture: first, the iOS device's camera transmits video feed. 2. Image processing: a frame of the video feed needs formatting (with cropping and orientation adjustments) before we can predict. 3. Prediction: the processed image is assigned a prediction from the Core ML model's classifier. Step 2 above is what's broken, and should be fixed this week. I highly recommend our docs <https://docs.lobe.ai/docs/improving/improving> if you'd like to learn more! Please let me know if you have any other questions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU2FGAHP27FQ7ROYWUS37TS6KXFBANCNFSM4XJ4TGUQ> .

[ellbosch]

Good question! Yes, we also format images selected from device library before sending to prediction (i.e. we need images formatted into a square aspect ratio).