Optimizing PaliGemma to TFLite / TorchScript

[DarioPTWR]

Description of the feature request:

Hi, I was wondering if anyone has a conversion script or method that can be used to convert PaliGemma from HuggingFace Transformers to TFLite or TorchScript? Am trying to use it in mobile devices but currently can't seem to find a method that works 100% for this model conversion. Any help is greatly appreciated, thanks!

What problem are you trying to solve with this feature?

No response

Any other information you'd like to share?

No response

[haozha111]

hi @DarioPTWR ,

please take a look at the AI Edge Torch generative API : https://github.com/google-ai-edge/ai-edge-torch/tree/main/ai_edge_torch/generative

[DarioPTWR]

hi @haozha111 have taken a look and tried it on my own but seemed to face significant errors in converting the model's layers, are there any working solutions for PaliGemma available that i could reference / get more help in detail? also seems like the generative API now only has working examples for LLMs, will it work for VLMs as well? Thanks and greatly appreciate

[haozha111]

hi @DarioPTWR

Yes we are actively working on adding multi-modal support for Generative API, and we aim to provide an update on this topic in the near future, thanks!

[DarioPTWR]

hi @haozha111 thanks for your update! looking forward to the added support, but also want to ask if you are familiar with / aware of any other existing methods to efficiently run PaliGemma (or any VLM) using on-device inference in Android? preferably in TFLite format. thanks!

[yichunk]

Hi @DarioPTWR

We are actively working on the image modality support for VLM. At the same time, you can also stay tuned for AiCore from Android https://developer.android.com/ai/aicore, which may provide VLM integration in the near future as well.