Supercharge the app with artificial intelligence (Tracker)

[teolemon]

What

Tasks

Beta Give feedback

1. Simplify the basic details input by using Robotoff predictions #3279
   Product addition ✏️ Editing ✨ enhancement 🎨 Mockups available 🎯 P1 🤖 Robotoff +6
2. We can add Robotoff questions filtered on labels in the labels editing section #3986
   ✏️ Editing 🎯 P1 🤖 Robotoff +3
   
3. We can add Robotoff questions filtered on category in the category editing section #3985
   ✏️ Editing 🎯 P1 🤖 Robotoff +3
4. We can add Robotoff questions filtered on brands in the basic details editing section #4069
   ✏️ Editing ✏️ Editing - Basic info input 🎯 P1 +3
5. Inline Robotoff questions to increase the answer rate #3326
   🎯 P0 🤖 Robotoff +2
   
6. Instrument Robotoff answers in Matomo #3775
   📈 Analytics 📈 Matomo +2
   
7. Offer in-line validation of Robotoff predictions for nutrition facts #3278
   ✏️ Editing - Nutrition input 🎨 Mockups available 🎯 P1 🤖 Robotoff 🤖 blocked on Robotoff +5
8. Random Robotoff insight UX suggestions #3340
   Hunger Games ✨ enhancement 🤖 Robotoff +3
   
9. Ensure Robotoff questions works offline (and local answers are dropped in the right cases after reconnection) #2457
   Offline Offline - Contribution 🤖 Robotoff +3
10. Add optional offline ingredient extraction using MLKIT (disabled on the F-Droid build)
11. Add offline packaging extraction using MLKIT (disabled on the F-Droid build)
12. Explore the feasability of bringing OCR analysis done by Robotoff offline #5654
    On-device ML 🤳 MLKit +2
13. Explore the feasability of bringing nutrition analysis done by Robotoff offline #5652
    On-device ML +1
14. Explore the feasability of bringing category prediction done by Robotoff offline #5653
    On-device ML +1
15. Explore segmentation of photos on device #5007
    ⭐ top issue On-device ML +2
16. Offer on-device translation of products using MLKit #5655
    On-device ML 🤳 MLKit +2
17. Use automatic language recognition to suggest additional languages to add, or suggest correcting product language before sending #5006
    On-device ML +1
18. Bring minimum on-device moderation using MLKit #5008
    On-device ML +1
19. Add custom fields to the nutrition input form based on the category (eg for waters, chocolate, alcohol) #5401
    ✏️ Editing - Nutrition input +1
20. Attempt getting Robotoff predictions during product addition #5402
    Product addition 🤖 Robotoff +2
21. Add an optional button on the product page to jump to Hunger Games crops and logos view for the product
22. Use on-device classification to detect the type of product (Food, Cosmetic, PetFood, Products, or more granular) and show a warning #5656
    On-device ML OxF 🐾 Open Pet Food Facts 📸 Open Products Facts 🧴 Open Beauty Facts +5
23. Automatic warning when adding an old receipt with today's date #5387
    On-device ML Prices 🤳 MLKit +3
24. Towards a try harder scan mode ? #5322
    On-device ML 🤳 MLKit +2
25. Create a custom Tensorflow model for MLKit to classify Nutrition facts, Packaging instruction, Front and Ingredients photo on device #5010
    On-device ML 🤳 MLKit +2
26. Use the DUMP API to make an offline Hunger Games ? https://openfoodfacts.github.io/robotoff/references/api/#tag/Insights/paths/~1insights~1dump/get

Part of

• 🎯 What can I work on ? #525

[teolemon]

@raphael0202 I added the idea to do on-device language recognition here

[teolemon]

What

Create a custom Tensorflow model for MLKit to classify Nutrition facts, Packaging instruction, Front and Ingredients photo on device
Create a custom Tensorflow model for MLKit to classify food, pet food, cosmetics and other products on device

How

https://developers.google.com/ml-kit/vision/image-labeling?hl=fr
https://developers.google.com/ml-kit/vision/face-detection?hl=fr

[teolemon]

Explore using the on-device LLM for various tasks

• Food intake journaling
• Ingredient analysis
• FAQ/Support (chat?)
• other use-cases TBD

How

• https://ai.google/build/project-guides/llm-on-android-with-keras-and-tensorflow-lite
• https://android-developers.googleblog.com/2023/12/leverage-generative-ai-in-your-android-apps.html
• https://developer.android.com/studio/preview/gemini-template