Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate nutrition extraction in the edit mode #10950

Open
teolemon opened this issue Oct 29, 2024 · 0 comments
Open

Integrate nutrition extraction in the edit mode #10950

teolemon opened this issue Oct 29, 2024 · 0 comments

Comments

@teolemon
Copy link
Member

teolemon commented Oct 29, 2024

Problem

  • Integrate nutrition extraction in the edit mode
  • The nutrient extraction model was deployed and integrated to Robotoff. For every new image, we run the model on it and generate a prediction. An insight is generated if in the extracted nutrient values, at least one value is not present in the current nutrients.
  • To get the insights:

GET https://robotoff.openfoodfacts.org/api/v1/insights?insight_types=nutrient_extraction?barcode={BARCODE}

Nutrient values are in insight.data. It contains:

  • entities: a subset of the extracted entities at different processing steps (raw, aggregated, postprocessed). We only have postprocessed here, it's more useful as debug information. We can ignore this field here.

  • nutrients: a dictionary mapping nutrient name to a dict containing:

    • value: the value to add, without the unit
    • unit: the unit (can be g, mg, µg or null). If it's null, it's because we couldn't extract it from the image (either it's missing, the model was wrong or the OCR result was not good enough). In such case I think we can safely use the "default" unit, which depends on the nutrient (as it's done on Product Opener).
    • score: the entity score. Maybe not really relevant here, as this score is not calibrated (most values are > 0.98).
    • char_start, char_end: start and end character offsets in the original text
    • start, end: start and end word offset in the original text
  • We would go for an "Extract Nutrition Facts" button, very much like ingredients, only enabled if we have a precomputer nutrition insight already available (otherwise greyed out). That's a way to keep the user in charge, and avoid potential backlash on "unwanted/unhelpful help" from Robotoff.

  • Additional values provided by the model would be orange, unchanged values (also provided by the model) would stay normal.

  • In the future, We might nudge the user for a new photo if we deem it too old, but the result won't be instant (photo, bg task, inference, reloading). We might use animations to provide hints that new values arrived.

(part of this bug courtesy of @raphael0202 )

Part of

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: To discuss and validate
Development

No branches or pull requests

1 participant