Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Added origin extraction #890

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
c65438d
Added origin extraction
ValentinRegnault Sep 8, 2022
e07cac9
Modifications to follow Semantic PRs
ValentinRegnault Sep 8, 2022
bf7dc9f
Removed unused import, and one useless statement
ValentinRegnault Sep 10, 2022
b77360d
Update robotoff/prediction/ocr/origin.py
ValentinRegnault Sep 13, 2022
eeb2c64
Update robotoff/prediction/ocr/origin.py
ValentinRegnault Sep 13, 2022
bb64958
Update robotoff/prediction/ocr/origin.py
ValentinRegnault Sep 13, 2022
6bb5767
Origin parser put in a class, with lazy initialization + small changes
ValentinRegnault Sep 14, 2022
668d2fc
Origin parser put in a class, with lazy initialization + small changes
ValentinRegnault Sep 14, 2022
31f07e5
repair last commit :/
ValentinRegnault Sep 14, 2022
ccba94c
small tweaks
ValentinRegnault Sep 14, 2022
c142504
Delete temp.txt
ValentinRegnault Sep 24, 2022
6d0af13
Delete temp2.txt
ValentinRegnault Sep 24, 2022
cc72baf
Small tweaks suggested by raphael0202
ValentinRegnault Sep 24, 2022
335e563
Revert .env
ValentinRegnault Sep 26, 2022
ee351bb
Added origin extraction
ValentinRegnault Oct 11, 2022
9806baa
remove useless comment
ValentinRegnault Oct 11, 2022
6228811
Merge branch 'master' of https://github.com/Pykorm/robotoff
ValentinRegnault Oct 11, 2022
64bf201
Revert "Merge branch 'master' of https://github.com/Pykorm/robotoff"
ValentinRegnault Oct 11, 2022
69c685d
Revert "remove useless comment"
ValentinRegnault Oct 11, 2022
8c48f12
Revert "Added origin extraction"
ValentinRegnault Oct 11, 2022
433ae15
Merge branch 'master' into master
ValentinRegnault Oct 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ PO_LOCAL_NET=po_default

# User uid - in dev align this with your own user uid / gid
# see https://gist.github.com/alexgarel/6e6158ee869d6db2192e0441fd58576e
# OFF_UID=1000
# OFF_GID=1000
OFF_UID=1000
OFF_GID=1000
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why have you introduced these changes?


# Robotoff
TAG=latest
Expand Down
10 changes: 5 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# nice way to have our .env in environment for use in makefile
# see https://lithic.tech/blog/2020-05/makefile-dot-env
# Note: this will mask environment variable as opposed to docker-compose priority
# Note: this will mask environment variable as opposed to docker compose priority
Copy link
Collaborator

@raphael0202 raphael0202 Sep 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why have you introduced these changes?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing, I made that because on my computer for unknown reason the "docker compose" don't exists, I have to use "docker-compose". This should be removed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, can you revert to the original version?

# yet most developper should'nt bump into this
ifneq (,$(wildcard ./.env))
-include .env
Expand All @@ -14,8 +14,8 @@ NAME = "robotoff"
ENV_FILE ?= .env
MOUNT_POINT ?= /mnt
HOSTS=127.0.0.1 robotoff.openfoodfacts.localhost
DOCKER_COMPOSE=docker-compose --env-file=${ENV_FILE}
DOCKER_COMPOSE_TEST=COMPOSE_PROJECT_NAME=robotoff_test PO_LOCAL_NET=po_test docker-compose --env-file=${ENV_FILE}
DOCKER_COMPOSE=docker compose --env-file=${ENV_FILE}
DOCKER_COMPOSE_TEST=COMPOSE_PROJECT_NAME=robotoff_test PO_LOCAL_NET=po_test docker compose --env-file=${ENV_FILE}

.DEFAULT_GOAL := dev
# avoid target corresponding to file names, to depends on them
Expand Down Expand Up @@ -50,7 +50,7 @@ edit_etc_hosts:
# Docker Compose #
#----------------#
up:
# creates a docker network and runs docker-compose
# creates a docker network and runs docker compose
@echo "🥫 Building and starting containers …"
docker network create po_default || true
${DOCKER_COMPOSE} up -d --build 2>&1
Expand All @@ -76,7 +76,7 @@ livecheck:
docker/docker-livecheck.sh

log:
@echo "🥫 Reading logs (docker-compose) …"
@echo "🥫 Reading logs (docker compose) …"
${DOCKER_COMPOSE} logs -f

#------------#
Expand Down
1 change: 1 addition & 0 deletions data/ocr/countries.json

Large diffs are not rendered by default.

200,748 changes: 200,747 additions & 1 deletion data/taxonomies/categories.full.json

Large diffs are not rendered by default.

307,005 changes: 307,004 additions & 1 deletion data/taxonomies/ingredients.full.json

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions doc/references/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,3 +163,19 @@ Generate spellcheck corrections. Either the barcode or the text to correct must
}
}
```

### Orgin prediction [/predict/origin]

#### Predict origin from OCR JSON [GET]

- ocr_url (str, required) - the url of the OCR JSON

- Response 200 (application/json)

```json
[{
"origin": "en:france",
"same_for_all_ingredients": False, //if true, all ingredients have the same origin
"concerned_ingredients": ["en:quinoa"] // if same_for_all_ingredients is True, this field is null
}]
```
Empty file modified docker-compose.yml
100644 → 100755
Empty file.
36 changes: 36 additions & 0 deletions robotoff/app/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,41 @@ def on_get(self, req: falcon.Request, resp: falcon.Response):
resp.media = {"nutrients": [p.to_dict() for p in predictions]}


class OriginPredictorResource:
def on_get(self, req: falcon.Request, resp: falcon.Response):
ocr_url = req.get_param("ocr_url", required=True)

if not ocr_url.endswith(".json"):
raise falcon.HTTPBadRequest("a JSON file is expected")

barcode = get_barcode_from_url(ocr_url)

if barcode is None:
raise falcon.HTTPBadRequest(f"invalid OCR URL: {ocr_url}")

try:
predictions = extract_ocr_predictions(
barcode, ocr_url, [PredictionType.origin]
alexgarel marked this conversation as resolved.
Show resolved Hide resolved
)

except requests.exceptions.RequestException:
resp.media = {
"error": "download_error",
"error_description": "an error occurred during OCR JSON download",
}
return

except OCRParsingException as e:
logger.error(e)
resp.media = {
"error": "invalid_ocr",
"error_description": "an error occurred during OCR parsing",
}
return

resp.media = {"origin": [p.to_dict() for p in predictions]}


class OCRInsightsPredictorResource:
def on_get(self, req: falcon.Request, resp: falcon.Response):
ocr_url = req.get_param("ocr_url", required=True)
Expand Down Expand Up @@ -1172,6 +1207,7 @@ def on_get(self, req: falcon.Request, resp: falcon.Response):
api.add_route("/api/v1/insights/annotate", AnnotateInsightResource())
api.add_route("/api/v1/predict/ingredients/spellcheck", IngredientSpellcheckResource())
api.add_route("/api/v1/predict/nutrient", NutrientPredictorResource())
api.add_route("/api/v1/predict/origin", OriginPredictorResource())
api.add_route("/api/v1/predict/ocr_insights", OCRInsightsPredictorResource())
api.add_route("/api/v1/predict/category", CategoryPredictorResource())
api.add_route("/api/v1/products/dataset", UpdateDatasetResource())
Expand Down
1 change: 1 addition & 0 deletions robotoff/insights/extraction.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
PredictionType.store,
PredictionType.packaging,
PredictionType.nutrient,
PredictionType.origin,
PredictionType.nutrient_mention,
PredictionType.image_lang,
PredictionType.image_orientation,
Expand Down
2 changes: 2 additions & 0 deletions robotoff/prediction/ocr/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from .label import find_labels
from .location import find_locations
from .nutrient import find_nutrient_mentions, find_nutrient_values
from .origin import origin_parser
from .packager_code import find_packager_codes
from .packaging import find_packaging
from .product_weight import find_product_weight
Expand All @@ -41,6 +42,7 @@
PredictionType.trace: find_traces,
PredictionType.nutrient: find_nutrient_values,
PredictionType.nutrient_mention: find_nutrient_mentions,
PredictionType.origin: origin_parser.find_origin,
PredictionType.brand: find_brands,
PredictionType.store: find_stores,
PredictionType.packaging: find_packaging,
Expand Down
Loading