Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetch non-food product data from Open Product Facts & Open Beauty Facts #114

Closed
raphodn opened this issue Jan 1, 2024 · 7 comments · Fixed by #264
Closed

Fetch non-food product data from Open Product Facts & Open Beauty Facts #114

raphodn opened this issue Jan 1, 2024 · 7 comments · Fixed by #264
Labels
OxF Product data sync ✅ Task

Comments

@raphodn
Copy link
Member

raphodn commented Jan 1, 2024 •
edited
Loading

Story

Currently we consider that a new barcode is necessarily a food product, and we fetch its data from Open Food Facts.

What if a user scans a book ? or a boardgame ? or a deodorant ?

Solution

Sync with the OBF & OPF databases.
How to manage duplicates ? Or products that switch from one db to another ? Or are deleted ?

Extra info

Link Number of products csv size
OFF https://world.openfoodfacts.org 3M+ 855 MB
OBF https://world.openbeautyfacts.org 38k 78 MB
OPFF https://world.openpetfoodfacts.org 10k 22 MB
OPF https://world.openproductsfacts.org 17k 39 MB
Obsolete products (OFF) see issue #281 24k
@teolemon teolemon moved this to Backlog in 💸 Open Prices Jan 25, 2024
@raphodn raphodn added the 🙅‍♀️ Wontfix This will not be worked on label Jan 25, 2024
@raphodn
Copy link
Member Author

raphodn commented Jan 25, 2024 •
edited
Loading

Status : not a priority, decided to concentrate our efforts on food products, while the OPF & OBF databases are revamped in 2024

Nevertheless, users are regularly adding non-food prices. It could be a good starting point to nudge users to add new products in the corresponding database 😉

@raphodn raphodn mentioned this issue Feb 28, 2024
Closed
@raphodn
Copy link
Member Author

raphodn commented Mar 28, 2024

Opened an issue in the Python package to figure out how to fetch easily the dumps of the other databases

openfoodfacts/openfoodfacts-python#220

@raphodn raphodn removed the 🙅‍♀️ Wontfix This will not be worked on label Mar 28, 2024
@raphodn
Copy link
Member Author

raphodn commented Mar 29, 2024

@raphael0202 is it possible to get dumps from OBF & co ?

@teolemon
Copy link
Member

https://world.openbeautyfacts.org/data
https://world.openpetfoodfacts.org/data
https://world.openproductsfacts.org/data

They should have similar URLS, but we probably don't have fancy new stuff like JSONL

@teolemon
Copy link
Member

https://static.openbeautyfacts.org/data/openbeautyfacts-products.jsonl.gz
https://static.openpetfoodfacts.org/data/openpetfoodfacts-products.jsonl.gz
https://static.openproductsfacts.org/data/openproductsfacts-products.jsonl.gz

@raphodn
Copy link
Member Author

raphodn commented Mar 29, 2024 •
edited
Loading

Ok cool !

Maybe i'll try to improve the openfoodfacts-python package first ? It seems limited to only fetching the OFF data dumps.

I'll see if I manage to get something working.

Issue opened : openfoodfacts/openfoodfacts-python#220

@teolemon teolemon added the OxF label Mar 29, 2024
@raphodn
Copy link
Member Author

raphodn commented Apr 1, 2024 •
edited
Loading

Ok cool ! Maybe i'll try to improve the openfoodfacts-python package first ? It seems limited to only fetching the OFF data dumps.

PR opened ! here: openfoodfacts/openfoodfacts-python#223

@raphodn raphodn linked a pull request Apr 1, 2024 that will close this issue
feat: sync ALL products (obf, opff, opf) #264
Merged
@github-project-automation github-project-automation bot moved this from To do to Done in 📸 Open Products Facts Apr 3, 2024
@github-project-automation github-project-automation bot moved this from To do to Done in 🧴 Open Beauty Facts Apr 3, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in 💸 Open Prices Apr 3, 2024
@github-project-automation github-project-automation bot moved this from To do to Done in 🐾 Open Pet Food Facts Apr 3, 2024
@raphodn raphodn mentioned this issue Apr 3, 2024
Closed
4 tasks
@raphodn raphodn linked a pull request May 6, 2024 that will close this issue
refactor(sync): log extra info #288
Closed
@raphodn raphodn mentioned this issue Oct 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
OxF Product data sync ✅ Task
Projects
🐾 Open Pet Food Facts
Archived in project
💸 Open Prices
Status: Done
📸 Open Products Facts
Archived in project
🧴 Open Beauty Facts
Status: Done
Milestone
No milestone
2 participants
@teolemon @raphodn