Obsolete products appear as duplicates in CSV export

[CharlesNepote]

As of today, 2600+ products are duplicates in the CSV export. It appears these products are obsolete products. Eg. https://world.openfoodfacts.org/product/7614800709822/duo-florentin-kambly

Ideally:

• we should not export them twice
• we should export obsolete field in the CSV export

[stephanegigandet]

To reproduce in local dev environment:

• create a user, add this users to %admins in lib/ProductOpener/Config_off.pm
• edit a product page, mark the product as obsolete
• run the scripts/export_database.pl script

:~/openfoodfacts-server$ docker exec -it po_off-backend-1 bash
www-data@4b87a47b2b52:/opt/product-opener$ ./scripts/export_database.pl 
[Mon Mar 18 14:52:59 2024] XSLoader.pm: Name "FFI::Platypus::keep" used only once: possible typo at /usr/lib/x86_64-linux-gnu/perl-base/XSLoader.pm line 111.
[Mon Mar 18 14:52:59 2024] XSLoader.pm: Name "FFI::Platypus::TypeParser::ffi_type" used only once: possible typo at /usr/lib/x86_64-linux-gnu/perl-base/XSLoader.pm line 111.
Write file: /opt/product-opener/html/data/en.openfoodfacts.localhost.products.csv.temp
Write file: /opt/product-opener/html/data/en.openfoodfacts.localhost.products.rdf.temp
Write file: /opt/product-opener/html/data/fr.openfoodfacts.localhost.products.csv.temp
Write file: /opt/product-opener/html/data/fr.openfoodfacts.localhost.products.rdf.temp

[komalverma04]

I am looking to work on it.

[deveshidwivedi]

@komalverma04 Are you still working on this one?

[komalverma04]

@deveshidwivedi you can work on this. Thank you

[deveshidwivedi]

Hi @stephanegigandet I am unable to log in as an admin. I have tried using the username from the existing admins and also tried to test by creating another user and adding them as admin. How could i resolve this to further go ahead?

Also, how could files like /opt/product-opener/html/data/en.openfoodfacts.localhost.products.csv.temp be viewed?

[hangy]

Hi @stephanegigandet I am unable to log in as an admin. I have tried using the username from the existing admins and also tried to test by creating another user and adding them as admin. How could i resolve this to further go ahead?

Being an admin is actually not enough. You have to make yourself a moderator (for which you need to be an admin). Go to your account parameters (http://world.openfoodfacts.localhost/cgi/user.pl?type=edit, make yourself a moderator . I believe you should then be able to mark the product as obsolete:

Also, how could files like /opt/product-opener/html/data/en.openfoodfacts.localhost.products.csv.temp be viewed?

This should be available as /opt/product-opener/html/data/en.openfoodfacts.localhost.products.csv after a successful run of the script. The files are stored in the docker volume po_off_podata by default, so the best way to read the files might be

~/openfoodfacts-server$ docker compose exec -it backend bash
www-data@edee90c7de7a:/opt/product-opener$ cat /opt/product-opener/html/data/en.openfoodfacts.localhost.products.csv

or you could copy the file somewhere else, ie.

~/openfoodfacts-server$ docker compose cp backend:/opt/product-opener/html/data/en.openfoodfacts.localhost.products.csv /tmp

[deveshidwivedi]

Thank you very much! @hangy It works now!