feat: Batch job - Spellcheck

[jeremyarancio]

Objective of the PR:

A new model based on Mistral-7B LLM has been developed to tackle the OCR-parsed ingredients spellcheck problem (see #spellcheck). While this solution doesn't fit Open Food Facts' production infrastructure, we've created a GPU-powered batch inference pipeline on GCP to process thousands of products.

See demo: Ingredient Spellcheck

What

• Set up Batch Job with GCP for Robotoff Tasks.
• We start with Ingredients-Spellcheck

Part of

• Spellcheck work (link)
• Discussion Robotoff (link)

[jeremyarancio]

Rest to do:

• Transfert batch launch from api endpoint to CLI
• Set up API security key for Batch Data import, triggered by GCP batch job once finished
• Improve InsightsImport to consider several use cases (see Robotoff doc)

[raphael0202]

Not needed anymore (as we're using an envvar)

[raphael0202]

Not needed anymore (as we're using an envvar)

[raphael0202]

Not needed anymore (as we're using an envvar)

[raphael0202]

Possibly more than 1, no?

[jeremyarancio]

There's only one correction, no?
Or maybe I didn't understand what it means

[raphael0202]

There can be one correction per language, so more than 1 correction for each product

[jeremyarancio]

Black fails on os.getenv("BATCH_JOB_KEY")}, which is normal since it is not added to Github secret
This job key is used by the import_batch endpoint to secure it. Only the batch job docker image owns it.
We will have to add a key to Github secrets for the API, but also during the deployment of the image in GCP (CI/CD to make things easier?)

[jeremyarancio]

Quick note about error handling:
Falcon doesn't seem to handle custom exception error (such APITokenError(Exception)) and return status 500 instead by default

[jeremyarancio]

I implemented the following changes:

• Add make deploy-spellcheck in the Makefile
• Modified the query to extract batch based on popularity_key
• Delve into Google credentials to remove the credentilal.json folder necessity. We'll use a service_account instead (Need to generate it for production with the correct roles: incoming)
• Added environment key to the batch job
• Update Importer (still a bug, needs to solve it)
• The overall process is tested, and everything looks fine. (Process 10000 products in 20 minutes)

[jeremyarancio]

Cannot explain why test fail with DeepSource, and not for other identical cases

[raphael0202]

Deepsource generates a lot of false positive. I thought about disabling it to be honest.

[jeremyarancio]

Improvement ideas:

• Generate an unique id for each data processed in the bucket (reason: enable debugging)

[raphael0202]

Good job! 🎉