feat: Use openfoodfacts-query service for facet queries instead of product_tags collection

[john-gom]

• Include unit tests for new functionality
• If you have multiple commits please combine them into one commit by squashing them.

What

Use the Postgres-based query service for counts and aggregations. Updated the product_tags refresh scripts to refresh this database instead

Related issue(s) and discussion

• Fixes Move products_tags to Postgres to improve performance #8676

[alexgarel]

LGTM.

Kudos @john-gom (for the off-query part firstly !).

[alexgarel]

there again, I think it's wiser to save $@ just after the eval block, to avoid a future mistake.

[alexgarel]

Is this query returning quickly ? (maybe it should)

[john-gom]

This does wait until the refresh has been completed before returning at the moment. It will be quick if it is an incremental update, but takes about 1.5 hours to do a full load.

[stephanegigandet]

http://world.openfoodfacts.localhost/country/united-kingdom/misc does work though

[john-gom]

I had to do a small change to run it in a dev environment in Linux:

host.docker.internal is not available in Linux by default (only Windows and Mac), but it could be added like this: (it may have unwanted side effects)

+++ b/docker/dev.yml
@@ -20,6 +20,10 @@ x-backend-conf: &backend-conf
     - ./docker/docker-entrypoint.sh:/docker-entrypoint.sh
     # we use this for debugging, from times to times
     - ./debug:/mnt/podata/debug/
+  # Allow the container to connect to the host
+  # this is needed to access the dev openfoodfacts-query running on the host
+  extra_hosts:
+    - "host.docker.internal:host-gateway"

Hi @stephanegigandet , just to clarify, you should only need this kind of configuration if something running inside docker needs to access a service running on the host. So this would either be the query service running in docker trying to access MongoDB running on the host (not in Docker) or Product Opener running in Docker trying to access the Query service running on the host (not in Docker). If the Query service is running in Docker and is accessing MongoDB running in docker then it should be able to access the MongoDB database using the "mongodb" host name as the Query service joins the po_default network. What scenario were you trying?

[john-gom]

For queries that include a country filter, the country filter is not taken into account.

e.g. http://uk.openfoodfacts.localhost/label/fair-trade/misc

[Nest] 1  - 09/14/2023, 8:46:11 AM   DEBUG [QueryService] Object:
[
  {
    "$match": {
      "labels_tags": "en:fair-trade",
      "countries_tags": "en:united-kingdom"
    }
  },
  {
    "$unwind": "$misc_tags"
  },
  {
    "$group": {
      "_id": "$misc_tags",
      "count": {
        "$sum": 1
      }
    }
  },
  {
    "$sort": {
      "count": -1
    }
  },
  {
    "$skip": 0
  },
  {
    "$limit": 10000
  }
]

[Nest] 1  - 09/14/2023, 8:46:11 AM   DEBUG [QueryService] select value _id, count(*) count from "query"."product_misc_tag" as "pt" where (not pt.obsolete) and (EXISTS (select "pt2".* from "query"."product_labels_tag" as "pt2" where (pt2.product_id = pt.product_id and pt2.value = 'en:fair-trade'))) group by "pt"."value" order by 2 desc limit 10000
[Nest] 1  - 09/14/2023, 8:46:11 AM   DEBUG [QueryService] Processed misc_tags where labels_tags == en:fair-trade in 21 ms. Returning 54 records

Hi @stephanegigandet . I hadn't realised that we supported multiple filters like this. I can enhance the query handling to support this relatively easily

[john-gom]

For queries that include a country filter, the country filter is not taken into account.
e.g. http://uk.openfoodfacts.localhost/label/fair-trade/misc

[Nest] 1  - 09/14/2023, 8:46:11 AM   DEBUG [QueryService] Object:
[
  {
    "$match": {
      "labels_tags": "en:fair-trade",
      "countries_tags": "en:united-kingdom"
    }
  },
  {
    "$unwind": "$misc_tags"
  },
  {
    "$group": {
      "_id": "$misc_tags",
      "count": {
        "$sum": 1
      }
    }
  },
  {
    "$sort": {
      "count": -1
    }
  },
  {
    "$skip": 0
  },
  {
    "$limit": 10000
  }
]

[Nest] 1  - 09/14/2023, 8:46:11 AM   DEBUG [QueryService] select value _id, count(*) count from "query"."product_misc_tag" as "pt" where (not pt.obsolete) and (EXISTS (select "pt2".* from "query"."product_labels_tag" as "pt2" where (pt2.product_id = pt.product_id and pt2.value = 'en:fair-trade'))) group by "pt"."value" order by 2 desc limit 10000
[Nest] 1  - 09/14/2023, 8:46:11 AM   DEBUG [QueryService] Processed misc_tags where labels_tags == en:fair-trade in 21 ms. Returning 54 records

Hi @stephanegigandet . I hadn't realised that we supported multiple filters like this. I can enhance the query handling to support this relatively easily

Hi @stephanegigandet , this should be fixed now

[codecov-commenter]

Codecov Report

Merging #8947 (f311104) into main (7e07fcb) will decrease coverage by 0.03%.
The diff coverage is 17.14%.

@@            Coverage Diff             @@
##             main    #8947      +/-   ##
==========================================
- Coverage   47.95%   47.93%   -0.03%     
==========================================
  Files          64       64              
  Lines       20013    20052      +39     
  Branches     4850     4868      +18     
==========================================
+ Hits         9597     9611      +14     
- Misses       9168     9186      +18     
- Partials     1248     1255       +7     

Files	Coverage Δ
lib/ProductOpener/Data.pm	76.00% <50.00%> (-8.22%)	⬇️
lib/ProductOpener/Display.pm	11.27% <10.34%> (+0.06%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[stephanegigandet]

What scenario were you trying?

I was following the instructions on https://github.com/openfoodfacts/openfoodfacts-query#prepare-for-development
I didn't realize that I already had a running query server inside docker, so I ran the npm commands that started another one on the host.

[john-gom]

Hi @stephanegigandet , @raphael0202 , @alexgarel , I have reverted the code so that the products_tags collection is used if the QUERY_URL is not available (or the call to the query service fails). Would you be happy to take a look at the latest changes to check you have no concerns.

Also, the tests are not passing but I can't figure out why my changes should have affected this. If anyone can help I would be vary grateful.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[stephanegigandet]

Also, the tests are not passing but I can't figure out why my changes should have affected this. If anyone can help I would be vary grateful.

You are getting a 404 because the returned product count is 0 I think.

[john-gom]

Also, the tests are not passing but I can't figure out why my changes should have affected this. If anyone can help I would be vary grateful.

You are getting a 404 because the returned product count is 0 I think.

Thanks. Figured it out, the code was still going to product_tags when no_cache=1

[john-gom]

Some comparisons. Local version connecting to quer.openfoodfacts.org, production still using product_tags collection:


Ecoscore by nutrition grade D:

Query counts seem lower - investigating...

[john-gom]

I added a "select" service to compare results but realised that my own local service was still using my local query service as I hadn't rebuilt and re-started my docker containers. Results are now as expected query service is slightly higher as it was refreshed more recently than product_tags: