Add report mechanism (storing, APIs and report build)

[antoineludeau]

Context

Our back-end system is composed of several components :

• Pre-processing component (id-fix)
• Processing component (ban-plateforme)

Our logs are separated and also are only written on server files.
Our system need to have a better visibility on the pre-processing and processing logs of the BAL sent to the BAN.

Enhancements

This PR aims to add :

• a new "processing-reports" collections to store the reports
• new APIs route to POST and GET the reports stored in the collection (max 5 per district)
• a build mechanism for reports that target asynchronous operations (such our pre-processing component sending data through the BAN APIs).

There are two ways to build an asynchronous report :

• GET the report (GET /api/report/district/{districtID} (or GET /api/report/district/cog/{cog})
• wait for the daily batch (worker job)

How to test

Need to start ban-plateforme and id-fix, connected locally.
Here is the corresponding id-fix PR : BaseAdresseNationale/id-fix#92

I. Legacy pre-processing report :

• Send a BAL through id-fix that is not part of the whitelist or that is not using ban-IDs
• The BAL is sent to ban-plateforme legacy API
• A pre-processing report is generated and sent to ban-plateforme
• The report that is generated in the processing-reports collection is the following one :

{
  "districtID": "a950efd3-69e7-41df-b5d8-a47dc660b66e",
  "preProcessingStatusKey": 0,
  "preProcessingStatus": "success",
  "preProcessingMessage": "Redirected to legacy : District id a950efd3-69e7-41df-b5d8-a47dc660b66e (cog: 59183) is not part of the whitelist.",
  "preProcessingResponse": {},
  "meta": {
    "targetedPlateform": "legacy",
    "revision": {...revisionData*},
    "cog": "59183"
  },
  "preProcessingDate": {
    "$date": "2024-06-21T09:39:19.027Z"
  }
}

*revisionData being the data from the last revision of api-de-dépôt.

II. Pre-processing reports - Success:

• Send a BAL throgh id-fix that is part of the whitelist
• The BAL is processed by id-fix and all the different components (addresses and common toponyms) are sent to BAN BAN-IDs APIs (/address /common-toponym)
• A pre-processing report is generated and sent to ban-plateforme
• The report that is generated in the processing-reports collection is the following one :

{
  "districtID": "a950efd3-69e7-41df-b5d8-a47dc660b66e",
  "preProcessingStatusKey": 0,
  "preProcessingStatus": "success",
  "preProcessingMessage": "Pre-processed : District id a950efd3-69e7-41df-b5d8-a47dc660b66e (cog: 59183) pre-processed and sent to BAN APIs.",
  "preProcessingResponse": {
    "addresses": {
      "update": [
        {
          "date": "2024-06-21T09:46:32.527Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/H5BY3S7AF",
          "response": {
            "statusID": "H5BY3S7AF"
          }
        }
      ],
      "delete": [
        {
          "date": "2024-06-21T09:46:32.531Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/94DX1NB9Q",
          "response": {
            "statusID": "94DX1NB9Q"
          }
        }
      ]
    },
    "commonToponyms": {
      "update": [
        {
          "date": "2024-06-21T09:46:32.524Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/26DRMPNQ1",
          "response": {
            "statusID": "26DRMPNQ1"
          }
        }
      ],
      "delete": [
        {
          "date": "2024-06-21T09:46:32.534Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/BDTT4AGXC",
          "response": {
            "statusID": "BDTT4AGXC"
          }
        }
      ]
    }
  },
  "meta": {
    "targetedPlateform": "ban",
    "revision": {...revisionData*},
    "cog": "59183"
  },
  "preProcessingDate": {
    "$date": "2024-06-21T09:46:32.535Z"
  }
}

*revisionData being the data from the last revision of api-de-dépôt.

This pre-processing report contains a pre-processing response with the different status ID from our asynchronous ban apis. This pre-processing report can be "rebuilt" in a final state by two mechanisms :

• make an API call to GET api/report/district/{districtID} (or /api/report/district/cog/{cog})
• make the daily batch run by changing the time between two jobs :
  In worker.js file, change 1d to 10s for example :
  queue('build-reports').add({}, {jobId: 'buildReportsJobId', repeat: {every: ms('10s')}, removeOnComplete: true})

III. Pre-processing reports - Error:

The pre-processing report can also sent pre-processing errors such as :

• a missing ID on a BAL line :

To test this behaviour :

• Send a BAL that is part of the whitelist, that have ban ids but that has an error on a line (delete an address ID for example).
• The BAL is processed by id-fix and the error is detected
• A pre-processing report is generated and sent to ban-plateforme
• The report that is generated in the processing-reports collection is the following one :

{
  "districtID": "a950efd3-69e7-41df-b5d8-a47dc660b66e",
  "preProcessingStatusKey": 1,
  "preProcessingStatus": "error",
  "preProcessingMessage": "Not Processed : Missing mainTopoID for bal address {\"objectid\":\"61419\",\"cle_interop\":\"59183_3577_00005\",\"commune_insee\":\"59183\",\"commune_nom\":\"Dunkerque\",\"commune_deleguee_insee\":\"59540\",\"commune_deleguee_nom\":\"Saint-Pol-sur-Mer\",\"voie_nom\":\"Square Delvallez\",\"lieudit_complement_nom\":\"\",\"numero\":5,\"suffixe\":\"\",\"position\":\"entrée\",\"x\":653301.9598168375,\"y\":7103868.546534107,\"long\":2.3358292732844284,\"lat\":51.02876869564352,\"cad_parcelles\":[\"591183540AV0192\",\"591183540AV0191\"],\"source\":\"BAN\",\"date_der_maj\":\"2024-01-04T00:00:00.000Z\",\"certification_commune\":false,\"id_ban_adresse\":\"b611de43-6578-4a07-9ac2-0f58d5b8f0ba\",\"id_ban_toponyme\":\"\",\"id_ban_commune\":\"a950efd3-69e7-41df-b5d8-a47dc660b66e\"}",
  "preProcessingResponse": {},
  "meta": {
    "revision": {...revisionData*},
    "cog": "59183"
  },
  "preProcessingDate": {
    "$date": "2024-06-21T10:12:49.077Z"
  }
}

*revisionData being the data from the last revision of api-de-dépôt.

[IGNFBourcier]

les erreurs à répétions et/sur les grosses communes vont difficilement passer dans un document (16mo) ? (raw rapport + rapport compiller)

[antoineludeau]

J'ai fait un test avec la BAL de Toulouse (la BAL la plus conséquente), en insérant une erreur à chaque ligne. Le rapport résultant (raw rapport + rapport compiller) fait 11.3 megabytes donc en effet, on s'approche de la limite du 16mo. Plusieurs options :

• on considère que c'est acceptable,
• on ajoute un calcule de la taille du document avant de l'insérer. Si il dépasse une certaine limite que l'on aura fixé, on ne l'insère pas ou on simplifie le rapport pour pouvoir l'insérer,
• on travail sur la structure des rapports d'erreurs pour optimiser la taille,
• on passe sur des rapports stockés sur S3.