First mechanism of export from internal DB (postgres) to exploitation DB (mongo)

[antoineludeau]

I. Context

We are currently using mongoDB as our unique DB to store data and to exploit them. We have implemented a PostgreSQL relational DB that stores the data from our new BAN-ID schema API. We now need to implement an export mechanism from PostgreSQL (internal DB) to MongoDB (exploitation DB).

II. Enhancement

1- We have decided to trigger the export of an entire district when : updating, creating and deleting an address or a common toponym from this district.
2- Each time that happens, a job is created (jobID = districtID) in a new queue : export-to-exploitation-db
3- The job is delayed for 10s. If a job with the same ID is created during this delay, it replaces the job waiting in the queue.
4- After the 10s, the job is consumed by a mechanism coded in the export-to-exploitation-db-consumer.js file. This mechanism is :

• Deleting all data from the district (district, common toponyms, addresses)
• Getting the district data from PostgreSQL DB
• Sending the district data to Mongo DB
• Getting paginated data from common toponyms
• Sending each page to Mongo DB
• Getting paginated data from addresses
• Sending each page to Mongo DB
  5- All of this process is done in a transaction that is locking the data of each pages (for common toponyms and addresses) before each export.

III. How to test

1- Push a district using the API : POST /district

Example :

[
  {
    "id": "7d40ffcc-57e2-4621-a538-e3ef20b74b8e",
    "labels": [
      {
        "isoCode": "fra",
        "value": "Toulouse"
      }
    ],
    "updateDate": "2023-01-01",
    "meta": {
        "insee": {
            "cog": "31555"
        }
    },
    "config": {
        "useBanId": true
    }
  }
]

2- After around 10 seconds, you should see the logs :

Exporting districtID 7d40ffcc-57e2-4621-a538-e3ef20b74b8e to exploitation DB...
2023-08-29 14:42:22 Exporting districtID 7d40ffcc-57e2-4621-a538-e3ef20b74b8e done

After a few seconds, check if the data has been exported to MongoDB in the new collection districts

{
  "id": "7d40ffcc-57e2-4621-a538-e3ef20b74b8e",
  "labels": [
    {
      "value": "Toulouse",
      "isoCode": "fra"
    }
  ],
  "updateDate": {
    "$date": "2023-01-01T00:00:00.000Z"
  },
  "config": {
    "useBanId": true
  },
  "meta": {
    "insee": {
      "cog": "31555"
    }
  },
  "createdAt": {
    "$date": "2023-08-28T16:05:16.528Z"
  },
  "updatedAt": {
    "$date": "2023-08-28T16:05:51.037Z"
  }
}