Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified job add or update mechanism #327

Merged
merged 1 commit into from
Nov 27, 2023

Conversation

antoineludeau
Copy link
Member

@antoineludeau antoineludeau commented Nov 21, 2023

I. Context

In order to transfert data from postgres to mongo, we send a job to the 'export-to-exploitation-db' worker queue. This job is delayed (30s) so that multiple incoming requests (put/post/delete) during the period of delay are gathered in one export for the concerned district.
For example, id-fix is making a lot of api call during a short period of time. We don't want each call to result in an export to mongo.

The first mechanism in place was to send a job to that queue with an ID corresponding to the districtID. Each time we want to add a job we were checking if the job (with the same ID) was already in the queue and if it was the case, we were trying to remove it and replace it by a new one (with the same delay).

The issue with the first mechanism was that, once the job was active (= in processing by the worker), we could not stop it and delete it. And we couldn't add a new job to the queue as there were sharing the same ID.

II. Enhancement

This PR aims to modify this first mechanism :

  • We now add a job not with a specific ID (we let 'bull' adding a unique ID on itself)
  • Each time we want to add a new job, we first checks the jobs that are in the "waiting" or "delayed" queue (for more information about bull lifecycle : https://optimalbits.github.io/bull/). We check if the job data has the same districtID. If it is the case, we don't add a new job. If not, it means either the job is 'active' or there is no job at all : as a consequence, we add it to the queue.

@antoineludeau antoineludeau force-pushed the antoineludeau/fix-queue-active-state branch from 580e9df to 0eb9c19 Compare November 27, 2023 15:23
@antoineludeau antoineludeau merged commit 9721699 into master Nov 27, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BAN-Plateforme] : Changement de mécanisme pour l'ajout des 'job' dans la queue d'export vers mongo
2 participants