Modified job add or update mechanism #327
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I. Context
In order to transfert data from postgres to mongo, we send a job to the 'export-to-exploitation-db' worker queue. This job is delayed (30s) so that multiple incoming requests (put/post/delete) during the period of delay are gathered in one export for the concerned district.
For example, id-fix is making a lot of api call during a short period of time. We don't want each call to result in an export to mongo.
The first mechanism in place was to send a job to that queue with an ID corresponding to the districtID. Each time we want to add a job we were checking if the job (with the same ID) was already in the queue and if it was the case, we were trying to remove it and replace it by a new one (with the same delay).
The issue with the first mechanism was that, once the job was active (= in processing by the worker), we could not stop it and delete it. And we couldn't add a new job to the queue as there were sharing the same ID.
II. Enhancement
This PR aims to modify this first mechanism :