Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[From Postgres to Mongo] : Added temp collections to lower data unavailability time #461

Merged

Conversation

antoineludeau
Copy link
Member

@antoineludeau antoineludeau commented Aug 9, 2024

Context

Currently, when transfering data from main db (postgresql) to our exploitation db (mongodb), we are, in this order, :

  • deleting old data from mongo
  • creating temp sql tables with code postal calculation
  • then, page per page (page = 100 items) we are :
    • getting data
    • formating data
    • inserting data into mongo

This can take a long time to processing especially for big cities like 'Toulouse' for example (around 15 minutes). As a consequence, the data in our exploitation db, that has been deleted at the beginning, is unavailable during that processing time.

Enhancement

To solve this issue, this PRs adds a mechanism of "temp collection" :

  • all the processed data will be inserted in those temp collection (temp_voies_${cog} and temp_numero_${cog})
  • when all the data is inserted, the old data is deleted and replaced by the data from those temp collections
  • the temp collections are deleted

@jbouhadoun jbouhadoun self-requested a review August 14, 2024 14:56
@antoineludeau antoineludeau force-pushed the antoineludeau/improve-export-with-temp-collection branch 3 times, most recently from 1f42624 to d2755f5 Compare September 10, 2024 13:33
@antoineludeau antoineludeau force-pushed the antoineludeau/improve-export-with-temp-collection branch from d2755f5 to 787ec7e Compare September 10, 2024 13:38
@antoineludeau antoineludeau merged commit e013897 into master Sep 18, 2024
1 check passed
@antoineludeau antoineludeau deleted the antoineludeau/improve-export-with-temp-collection branch September 18, 2024 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants