Managing data across deployments

[fridex]

Is your feature request related to a problem? Please describe.

Let's check and document how we could manage data in multiple deployments.

Describe the solution you'd like

• have document-sync-job deployed in the stage environment and let it sync data to the required deployments

• let's see how document-sync-job works in deployments to automatically manage data

  • there might be needed an optimization to sync only data computed since the last sync given the volume of data
  • the sync should be a background job so that it requires minimal human effort and data are ready on each release
  • have metrics exposed Add metrics to document sync job when syncing documents document-sync-job#6

The bottom two checklist would be take care by: #2513

• check if we can automate PostgreSQL dumps
• document how we would like to manage data in deployments
  • how stage data are populated, what needs to be done automatically and what is manual

Related: #2268

[harshad16]

Data migration based on the postgresql db dumps: #2216
/triage accepted
/priority important-soon

[harshad16]

Component document-sync-job was created to accommodate the needs.
The deployment of the document-sync-job is created via the following issues:

• Set up document sync job for the thoth-station #2286
• Increase document-sync deadline to 12 hours #2319
• Perform syncs of documents that are recent #2323
• Increase document sync to 3 days #2324
• Increase memory for document-sync-job #2327
• Bump document-sync-job to v0.3.1 in stage #2332
• switch on the document-sync in stage #2476

This issue is worked and completed.
Thanks for all the efforts @fridex 💯