Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bigpicture] data replication services #365

Merged
merged 34 commits into from
Nov 17, 2023
Merged

[Bigpicture] data replication services #365

merged 34 commits into from
Nov 17, 2023

Conversation

jbygdell
Copy link
Collaborator

@jbygdell jbygdell commented Oct 20, 2023 •
edited
Loading

This PR adds the standalone data replication service (sda-backup) into the merged code base under the name sync
The sync-api originally comes from here, modified to be part of the merged code base. The functionality to send POST requests have been moved to the sync service.

  • sync
    • Reads messages from the mapping_stream
    • Checks if the dataset is generated locally or remotely, remotely generated datasets are ignored.
    • Copies files from the archive, attaches and re-encrypts the header with the remote sites public key so that the complete files can be ingested as if they where uploaded by an end user.
    • Sends JSON data to the remote API host with information about all files in a dataset.
  • sync-api
    • Parses the incoming JSON data and sends messages to ingest the files, assign file IDs and dataset ID

@jbygdell jbygdell changed the title Bigpicture data replication conponents Bigpicture data replication components Oct 20, 2023
@jbygdell jbygdell force-pushed the bigpicture_sync branch 3 times, most recently from 704a1e6 to ee7924b Compare October 20, 2023 13:26
@jbygdell jbygdell force-pushed the merge_sda_pipeline_4 branch 3 times, most recently from bca5c0a to bf13c15 Compare October 30, 2023 12:35
@jbygdell jbygdell changed the title Bigpicture data replication components Bigpicture data replication service Oct 30, 2023
@jbygdell jbygdell force-pushed the merge_sda_pipeline_4 branch from bf13c15 to 9297a92 Compare October 31, 2023 06:40
Base automatically changed from merge_sda_pipeline_4 to main October 31, 2023 10:00
@jbygdell jbygdell force-pushed the bigpicture_sync branch 3 times, most recently from 624c9af to e9ea81d Compare November 3, 2023 08:01
@jbygdell jbygdell changed the title Bigpicture data replication service [Bigpicture] data replication service Nov 3, 2023
@jbygdell jbygdell self-assigned this Nov 3, 2023
@jbygdell jbygdell requested a review from a team November 3, 2023 08:38
@jbygdell jbygdell marked this pull request as ready for review November 3, 2023 08:38
@jbygdell jbygdell marked this pull request as draft November 3, 2023 09:47
@aaperis aaperis mentioned this pull request Nov 6, 2023
Merged
7 tasks
@jbygdell jbygdell force-pushed the bigpicture_sync branch 5 times, most recently from ce54285 to 7327d1f Compare November 9, 2023 11:41
@jbygdell jbygdell changed the title [Bigpicture] data replication service [Bigpicture] data replication services Nov 9, 2023
@jbygdell
[sync] cleanup error messages
8a761c8
@jbygdell
[sync] No need to send an message after completing the work
07e6e41
@jbygdell
[Integration test] Add case for sync
605dadd
@jbygdell
[sync] cleanup Readme.md
ac6cb63
@jbygdell
Copy sync-api related files from old sda-pipline repo
68ad267
@jbygdell
[sync-api] make it work with the merged code base
e4af272
@jbygdell
[sync-api][test] convert to run as suite.
9d68a9b 
* Starts a real MQ and DB  using `ory/dockertests`
* Replaces mock SQL backend with real DB
@jbygdell
[sync] switch to read messages from the mapping_stream
a09a80a
@jbygdell
[sync] send POST after files have been synced
56129a7
@jbygdell
[sync] Add dataset ID validation
3d5c27f 
By checking the prefix of the dataset ID in the message we can skip processing datasets that originate from another center.
@jbygdell
[sync-api] remove everything database related
4356d49
@jbygdell
[config] remove remote end from sync api
5fe6fab
@jbygdell
[sync-api] cleanup error messages
4b39d4b
@jbygdell
[config] fix sync and sync-api
fa15d0e
@jbygdell
[schemas] encrypted checksums are not mandatory for ingestion-trigger
41ebaf7
@jbygdell
[schemas] set correct url in json files
788a151
@jbygdell
[finalize] nack message if GetFilestatus fails
8187bdc
@jbygdell
[Go.mod] add missing library
1c371f2
@jbygdell
[integration test] update test for data sync
cbad793
@jbygdell
[integration test] add case for sync-api
546f740
@jbygdell
Cleanup after rebase
40fd512
@jbygdell
[Integration test] fix after rebase
8d346ea
@jbygdell
[sync] [sync-api] Update readme
3c292b8
@jbygdell
[json] for isolated setup we don't care for MD5 sums
7fd60da
@jbygdell
[sync-api] create correct correlation ids
d8a41cc
@jbygdell
[sync-api] fix comment
a1db42b
@jbygdell jbygdell merged commit 02187f9 into main Nov 17, 2023
28 checks passed
@jbygdell jbygdell deleted the bigpicture_sync branch November 17, 2023 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aaperis aaperis aaperis approved these changes

@dbampalikis dbampalikis dbampalikis approved these changes

Assignees

@jbygdell jbygdell

Labels
BigPicture
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jbygdell @dbampalikis @aaperis @viklund