Add new workflow to move validated data from a staging location to an archive location #14

pranavanba · 2023-11-10T22:17:45Z

Major Changes

New script for workflow to sync selected validated data from s3 location to archive s3 location and then index in synapse

Create new prod param to indicate path to use for syncing to/from for staging to archive ops
Remove unused, commented code
Borrow necessary function from other file in new file

…rod params

…ontinuing

…emporary dirs and vars

pranavanba added 9 commits November 10, 2023 20:32

Remove unused, commented code

cf46e20

Initial commit

b672150

Get s3 URIs for staging and archive and replace staging params with p…

12b642e

…rod params

Move selected staging folder to archive and then index in synapse

494f131

Get staging params directly from config file

a18ea1d

Check if selected folder exists in staging folder in Synapse before c…

9a47179

…ontinuing

Sync validated date s3 folder to local to archive s3 folder; remove t…

80787bd

…emporary dirs and vars

New param for path to use for syncing to/from for staging to archive ops

3fc72f1

Add necessary function

6a223cf

pranavanba self-assigned this Nov 10, 2023

pranavanba merged commit c2383c1 into Sage-Bionetworks:main Nov 10, 2023
2 checks passed