You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Related to #307, we want to support users who run the ingest workflow on different cloud platforms like Google Cloud (for example, through Terra). Although we can bypass the ingest workflow's default upload target that expects an S3 bucket as its destination in WDL workflows on Terra and upload directly to GS from the WDL workflow, we would probably benefit in the long run from native support for uploads to other cloud platforms besides S3.
Description
As in #307, we should allow users to specify a generic "destination" URI corresponding to at least an S3 or GS bucket such that the upload rule uploads all data from the workflow to the given bucket. This means the remainder of the S3-specific logic in the workflow needs to be replaced with generic remote file support.
We might be able to reuse the remote file handling logic from ncov to support writing files to valid remote URIs. However, we may also want to compress files in the workflow prior to upload instead of compressing during the upload process.
The text was updated successfully, but these errors were encountered:
Context
Related to #307, we want to support users who run the ingest workflow on different cloud platforms like Google Cloud (for example, through Terra). Although we can bypass the ingest workflow's default
upload
target that expects an S3 bucket as its destination in WDL workflows on Terra and upload directly to GS from the WDL workflow, we would probably benefit in the long run from native support for uploads to other cloud platforms besides S3.Description
As in #307, we should allow users to specify a generic "destination" URI corresponding to at least an S3 or GS bucket such that the
upload
rule uploads all data from the workflow to the given bucket. This means the remainder of the S3-specific logic in the workflow needs to be replaced with generic remote file support.We might be able to reuse the remote file handling logic from ncov to support writing files to valid remote URIs. However, we may also want to compress files in the workflow prior to upload instead of compressing during the upload process.
The text was updated successfully, but these errors were encountered: