Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change data fetch method #30

Merged
merged 5 commits into from
Jul 12, 2024
Merged

Change data fetch method #30

merged 5 commits into from
Jul 12, 2024

Conversation

pranavanba
Copy link
Collaborator

Major Changes

  1. Significantly speed up pipeline execution time by connecting directly to S3 bucket, instead of syncing S3 bucket objects to local directory
  2. Read datasets directly from S3 bucket instead of from local file paths

Minor Changes

  1. Remove config parameters that are no longer needed due to Major Change # 1
  2. Make progress print statements prettier

@pranavanba
Significantly speed up pipeline run time by connecting directly to S3…
3b1d210 
… bucket instead of syncing files from S3 storage to local directory
@pranavanba
Remove downloadLocation and deleteExistingDir config parameters since…
5d18968 
… they are no longer needed due to connecting directly to S3 bucket instead of syncing files from the bucket to a local directory
@pranavanba
Pass S3 object path instead of local file path since S3 objects are n…
612d859 
…o longer synced to local directory and are instead directly read from S3 bucket connection
@pranavanba
Lint
a49a388
@pranavanba
Change progress print statements
03f46af
@pranavanba pranavanba self-assigned this Jul 12, 2024
@pranavanba pranavanba merged commit 21f3e7a into Sage-Bionetworks:main Jul 12, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@pranavanba pranavanba

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pranavanba