Skip to content

VasileiosNt/PSQL_to_RS

Folders and files

NameName
Last commit message
Last commit date

Latest commit

author
vasilis
May 24, 2022
89f8e8d · May 24, 2022

History

6 Commits
May 24, 2022
May 24, 2022
May 20, 2022
May 20, 2022

Repository files navigation

PSQL_to_RS

Simple example on loading data from a local postgresql to a redshift cluster on AWS.

Preresequites

  • poetry
  • python >= 3.7
  • AWS credentials under ~/.aws/ with full access on S3 and Redshift
  • docker and docker-compose

Execution

  • Install dependencies make install
  • Start postgresql make run-psql
  • Create redshift cluster make create-rs-cluster
  • Generate and load 5 million dummy data into postgresql make load-data-to-psql
  • Save table as a csv make extract-table
  • Upload to s3 bucket make upload-to-s3
  • Get redshift URI make get-uri-rs
  • Load to redshift make migrate-to-redshift

Notes

  • The example should not be used for production environments. Use dedicated IAM roles for the redshift cluster.
  • You can assign your preffered node type for the redshift cluster under ./infrastructure/redshift.yml
  • For postgresql configuration modify /infrastructure/env.psql
  • For redshift configuration modify /infrastructure/vars/rs_vars.yaml

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published