Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: dump postgres records #709

Merged
merged 8 commits into from
Nov 26, 2024
Merged

feat: dump postgres records #709

merged 8 commits into from
Nov 26, 2024

Conversation

mmalenic
Copy link
Member

Closes #705

Changes

  • Adds pg-dd command that can dump the local database, upload/download dumps to S3, and load dumps into the database.
    • This is based on copy ... from/to stdin/stdout SQL command. The reason this is used of pg_dump is because it's easier to embed in a Lambda function, and it allows executing custom sql queries to fetch records (which is used for filemanager).
    • Using Python because psycopg has good support for this compared to TypeScript.
  • Deploy this code as a Lambda function which can be used to dump records from the running dev RDS instance to S3.

@mmalenic mmalenic self-assigned this Nov 20, 2024
@mmalenic mmalenic added feature New feature platform labels Nov 20, 2024
Copy link
Member

@victorskl victorskl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I will give it a try soon.

@mmalenic
Copy link
Member Author

mmalenic commented Nov 21, 2024

Further Updates

  • Dockerfile is simplified to call poetry run directly instead of trying to create a multi-stage build, in order to increase compatibility.
  • The upload and download commands are more intuitive, they now only read from/write to the output directory, they don't actually connect to the database.
    • There is now an extra option for the upload subcommand --dump-db that allows dumping the database to the local directory first before uploading.

Copy link
Member

@williamputraintan williamputraintan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mmalenic
Copy link
Member Author

mmalenic commented Nov 26, 2024

No idea why, but for some reason when writing the object with cli download, it gets compressed twice. Fixed with recent commit.

@williamputraintan
Copy link
Member

Fixed with recent commit.

Could confirm the fix here, thanks!

@mmalenic mmalenic merged commit 446eb0f into main Nov 26, 2024
6 checks passed
@mmalenic mmalenic deleted the feat/pg-dd branch November 26, 2024 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: add Lambda function to dump development database records
3 participants