Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use psycopg2 for Connection Pooling and Implement Global Engine with Context Manager for Session Management #837

Merged
merged 27 commits into from
Dec 17, 2024

Conversation

qcdyx
Copy link
Contributor

@qcdyx qcdyx commented Nov 27, 2024

Summary:

Closes #826

  • Updated the database connection setup to use psycopg2 for connection pooling.
  • Refactored the code to use a global SQLAlchemy engine.
  • Implemented a context manager for managing database sessions.
  • Ensured session isolation for each request to maintain data integrity and prevent conflicts.

Expected behavior:

  • database connections are reused, reducing the overhead of creating new connections and improving performance.
  • the context manager ensures that sessions are properly opened, committed, rolled back, and closed, providing robust transaction management.
  • ensured that each request gets its own isolated session.

Load test shows that ramping to 5000 users at a rate of 50.00 per second uses 22 active connections - we have max 100 connections.
image

I'll work on a knowledge share sub-issue and explain all the changes in details.

Testing tips:

Provide tips, procedures and sample files on how to test the feature.
Testers are invited to follow the tips AND to try anything they deem relevant outside the bounds of the testing tips.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@qcdyx qcdyx changed the title 293 psycopg feat: Use psycopg2 for Connection Pooling and Implement Global Engine with Context Manager for Session Management Nov 27, 2024
@qcdyx qcdyx requested review from davidgamez and cka-y December 2, 2024 22:03
api/src/database/database.py Show resolved Hide resolved
api/src/database/database.py Outdated Show resolved Hide resolved
api/src/database/database.py Show resolved Hide resolved
api/src/scripts/populate_db_gtfs.py Show resolved Hide resolved
api/tests/test_utils/db_utils.py Show resolved Hide resolved
api/src/scripts/populate_db_gtfs.py Show resolved Hide resolved
Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing improvement!

@qcdyx qcdyx merged commit 2aec616 into main Dec 17, 2024
3 checks passed
@qcdyx qcdyx deleted the 293-Psycopg branch December 17, 2024 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use a global SQLAlchemy engine for db connection pooling with psycopg2
3 participants