Enabling PostgreSQL support

[AlexCuadron]

This PR addresses issues #177 and #1138 by enabling SkyPilot to support PostgreSQL in addition to SQLite. While it introduces PostgreSQL as the default database, the implementation ensures backward compatibility with SQLite. If a PostgreSQL connection cannot be established, SkyPilot will automatically revert to using SQLite. This dual-database approach allows for versatile usage: SQLite can be utilized within deployed VMs, while PostgreSQL can be employed in the environment where the Sky CLI operates.

We have duplicated and adapted all smoke tests to verify SkyPilot's compatibility with PostgreSQL. This process is carried out in test_postgresql.py, wherein a PostgreSQL instance is launched within a Docker container specifically for testing. The tests then connect to this PostgreSQL instance and operate it. These tests are meticulously designed to yield output identical to their SQLite equivalents. After each test is completed, the PostgreSQL Docker instance is terminated, ensuring a clean environment for the subsequent test.

Additionally, we have executed all standard smoke tests to ensure comprehensive coverage.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  All the new tests inside test_postgresql.py
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[github-actions]

This PR is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was closed because it has been stalled for 10 days with no activity.

[gufranmirza]

@Michaelvll @romilbhardwaj, I was wondering if you are still interested in adding support for Postgres. I can work on this PR to add it.

[Michaelvll]

Hi @gufranmirza, thanks for your interest! Yes, we still want to support a different more scalable db backend than sqlite, but we would prefer to have it more general, ie it can be supported with different db backend. One proposal is to use sqlalchemy. Wdyt?

[gufranmirza]

@Michaelvll - It makes sense to use SQLAlchemy as it can support different database backends.

[AlexCuadron]

@gufranmirza Thanks for reactivating the discussion on this PR. @Michaelvll think using SQLAlquemy makes sense. I'll refactor the PR to use it

[Michaelvll]

Thanks @AlexCuadron for picking this up! As the first step, using sqlalchemy to support the backend of sqlite and postgreSQL (AWS RDS) should be good enough.