[pgmq-python] Merge pgmq-sqlalchemy with current python client package

[jason810496]

Hi, after discussing with @ChuckHend,
I will merge pgmq-sqlalchemy with the current Python client package.

The process of merging these two Python packages might include:

• Migrating documentation:
  • Add RST format docs for client class methods.
  • Add documentation with example usage.
• Migrating unit tests.

Other discussions include:

• For the pgmq-sqlalchemy client (to be merged into tembo-pgmq-python):
  • Add transaction support (allow injecting db_session).
• For the current tembo-pgmq-python client:
  • Refactor with the Python DBAPI interface to support psycopg2 and psycopg3.
• Designing the package hierarchy.

The overall process might result in two client classes:
one with SQLAlchemy support and another that directly interacts with the DBAPI.

Are there any concerns that should be addressed during this migration?

Tasks

Preview Give feedback

No tasks being tracked yet.

[ChuckHend]

I like pgmq-sqlalchemy because it has better support for the various SQL client libraries in the python ecosystem, and sync/async. tembo-pgmq-python only supports psycopg3 at the moment, and I think most of industry still uses psycopg2. It would be much simpler to have a single recognized client per language, so I am all for merging the two projects.

I will merge pgmq-sqlalchemy with the current Python client package.

That links to tembo-pgmq-python, but what we are discussing here is merging pgmq-sqlalachemy into tembo-pgmq-python, correct?

---

From what I can tell, pgmq-sqlalchemy and have very similar APIs, e.g. pgmq.send(), pgmq.send_batch(). class init is different, since tembo-pgmq-python only handles psycopg3. @jason810496 do you see any places across the two APIs that are vastly different? If they are the same, then maybe we can get away with a single class. Otherwise, yeah two classes would be fine to start then we could aim to collapse them in a later release.

[jason810496]

There isn’t any significant difference between the two packages.

I’m just confirming the final details; then I will start the process.
After that, there will be a large PR that needs to be reviewed 🙏

Extract pgmq Functions from the Queue Class

I think we should extract all the functions from the Queue class into a func module, which will also support inflight transactions.

For example:

from tembo_pgmq_python import func 

# inflight db session 

msg_id = func.send(db_session, msg)
# other business logic 
db_session.commit()

Consolidate into One Queue Class

Additionally, the Queue class should be constructed using either the SQLAlchemy or psycopg interface.
The Queue class will also use the func module.

from . import func 

class Queue:
    # …
    def send(self, msg):
        # distinguish between SQLAlchemy or psycopg 
        func.send(db_session, msg)

The overall structure will look like this:

.
├── func    # pgmq functions extracted from the previous Queue class
├── queue 
└── schema

[ChuckHend]

@jason810496 , am I correct in assuming this will be a non-breaking change for the existing psycopg3 users? Seems like that will be true but I am not sure.

[jason810496]

Yes, they just need to change the way the client is constructed; all the methods provided by the client class remain unchanged.
The refactored version is forward-compatible. I can also provide some migration examples in the documentation.

[ChuckHend]

@tavallaie - could you elaborate on the concerns you voiced about merging these two projects? I don't understand how PGMQ is currently used with PeeWee or Celery, which means I also don't understand what problems this will introduce.

[tavallaie]

Django and Celery both use their own database backends, relying on psycopg to manage Postgres connections. Similarly, peewee is another ORM that also uses psycopg . By integrating sqlalchemy directly into pgmq , we're introducing an additional dependency that isn't necessary or utilized.
we can have it like tembo-pgmq-python[sqlalchemy] same as async

[ChuckHend]

I think that is a good idea, tembo-pgmq-python[sqlalchemy], tembo-pgmq-python[sqlalchemy], tembo-pgmq-python[psycopg-async], etc.

If this is implemented this way, does that resolve your concerns @tavallaie ?

[tavallaie]

I think so, I mean I don't see any problem in that way.