Psycopg2 dependency introduced in 1.8.0

[padbk]

This PR introduced Psycopg2 as a dependecy for dbt-postgres.

I only use dbt-postgres as it is a depedency of dbt-redshift

And dbt-redshift can function perfectly well without Psycopg2 as it uses redshift-connector

Therefore when, inside a docker container, I installed dbt-redshift 1.8.0 and dbt-postgres 1.8.0, Psycopg2 was also attempted to be installed, when I have not needed it previously. This installation then failed as the docker container I was using did not have the build prerequisites to build Psycopg2 from source code.

I changed the Dockerfile to the following using an alpine image:

FROM python:3.11-alpine

# Psycopg2 build dependencies
RUN apk update 
RUN apk add --no-cache libpq
RUN apk add --virtual temp1 --no-cache gcc python3-dev musl-dev postgresql-dev libffi-dev

# Upgrade pip and install Poetry
RUN pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir poetry

# Generate the requirements file using Poetry
COPY pyproject.toml pyproject.toml
RUN poetry export --without-hashes -f requirements.txt --output requirements.txt

# Install the requirements
RUN pip install --no-cache-dir -r requirements.txt

This works, but I have outstanding questions:

1. Is Psycopg2 really needed as a depedency of dbt-postgres?
2. If it is needed, is there a way to skip installation of Psycopg2 when using dbt-redshift only for example?
3. If it is not possible to skip installation, is there a way to specify to use Psycopg2-binary instead of having to build from source when using Linux?

[jtcohen6]

@padbk Thanks for starting the discussion!

I left a reply on the regression report issue:

• [Regression] Forced psycopg2 dependency on Linux causes sudden build failures #96 (comment)

But I think the ultimate resolution to this question ("why do I need to install psycopg2 in order to use dbt-redshift") is for us to remove dbt-postgres as a dependency of dbt-redshift — now that we've migrated to use redshift-connector instead, and in line with Redshift's slow-but-steady evolution away from Postgres-like syntax/functionality.

• [Feature] Remove dbt-postgres dependency dbt-redshift#811

[padbk]

@jtcohen6 Agree, thanks.

[ssillaots-boku]

@jtcohen6
I've now stumbled upon this issue as well.

But I think the ultimate resolution to this question is for us to remove dbt-postgres as a dependency of dbt-redshift

Could you say if the resolution will be released any time soon?

[jtcohen6]

Hey @ssillaots-boku - we've started on this work, and we've identified a blocker that will require a dbt-core change before we can safely release this:

• [Feature] Remove dbt-postgres dependency dbt-redshift#811 (comment)

Meanwhile, we're going to revert the change to dbt-postgres install dependencies, which should avoid the immediate issue for most users in the short term:

• [Regression] Fix psycopg2 version install #113

[ssillaots-boku]

Awesome, thanks!