upgrade error

[kirmorozov1992]

Bug description

Hi
I want to upgrade Superset 4.0.2 to 4.1.1 version using Helm.
helm upgrade --install superset superset/superset -f values.yaml
I got issue with psycopg2. Help, please.

Screenshots/recordings

Defaulted container "superset-init-db" out of: superset-init-db, wait-for-postgres (init)
Upgrading DB schema...
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
2024-11-22 03:34:11,278:ERROR:superset.app:Failed to create app
Traceback (most recent call last):
File "/app/superset/app.py", line 40, in create_app
app_initializer.init_app()
File "/app/superset/initialization/init.py", line 476, in init_app
self.setup_db()
File "/app/superset/initialization/init.py", line 667, in setup_db
pessimistic_connection_handling(db.engine)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 998, in engine
return self.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1017, in get_engine
return connector.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 594, in get_engine
self._engine = rv = self._sa.create_engine(sa_url, options)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1027, in create_engine
return sqlalchemy.create_engine(sa_url, **engine_opts)
File "", line 2, in create_engine
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 811, in dbapi
import psycopg2
ModuleNotFoundError: No module named 'psycopg2'
Traceback (most recent call last):
File "/usr/local/bin/superset", line 8, in
sys.exit(superset())
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1685, in invoke
super().invoke(ctx)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 355, in decorator
app = __ctx.ensure_object(ScriptInfo).load_app()
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 309, in load_app
app = locate_app(import_name, name)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 238, in locate_app
return find_app_by_string(module, app_name)
File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 166, in find_app_by_string
app = attr(*args, **kwargs)
File "/app/superset/app.py", line 40, in create_app
app_initializer.init_app()
File "/app/superset/initialization/init.py", line 476, in init_app
self.setup_db()
File "/app/superset/initialization/init.py", line 667, in setup_db
pessimistic_connection_handling(db.engine)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 998, in engine
return self.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1017, in get_engine
return connector.get_engine()
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 594, in get_engine
self._engine = rv = self._sa.create_engine(sa_url, options)
File "/usr/local/lib/python3.10/site-packages/flask_sqlalchemy/init.py", line 1027, in create_engine
return sqlalchemy.create_engine(sa_url, **engine_opts)
File "", line 2, in create_engine
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 375, in warned
return fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 544, in create_engine
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 811, in dbapi
import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

Superset version

master / latest-dev

Python version

3.9

Node version

16

Browser

Chrome

Additional context

No response

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hello @kirmorozov1992! I'm here to assist you with any bugs, questions, or contributions you have. Let's work together to solve the issue you're facing.

To resolve the "No module named 'psycopg2'" error when upgrading Apache Superset using Helm, you need to ensure that the psycopg2 package is installed as part of the bootstrap script in your Helm chart configuration. You can do this by adding the installation command for psycopg2 in the bootstrapScript section of your values.yaml file. Here is an example configuration:

bootstrapScript: |
  #!/bin/bash
  pip install psycopg2==2.9.6 \
    sqlalchemy-bigquery==1.6.1 \
    elasticsearch-dbapi==0.2.5 &&\
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

This script ensures that psycopg2 is installed when the Superset application starts, which should resolve the module not found error ^[1].

To continue talking to Dosu, mention @dosu.

[kirmorozov1992]

@dosu Unfortunately it does not help me. Somebody has other advice?

[dosubot]

This is a tough nut to crack, could really use your help on this, @rusackas.

To continue talking to Dosu, mention @dosu.

[DavidProdinger]

There is the same issue for the docker compose version (docker-compose-image-tag.yml), not limited to Helm.

[paulchoi]

This worked for me. You need psycopg2-binary.
You can ignore clickhouse-connect.

bootstrapScript: |
  #!/bin/bash
  pip install \
    "clickhouse-connect>=0.6.8" \
    "psycopg2-binary>=2.9.10" \
    &&\
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

[kirmorozov1992]

@paulchoi Hi! Do you check this with Docker Compose? I`ve tried many times to change config and reloading, but it is unhelpful for me. I use Minikube.

[DavidProdinger]

I use it with Docker Compose

just add in your docker/requirements-local.txt file this content:

# database drivers
pymysql
psycopg2-binary

Sadly the non-binary version of psycopg2 cant be installed, as well as mysqlclient.
Therefore I use pymysql with the URL mariadb+pymysql://...

[richard-fairthorne]

Put this in your bootstrapScript, before pip install:

apt-get update && apt-get install -y build-essential

I am surprised this is not included in the documentation.