Add database connection settings & psycopg library (#576)

* add database connection settings & psycopg library

* update docker file

* add startup script

* update waffle command

* add pyscopg binary

* add loopback details to READ.me

* misc settings update

* update readme

* update package settings

* formatting changes

* fix incorrect port forward pod port

* remove ntlk.downloader

* various updates to guidance

Dockerfile

@@ -31,6 +31,10 @@ RUN poetry install --without dev --no-root && rm -rf $POETRY_CACHE_DIR
 # The runtime image, used to just run the code provided its virtual environment
 FROM python:3.11-slim-buster as runtime
 
+# Update and Install Netcat
+RUN apt-get update && \
+    apt-get install -y netcat
+
 WORKDIR /app
 
 ENV VIRTUAL_ENV=/app/.venv \
@@ -41,20 +45,17 @@ COPY . .
 COPY --from=builder /app/static ./static
 COPY --from=builder ${VIRTUAL_ENV} ${VIRTUAL_ENV}
 
+RUN chmod +x ./scripts/app-entrypoint.sh
 
 RUN python manage.py collectstatic --noinput
-RUN python manage.py migrate
-
-# create switch with default setting
-RUN ./manage.py waffle_switch search-sort-radio-buttons off --create
 
 # Use a non-root user
 RUN addgroup --gid 31337 --system appuser \
   && adduser --uid 31337 --system appuser --ingroup appuser
 RUN chown --recursive appuser:appuser /app
-USER 31337
 
+USER 31337
 
 EXPOSE 8000
 
-ENTRYPOINT ["gunicorn", "--bind", "0.0.0.0:8000", "core.wsgi:application"]
+ENTRYPOINT [ "./scripts/app-entrypoint.sh" ]

README.md

@@ -8,21 +8,135 @@ Please refer to Prerequisites for dependencies and installation instructions
 
 1. Run `poetry install` to install python dependencies
 1. Run `npm install` to download frontend static dependencies.
-1. Run `poetry run python -m nltk.downloader punkt` to install nltk data
 1. Set the `ENV` var to `local` i.e. `export ENV=local`
 1. Run `op inject --in-file .env.tpl --out-file .env` to generate a compatible `.env` file
 1. Optionally substitute value for `CATALOGUE_TOKEN` var in .env with your own PAT value to be able to retrieve search data.
+1. Run `poetry run python manage.py collectstatic --noinput` to collect static files
+1. Run `poetry run python manage.py migrate` this will create a local sqlite3 database and migrate any tables
+1. Run `poetry run python manage.py waffle_switch search-sort-radio-buttons off --create` to setup the waffle switch tables
 1. Run `poetry run python manage.py runserver`
 
 ```sh
 poetry install --no-root
 npm install
-poetry run python -m nltk.downloader punkt
 export ENV=local
 op inject --in-file .env.tpl --out-file .env
+poetry run python manage.py collectstatic --noinput
+poetry run python manage.py migrate
+poetry run python manage.py waffle_switch search-sort-radio-buttons off --create
 poetry run python manage.py runserver
 ```
 
+# Running the app against the RDS database(s)
+The database settings in the application are such that for local development an sqlite db will be created.
+However if there is a usecase to target one of the RDS environments then you will need to carry out the following steps.
+It is important to note that to access the rds environment you will need to create a loopback connection which means opening a local free
+port on your machine for the postgres connection, the loopback connection then forwards matching traffic to a dedicated port forward pod, which inturn forwards
+to the specified rds environment. Please pay particular attention to the port numbers that have been specified in the examples. We have used port 1234 for
+the local port.
+
+1) Create a loop back pod for the given namespace. Note all ports are standard postgres port 5432 for this command.
+    ```
+    kubectl -n data-platform-find-moj-data-dev \
+    run port-forward-pod \
+    --image=ministryofjustice/port-forward \
+    --port=5432 \
+    --env="REMOTE_HOST=cloud-platform-2d5acdf1ab5379e3.cdwm328dlye6.eu-west-2.rds.amazonaws.com" \
+    --env="LOCAL_PORT=5432" \
+    --env="REMOTE_PORT=5432"
+    ```
+2) Forward traffic from your local host to the remote pod and keep the connection open. Note the use of local port of 1234.
+
+    ```
+    kubectl -n data-platform-find-moj-data-dev port-forward port-forward-pod 1234:5432
+    ```
+3) You can test connectivity as follows using postgres utility psql.Note the use of localhost and local port 1234.
+
+    ```
+    psql postgres://< Database Username >:< Database Password >@localhost:1234/< Database Name >
+    ```
+    ```
+    psql (14.11 (Homebrew), server 16.3)
+    WARNING: psql major version 14, server major version 16.
+            Some psql features might not work.
+    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
+    Type "help" for help.
+
+    db2d5acdf1ab5379e3=>
+    ```
+
+4) Via Netcat. Note the use of localhost and local port 1234.
+
+    ```
+    nc -z localhost 1234
+    ```
+    ```
+    Connection to localhost port 1234 [tcp/search-agent] succeeded!
+    ```
+
+5) Optionally populate the .env file if you dont already have one. Add additional environment variables required for RDS.
+Note the value of `RDS_INSTANCE_ADDRESS` as `docker.for.mac.host.internal`. This is a special requirment for accessing local connections through docker containers on MAC OS.
+
+    ```
+    op inject --in-file .env.tpl --out-file .env
+    ```
+
+    ```
+    RDS_INSTANCE_ADDRESS=docker.for.mac.host.internal
+    DATABASE_NAME=< 1pass >
+    DATABASE_USERNAME=< 1pass >
+    DATABASE_PASSWORD=< 1pass >
+    ```
+
+6) In order for the application to utilise the loopback connection you have created in the steps above, you will need to change the postgres port number in the Django settings file `settings.py` and addionally in the startup script ```./scripts/app-entrypoint.sh` if you are running as a docker image, to match the local port value used for the loopback connection i.e. `1234` in our examples.
+    ```
+    "default": {
+        "ENGINE": (
+            "django.db.backends.postgresql"
+            if os.environ.get("RDS_INSTANCE_ADDRESS")
+            else "django.db.backends.sqlite3"
+        ),
+        "NAME": os.environ.get("DATABASE_NAME", BASE_DIR / "db.sqlite3"),
+        "USER": os.environ.get("DATABASE_USERNAME", ""),
+        "PASSWORD": os.environ.get("DATABASE_PASSWORD", ""),
+        "HOST": os.environ.get("RDS_INSTANCE_ADDRESS", ""),
+        "PORT": "1234",
+    }
+    ```
+    If running the app on the development server rather than as a docker image, ignore changing the startup script value as below.
+
+    ```
+    if [ -n "$RDS_INSTANCE_ADDRESS" ]; then
+      echo "Waiting for postgres..."
+
+      while ! nc -z $RDS_INSTANCE_ADDRESS 1234; do
+        sleep 0.1
+      done
+
+      echo "PostgreSQL started"
+    fi
+    ```
+
+7) Building and running as a Docker image.
+
+    ```
+    docker build -t find-moj-data:latest . && docker run --env-file .env -it -p 8000:8000 find-moj-data:latest
+    ```
+
+8) Alternatively run the development server
+    ```
+    poetry run python manage.py collectstatic --noinput
+    poetry run python manage.py migrate
+    poetry run python manage.py waffle_switch search-sort-radio-buttons off --create
+    poetry run python manage.py runserver
+    ```
+
+8) The app should be running at http://localhost:8000
+
+9) Delete the port forward pod
+
+    ```kubectl delete pod port-forward-pod -n data-platform-find-moj-data-dev```
+
 # Prerequisites
 
 ## Npm

core/settings.py

@@ -146,10 +146,19 @@
 
 DATABASES = {
     "default": {
-        "ENGINE": "django.db.backends.sqlite3",
-        "NAME": "db.sqlite3",
+        "ENGINE": (
+            "django.db.backends.postgresql"
+            if os.environ.get("RDS_INSTANCE_ADDRESS")
+            else "django.db.backends.sqlite3"
+        ),
+        "NAME": os.environ.get("DATABASE_NAME", BASE_DIR / "db.sqlite3"),
+        "USER": os.environ.get("DATABASE_USERNAME", ""),
+        "PASSWORD": os.environ.get("DATABASE_PASSWORD", ""),
+        "HOST": os.environ.get("RDS_INSTANCE_ADDRESS", ""),
+        "PORT": "5432",
     }
 }
+
 # Define a service name setting for page titles
 SERVICE_NAME = "Find MOJ data"
 GOV_UK_SUFFIX = "GOV.UK"