Skip to content

ministryofjustice/find-moj-data

Folders and files

NameName
Last commit message
Last commit date

Latest commit

3fc3716 · Aug 6, 2024
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Find MOJ Data

repo standards badge

Find MOJ Data is an internal service for finding and sharing data across the Ministry of Justice. It is powered by Datahub, an open source data catalogue. This repository contains a custom UI that uses the GOV.UK design system, and focuses on data discovery.

Screenshot of the service showing the search page

Prerequisites

Npm

Required for building the front end javascript dependencies

Poetry

Required for managing python package dependencies. Follow installation instructions here https://python-poetry.org/docs/#installation

1Password

Organisational level tool for storing application secrets and passwords securely. There are a number of 1password utilities available to manage credentials from cli and desktop environments.

  1. Install the 1Password desktop app - https://support.1password.com/get-the-apps/
  2. Install the 1Password CLI app - https://developer.1password.com/docs/cli/get-started/
  3. Follow the steps to turn on and test the 1password desktop app integration

Chromedriver

Tests will require chromedriver at a version compatible with your chrome browser (https://googlechromelabs.github.io/chrome-for-testing/)

Quick start

Please refer to Prerequisites for dependencies and installation instructions

  1. Export a local var with the environment name export ENV=local
  2. Run make build to install dependencies and build the app
    1. Make any needed customizations required to .env file
  3. Run make test to run unit tests for the app. This step requires chromedriver up-to-date with your google chrome version: https://googlechromelabs.github.io/chrome-for-testing/
  4. Run make run to run the app locally on http://localhost:8000
export ENV=local
make build
make test
make run

Running the app against the RDS database(s)

The database settings in the application are such that for local development an sqlite db will be created. However if there is a usecase to target one of the RDS environments then you will need to carry out the following steps. It is important to note that to access the rds environment you will need to create a loopback connection which means opening a local free port on your machine for the postgres connection, the loopback connection then forwards matching traffic to a dedicated port forward pod, which inturn forwards to the specified rds environment. Please pay particular attention to the port numbers that have been specified in the examples. We have used port 1234 for the local port.

  1. Create a loop back pod for the given namespace. Note all ports are standard postgres port 5432 for this command.

    kubectl -n data-platform-find-moj-data-dev \
    run port-forward-pod \
    --image=ministryofjustice/port-forward \
    --port=5432 \
    --env="REMOTE_HOST=cloud-platform-2d5acdf1ab5379e3.cdwm328dlye6.eu-west-2.rds.amazonaws.com" \
    --env="LOCAL_PORT=5432" \
    --env="REMOTE_PORT=5432"
    
  2. Forward traffic from your local host to the remote pod and keep the connection open. Note the use of local port of 1234.

    kubectl -n data-platform-find-moj-data-dev port-forward port-forward-pod 1234:5432
    
  3. You can test connectivity as follows using postgres utility psql.Note the use of localhost and local port 1234.

    psql postgres://< Database Username >:< Database Password >@localhost:1234/< Database Name >
    
    psql (14.11 (Homebrew), server 16.3)
    WARNING: psql major version 14, server major version 16.
            Some psql features might not work.
    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
    Type "help" for help.
    
    db2d5acdf1ab5379e3=>
    
  4. Via Netcat. Note the use of localhost and local port 1234.

    nc -z localhost 1234
    
    Connection to localhost port 1234 [tcp/search-agent] succeeded!
    
  5. Optionally populate the .env file if you dont already have one. Add additional environment variables required for RDS. Note the value of RDS_INSTANCE_ADDRESS as docker.for.mac.host.internal. This is a special requirment for accessing local connections through docker containers on MAC OS.

    ```
    op inject --in-file .env.tpl --out-file .env
    ```
    
    ```
    RDS_INSTANCE_ADDRESS=docker.for.mac.host.internal
    DATABASE_NAME=< 1pass >
    DATABASE_USERNAME=< 1pass >
    DATABASE_PASSWORD=< 1pass >
    ```
    
  6. In order for the application to utilise the loopback connection you have created in the steps above, you will need to change the postgres port number in the Django settings file settings.py and addionally in the startup script ```./scripts/app-entrypoint.shif you are running as a docker image, to match the local port value used for the loopback connection i.e.1234` in our examples.

    "default": {
        "ENGINE": (
            "django.db.backends.postgresql"
            if os.environ.get("RDS_INSTANCE_ADDRESS")
            else "django.db.backends.sqlite3"
        ),
        "NAME": os.environ.get("DATABASE_NAME", BASE_DIR / "db.sqlite3"),
        "USER": os.environ.get("DATABASE_USERNAME", ""),
        "PASSWORD": os.environ.get("DATABASE_PASSWORD", ""),
        "HOST": os.environ.get("RDS_INSTANCE_ADDRESS", ""),
        "PORT": "1234",
    }
    

    If running the app on the development server rather than as a docker image, ignore changing the startup script value as below.

    if [ -n "$RDS_INSTANCE_ADDRESS" ]; then
      echo "Waiting for postgres..."
    
      while ! nc -z $RDS_INSTANCE_ADDRESS 1234; do
        sleep 0.1
      done
    
      echo "PostgreSQL started"
    fi
    
  7. Building and running as a Docker image.

    docker build -t find-moj-data:latest . && docker run --env-file .env -it -p 8000:8000 find-moj-data:latest
    
  8. Alternatively run the development server

    poetry run python manage.py collectstatic --noinput
    poetry run python manage.py migrate
    poetry run python manage.py waffle_switch search-sort-radio-buttons off --create
    poetry run python manage.py runserver
    
  9. The app should be running at http://localhost:8000

  10. Delete the port forward pod

    kubectl delete pod port-forward-pod -n data-platform-find-moj-data-dev

Contributing

Run pre-commit install from inside the poetry environment to set up pre commit hooks.

  • Linting and formatting handled by black, flake8, pre-commit, and isort
    • isort is configured in pyproject.toml
  • detect-secrets is used to prevent leakage of secrets
  • sync_with_poetry ensures the versions of the modules in the pre-commit specification are kept in line with those in the pyproject.toml config.

Testing

  • Python unit tests: pytest -m 'not slow'
  • Javascript unit tests: npm test
  • Selenium tests: pytest -m tests/selenium
  • Search benchmarks (these query the real Datahub backend): pytest tests/benchmarks

Selenium makes use of chromedriver to run a headless browser. As either the chrome browser or chromedriver are updated, the local version of chromedriver and chrome may drift apart. If so, update both to the latest version: update your local chrome and run npm install -g chromedriver chromedriver@latest to install the latest chromedriver.

Frontend styling

If making changes to the scss, to ensure your changes are reflected in local deployments, run: make build to update the css files.

Copy editing

Find MOJ data is only available in english, but we use django's translation system so that we can review and edit the copy in one place, the message file.

When adding new copy to the service, developers should mark it up with translation strings, before running make messages (or manage.py makemessages).

To change a message in github:

  1. Open the message file in github
  2. Click the pencil icon (edit this file)
  3. Change the msgstring values (leave msgid unchanged)
  4. Click the green "Commit changes" button
  5. Add a description of your changes, and then click "Propose changes"

Feature Switches

We have enabled feature switches in find-moj-data. These have been enabled using a package called django-waffle.

Switches can be managed via the cli or via the django admin page.

Our switches are created via the cli; the commands can be seen in the Dockerfile.

Current swicthes and default settings:

  • search-sort-radio-buttons off - switches on/off radio selection buttons for sort order of search results.
  • display-result-tags off - switches on/off the display of tags in search and results pages