DBT Platform migration (#6608)

* Upgrade postgres, open search and redis (#6605)

* Upgrade postgres, open search and redis

* Rename /healthcheck end point to /pingdom

* Migration/dpm 198 data hub frontend logging (#6672)

* Add ASIM format to logger in production. Tag logs as express startup/request.

* DPM-344 Rename AWS_ keys to match API. (#6704)

* Skip failing activity stream tests

* Add missing variables for legal basis

* Use migration branch

* Add missing DB variable

* ci: build configuration

* Add default parameter to metadata requests to fix Hawk issue. (#6758)

* Add default parameter to end point for metadata requests to fix Hawk issue.

* DPM 616 filters on investment profiles (#6759)

* Use urls.metadata for investment profile filter options.
* Exclude end point from a11y check

* Use Data Hub API for events CollectionList (#6802)

* Refactor CollectionItem to allow tags to be rendered in the same way as in activity cards

* Refactor events collection page to use the API instead of ActivityStream

* Update tests

* Cleanup dead code

* add default params to metadata api calls for inline resources, resource options field and get metadata with context option

* fix failing functional test

* fix audit-spec e2e test

* put default params onto ResourceOptionsField to allow metadata api calls

* fix failing e2e tests due to new params on resource options field resources

* fix add-export e2e test

* Use Data Hub API for events CollectionList (#6802)

* Refactor CollectionItem to allow tags to be rendered in the same way as in activity cards

* Refactor events collection page to use the API instead of ActivityStream

* Update tests

* Cleanup dead code

* Allow for parameters for metadata API calls

* Fix incorrect URLs, remove mapping and add additional proptype

* Update incorrect URL in urlExclusions

* Stop using ActivityStream in company activities (#6803)

* Stop using ActivityStream in company activities

* Refactor tests

* Fixup! review feedback

* remove vcap

* Remove ActivityFeed from contact activity collection (#6917)

* Remove ActivityFeed from contact activity collection

* Update tests

* Fix broken tests (#6933)

* Fix event filter-spec

* Fix export pipeline filter-spec

* Refactor overview activity cards to call Data Hub API (#6929)

* Refactor overview activity cards to call DH API

* Update tests

* Add default parameter for administrative areas

* Remove ActivityStream remnants (#6948)

* Move ActivityFeedCheckbox to MyInvestmentProjects and rename it

* Move code used by company activity tab to CompanyActivity directory

* Remove deprecated ActivityStream code from components directory

* Remove deprecated company ActivityFeed code

* Remove deprecated contact activity code

* Remove deprecated code related to events

* Remove deprecated code related to interactions

* Clear out Sandbox and other remnants

* Skip tests for AS-reliant pages

* Render buttons as <a> instead of govukreact Link

* Fix export-data test

* update SOP

Update README.md

Update README.md

---------

Co-authored-by: Christopher <[email protected]>
Co-authored-by: Lawrence Goldstien <[email protected]>
Co-authored-by: Richard Pentecost <[email protected]>
Co-authored-by: Dean Elliott <[email protected]>
Co-authored-by: Bau Nguyen <[email protected]>

.copilot/config.yml

@@ -0,0 +1,4 @@
+repository: data-hub-frontend
+builder:
+  name: paketobuildpacks/builder-jammy-full
+  version: 0.3.339

README.md

@@ -114,12 +114,17 @@ This method is recommended if you are only making frontend changes.
 
 5.  Go to the [Playbook](https://readme.trade.gov.uk/docs/playbooks/datahub.html#environments), find the environment you want to use in the Admin URLs section, and set this URL to the environment variable `API_ROOT`. You will need to remove `/admin` from the end of the API path.
 
-6.  Go to Vault, look in datahub-fe, and click on the environment you want to use. Change the following environment variables in your `.env` file to the ones specified in Vault:
+6.  Go to the parameter store in AWS (https://eu-west-2.console.aws.amazon.com/systems-manager/parameters/?region=eu-west-2&tab=Table). Change the following environment variables in your `.env` file to the ones specified in the parameter store. Remember to select the correct envriorment (e.g dev, uat, staging) you are using.
 
-    ```
-    DATA_HUB_BACKEND_ACCESS_KEY_ID=frontend-key-id
-    DATA_HUB_BACKEND_SECRET_ACCESS_KEY=frontend-key
-    ```
+> **Warning**
+> Gov PaaS will be deprecated from July 2024 and keys will be moved from Vault to Parameter Store.
+>
+> Go to Vault, look in datahub-fe, and click on the environment you want to use. Change the following environment variables in your `.env` file to the ones specified in Vault:
+
+```
+DATA_HUB_BACKEND_ACCESS_KEY_ID=frontend-key-id
+DATA_HUB_BACKEND_SECRET_ACCESS_KEY=frontend-key
+```
 
 7.  The environment variables copied from `sample.env` are set up for running both the frontend and the API using the docker. To run the frontend natively, the following variables will need to be changed to:
 
@@ -190,7 +195,7 @@ These instructions work for both the dockerised environment and the native envir
 
 List of all environment variables can be found in the source code of [envSchema.js](./src/config/envSchema.js).
 
-Check Vault for environment variables that point to other environments, such as staging and dev.
+Check AWS parameter store for environment variables that point to other environments, such as staging and dev.
 
 ## Making changes
 

docker-compose.base.yml

@@ -26,4 +26,4 @@ services:
       NODE_ENV: test
     command: bash -c 'npm run build:for-test-coverage && npm run start:coverage'
   redis:
-    image: redis:6.2.6
+    image: redis:7.2.4

docker-compose.e2e.backend.yml

@@ -5,7 +5,7 @@ services:
       - api
 
   api:
-    image: gcr.io/sre-docker-registry/github.com/uktrade/data-hub-api:main
+    image: gcr.io/sre-docker-registry/github.com/uktrade/data-hub-api:migration-deploy
     env_file: .env
     environment:
       DEBUG: 'False'
@@ -20,9 +20,9 @@ services:
       ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID: incoming-some-id-1
       ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY: incoming-some-secret-1
       ACTIVITY_STREAM_ACCESS_KEY_ID: some-id
-      ACTIVITY_STREAM_INCOMING_ACCESS_KEY_ID: some-id
       ACTIVITY_STREAM_SECRET_ACCESS_KEY: some-secret
       ACTIVITY_STREAM_INCOMING_SECRET_ACCESS_KEY: some-secret
+      DATABASE_CREDENTIALS: '{"username": "postgres", "password": "datahub", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "postgres", "dbInstanceIdentifier": "db-instance"}'
     ports:
       - 8000:8000
     depends_on:
@@ -35,30 +35,29 @@ services:
     command: /app/setup-uat.sh || echo "all good"
 
   rq:
-    image: gcr.io/sre-docker-registry/github.com/uktrade/data-hub-api:main
+    image: gcr.io/sre-docker-registry/github.com/uktrade/data-hub-api:migration-deploy
     env_file: .env
     depends_on:
       - postgres
       - opensearch
       - redis
     entrypoint: dockerize -wait tcp://postgres:5432 -wait tcp://opensearch:9200 -wait tcp://redis:6379 -timeout 5m
     command: python short-running-worker.py long-running-worker.py
-    logging:
-      driver: none
+    environment:
+      DATABASE_CREDENTIALS: '{"username": "postgres", "password": "datahub", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "postgres", "dbInstanceIdentifier": "db-instance"}'
 
   postgres:
-    image: postgres:12
+    image: postgres:16
     ports:
-      - "5432:5432"
+      - '5432:5432'
     environment:
       POSTGRES_DB: datahub
       POSTGRES_USER: user
       POSTGRES_PASSWORD: password
-    logging:
-      driver: none
+      DATABASE_CREDENTIALS: '{"username": "postgres", "password": "datahub", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "postgres", "dbInstanceIdentifier": "db-instance"}'
 
   opensearch:
-    image: opensearchproject/opensearch:1.2.4
+    image: opensearchproject/opensearch:2.11.0
     ports:
       - '9200:9200'
       - '9300:9300'
@@ -75,7 +74,7 @@ services:
   activity-feed-reverseproxy:
     build: ./test/end-to-end/proxy
     ports:
-        - 8081:8081
+      - 8081:8081
     depends_on:
       - activity-feed
     logging:
@@ -92,7 +91,8 @@ services:
       - 8082:8082
     environment:
       - PORT=8082
-      - 'VCAP_SERVICES={"redis":[{"credentials":{"uri":"redis://redis:6379"}}],"opensearch":[{"credentials":{"uri":"http://some-id:some-secret@opensearch:9200"}, "plan": "123-123-123" } ] }'
+      - REDIS_URL=redis://redis:6379
+      - OPENSEARCH=http://some-id:some-secret@opensearch:9200
       - SENTRY_ENVIRONMENT=test
       - SENTRY_DSN=http://abc:cvb@localhost:9872/123
       - FEEDS__1__UNIQUE_ID=verification_feed_app

docker-compose.services.yml

@@ -33,22 +33,21 @@ services:
       MAXEMAIL_USERNAME: dummyUser
       MAXEMAIL_PASSWORD: dummyPassword
       ELASTIC_APM_DISABLE_SEND: 1
+      DYNAMICS_INSTANCE_URI: https://localhost
+      DYNAMICS_TENANT_ID: xxx
+      DYNAMICS_APPLICATION_ID: xxx
+      DYNAMICS_CLIENT_SECRET: xxx
     ports:
       - 8001:8001
     depends_on:
       - lbdb
-    command:  >
-      bash -c "sleep 5 && python manage.py migrate &&
-      python manage.py collectstatic --noinput &&
-      python -Wd manage.py runserver 0.0.0.0:8001"
-    logging:
-      driver: none
+    command: >
+      bash -c "sleep 5 && python manage.py migrate && python manage.py collectstatic --noinput && python -Wd manage.py runserver 0.0.0.0:8001"
 
   lbdb:
-    image: postgres:9.6.9-alpine
+    image: postgres:16
     restart: unless-stopped
     environment:
       POSTGRES_USER: postgres
       POSTGRES_DB: legal-basis
-    logging:
-      driver: none
+      POSTGRES_PASSWORD: password

manifest.yml

@@ -3,7 +3,7 @@ applications:
   - buildpacks:
       - https://github.com/cloudfoundry/nodejs-buildpack.git#v1.8.23
     health-check-type: http
-    health-check-http-endpoint: /healthcheck
+    health-check-http-endpoint: /pingdom
     memory: 5G
     disk_quota: 8G
     stack: cflinuxfs4

package.json

@@ -23,8 +23,8 @@
     "lint:js": "npx eslint --ext .jsx,.js ./test ./src ./assets",
     "lint:js:fix": "npx eslint --ext .jsx,.js ./test ./src ./assets --fix",
     "test": "run-s test:*",
-    "test:unit:files": "f() { NODE_ENV=test API_ROOT=http://test LOG_LEVEL=silent MOCHA_FILE=junit/test-results.xml npx mocha \"${@:-./src/**/*.test.js}\"; }; f",
-    "test:unit": "NODE_ENV=test API_ROOT=http://test MOCHA_FILE=junit/test-results.xml npx mocha ./src/**/*.test.js --exit",
+    "test:unit:files": "f() { NODE_ENV=test API_ROOT=http://test LOG_LEVEL=info MOCHA_FILE=junit/test-results.xml npx mocha \"${@:-./src/**/*.test.js}\"; }; f",
+    "test:unit": "NODE_ENV=test API_ROOT=http://test LOG_LEVEL=info MOCHA_FILE=junit/test-results.xml npx mocha ./src/**/*.test.js --exit",
     "test:functional": "CYPRESS_retries=3 nyc cypress run --browser chrome",
     "test:a11y": "CYPRESS_coverage=false cypress run --config-file cypress.a11y.config.js --browser chrome",
     "test:a11y:watch": "CYPRESS_coverage=false cypress open --config-file cypress.a11y.config.js --e2e --browser chrome",
@@ -211,6 +211,7 @@
     "sharp": "^0.33.4",
     "sinon": "^18.0.0",
     "sinon-chai": "^3.7.0",
+    "std-mocks": "^2.0.0",
     "storybook": "^8.1.11",
     "supertest": "^7.0.0",
     "webpack": "^5.92.0",

sample.env

@@ -44,9 +44,9 @@ ACTIVITY_STREAM_OUTGOING_URL=https://lookInVault.com/
 ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID=some-id
 ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY=some-secret
 ADMIN_OAUTH2_ENABLED=false
-AWS_ACCESS_KEY_ID=foo
-AWS_DEFAULT_REGION=eu-west-2
-AWS_SECRET_ACCESS_KEY=bar
+DEFAULT_BUCKET_AWS_ACCESS_KEY_ID=foo
+DEFAULT_BUCKET_AWS_DEFAULT_REGION=eu-west-2
+DEFAULT_BUCKET_AWS_SECRET_ACCESS_KEY=bar
 # Workaround for Docker/CircleCI compatibility problem with Python 3.8
 COLUMNS=80
 DATA_HUB_FRONTEND_ACCESS_KEY_ID=frontend-key-id