Merge branch 'develop'

Author: myieye

.github/pull_request_template.md

@@ -1,23 +1,12 @@
-Fixes # (issue)
+### Fixes #_issue_
 
 ## Description
 
-Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
-
-### Type of Change
-
-Only keep lines below that describe this change, then delete the rest.
-
-- Bug fix (non-breaking change which fixes an issue)
-- New feature (non-breaking change which adds functionality)
-- Breaking change (fix or feature that would cause existing functionality to not work as expected)
-- This change requires a documentation update
-- migration required (a model change requiring a mongodb script or migration script)
-- UI change
+_A summary of the change including motivation and context._
 
 ## Screenshots
 
-Please provide screenshots / animations for any change that involves the UI. Please provide animations to demonstrate user interaction / behavior changes
+_Demonstrate any UI / behavioral changes with screenshots or animations._
 
 ## Checklist
 
@@ -28,13 +17,10 @@ Please provide screenshots / animations for any change that involves the UI. Ple
 - [ ] I have added tests that prove my fix is effective or that my feature works
 - [ ] I have enabled auto-merge (optional)
 
-## How to test
+## QA testing
 
-Please describe how to test and/or verify your changes. Provide instructions so we can reproduce. Please also provide relevant test data as necessary. These instructions will be used for QA testing below.
+Testers, use the following instructions on [qa.languageforge.org](https://qa.languageforge.org). Post your findings as a comment and include any meaningful screenshots, etc.
 
-- Test A
-- Test B
-
-## qa.languageforge.org testing
+_Describe how to verify your changes and provide any necessary test data._
 
-Testers should add his/her findings to end of the PR in a comment and include screenshots, files, etc that are beneficial.
+- Test A

.github/workflows/build-and-publish-base-php.yml

@@ -15,15 +15,17 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
-      - name: Build and tag base image
-        run: docker build -t ${{ env.IMAGE }} -f docker/base-php/Dockerfile .
+      - uses: docker/setup-qemu-action@v2
+
+      - uses: docker/setup-buildx-action@v2
+        with:
+          platforms: linux/amd64,linux/arm64
 
       - name: Log in to Docker Hub
         uses: docker/login-action@v2
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
 
-      - name: Publish image
-        run: |
-          docker push ${{ env.IMAGE }}
+      - name: Build, tag and push image
+        run: docker buildx build --push --platform linux/amd64,linux/arm64 -t ${{ env.IMAGE }} -f docker/base-php/Dockerfile .

.github/workflows/e2e-tests.yml

@@ -0,0 +1,42 @@
+name: Run E2E tests
+
+on:
+  workflow_call:
+
+jobs:
+  e2e-tests:
+    name: ${{matrix.browser}} (${{ matrix.shard }}/${{ matrix.shards }})
+    strategy:
+      fail-fast: false
+      matrix:
+        browser: [chromium, firefox]
+        shard: [1, 2, 3, 4, 5, 6]
+        shards: [6]
+
+    runs-on: ubuntu-latest
+
+    steps:
+      -
+        uses: actions/checkout@v3
+      -
+        name: Playwright E2E Tests
+        run: make e2e-tests-ci browser=${{ matrix.browser }} shard="${{ matrix.shard }}/${{ matrix.shards }}"
+      -
+        name: Upload Playwright test results
+        if: always()
+        uses: actions/upload-artifact@v3
+        with:
+          name: test-results-${{ matrix.browser }}-${{ matrix.shard }}
+          path: test/e2e/test-results
+          if-no-files-found: error
+
+  e2e-tests-result:
+    if: always()
+    runs-on: ubuntu-latest
+    needs: e2e-tests
+    steps:
+      - name: Check aggregated E2E result
+        if: ${{ needs.e2e-tests.result != 'success' }}
+        run: |
+          echo "Some E2E tests failed"
+          exit 1

.github/workflows/integrate-and-deploy.yml

@@ -74,17 +74,6 @@ jobs:
       -
         name: Unit Tests
         run: make unit-tests-ci
-      -
-        name: Playwright E2E Tests
-        run: make e2e-tests-ci
-      -
-        name: Upload Playwright test results
-        if: always()
-        uses: actions/upload-artifact@v3
-        with:
-          name: test-results
-          path: test/e2e/test-results
-          if-no-files-found: error
       -
         name: Log in to Docker Hub
         uses: docker/login-action@v2
@@ -109,10 +98,13 @@ jobs:
       IMAGE_NEXT_APP: ${{ steps.image.outputs.NAMESPACE }}:${{ steps.image.outputs.TAG_NEXT_APP }}
       IMAGE_LFMERGE: ${{ steps.image.outputs.LFMERGE_NAMESPACE }}:${{ steps.image.outputs.TAG_LFMERGE }}
 
+  e2e-tests:
+    uses: ./.github/workflows/e2e-tests.yml
+
   deploy:
     runs-on: [self-hosted, languageforge]
 
-    needs: integrate
+    needs: [integrate, e2e-tests]
 
     steps:
       -

.github/workflows/pull-request.yml

@@ -54,45 +54,8 @@ jobs:
           github_token: ${{ github.token }}
           junit_files: PhpUnitTests.xml
 
-  build-app-run-e2e-tests:
-    runs-on: ubuntu-latest
-
-    steps:
-      -
-        uses: actions/checkout@v3
-      -
-        uses: actions/setup-node@v3
-        with:
-          node-version: '16.14.0'
-          cache: 'npm'
-      -
-        name: Get installed Playwright version
-        id: playwright-version
-        run: echo -n "version=$(npm ls @playwright/test --json | jq --raw-output '.version')" >> $GITHUB_OUTPUT
-      -
-        name: Cache playwright browsers
-        uses: actions/cache@v3
-        id: playwright-cache
-        with:
-          path: '~/.cache/ms-playwright'
-          # Make each playwright version use its own cache in case it uses different browser versions
-          # Cache entries are deleted if not accessed for 7 days
-          # https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy
-          key: 'playwright-${{ steps.playwright-version.outputs.version }}'
-
-      -
-        name: Playwright E2E Tests
-        # see https://playwright.dev/docs/ci#github-actions
-        run: make e2e-tests-ci
-
-      -
-        name: Upload Playwright test results
-        if: always()
-        uses: actions/upload-artifact@v3
-        with:
-          name: test-results
-          path: test/e2e/test-results
-          if-no-files-found: error
+  e2e-tests:
+    uses: ./.github/workflows/e2e-tests.yml
 
   check-code-formatting:
     runs-on: ubuntu-latest

.prettierignore

@@ -38,7 +38,6 @@ next-app/**/.env.*
 
 ## Extras
 # Libraries and generated code
-src/js/lib/
 src/Site/views/languageforge/theme/default/page/home/assets/
 *.generated-data.*
 

Makefile

@@ -16,14 +16,14 @@ ui-builder:
 e2e-tests-ci:
 	npm ci
 	$(MAKE) e2e-app
-	npx playwright install chromium
-	npx playwright test -c ./test/e2e/playwright.config.ts
+	npx playwright install ${browser} --with-deps
+	npx playwright test -c ./test/e2e/playwright.config.ts --project=${browser} --shard=${shard}
 
 .PHONY: e2e-tests
 e2e-tests: ui-builder
 	npm install
 	$(MAKE) e2e-app
-	npx playwright install chromium
+	npx playwright install chromium firefox
 	npx playwright test -c ./test/e2e/playwright.config.ts $(params)
 
 .PHONY: e2e-app
@@ -67,6 +67,11 @@ build-next:
 build-base-php:
 	docker build -t sillsdev/web-languageforge:base-php -f docker/base-php/Dockerfile .
 
+.PHONY: composer-dev
+composer-dev:
+	docker compose build composer-dev
+	docker compose run composer-dev
+
 .PHONY: clean
 clean:
 	docker compose down

docker-compose.yml

@@ -30,7 +30,6 @@ services:
         - ENVIRONMENT=development
     image: lf-app
     container_name: lf-app
-    platform: linux/amd64
     depends_on:
       - db
       - mail
@@ -243,7 +242,6 @@ services:
         - ENVIRONMENT=development
     image: lf-e2e-app
     container_name: e2e-app
-    platform: linux/amd64
     depends_on:
       - db
       - mail
@@ -288,19 +286,26 @@ services:
       - MONGODB_CONN=mongodb://db:27017
       - MAIL_HOST=mail
       - LANGUAGE_DEPOT_API_TOKEN=bogus-development-token
-      # Note: Uncomment to enable XDebug in Unit Tests.  The default mode is "off".
-      # These lines are commented out as this XDebug setup causes CI failures.
-      # XDebug is intended for local development only.  Uncomment the following 3 lines:
-      # - XDEBUG_MODE=develop,debug
-    # extra_hosts:
-      # - "host.docker.internal:host-gateway"
+      - XDEBUG_MODE=develop,debug
+    # used by XDebug
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     command: sh -c "/wait && /run.sh"
     volumes:
       # for developer convenience
       - ./test:/var/www/test
       - ./src/Api:/var/www/src/Api
       - ./src/Site:/var/www/src/Site
 
+  composer-dev:
+    build:
+      dockerfile: docker/composer-dev/Dockerfile
+    image: lf-composer-dev
+    container_name: lf-composer-dev
+    volumes:
+      - ./src/composer.json:/work/composer.json
+      - ./src/composer.lock:/work/composer.lock
+
 volumes:
   lf-caddy-config:
   lf-caddy-data:

docker/base-php/Dockerfile

@@ -10,7 +10,7 @@ RUN apt-get update && apt-get -y install p7zip-full unzip curl tini ffmpeg && rm
 # see https://github.com/mlocati/docker-php-extension-installer
 # PHP extensions required by the LF application
 COPY --from=mlocati/php-extension-installer /usr/bin/install-php-extensions /usr/local/bin/
-RUN install-php-extensions gd mongodb intl
+RUN install-php-extensions mongodb intl
 
 # php customizations
 COPY docker/base-php/customizations.php.ini $PHP_INI_DIR/conf.d/

docker/composer-dev/Dockerfile

@@ -0,0 +1,10 @@
+FROM sillsdev/web-languageforge:base-php
+
+
+WORKDIR /work
+ENV COMPOSER_ALLOW_SUPERUSER=1
+RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/* \
+    && install-php-extensions @composer
+COPY src/composer.json src/composer.lock /work/
+
+CMD ["bash"]

docker/db/README.md

@@ -16,7 +16,7 @@
 
 > https://www.mongodb.com/docs/v4.4/mongo
 
-`mongo` within `db` container
+`mongosh` within `db` container
 
 > https://www.mongodb.com/docs/manual/reference/mongo-shell/#command-helpers
 

docker/test-php/Dockerfile

@@ -9,9 +9,8 @@ RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/* \
 COPY src/composer.json src/composer.lock /var/www/html/
 RUN composer install
 
-# uncomment if you want xdebug in your local image
-#RUN install-php-extensions xdebug
-#COPY docker/app/docker-php-ext-xdebug.ini /usr/local/etc/php/conf.d
+RUN install-php-extensions xdebug
+COPY docker/app/docker-php-ext-xdebug.ini /usr/local/etc/php/conf.d
 
 RUN mv $PHP_INI_DIR/php.ini-development $PHP_INI_DIR/php.ini
 COPY --from=sillsdev/web-languageforge:wait-latest /wait /wait

docker/ui-builder/Dockerfile

@@ -13,7 +13,6 @@ COPY webpack.config.js webpack-dev.config.js webpack-prd.config.js tsconfig.json
 # copy in src local files
 COPY src/angular-app ./src/angular-app
 COPY src/appManifest ./src/appManifest
-COPY src/js ./src/js
 COPY src/json ./src/json
 COPY src/sass ./src/sass
 COPY src/Site/views ./src/Site/views

docs/DEVELOPER.md

@@ -151,14 +151,6 @@ A [tutorial on YouTube is available showing how to use XDebug and VSCode](https:
 
 To debug the PHP tests, follow these steps:
 
-- uncomment the 3 lines in the docker-compose.yml file related to XDebug under the service section `test-php`:
-
-```
-       - XDEBUG_MODE=develop,debug
-     extra_hosts:
-       - "host.docker.internal:host-gateway
-```
-
 - In VS Code, set a breakpoint on a line of code in one of the PHP tests (in the `test/php` folder)
 - Click on the `Run and Debug` area of VS Code, then click the green play icon next to `XDebug` in the configuration dropdown.
 

docs/RELEASE.md

@@ -61,7 +61,7 @@ Since database upgrades are so infrequent, require extra care and a brief outage
 1. `make scale-down` and ensure all containers are stopped. It's also a good idea to watch the logs for the `db` container ensuring the shutdown was "clean".
 1. `make deploy-db` and ensure configs are applied to the deployment
 1. `make scale-up` and ensure all containers start back up. It's also a good idea to watch the logs for the `db` container ensuring the startup was "clean" and the new version is actually running.
-1. Verify that the running MongoDB version is in fact the version you expect. Connecting to QA or PROD mongo instance using the `mongo` client should confirm this (the server version is printed when you connect). You can also check the image name of the deployment you are inspecting.
+1. Verify that the running MongoDB version is in fact the version you expect. Connecting to QA or PROD mongo instance using the `mongosh` client should confirm this (the server version is printed when you connect). You can also check the image name of the deployment you are inspecting.
 1. Run the `db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )` command to verify that feature version is set to the previous db version
 1. Ensure a recent backup of the mongo databases was successful and is available in case a restore/revert is necessary.
 1. To perform the actual upgrade run `db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } )` Change the version to the desired new version number. This operation may take some time to complete as it may trigger an internal data migration on existing collections.

next-app/README-Tech.md

@@ -71,13 +71,11 @@ Of course choosing tailwindcss comes with a number of criticisms but we believe
 
 1. The codebase can be kept free of class clutter by simply utlizing `@apply` in the `<style>` section of our `.svelte` files but when doing this a couple of things need to be taken into consideration:
 
-> modifiers are not supported in `@apply` without additional `preprocess` and `postcss` configuration and increased specificity can sometimes cause problems when class overrides are needed on an element directly.
+> modifiers are not supported in `@apply` without additional `preprocess` and `postcss` configuration and increased specificity can sometimes cause problems when class overrides are needed on an element directly. See [Warn: Unused CSS selector](https://github.com/saadeghi/daisyui/discussions/1490) for some elaboration. We do still use the `@apply` in a component but not scoped, simply for an organizational benefit, see `/next-app/src/lib/forms/Form.svelte`
 
 2. We will inevitably have situations where parent components or views will want to pass styles down to children. This is very common and requires writing global classes anyway. Using something like tailwindcss or bootstrap takes all of that work off of our plate. Unused CSS will still get purged by the compiler so the app doesn't take on unnecessary bloat.
 3. daisyUI is built upon tailwindcss anyway so we benefit from using it in both situations.
 
-[ ] Confirm no additional configuration is required to purge unused CSS.
-
 #### daisyUI
 
 With such a small team, we were always going to need a well thought-out UI library to keep us from having to make a bunch of design decisions along the way. While there are a handful of Svelte-based libraries, none of them met the criteria necessary to help make our team as efficient as possible. There were many mature UI systems already in place built upon tailwindcss and daisyUI represents one of the best of the ones we looked at in it's goal of further simplifying the use of tailwindcss. Additionally, daisyUI has already integrated many best practices and opinions in its design decisions.