chore: Update helm chart to 7.0.3

.gitignore

@@ -9,3 +9,5 @@ kustomize/base/resources.yaml
 jobs/reindex/values-reindex-*.yaml
 completed.txt
 ids.json
+alfresco-content-services/
+s3-config/

README.md

@@ -1,102 +1,49 @@
-### Purpose
+# hmpps-delius-alfresco
 
-These example instsructions explain how you would customise one particular dependant Helm chart and publish a newer version to GitHub pages. Instructions assume GitHub pages are already configured for your repo; see the reference section below
+## Helm + Kustomize
 
-### Start services
-In order to start the alfresco-repository service, we need to make a valid  license available in the namespace. A secret containing the license needs to be created:
-```bash
-ACS_NAMESPACE=hmpps-delius-alfrsco-poc
-kubectl create secret generic alfresco-license \
-  --namespace $ACS_NAMESPACE \
-  --from-file /example/path/to/license/file.lic
-```
+This repository contains the Helm and Kustomize configuration for the Delius Alfresco deployment.
 
-Next We will need to ensure all services are up and running.
-Start k8s services by executing helm command _(Helm will complain if a random secret is not created)_
+Rather than using/modifying the Helm chart directly, we use Kustomize to overlay the Helm chart with our custom configuration.
+This allows us to keep the Helm chart as a dependency and only modify the configuration that we need to.
 
-```bash
-cd hmpps-delius-alfresco-poc/alfresco-content-services
-export SECRET=$(openssl rand -base64 20)
-export BUCKET_NAME=$(awk '{print substr($0, 0)}' <<< $(kubectl get secrets s3-bucket-output -o jsonpath='{.data.BUCKET_NAME}' | base64 -d))
-helm install alfresco-content-services . --values=./values.yaml \
---set s3connector.config.bucketName=$BUCKET_NAME \
---set global.tracking.sharedsecret=$SECRET
-```
+### Usage
+
+Note: we use taskfile to simplify the commands. You can install taskfile by running `brew install go-task/tap/go-task`.
+
+
+To deploy the Delius Alfresco stack, you can use the following command:
 
-### Check the chart file for dependent charts and pull the required version
-For the purpose of this demo, we will select the following service from the `Chart.yaml` file
-```yaml
-- condition: alfresco-sync-service.enabled
-  name: alfresco-sync-service
-  repository: https://alfresco.github.io/alfresco-helm-charts/
-  version: 4.1.0
 ```
+task helm_upgrade ENV=<dev|test|stage|preprod|prod> DEBUG=<true|false>
 ```
-1. Delete existing tar file for the sync service. There will be an error pulling the chart otherwise
-rm -rf charts/alfresco-sync-service-4.1.0.tgz
 
-2. Pull a particular version of `alfresco-sync-service` chart
-helm pull alfresco-sync-service --repo  https://alfresco.github.io/alfresco-helm-charts --version 4.1.0 -d charts --untar
+This will deploy the Delius Alfresco stack to the specified environment.
+The `DEBUG` flag can be used to enable debug mode, which will enable helm verbose logging + output the templated,
+rendered and kustomized manifests to the environment directory.
 
-3. The above command will pull a tar file called `charts/alfresco-sync-service-4.1.0.tgz` and then untar it into a directory called `alfresco-sync-service`. Delete the tar file
-rm -rf charts/alfresco-sync-service-4.1.0.tgz
-```
 
-### Modify charts
-
-1. Change the chart version in the newly pulled chart. For example change is from `4.1.0` to `4.1.1`
-2. Make your changes and then test them by upgrading Helm release
-   ```
-   - export SECRET=$(awk '{print substr($0, 19)}' <<< $(kubectl get secrets alfresco-content-services-alfresco-repository-properties-secret -o jsonpath='{.data.alfresco-global\.properties}' | base64 -d))
-   - export BUCKET_NAME=$(awk '{print substr($0, 0)}' <<< $(kubectl get secrets s3-bucket-output -o jsonpath='{.data.BUCKET_NAME}' | base64 -d))
-   - helm upgrade alfresco-content-services . --values=./values.yaml --set s3connector.config.bucketName=$BUCKET_NAME --set global.tracking.sharedsecret=$SECRET
-   - NOTE: For the release upgrade, use the existing secret. You will otherwise have to restart pods consuming those secrets
-   ```
-4. Once satisfied with your changes, create a package and add it to the docs directory
-   - "helm package charts/alfresco-sync-service -d ../docs"
-5. Create / update an index file in docs directory
-   - "helm repo index ../docs --url https://ministryofjustice.github.io/hmpps-delius-alfresco-poc"
-
-
-### Update the lock file and commit changes
-Locate the `Chart.yaml` file and modify the repository URL and version. It should now look like the code snippet below after the change:
-```yaml
-- condition: alfresco-sync-service.enabled
-  name: alfresco-sync-service
-  repository: https://ministryofjustice.github.io/hmpps-delius-alfresco-poc/
-  version: 4.1.1
-```
+### Configuration
 
-1. Delete `charts/alfresco-sync-service` directory as it is no longer needed
-2. Push your changes / docs directory to the feature branch
-3. Update your GitHub pages settings so that the `source branch` is pointing to your feature branch
-4. Update helm dependencies which will pull the updated charts and will update the lock file
-   - `helm dependency update .`
-5. Push the lock file and charts dirctory to the feature branch and get merge approval
-6. Merge into main branch
-7. Update your GitHub pages settings so that the `source branch` is pointing to your main branch
-8. Upgrade the helm release for the changes to be updated in kubernetes cluster
-   ```
-   - export SECRET=$(awk '{print substr($0, 19)}' <<< $(kubectl get secrets alfresco-content-services-alfresco-repository-properties-secret -o jsonpath='{.data.alfresco-global\.properties}' | base64 -d))
-   - helm upgrade alfresco-content-services . --values=./values.yaml --set global.tracking.sharedsecret=$SECRET
-   - NOTE: For the release upgrade, use the existing secret. You will otherwise have to restart pods consuming those secrets
-   ```
-
-### Alternatively, pull a particular chart either directly from repository URL or by adding it in the local repo
-
-1. Pull a chart with a particular version direcly from the GitHub pages
-- `helm pull alfresco-sync-service --repo  https://ministryofjustice.github.io/hmpps-delius-alfresco-poc/ --version 4.1.1 -d charts --untar`
-
-2. Or add the updated chart in a local helm repo
-```
-helm repo add alfresco-sync-service https://ministryofjustice.github.io/hmpps-delius-alfresco-poc/
+1. Helm values
+The base helm values are stored in the `kustomize/base/values.yaml` file.
+Each environment has its own values file, which is stored in the `kustomize/environments/<env>/values.yaml` file.
+These values are combined when deploying the stack, with the environment values taking precedence.
 
-helm search repo alfresco-sync-service
-NAME                                            CHART VERSION   APP VERSION     DESCRIPTION
-alfresco-sync-service/alfresco-sync-service     4.1.1           3.9.0           Alfresco Sync Service
-```
+2. Kustomize
+The kustomize overlays are stored in the `kustomize/environments/<env>` directory.
+These overlays are applied to the Helm chart's resources to modify the configuration as needed for the environment.
+
+
+### Secrets
+
+A number of secrets are required to deploy the Delius Alfresco stack. Some of these are set by the cloud-platform-environments repository, while others are set manually.
 
-### References to various docs that explain how to set up GitHub pages and how to publish Helm charts
-- [The Chart Repository Guide](https://helm.sh/docs/topics/chart_repository/#github-pages-example)
-- [Chart Releaser Action to Automate GitHub Page Charts ](https://helm.sh/docs/howto/chart_releaser_action/#github-actions-workflow)
-- [Example on how to publish a chart on GitHub pages](https://github.com/technosophos/tscharts)
+Table:
+| Secret Name | Description | Set By | example/required keys |
+| --- | --- | --- | --- |
+| amazon-mq-broker-secret | The secret for the Amazon MQ broker | cloud-platform-environments |  see [cloud-platform-environments](https://github.com/ministryofjustice/cloud-platform-environments/blob/7968f9c66f6914d33db35b68209c55b2dcb25d7d/namespaces/live.cloud-platform.service.justice.gov.uk/hmpps-delius-alfresco-stage/resources/amq.tf#L218) |
+| alfresco-license | The Alfresco license file | manual | `<alfresco-license-file-name> : <base64-encoded-alfresco-license-file>`
+| legacy-rds-instance | The RDS instance for the legacy Delius Alfresco stack | manual | `DATABASE_NAME: <database-name>, DATABASE_USERNAME: <database-username>, DATABASE_PASSWORD: <database-password>, RDS_INSTANCE_ADDRESS: <rds-instance-address>` |
+| rds-instance-outpur | The RDS instance for the CP Delius Alfresco stack | cloud-platform-environments | see [cloud-platform-environments](https://github.com/ministryofjustice/cloud-platform-environments/blob/7968f9c66f6914d33db35b68209c55b2dcb25d7d/namespaces/live.cloud-platform.service.justice.gov.uk/hmpps-delius-alfresco-stage/resources/rds.tf#L35) |
+| quay-registry-secret | The secret for the Quay registry | manual | `.dockerconfigjson: {"auths":{"quay.io":{"username":"<quay-username>","password":"<quay-password>","email":"<quay-email>","auth":"<base64-encoded-auth>"}}}` |

Taskfile.yml

@@ -10,6 +10,11 @@ vars:
     sh: kubectl get svc --namespace {{.NAMESPACE}} | grep 'opensearch-proxy-service-cloud-platform' | awk '{ print $1 }'
   OPENSEARCH_HOST:
     sh: echo "{{.OPEN_SEARCH_PREFIX}}.{{.NAMESPACE}}.svc.cluster.local"
+  MESSAGEBROKER_URL:
+    sh: |
+      kubectl get secrets amazon-mq-broker-secret -o json | \
+      jq -r ".data | map_values(@base64d) | .BROKER_URL" | \
+      sed -e 's/(/\\(/g' -e 's/)/\\)/g' -e 's/,/\\,/g'
   ALLOWLIST:
     sh: yq 'join(",")' ./kustomize/{{.ENV}}/allowlist.yaml
   RDS_JDBC_URL:
@@ -22,7 +27,7 @@ vars:
   ATOMIC: "true"
   ATOMIC_FLAG:
     sh: if [ "{{.ATOMIC}}" = "true" ]; then echo "--atomic"; else echo ""; fi
-  CHART_VERSION: "6.0.2"
+  CHART_VERSION: "7.0.3"
 
 tasks:
   helm_upgrade:
@@ -82,11 +87,12 @@ tasks:
     cmds:
       - |
         helm upgrade --install alfresco-content-services alfresco/alfresco-content-services --version {{.CHART_VERSION}} --namespace {{.NAMESPACE}} \
-        --values=../base/values.yaml --values=values.yaml \
+        --values=../base/values703.yaml --values=../base/values-versions.yaml --values=values.yaml \
         --set s3connector.config.bucketName={{.BUCKET_NAME}} \
         --set database.url={{.RDS_JDBC_URL}} \
-        --set global.elasticsearch.host={{.OPENSEARCH_HOST}} \
-        --set alfresco-search-enterprise.searchIndex.host={{.OPENSEARCH_HOST}} \
+        --set-string messageBroker.url="{{.MESSAGEBROKER_URL}}" \
+        --set global.search.url=http://{{.OPENSEARCH_HOST}}:8080 \
+        --set global.search.host={{.OPENSEARCH_HOST}} \
         --wait --timeout=60m \
         --post-renderer ../kustomizer.sh --post-renderer-args "{{.HELM_POST_RENDERER_ARGS}}" \
         {{.DEBUG_FLAG}} {{.ATOMIC_FLAG}}
@@ -103,15 +109,15 @@ tasks:
     cmds:
       - yq '.metadata.annotations."nginx.ingress.kubernetes.io/whitelist-source-range" = "placeholder"' -i patch-ingress-repository.yaml
       - yq '.metadata.annotations."nginx.ingress.kubernetes.io/whitelist-source-range" = "placeholder"' -i patch-ingress-share.yaml
-  
+
   simple_reindex:
     cmds:
-      - | 
-        helm install "reindex-default-$(openssl rand -hex 4)" ./jobs/reindex --set "global.elasticsearch.host={{.OPENSEARCH_HOST}}" --set "fromId=27451380" --set "toId=27908429" --namespace {{.NAMESPACE}}  
+      - |
+        helm install "reindex-default-$(openssl rand -hex 4)" ./jobs/reindex --set "global.elasticsearch.host={{.OPENSEARCH_HOST}}" --set "fromId=150000000" --set "toId=200000000" --namespace {{.NAMESPACE}}
 
   reindex_list:
     cmds:
-      - | 
+      - |
         # Set your batch size (you can adjust this number as needed)
         BATCH_SIZE=40
 
@@ -178,23 +184,20 @@ tasks:
         if [ "$index" -gt 0 ]; then
             create_helm_job "${batch[@]}"
         fi
-        
+
         echo "All jobs have been created!"
         echo "Cleaning up..."
         helm uninstall "reindex-list-${RANDOM_ID}" --namespace {{.NAMESPACE}}
         echo "Cleanup complete!"
 
-
-
-
   simple_reindex_date:
     cmds:
-      - | 
-        helm install "reindex-default-$(openssl rand -hex 4)" ./jobs/reindex_date --set "global.elasticsearch.host={{.OPENSEARCH_HOST}}" --set "fromTime=201707060001" --set "toTime=201707070001" --namespace {{.NAMESPACE}}
+      - |
+        helm install "reindex-default-$(openssl rand -hex 4)" ./jobs/reindex_date --set "global.elasticsearch.host={{.OPENSEARCH_HOST}}" --set "fromTime=201707030001" --set "toTime=201707121301" --namespace {{.NAMESPACE}}
 
   simple_reindex_date_metadata-only:
     cmds:
-      - | 
+      - |
         helm install "reindex-default-date-meta" ./jobs/reindex_date --set "global.elasticsearch.host={{.OPENSEARCH_HOST}}" --set "fromTime=202402010100" --set "toTime=202402100100" --set "content=false" --namespace {{.NAMESPACE}}
 
   batch_reindex:
@@ -249,7 +252,7 @@ tasks:
         for item in $previous_completed; do
           pending=$(echo "$pending" | grep -v "$item")
         done
-        
+
         total_items=$(echo "$pending" | wc -l)
         echo "Total items: $total_items"
 
@@ -275,11 +278,11 @@ tasks:
             if [ $running_jobs -ge {{.CONCURRENCY}} ]; then
                 echo "No available slots, waiting for 5 seconds"
                 sleep 5
-            else 
+            else
               echo "Found at least 1 available slot!"
               echo "Available slots left: $(({{.CONCURRENCY}} - $running_jobs))"
               # run the job
-              echo "helm install reindex-${start}-${end} ./jobs/reindex --set global.elasticsearch.host={{.OPENSEARCH_HOST}} --set fromId=${start} --set toId=${end} --namespace {{.NAMESPACE}}"  
+              echo "helm install reindex-${start}-${end} ./jobs/reindex --set global.elasticsearch.host={{.OPENSEARCH_HOST}} --set fromId=${start} --set toId=${end} --namespace {{.NAMESPACE}}"
               helm install "reindex-${start}-${end}" ./jobs/reindex --set "global.elasticsearch.host={{.OPENSEARCH_HOST}}" --set "fromId=${start}" --set "toId=${end}" --namespace {{.NAMESPACE}}
               # Remove the item from the list
               pending=$(echo "$pending" | tail -n +2)
@@ -330,4 +333,3 @@ tasks:
           echo "Uninstalling release: $release"
           helm uninstall "$release" -n "$NAMESPACE"
         done
-

jobs/reindex-list/templates/reindexing-config.yaml

@@ -5,8 +5,8 @@ metadata:
 data:
   ELASTICSEARCH_INDEXNAME: "{{ .Values.indexName }}"
   {{ template "alfresco-search-enterprise.config.spring" . }}
-  ALFRESCO_SHAREDFILESTORE_BASEURL: http://alfresco-content-services-alfresco-filestore:80/alfresco/api/-default-/private/sfs/versions/1/file/
-  ALFRESCO_ACCEPTEDCONTENTMEDIATYPESCACHE_BASEURL: http://alfresco-content-services-alfresco-router/transform/config
+  ALFRESCO_SHAREDFILESTORE_BASEURL: https://alf-sfs.preprod.delius-core.hmpps-preproduction.modernisation-platform.service.justice.gov.uk/alfresco/api/-default-/private/sfs/versions/1/file/
+  ALFRESCO_ACCEPTEDCONTENTMEDIATYPESCACHE_BASEURL: http://alfresco-content-services-transform-router/transform/config
   ALFRESCO_REINDEX_PATHINDEXINGENABLED: {{ .Values.pathIndexingEnabled | quote }}
   SPRING_DATASOURCE_URL: {{ .Values.postgresql.url }}
   {{- if .Values.environment }}

jobs/reindex/templates/reindexing-config.yaml

@@ -5,8 +5,8 @@ metadata:
 data:
   ELASTICSEARCH_INDEXNAME: "{{ .Values.indexName }}"
   {{ template "alfresco-search-enterprise.config.spring" . }}
-  ALFRESCO_SHAREDFILESTORE_BASEURL: http://alfresco-content-services-alfresco-filestore:80/alfresco/api/-default-/private/sfs/versions/1/file/
-  ALFRESCO_ACCEPTEDCONTENTMEDIATYPESCACHE_BASEURL: http://alfresco-content-services-alfresco-router/transform/config
+  ALFRESCO_SHAREDFILESTORE_BASEURL: https://alf-sfs.preprod.delius-core.hmpps-preproduction.modernisation-platform.service.justice.gov.uk/alfresco/api/-default-/private/sfs/versions/1/file/
+  ALFRESCO_ACCEPTEDCONTENTMEDIATYPESCACHE_BASEURL: http://alfresco-content-services-transform-router/transform/config
   ALFRESCO_REINDEX_PATHINDEXINGENABLED: {{ .Values.pathIndexingEnabled | quote }}
   SPRING_DATASOURCE_URL: {{ .Values.postgresql.url }}
   {{- if .Values.environment }}

jobs/reindex_date/templates/reindexing-config.yaml

@@ -5,8 +5,8 @@ metadata:
 data:
   ELASTICSEARCH_INDEXNAME: "{{ .Values.indexName }}"
   {{ template "alfresco-search-enterprise.config.spring" . }}
-  ALFRESCO_SHAREDFILESTORE_BASEURL: http://alfresco-content-services-alfresco-filestore:80/alfresco/api/-default-/private/sfs/versions/1/file/
-  ALFRESCO_ACCEPTEDCONTENTMEDIATYPESCACHE_BASEURL: http://alfresco-content-services-alfresco-router/transform/config
+  ALFRESCO_SHAREDFILESTORE_BASEURL: https://alf-sfs.preprod.delius-core.hmpps-preproduction.modernisation-platform.service.justice.gov.uk/alfresco/api/-default-/private/sfs/versions/1/file/
+  ALFRESCO_ACCEPTEDCONTENTMEDIATYPESCACHE_BASEURL: http://alfresco-content-services-transform-router/transform/config
   ALFRESCO_REINDEX_PATHINDEXINGENABLED: {{ .Values.pathIndexingEnabled | quote }}
   SPRING_DATASOURCE_URL: {{ .Values.postgresql.url }}
   {{- if .Values.environment }}

kustomize/base/kustomization.yaml

@@ -7,10 +7,12 @@ resources:
 patches:
   - path: patch-ingress-repository.yaml
   - path: patch-ingress-share.yaml
-  - path: patch-delete-reindexing-config-map.yaml
-  - path: patch-live-path-indexing.yaml
-  - path: patch-live-metadata-indexing.yaml
-  - path: patch-live-content-indexing.yaml
-  - path: patch-live-mediation-indexing.yaml
-  - path: patch-shared-filestore.yaml
+  - path: patch-live-indexing-path.yaml
+  - path: patch-live-indexing-metadata.yaml
+  - path: patch-live-indexing-mediation.yaml
+  - path: patch-live-indexing-content.yaml
   - path: patch-router.yaml
+  - path: patch-tika.yaml
+  - path: patch-transform-misc.yaml
+  - path: patch-transform-libreoffice.yaml
+  - path: patch-transform-imagemagick.yaml

kustomize/base/patch-ingress-repository.yaml

@@ -1,7 +1,7 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
-  name: alfresco-content-services-alfresco-cs-repository
+  name:  alfresco-content-services-alfresco-repository
   annotations:
     external-dns.alpha.kubernetes.io/aws-weight: "100"
     kubernetes.io/ingress.class: default

kustomize/base/patch-ingress-share.yaml

@@ -1,7 +1,7 @@
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
-  name: alfresco-content-services-alfresco-cs-share
+  name: alfresco-content-services-share
   annotations:
     external-dns.alpha.kubernetes.io/aws-weight: "100"
     kubernetes.io/ingress.class: default

kustomize/base/patch-live-content-indexing.yaml → kustomize/base/patch-live-indexing-content.yaml

@@ -18,6 +18,9 @@ spec:
               value: "60000"          # 60 seconds expiry timeout
             - name: JAVA_OPTS
               value: "-Dspring.activemq.packages.trustAll=true"
-            # - name: INPUT_ALFRESCO_CONTENT_BATCH_EVENT_CHANNEL
-            #   value: sjms-batch:contentstore.event?completionTimeout=1000&completionSize=10&aggregationStrategy=#eventAggregator&?consumerCount=20
+            - name: ALFRESCO_SHAREDFILESTORE_BASEURL
+              valueFrom:
+                configMapKeyRef:
+                  name: alfresco-infrastructure
+                  key: SFS_URL_API