Merge pull request #3804 from uselagoon/docs/221_updates

docs: Add docs on "additional volumes"

docs/concepts-advanced/service-types.md

@@ -2,6 +2,8 @@
 
 The below lists all service types that can be defined via `lagoon.type` within a [`docker-compose.yml` file](../concepts-basics/docker-compose-yml.md).
 
+For more information on the `lagoon.volumes.X.path` label, please see [Additional Volumes](../concepts-basics/docker-compose-yml.md/#additional-volumes)
+
 !!! Warning
     Once a `lagoon.type` is defined and the environment is deployed, changing it to a different type is not supported and could result in a broken environment.
 
@@ -11,31 +13,31 @@ Basic container, good to use for most applications that don't have an existing t
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| TCP connection on `3000` | `3000` | Yes | No | `lagoon.service.port`, `lagoon.autogeneratedroute` |
+| TCP connection on `3000` | `3000` | Yes | No | `lagoon.service.port`, `lagoon.autogeneratedroute`, `lagoon.volumes.X.path` |
 
 ## `basic-persistent`
 
 Like `basic`. Will also generate persistent storage, defines mount location via `lagoon.persistent`.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| TCP connection on `3000` | `3000` | Yes | Yes |  `lagoon.service.port`, `lagoon.autogeneratedroute`, `lagoon.persistent`, `lagoon.persistent.name`, `lagoon.persistent.size`, `lagoon.persistent.class` |
+| TCP connection on `3000` | `3000` | Yes | Yes |  `lagoon.service.port`, `lagoon.autogeneratedroute`, `lagoon.persistent`, `lagoon.persistent.name`, `lagoon.persistent.size`, `lagoon.persistent.class` , `lagoon.volumes.X.path` |
 
 ## `cli`
 
 Use for any kind of CLI container \(like PHP, Node.js, etc\). Automatically gets the customer SSH private key that is mounted in `/var/run/secrets/lagoon/sshkey/ssh-privatekey`.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| - | No | No | No | - |
+| - | No | No | No | `lagoon.volumes.X.path` |
 
 ## `cli-persistent`
 
 Like `cli`, expects `lagoon.persistent.name` to be given the name of a service that has persistent storage, which will be mounted under defined `lagoon.persistent` label. Does NOT generate its own persistent storage, only used to mount another service's persistent storage.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| - | No | No | Yes | `lagoon.persistent.name`, `lagoon.persistent` |
+| - | No | No | Yes | `lagoon.persistent.name`, `lagoon.persistent`, `lagoon.volumes.X.path` |
 
 ## `elasticsearch`
 
@@ -115,39 +117,39 @@ NGINX container. No persistent storage.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| `localhost:50000/nginx_status` | `8080` | Yes | No | `lagoon.autogeneratedroute` |
+| `localhost:50000/nginx_status` | `8080` | Yes | No | `lagoon.autogeneratedroute`, `lagoon.volumes.X.path` |
 
 ## `nginx-php`
 
 Like `nginx`, but additionally a `php` container.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| NGINX: `localhost:50000/nginx_status`, PHP: `/usr/sbin/check_fcgi` | `8080` | Yes | No | `lagoon.autogeneratedroute` |
+| NGINX: `localhost:50000/nginx_status`, PHP: `/usr/sbin/check_fcgi` | `8080` | Yes | No | `lagoon.autogeneratedroute`, `lagoon.volumes.X.path` |
 
 ## `nginx-php-persistent`
 
 Like `nginx-php.` Will generate persistent storage, defines mount location via `lagoon.persistent`.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| NGINX: `localhost:50000/nginx_status`, PHP: `/usr/sbin/check_fcgi` | http on `8080` | Yes | Yes | `lagoon.autogeneratedroute`, `lagoon.persistent`, `lagoon.persistent.name`, `lagoon.persistent.size`, `lagoon.persistent.class` |
+| NGINX: `localhost:50000/nginx_status`, PHP: `/usr/sbin/check_fcgi` | http on `8080` | Yes | Yes | `lagoon.autogeneratedroute`, `lagoon.persistent`, `lagoon.persistent.name`, `lagoon.persistent.size`, `lagoon.persistent.class`, `lagoon.volumes.X.path` |
 
 ## `node`
 
 Node.js container. No persistent storage.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| TCP connection on `3000` | `3000` | Yes | No | `lagoon.autogeneratedroute` |
+| TCP connection on `3000` | `3000` | Yes | No | `lagoon.autogeneratedroute`, `lagoon.volumes.X.path` |
 
 ## `node-persistent`
 
 Like `node`. Will generate persistent storage, defines mount location via `lagoon.persistent`.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| TCP connection on `3000` | `3000` | Yes | Yes | `lagoon.autogeneratedroute`, `lagoon.persistent`, `lagoon.persistent.name`, `lagoon.persistent.size`, `lagoon.persistent.class` |
+| TCP connection on `3000` | `3000` | Yes | Yes | `lagoon.autogeneratedroute`, `lagoon.persistent`, `lagoon.persistent.name`, `lagoon.persistent.size`, `lagoon.persistent.class`, `lagoon.volumes.X.path` |
 
 ## `none`
 
@@ -195,15 +197,15 @@ Python container. No persistent storage.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| HTTP connection on `8800` | `8800` | Yes | No | `lagoon.autogeneratedroute` |
+| HTTP connection on `8800` | `8800` | Yes | No | `lagoon.autogeneratedroute`, `lagoon.volumes.X.path` |
 
 ## `python-persistent`
 
 Python container. With persistent storage.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| HTTP connection on `8800` | `8800` | Yes | Yes | `lagoon.autogeneratedroute` |
+| HTTP connection on `8800` | `8800` | Yes | Yes | `lagoon.autogeneratedroute`, `lagoon.volumes.X.path` |
 
 ## `redis`
 
@@ -251,12 +253,12 @@ Use for any kind of worker container \(like queue workers, etc.\) where there is
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| - | No | No | No | - |
+| - | No | No | No | `lagoon.volumes.X.path` |
 
 ## `worker-persistent`
 
 Like `worker`, expects `lagoon.persistent.name` to be given the name of a service that has persistent storage, which will be mounted under defined `lagoon.persistent` label. Does NOT generate its own persistent storage, only used to mount another service's persistent storage.
 
 | Healthcheck | Exposed Ports | Auto Generated Routes | Storage | Additional customization parameter |
 | :--- | :--- | :--- | :--- | :--- |
-| - | No | No | Yes | `lagoon.persistent.name`, `lagoon.persistent` |
+| - | No | No | Yes | `lagoon.persistent.name`, `lagoon.persistent`, `lagoon.volumes.X.path` |

docs/concepts-basics/docker-compose-yml.md

@@ -16,8 +16,6 @@ This is intentional, as the `docker-compose` file is there to define your local
 Here a straightforward example of a `docker-compose.yml` file for Drupal:
 
 ```yaml title="docker-compose.yml"
-version: '2.3'
-
 x-lagoon-project:
   # Lagoon project name (leave `&lagoon-project` when you edit this)
   &lagoon-project drupal-example
@@ -148,12 +146,79 @@ In many cases, Lagoon knows where that persistent storage needs to go. For examp
 * `lagoon.persistent.size` - The size of persistent storage you require (Lagoon usually gives you minimum 5G of persistent storage, if you need more, define it here).
 * `lagoon.persistent.class` - By default Lagoon automatically assigns the right storage class for your service (like SSDs for MySQL, bulk storage for Nginx, etc.). If you need to overwrite this, you can do so here. This is highly dependent on the underlying Kubernetes/OpenShift that Lagoon runs on. Contact {{ defaults.helpstring }} for this.
 
+See the [Additional Volumes](#additional-volumes) section on how to add more volumes
+
 ### Auto-generated Routes
 
 The docker-compose.yml file also supports per-service enabling and disabling of [autogenerated routes](./lagoon-yml.md#routes)
 
 * `lagoon.autogeneratedroute: false` label will stop a route from being automatically created for that service. It can be applied to all services with autogenerated routes, but is mostly useful for the [`basic`](../concepts-advanced/service-types.md#basic) and [`basic-persistent`](../concepts-advanced/service-types.md#basic-persistent) service types when used to create an additional internal-facing service for a database service or similar. The inverse is also true - it will enable an auto-generated route for a service when the .lagoon.yml file [disables them](lagoon-yml.md#routesautogenerate).
 
+## Additional Volumes
+
+In situations where more than one persistent volume is required by a service (or services), Lagoon can support the creation of multiple additional volumes.
+
+To do this, the volume should be created in the docker-compose.yml with a `lagoon.type` label - currently only "persistent" is supported.
+
+```yaml title="docker-compose.yml volumes snippet"
+volumes:
+  extravol:
+    labels:
+      lagoon.type: persistent
+```
+
+This volume can then be referenced in each service that requires it, along with the path
+
+```yaml title="docker-compose.yml volume usage snippet"
+services:
+  basic:
+    build:
+      context: .
+      dockerfile: basic.dockerfile
+    labels:
+      lagoon.type: basic-persistent
+      lagoon.persistent: /data # basic-persistent provides a default volume, this needs to be defined
+      lagoon.volumes.extravol.path: /extra
+    volumes: # these volumes are ignored by lagoon, only the labels above are consumed to handle volume mapping
+      - ./data:/data:delegated
+      - extravol:/extra
+```
+Note that the original volume created for this service still exists.
+
+### Supported persistent volume flags
+* The default size is 5Gi - but can be modified with a `lagoon.persistent.size: XXGi` label on the volume.
+* By default, Lagoon will back up all persistent volumes, but this can be disabled with a `lagoon.backup: false` label on the volume.
+
+```yaml title="docker-compose.yml volumes snippet"
+volumes:
+  extravol:
+    labels:
+      lagoon.type: persistent
+      lagoon.persistent.size: 10Gi
+      lagoon.backup: false
+```
+
+### Default persistent volumes
+* If a service type provides a default persistent volume (nginx-php-persistent, basic-persistent, node-persistent, python-persistent), it will be created and named as the name of the service (unless the lagoon.persistent.name label is assigned).
+* If a volume defined in the volumes block is also named the same as a default volume would be, it will be ignored and only the default volume will be created.
+* If a lagoon.volumes.${volume_name}.path is also defined on a service that has a default volume (or a volume linked to it using the lagoon.persistent.name label, it will also be ignored and the value of the lagoon.persistent path will be used.
+### Ignored volume
+* If a volume defined in the volumes block has a label lagoon.type: none it will not be created.
+* If a volume defined in the volumes block has a label lagoon.type of a type (currently only persistent), but it is not consumed by any service using a lagoon.volumes.${volume_name}.path, then the volume will not be created.
+### Maximum number of volumes
+* Currently a hard-coded limit of 6 volumes is in place, this may be made configurable in the future.
+### Supported service types
+Currently, the only service types that support additional volumes are the following
+
+* basic / basic-persistent
+* worker / worker-persistent
+* node / node-persistent
+* python / python-persistent
+* nginx / nginx-php / nginx-php-persistent
+* cli / cli-persistent
+
+Adding a volume to any other type will result in an error during a build.
+
 ## Multi-Container Pods
 
 Kubernetes and OpenShift don't deploy plain containers. Instead, they deploy pods, with each one or more containers. Usually Lagoon creates a single pod with a container inside for each defined `docker-compose` service. For some cases, we need to put two containers inside a single pod, as these containers are so dependent on each other that they should always stay together. An example for such a situation is the PHP and NGINX containers that both contain PHP code of a web application like Drupal.
@@ -244,7 +309,7 @@ no such service: container:amazeeio-ssh-agent
 
 BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner.
 
-With the release of Lagoon v2.11.0, Lagoon now provides support for more advanced BuildKit-based docker compose builds. To enable BuildKit for your Project or Environment, add `DOCKER_BUILDKIT=1` as a build-time variable to your Lagoon project or environment.
+With the release of Lagoon v2.11.0, Lagoon now provides support for more advanced BuildKit-based docker compose builds, and in Lagoon v2.21.0, BuildKit is now enabled by default for all builds. To specifically disable BuildKit for your Project or Environment, add `DOCKER_BUILDKIT=0` as a build-time variable to your Lagoon project or environment.
 
 ## Docker Compose Errors in Lagoon Builds
 

docs/docker-images/deprecated-images.md

@@ -4,8 +4,20 @@ From time to time, the Lagoon team may need to mark images as deprecated.
 
 This is conveyed in a "sh.lagoon.image.deprecated" . It can be viewed in Docker Desktop, via a `docker inspect` command, or in future releases of Lagoon, highlighted in a build.
 
+![Deprecated Images in a build:](../images/deprecated-images-build.png)
+
 If the image has a suggested replacement, it will be conveyed in a "sh.lagoon.image.deprecated.suggested" label attached the deprecated image.
 
+```shell title="docker inspect output showing deprecated image"
+$ docker inspect amazeeio/mongo:latest
+...
+{
+  ...
+  "sh.lagoon.image.deprecated.status": "replaced",
+  "sh.lagoon.image.deprecated.suggested": "docker.io/uselagoon/mongo-4"
+}
+```
+
 ## Changing the image
 
 In all cases, changing to a suggested or updated image will require a change in your codebase. Any reference to the image in a Dockerfile, or in your docker-compose.yml will need to be updated.

docs/releases/2.21.0.md

@@ -2,17 +2,24 @@
 * lagoon [v2.21.0](https://github.com/uselagoon/lagoon/releases/tag/v2.21.0)
 * lagoon-ui [core-v2.21.0](https://github.com/uselagoon/lagoon-ui/releases/tag/core-v2.21.0)
 * lagoon-build-deploy [core-v2.21.0](https://github.com/uselagoon/build-deploy-tool/releases/tag/core-v2.21.0)
-* lagoon-core chart [x.xx.x](https://github.com/uselagoon/lagoon-charts/releases/tag/lagoon-core-x.xx.0)
+* lagoon-core chart [1.48.0](https://github.com/uselagoon/lagoon-charts/releases/tag/lagoon-core-1.48.0)
+* lagoon-remote chart [0.94.0](https://github.com/uselagoon/lagoon-charts/releases/tag/lagoon-remote-0.94.0)
 
 ## Release Notes
 
-tbc
+This release is a standard Lagoon release.
 
 ## Upgrades
 
 ### Upgrade to v2.20.x first
 You *must* upgrade to Lagoon v2.20.x before upgrading to v2.21.0. Ensure you read all the linked release notes.
+The upgrade to v2.20.x is to ensure that the functionality used to enable the feature flags in the broker service is present prior to being called as a pre-upgrade job in v2.22.x
+
+### Keycloak 24
+This release includes an update to Keycloak 24. This should be a seamless experience for all users.
 
 ## Deprecations
 
-tbc
+### Organization & Notification GraphQL commands
+Some of the GraphQL commands used to modify organization user roles have changed. The API schema documents them.
+Some of the GRaphQL command inputs used to interact with Notifications may have changed. The API schema documents them.

docs/releases/2.22.0.md

@@ -0,0 +1,18 @@
+#### Release Links
+* lagoon [v2.22.0](https://github.com/uselagoon/lagoon/releases/tag/v2.22.0)
+* lagoon-ui [core-v2.22.0](https://github.com/uselagoon/lagoon-ui/releases/tag/core-v2.22.0)
+* lagoon-build-deploy [core-v2.22.0](https://github.com/uselagoon/build-deploy-tool/releases/tag/core-v2.22.0)
+* lagoon-core chart [x.xx.x](https://github.com/uselagoon/lagoon-charts/releases/tag/lagoon-core-x.xx.0)
+* lagoon-remote chart [x.xx.x](https://github.com/uselagoon/lagoon-charts/releases/tag/lagoon-remote-x.xx.x)
+
+## Release Notes
+
+tbc - Not released yet
+
+## Upgrades
+
+tbc
+
+## Deprecations
+
+tbc

docs/using-lagoon-the-basics/lagoon-build-errors-and-warnings.md

@@ -27,3 +27,9 @@ There is an error in your Docker Compose file, most likely relating to a malform
 ```
 
 There is an error in your Docker Compose file, most likely relating to a malformed or misused alias or anchor. The error message should help you understand where.
+
+## Deprecated Images
+
+Please see the [dedicated page](../docker-images/deprecated-images.md) on how to handle this warning
+
+![Deprecated Images in a build:](../images/deprecated-images-build.png)