From 8bf75bd4f28b27483ba2d7954ab61f406d8a7db5 Mon Sep 17 00:00:00 2001 From: Nolan Brubaker Date: Fri, 8 May 2020 18:40:29 -0400 Subject: [PATCH] Add change log and docs site for v1.4.0-beta.1 (#2523) * Update site for v1.4.0-beta.1 Signed-off-by: Nolan Brubaker * Add v1.4.0-beta.1 changelogs Signed-off-by: Nolan Brubaker * Update upgrade link to v1.4 Signed-off-by: Nolan Brubaker * Correct docs links in changelog Signed-off-by: Nolan Brubaker --- CHANGELOG.md | 2 + changelogs/CHANGELOG-1.4.md | 53 +++ site/_config.yml | 7 + site/_data/toc-mapping.yml | 1 + site/_data/v1-4-pre-toc.yml | 87 ++++ site/docs/v1.4-pre/README.md | 52 +++ site/docs/v1.4-pre/api-types/README.md | 18 + site/docs/v1.4-pre/api-types/backup.md | 143 ++++++ .../api-types/backupstoragelocation.md | 45 ++ site/docs/v1.4-pre/api-types/restore.md | 89 ++++ site/docs/v1.4-pre/api-types/schedule.md | 132 ++++++ .../api-types/volumesnapshotlocation.md | 38 ++ site/docs/v1.4-pre/backup-reference.md | 9 + site/docs/v1.4-pre/basic-install.md | 64 +++ site/docs/v1.4-pre/build-from-source.md | 158 +++++++ site/docs/v1.4-pre/code-standards.md | 133 ++++++ .../docs/v1.4-pre/contributions/ibm-config.md | 96 ++++ site/docs/v1.4-pre/contributions/minio.md | 267 +++++++++++ .../v1.4-pre/contributions/oracle-config.md | 245 ++++++++++ site/docs/v1.4-pre/csi.md | 32 ++ site/docs/v1.4-pre/custom-plugins.md | 91 ++++ site/docs/v1.4-pre/customize-installation.md | 298 ++++++++++++ site/docs/v1.4-pre/debugging-install.md | 71 +++ site/docs/v1.4-pre/debugging-restores.md | 106 +++++ site/docs/v1.4-pre/development.md | 30 ++ site/docs/v1.4-pre/disaster-case.md | 41 ++ site/docs/v1.4-pre/examples.md | 63 +++ site/docs/v1.4-pre/faq.md | 44 ++ site/docs/v1.4-pre/hooks.md | 89 ++++ site/docs/v1.4-pre/how-velero-works.md | 82 ++++ site/docs/v1.4-pre/image-tagging.md | 21 + site/docs/v1.4-pre/img/README.md | 1 + site/docs/v1.4-pre/img/backup-process.png | Bin 0 -> 33630 bytes site/docs/v1.4-pre/img/velero.png | Bin 0 -> 45564 bytes site/docs/v1.4-pre/locations.md | 164 +++++++ site/docs/v1.4-pre/migration-case.md | 50 ++ site/docs/v1.4-pre/namespace.md | 19 + site/docs/v1.4-pre/on-premises.md | 24 + site/docs/v1.4-pre/output-file-format.md | 222 +++++++++ site/docs/v1.4-pre/overview-plugins.md | 27 ++ site/docs/v1.4-pre/rbac.md | 47 ++ site/docs/v1.4-pre/release-instructions.md | 83 ++++ site/docs/v1.4-pre/restic.md | 436 ++++++++++++++++++ site/docs/v1.4-pre/restore-reference.md | 90 ++++ site/docs/v1.4-pre/run-locally.md | 49 ++ .../docs/v1.4-pre/self-signed-certificates.md | 31 ++ site/docs/v1.4-pre/start-contributing.md | 21 + site/docs/v1.4-pre/support-process.md | 41 ++ site/docs/v1.4-pre/supported-providers.md | 85 ++++ site/docs/v1.4-pre/troubleshooting.md | 123 +++++ site/docs/v1.4-pre/uninstalling.md | 8 + site/docs/v1.4-pre/upgrade-to-1.4.md | 71 +++ site/docs/v1.4-pre/velero-install.md | 45 ++ site/docs/v1.4-pre/vendoring-dependencies.md | 18 + site/docs/v1.4-pre/website-guidelines.md | 46 ++ site/docs/v1.4-pre/zenhub.md | 15 + 56 files changed, 4323 insertions(+) create mode 100644 changelogs/CHANGELOG-1.4.md create mode 100644 site/_data/v1-4-pre-toc.yml create mode 100644 site/docs/v1.4-pre/README.md create mode 100644 site/docs/v1.4-pre/api-types/README.md create mode 100644 site/docs/v1.4-pre/api-types/backup.md create mode 100644 site/docs/v1.4-pre/api-types/backupstoragelocation.md create mode 100644 site/docs/v1.4-pre/api-types/restore.md create mode 100644 site/docs/v1.4-pre/api-types/schedule.md create mode 100644 site/docs/v1.4-pre/api-types/volumesnapshotlocation.md create mode 100644 site/docs/v1.4-pre/backup-reference.md create mode 100644 site/docs/v1.4-pre/basic-install.md create mode 100644 site/docs/v1.4-pre/build-from-source.md create mode 100644 site/docs/v1.4-pre/code-standards.md create mode 100644 site/docs/v1.4-pre/contributions/ibm-config.md create mode 100644 site/docs/v1.4-pre/contributions/minio.md create mode 100644 site/docs/v1.4-pre/contributions/oracle-config.md create mode 100644 site/docs/v1.4-pre/csi.md create mode 100644 site/docs/v1.4-pre/custom-plugins.md create mode 100644 site/docs/v1.4-pre/customize-installation.md create mode 100644 site/docs/v1.4-pre/debugging-install.md create mode 100644 site/docs/v1.4-pre/debugging-restores.md create mode 100644 site/docs/v1.4-pre/development.md create mode 100644 site/docs/v1.4-pre/disaster-case.md create mode 100644 site/docs/v1.4-pre/examples.md create mode 100644 site/docs/v1.4-pre/faq.md create mode 100644 site/docs/v1.4-pre/hooks.md create mode 100644 site/docs/v1.4-pre/how-velero-works.md create mode 100644 site/docs/v1.4-pre/image-tagging.md create mode 100644 site/docs/v1.4-pre/img/README.md create mode 100644 site/docs/v1.4-pre/img/backup-process.png create mode 100644 site/docs/v1.4-pre/img/velero.png create mode 100644 site/docs/v1.4-pre/locations.md create mode 100644 site/docs/v1.4-pre/migration-case.md create mode 100644 site/docs/v1.4-pre/namespace.md create mode 100644 site/docs/v1.4-pre/on-premises.md create mode 100644 site/docs/v1.4-pre/output-file-format.md create mode 100644 site/docs/v1.4-pre/overview-plugins.md create mode 100644 site/docs/v1.4-pre/rbac.md create mode 100644 site/docs/v1.4-pre/release-instructions.md create mode 100644 site/docs/v1.4-pre/restic.md create mode 100644 site/docs/v1.4-pre/restore-reference.md create mode 100644 site/docs/v1.4-pre/run-locally.md create mode 100644 site/docs/v1.4-pre/self-signed-certificates.md create mode 100644 site/docs/v1.4-pre/start-contributing.md create mode 100644 site/docs/v1.4-pre/support-process.md create mode 100644 site/docs/v1.4-pre/supported-providers.md create mode 100644 site/docs/v1.4-pre/troubleshooting.md create mode 100644 site/docs/v1.4-pre/uninstalling.md create mode 100644 site/docs/v1.4-pre/upgrade-to-1.4.md create mode 100644 site/docs/v1.4-pre/velero-install.md create mode 100644 site/docs/v1.4-pre/vendoring-dependencies.md create mode 100644 site/docs/v1.4-pre/website-guidelines.md create mode 100644 site/docs/v1.4-pre/zenhub.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a6307c947..f4ee57e346 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ * [CHANGELOG-1.3.md][13] ## Development release: + * [CHANGELOG-1.4.md][14] * [Unreleased Changes][0] ## Older releases: @@ -19,6 +20,7 @@ * [CHANGELOG-0.3.md][1] +[14]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.4.md [13]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.3.md [12]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.2.md [11]: https://github.com/vmware-tanzu/velero/blob/master/changelogs/CHANGELOG-1.1.md diff --git a/changelogs/CHANGELOG-1.4.md b/changelogs/CHANGELOG-1.4.md new file mode 100644 index 0000000000..5d0cee640c --- /dev/null +++ b/changelogs/CHANGELOG-1.4.md @@ -0,0 +1,53 @@ +## v1.4.0-beta.1 +### 2020-05-08 + +### Download +https://github.com/vmware-tanzu/velero/releases/tag/v1.4.0-beta.1 + +### Container Image +`velero/velero:v1.4.0-beta.1` + +### Documentation +https://velero.io/docs/v1.4-pre/ + +### Upgrading +https://velero.io/docs/v1.4-pre/upgrade-to-1.4/ + +### Download +https://github.com/vmware-tanzu/velero/releases/tag/v1.4.0-beta.1 + +### Highlights + + * Added beta-level CSI support + * Added custom CA certificate support + * Backup progress reporting + +### All Changes + * Add details of CSI volumesnapshotcontents associated with a backup to `velero backup describe` when the `EnableCSI` feature flag is given on the velero client. (#2448, @nrb) + * report backup progress (number of items backed up so far out of an estimated total number of items) during backup in the logs and as status fields on the Backup custom resource (#2440, @skriss) + * allow feature flags to be passed from install CLI (#2503, @ashish) + * sync backups' CSI API objects into the cluster as part of the backup sync controller (#2496, @ashish) + * bug fix: in error location logging hook, if the item logged under the `error` key doesn't implement the `error` interface, don't return an error since this is a valid scenario (#2487, @skriss) + * bug fix: in CRD restore plugin, don't use runtime.DefaultUnstructuredConverter.FromUnstructured(...) to avoid conversion issues when float64 fields contain int values (#2484, @skriss) + * during backup deletion also delete CSI volumesnapshotcontents that were created as a part of the backup but the associated volumesnapshot object does not exist (#2480, @ashish) + * If plugins don't support the `--features` flag, don't pass it to them. Also, update the standard plugin server to ignore unknown flags. (#2479, @skriss) + * At backup time, if a CustomResourceDefinition appears to have been created via the v1beta1 endpoint, retrieve it from the v1beta1 endpoint instead of simply changing the APIVersion. (#2478, @nrb) + * update container base images from ubuntu:bionic to ubuntu:focal (#2471, @skriss) + * bug fix: when a resource includes/excludes list contains unresolvable items, don't remove them from the list, so that the list doesn't inadvertently end up matching *all* resources. (#2462, @skriss) + * Azure: add support for getting storage account key for restic directly from an environment variable (#2455, @jaygridley) + * Support to skip VSL validation for the backup having SnapshotVolumes set to false or created with `--snapshot-volumes=false` (#2450, @mynktl) + * during backup deletion also delete CSI volumesnapshots that were created as a part of the backup (#2411, @ashish) + * clarify the wording for restore describe for namespaces included Instead of showing it as "*" explicitly mention that all the namespaces from the backup object are included. (#1918, @raghavendrabhat) + * When the EnableCSI feature flag is provided, upload CSI VolumeSnapshots and VolumeSnapshotContents to object storage as gzipped JSON. (#2323, @nrb) + * bug fix: populate namespace in logs for backup errors (#2438, @skriss) + * bug fix: save PodVolumeBackup manifests to object storage even if the volume was empty, so that on restore, the PV is dynamically reprovisioned if applicable (#2390, @skriss) + * adding annotations on backup CRD for k8s major, minor and git versions (#2346, @brito) + * bump Kubernetes module dependencies to v0.17.4 to get fix for https://github.com/kubernetes/kubernetes/issues/86149 (#2407, @skriss) + * Adding new restoreItemAction for PVC to update the selected-node annotation (#2377, @mynktl) + * Added a --cacert flag to the install command to provide the CA bundle to use when verifying TLS connections to object storage (#2368, @mansam) + * Added a `--cacert` flag to the velero client describe, download, and logs commands to allow passing a path to a certificate to use when verifying TLS connections to object storage. Also added a corresponding client config option called `cacert` which takes a path to a certificate bundle to use as a default when `--cacert` is not specified. (#2364, @mansam) + * support setting a custom CA certificate on a BSL to use when verifying TLS connections (#2353, @mansam) + * add CSI snapshot API types into default restore priorities (#2318, @ashish) + * refactoring: wait for all informer caches to sync before running controllers (#2299, @skriss) + * refactor restore code to lazily resolve resources via discovery and eliminate second restore loop for instances of restored CRDs (#2248, @skriss) + * upgrade to go 1.14 and migrate from `dep` to go modules (#2214, @skriss) diff --git a/site/_config.yml b/site/_config.yml index 66ee3030ba..a0a8c56ad3 100644 --- a/site/_config.yml +++ b/site/_config.yml @@ -55,6 +55,12 @@ defaults: version: master gh: https://github.com/vmware-tanzu/velero/tree/master layout: "docs" + - scope: + path: docs/v1.4-pre + values: + version: v1.4-pre + gh: https://github.com/vmware-tanzu/velero/tree/v1.4.0-beta.1 + layout: "docs" - scope: path: docs/v1.3.2 values: @@ -173,6 +179,7 @@ versioning: true latest: v1.3.2 versions: - master +- v1.4-pre - v1.3.2 - v1.3.1 - v1.3.0 diff --git a/site/_data/toc-mapping.yml b/site/_data/toc-mapping.yml index 4229d6f62d..a736117cf2 100644 --- a/site/_data/toc-mapping.yml +++ b/site/_data/toc-mapping.yml @@ -3,6 +3,7 @@ # that the navigation for older versions still work. master: master-toc +v1.4-pre: v1-4-pre-toc v1.3.2: v1-3-2-toc v1.3.1: v1-3-1-toc v1.3.0: v1-3-0-toc diff --git a/site/_data/v1-4-pre-toc.yml b/site/_data/v1-4-pre-toc.yml new file mode 100644 index 0000000000..42b47b0170 --- /dev/null +++ b/site/_data/v1-4-pre-toc.yml @@ -0,0 +1,87 @@ +toc: + - title: Introduction + subfolderitems: + - page: About Velero + url: /index.html + - page: How Velero works + url: /how-velero-works + - page: About locations + url: /locations + - title: Install + subfolderitems: + - page: Basic Install + url: /basic-install + - page: Customize Installation + url: /customize-installation + - page: Upgrade to 1.4 + url: /upgrade-to-1.4 + - page: Supported providers + url: /supported-providers + - page: Evaluation install + url: /contributions/minio + - page: Restic integration + url: /restic + - page: Examples + url: /examples + - page: Uninstalling + url: /uninstalling + - title: Use + subfolderitems: + - page: Disaster recovery + url: /disaster-case + - page: Cluster migration + url: /migration-case + - page: Backup reference + url: /backup-reference + - page: Restore reference + url: /restore-reference + - page: Run in any namespace + url: /namespace + - page: Extend with hooks + url: /hooks + - page: CSI Support (beta) + url: /csi + - page: Verifying Self-signed Certificates + url: /self-signed-certificates + - title: Plugins + subfolderitems: + - page: Overview + url: /overview-plugins + - page: Custom plugins + url: /custom-plugins + - title: Troubleshoot + subfolderitems: + - page: Troubleshooting + url: /troubleshooting + - page: Troubleshoot an install or setup + url: /debugging-install + - page: Troubleshoot a restore + url: /debugging-restores + - page: Troubleshoot Restic + url: /restic#troubleshooting + - title: Contribute + subfolderitems: + - page: Start Contributing + url: /start-contributing + - page: Development + url: /development + - page: Build from source + url: /build-from-source + - page: Run locally + url: /run-locally + - page: Code standards + url: /code-standards + - page: Website guidelines + url: /website-guidelines + - title: More information + subfolderitems: + - page: Backup file format + url: /output-file-format + - page: API types + url: /api-types + - page: FAQ + url: /faq + - page: ZenHub + url: /zenhub + - page: Support Process + url: /support-process diff --git a/site/docs/v1.4-pre/README.md b/site/docs/v1.4-pre/README.md new file mode 100644 index 0000000000..6f251c47de --- /dev/null +++ b/site/docs/v1.4-pre/README.md @@ -0,0 +1,52 @@ +![100] + +[![Build Status][1]][2] + +## Overview + +Velero (formerly Heptio Ark) gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes. You can run Velero with a cloud provider or on-premises. Velero lets you: + +* Take backups of your cluster and restore in case of loss. +* Migrate cluster resources to other clusters. +* Replicate your production cluster to development and testing clusters. + +Velero consists of: + +* A server that runs on your cluster +* A command-line client that runs locally + +## Documentation + +This site is our documentation home with installation instructions, plus information about customizing Velero for your needs, architecture, extending Velero, contributing to Velero and more. + +Please use the version selector at the top of the site to ensure you are using the appropriate documentation for your version of Velero. + +## Troubleshooting + +If you encounter issues, review the [troubleshooting docs][30], [file an issue][4], or talk to us on the [#velero channel][25] on the Kubernetes Slack server. + +## Contributing + +If you are ready to jump in and test, add code, or help with documentation, follow the instructions on our [Start contributing](https://velero.io/docs/v1.4.0-beta.1/start-contributing/) documentation for guidance on how to setup Velero for development. + +## Changelog + +See [the list of releases][6] to find out about feature changes. + +[1]: https://travis-ci.org/vmware-tanzu/velero.svg?branch=master +[2]: https://travis-ci.org/vmware-tanzu/velero + +[4]: https://github.com/vmware-tanzu/velero/issues +[6]: https://github.com/vmware-tanzu/velero/releases + +[9]: https://kubernetes.io/docs/setup/ +[10]: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-homebrew-on-macos +[11]: https://kubernetes.io/docs/tasks/tools/install-kubectl/#tabset-1 +[12]: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/README.md +[14]: https://github.com/kubernetes/kubernetes +[24]: https://groups.google.com/forum/#!forum/projectvelero +[25]: https://kubernetes.slack.com/messages/velero + +[30]: troubleshooting.md + +[100]: img/velero.png diff --git a/site/docs/v1.4-pre/api-types/README.md b/site/docs/v1.4-pre/api-types/README.md new file mode 100644 index 0000000000..2ba83cc5e1 --- /dev/null +++ b/site/docs/v1.4-pre/api-types/README.md @@ -0,0 +1,18 @@ +# Table of Contents + +## API types + +Here we list the API types that have some functionality that you can only configure via json/yaml vs the `velero` cli +(hooks) + +* [Backup][1] +* [Restore][2] +* [Schedule][3] +* [BackupStorageLocation][4] +* [VolumeSnapshotLocation][5] + +[1]: backup.md +[2]: restore.md +[3]: schedule.md +[4]: backupstoragelocation.md +[5]: volumesnapshotlocation.md diff --git a/site/docs/v1.4-pre/api-types/backup.md b/site/docs/v1.4-pre/api-types/backup.md new file mode 100644 index 0000000000..70120a9210 --- /dev/null +++ b/site/docs/v1.4-pre/api-types/backup.md @@ -0,0 +1,143 @@ +# Backup API Type + +## Use + +The `Backup` API type is used as a request for the Velero server to perform a backup. Once created, the +Velero Server immediately starts the backup process. + +## API GroupVersion + +Backup belongs to the API group version `velero.io/v1`. + +## Definition + +Here is a sample `Backup` object with each of the fields documented: + +```yaml +# Standard Kubernetes API Version declaration. Required. +apiVersion: velero.io/v1 +# Standard Kubernetes Kind declaration. Required. +kind: Backup +# Standard Kubernetes metadata. Required. +metadata: + # Backup name. May be any valid Kubernetes object name. Required. + name: a + # Backup namespace. Must be the namespace of the Velero server. Required. + namespace: velero +# Parameters about the backup. Required. +spec: + # Array of namespaces to include in the backup. If unspecified, all namespaces are included. + # Optional. + includedNamespaces: + - '*' + # Array of namespaces to exclude from the backup. Optional. + excludedNamespaces: + - some-namespace + # Array of resources to include in the backup. Resources may be shortcuts (e.g. 'po' for 'pods') + # or fully-qualified. If unspecified, all resources are included. Optional. + includedResources: + - '*' + # Array of resources to exclude from the backup. Resources may be shortcuts (e.g. 'po' for 'pods') + # or fully-qualified. Optional. + excludedResources: + - storageclasses.storage.k8s.io + # Whether or not to include cluster-scoped resources. Valid values are true, false, and + # null/unset. If true, all cluster-scoped resources are included (subject to included/excluded + # resources and the label selector). If false, no cluster-scoped resources are included. If unset, + # all cluster-scoped resources are included if and only if all namespaces are included and there are + # no excluded namespaces. Otherwise, if there is at least one namespace specified in either + # includedNamespaces or excludedNamespaces, then the only cluster-scoped resources that are backed + # up are those associated with namespace-scoped resources included in the backup. For example, if a + # PersistentVolumeClaim is included in the backup, its associated PersistentVolume (which is + # cluster-scoped) would also be backed up. + includeClusterResources: null + # Individual objects must match this label selector to be included in the backup. Optional. + labelSelector: + matchLabels: + app: velero + component: server + # Whether or not to snapshot volumes. This only applies to PersistentVolumes for Azure, GCE, and + # AWS. Valid values are true, false, and null/unset. If unset, Velero performs snapshots as long as + # a persistent volume provider is configured for Velero. + snapshotVolumes: null + # Where to store the tarball and logs. + storageLocation: aws-primary + # The list of locations in which to store volume snapshots created for this backup. + volumeSnapshotLocations: + - aws-primary + - gcp-primary + # The amount of time before this backup is eligible for garbage collection. If not specified, + # a default value of 30 days will be used. The default can be configured on the velero server + # by passing the flag --default-backup-ttl. + ttl: 24h0m0s + # Actions to perform at different times during a backup. The only hook currently supported is + # executing a command in a container in a pod using the pod exec API. Optional. + hooks: + # Array of hooks that are applicable to specific resources. Optional. + resources: + - + # Name of the hook. Will be displayed in backup log. + name: my-hook + # Array of namespaces to which this hook applies. If unspecified, the hook applies to all + # namespaces. Optional. + includedNamespaces: + - '*' + # Array of namespaces to which this hook does not apply. Optional. + excludedNamespaces: + - some-namespace + # Array of resources to which this hook applies. The only resource supported at this time is + # pods. + includedResources: + - pods + # Array of resources to which this hook does not apply. Optional. + excludedResources: [] + # This hook only applies to objects matching this label selector. Optional. + labelSelector: + matchLabels: + app: velero + component: server + # An array of hooks to run before executing custom actions. Currently only "exec" hooks are supported. + pre: + - + # The type of hook. This must be "exec". + exec: + # The name of the container where the command will be executed. If unspecified, the + # first container in the pod will be used. Optional. + container: my-container + # The command to execute, specified as an array. Required. + command: + - /bin/uname + - -a + # How to handle an error executing the command. Valid values are Fail and Continue. + # Defaults to Fail. Optional. + onError: Fail + # How long to wait for the command to finish executing. Defaults to 30 seconds. Optional. + timeout: 10s + # An array of hooks to run after all custom actions and additional items have been + # processed. Currently only "exec" hooks are supported. + post: + # Same content as pre above. +# Status about the Backup. Users should not set any data here. +status: + # The version of this Backup. The only version currently supported is 1. + version: 1 + # The date and time when the Backup is eligible for garbage collection. + expiration: null + # The current phase. Valid values are New, FailedValidation, InProgress, Completed, PartiallyFailed, Failed. + phase: "" + # An array of any validation errors encountered. + validationErrors: null + # Date/time when the backup started being processed. + startTimestamp: 2019-04-29T15:58:43Z + # Date/time when the backup finished being processed. + completionTimestamp: 2019-04-29T15:58:56Z + # Number of volume snapshots that Velero tried to create for this backup. + volumeSnapshotsAttempted: 2 + # Number of volume snapshots that Velero successfully created for this backup. + volumeSnapshotsCompleted: 1 + # Number of warnings that were logged by the backup. + warnings: 2 + # Number of errors that were logged by the backup. + errors: 0 + +``` diff --git a/site/docs/v1.4-pre/api-types/backupstoragelocation.md b/site/docs/v1.4-pre/api-types/backupstoragelocation.md new file mode 100644 index 0000000000..8fb3a0974f --- /dev/null +++ b/site/docs/v1.4-pre/api-types/backupstoragelocation.md @@ -0,0 +1,45 @@ +# Velero Backup Storage Locations + +## Backup Storage Location + +Velero can store backups in a number of locations. These are represented in the cluster via the `BackupStorageLocation` CRD. + +Velero must have at least one `BackupStorageLocation`. By default, this is expected to be named `default`, however the name can be changed by specifying `--default-backup-storage-location` on `velero server`. Backups that do not explicitly specify a storage location will be saved to this `BackupStorageLocation`. + +A sample YAML `BackupStorageLocation` looks like the following: + +```yaml +apiVersion: velero.io/v1 +kind: BackupStorageLocation +metadata: + name: default + namespace: velero +spec: + backupSyncPeriod: 2m0s + provider: aws + objectStorage: + bucket: myBucket + config: + region: us-west-2 + profile: "default" +``` + +### Parameter Reference + +The configurable parameters are as follows: + +#### Main config parameters + +| Key | Type | Default | Meaning | +| --- | --- | --- | --- | +| `provider` | String | Required Field | The name for whichever object storage provider will be used to store the backups. See [your object storage provider's plugin documentation][0] for the appropriate value to use. | +| `objectStorage` | ObjectStorageLocation | Required Field | Specification of the object storage for the given provider. | +| `objectStorage/bucket` | String | Required Field | The storage bucket where backups are to be uploaded. | +| `objectStorage/prefix` | String | Optional Field | The directory inside a storage bucket where backups are to be uploaded. | +| `objectStorage/caCert` | String | Optional Field | A base64 encoded CA bundle to be used when verifying TLS connections | +| `config` | map[string]string | None (Optional) | Provider-specific configuration keys/values to be passed to the object store plugin. See [your object storage provider's plugin documentation][0] for details. | +| `accessMode` | String | `ReadWrite` | How Velero can access the backup storage location. Valid values are `ReadWrite`, `ReadOnly`. | +| `backupSyncPeriod` | metav1.Duration | Optional Field | How frequently Velero should synchronize backups in object storage. Default is Velero's server backup sync period. Set this to `0s` to disable sync. | + + +[0]: ../supported-providers.md diff --git a/site/docs/v1.4-pre/api-types/restore.md b/site/docs/v1.4-pre/api-types/restore.md new file mode 100644 index 0000000000..2044d47253 --- /dev/null +++ b/site/docs/v1.4-pre/api-types/restore.md @@ -0,0 +1,89 @@ +# Restore API Type + +## Use + +The `Restore` API type is used as a request for the Velero server to perform a Restore. Once created, the +Velero Server immediately starts the Restore process. + +## API GroupVersion + +Restore belongs to the API group version `velero.io/v1`. + +## Definition + +Here is a sample `Restore` object with each of the fields documented: + +```yaml +# Standard Kubernetes API Version declaration. Required. +apiVersion: velero.io/v1 +# Standard Kubernetes Kind declaration. Required. +kind: Restore +# Standard Kubernetes metadata. Required. +metadata: + # Restore name. May be any valid Kubernetes object name. Required. + name: a-very-special-backup-0000111122223333 + # Restore namespace. Must be the namespace of the Velero server. Required. + namespace: velero +# Parameters about the restore. Required. +spec: + # BackupName is the unique name of the Velero backup to restore from. + backupName: a-very-special-backup + # Array of namespaces to include in the restore. If unspecified, all namespaces are included. + # Optional. + includedNamespaces: + - '*' + # Array of namespaces to exclude from the restore. Optional. + excludedNamespaces: + - some-namespace + # Array of resources to include in the restore. Resources may be shortcuts (e.g. 'po' for 'pods') + # or fully-qualified. If unspecified, all resources are included. Optional. + includedResources: + - '*' + # Array of resources to exclude from the restore. Resources may be shortcuts (e.g. 'po' for 'pods') + # or fully-qualified. Optional. + excludedResources: + - storageclasses.storage.k8s.io + # Whether or not to include cluster-scoped resources. Valid values are true, false, and + # null/unset. If true, all cluster-scoped resources are included (subject to included/excluded + # resources and the label selector). If false, no cluster-scoped resources are included. If unset, + # all cluster-scoped resources are included if and only if all namespaces are included and there are + # no excluded namespaces. Otherwise, if there is at least one namespace specified in either + # includedNamespaces or excludedNamespaces, then the only cluster-scoped resources that are backed + # up are those associated with namespace-scoped resources included in the restore. For example, if a + # PersistentVolumeClaim is included in the restore, its associated PersistentVolume (which is + # cluster-scoped) would also be backed up. + includeClusterResources: null + # Individual objects must match this label selector to be included in the restore. Optional. + labelSelector: + matchLabels: + app: velero + component: server + # NamespaceMapping is a map of source namespace names to + # target namespace names to restore into. Any source namespaces not + # included in the map will be restored into namespaces of the same name. + namespaceMapping: + namespace-backup-from: namespace-to-restore-to + # RestorePVs specifies whether to restore all included PVs + # from snapshot (via the cloudprovider). + restorePVs: true + # ScheduleName is the unique name of the Velero schedule + # to restore from. If specified, and BackupName is empty, Velero will + # restore from the most recent successful backup created from this schedule. + scheduleName: my-scheduled-backup-name +# RestoreStatus captures the current status of a Velero restore. Users should not set any data here. +status: + # The current phase. Valid values are New, FailedValidation, InProgress, Completed, PartiallyFailed, Failed. + phase: "" + # An array of any validation errors encountered. + validationErrors: null + # Number of warnings that were logged by the restore. + warnings: 2 + # Errors is a count of all error messages that were generated + # during execution of the restore. The actual errors are stored in object + # storage. + errors: 0 + # FailureReason is an error that caused the entire restore + # to fail. + failureReason: + +``` diff --git a/site/docs/v1.4-pre/api-types/schedule.md b/site/docs/v1.4-pre/api-types/schedule.md new file mode 100644 index 0000000000..ec7e471a6a --- /dev/null +++ b/site/docs/v1.4-pre/api-types/schedule.md @@ -0,0 +1,132 @@ +# Schedule API Type + +## Use + +The `Schedule` API type is used as a repeatable request for the Velero server to perform a backup for a given cron notation. Once created, the +Velero Server will start the backup process. It will then wait for the next valid point of the given cron expression and execute the backup +process on a repeating basis. + +## API GroupVersion + +Schedule belongs to the API group version `velero.io/v1`. + +## Definition + +Here is a sample `Schedule` object with each of the fields documented: + +```yaml +# Standard Kubernetes API Version declaration. Required. +apiVersion: velero.io/v1 +# Standard Kubernetes Kind declaration. Required. +kind: Schedule +# Standard Kubernetes metadata. Required. +metadata: + # Schedule name. May be any valid Kubernetes object name. Required. + name: a + # Schedule namespace. Must be the namespace of the Velero server. Required. + namespace: velero +# Parameters about the scheduled backup. Required. +spec: + # Schedule is a Cron expression defining when to run the Backup + schedule: 0 7 * * * + # Template is the spec that should be used for each backup triggered by this schedule. + template: + # Array of namespaces to include in the scheduled backup. If unspecified, all namespaces are included. + # Optional. + includedNamespaces: + - '*' + # Array of namespaces to exclude from the scheduled backup. Optional. + excludedNamespaces: + - some-namespace + # Array of resources to include in the scheduled backup. Resources may be shortcuts (e.g. 'po' for 'pods') + # or fully-qualified. If unspecified, all resources are included. Optional. + includedResources: + - '*' + # Array of resources to exclude from the scheduled backup. Resources may be shortcuts (e.g. 'po' for 'pods') + # or fully-qualified. Optional. + excludedResources: + - storageclasses.storage.k8s.io + # Whether or not to include cluster-scoped resources. Valid values are true, false, and + # null/unset. If true, all cluster-scoped resources are included (subject to included/excluded + # resources and the label selector). If false, no cluster-scoped resources are included. If unset, + # all cluster-scoped resources are included if and only if all namespaces are included and there are + # no excluded namespaces. Otherwise, if there is at least one namespace specified in either + # includedNamespaces or excludedNamespaces, then the only cluster-scoped resources that are backed + # up are those associated with namespace-scoped resources included in the scheduled backup. For example, if a + # PersistentVolumeClaim is included in the backup, its associated PersistentVolume (which is + # cluster-scoped) would also be backed up. + includeClusterResources: null + # Individual objects must match this label selector to be included in the scheduled backup. Optional. + labelSelector: + matchLabels: + app: velero + component: server + # Whether or not to snapshot volumes. This only applies to PersistentVolumes for Azure, GCE, and + # AWS. Valid values are true, false, and null/unset. If unset, Velero performs snapshots as long as + # a persistent volume provider is configured for Velero. + snapshotVolumes: null + # Where to store the tarball and logs. + storageLocation: aws-primary + # The list of locations in which to store volume snapshots created for backups under this schedule. + volumeSnapshotLocations: + - aws-primary + - gcp-primary + # The amount of time before backups created on this schedule are eligible for garbage collection. If not specified, + # a default value of 30 days will be used. The default can be configured on the velero server + # by passing the flag --default-backup-ttl. + ttl: 24h0m0s + # Actions to perform at different times during a backup. The only hook currently supported is + # executing a command in a container in a pod using the pod exec API. Optional. + hooks: + # Array of hooks that are applicable to specific resources. Optional. + resources: + - + # Name of the hook. Will be displayed in backup log. + name: my-hook + # Array of namespaces to which this hook applies. If unspecified, the hook applies to all + # namespaces. Optional. + includedNamespaces: + - '*' + # Array of namespaces to which this hook does not apply. Optional. + excludedNamespaces: + - some-namespace + # Array of resources to which this hook applies. The only resource supported at this time is + # pods. + includedResources: + - pods + # Array of resources to which this hook does not apply. Optional. + excludedResources: [] + # This hook only applies to objects matching this label selector. Optional. + labelSelector: + matchLabels: + app: velero + component: server + # An array of hooks to run before executing custom actions. Currently only "exec" hooks are supported. + pre: + - + # The type of hook. This must be "exec". + exec: + # The name of the container where the command will be executed. If unspecified, the + # first container in the pod will be used. Optional. + container: my-container + # The command to execute, specified as an array. Required. + command: + - /bin/uname + - -a + # How to handle an error executing the command. Valid values are Fail and Continue. + # Defaults to Fail. Optional. + onError: Fail + # How long to wait for the command to finish executing. Defaults to 30 seconds. Optional. + timeout: 10s + # An array of hooks to run after all custom actions and additional items have been + # processed. Currently only "exec" hooks are supported. + post: + # Same content as pre above. +status: + # The current phase of the latest scheduled backup. Valid values are New, FailedValidation, InProgress, Completed, PartiallyFailed, Failed. + phase: "" + # Date/time of the last backup for a given schedule + lastBackup: + # An array of any validation errors encountered. + validationErrors: +``` diff --git a/site/docs/v1.4-pre/api-types/volumesnapshotlocation.md b/site/docs/v1.4-pre/api-types/volumesnapshotlocation.md new file mode 100644 index 0000000000..8f3588dcbc --- /dev/null +++ b/site/docs/v1.4-pre/api-types/volumesnapshotlocation.md @@ -0,0 +1,38 @@ +# Velero Volume Snapshot Location + +## Volume Snapshot Location + +A volume snapshot location is the location in which to store the volume snapshots created for a backup. + +Velero can be configured to take snapshots of volumes from multiple providers. Velero also allows you to configure multiple possible `VolumeSnapshotLocation` per provider, although you can only select one location per provider at backup time. + +Each VolumeSnapshotLocation describes a provider + location. These are represented in the cluster via the `VolumeSnapshotLocation` CRD. Velero must have at least one `VolumeSnapshotLocation` per cloud provider. + +A sample YAML `VolumeSnapshotLocation` looks like the following: + +```yaml +apiVersion: velero.io/v1 +kind: VolumeSnapshotLocation +metadata: + name: aws-default + namespace: velero +spec: + provider: aws + config: + region: us-west-2 + profile: "default" +``` + +### Parameter Reference + +The configurable parameters are as follows: + +#### Main config parameters + +| Key | Type | Default | Meaning | +| --- | --- | --- | --- | +| `provider` | String | Required Field | The name for whichever storage provider will be used to create/store the volume snapshots. See [your volume snapshot provider's plugin documentation][0] for the appropriate value to use. | +| `config` | map[string]string | None (Optional) | Provider-specific configuration keys/values to be passed to the volume snapshotter plugin. See [your volume snapshot provider's plugin documentation][0] for details. | + + +[0]: ../supported-providers.md diff --git a/site/docs/v1.4-pre/backup-reference.md b/site/docs/v1.4-pre/backup-reference.md new file mode 100644 index 0000000000..797230d674 --- /dev/null +++ b/site/docs/v1.4-pre/backup-reference.md @@ -0,0 +1,9 @@ +# Backup Reference + +## Exclude Specific Items from Backup + +It is possible to exclude individual items from being backed up, even if they match the resource/namespace/label selectors defined in the backup spec. To do this, label the item as follows: + +```bash +kubectl label -n / velero.io/exclude-from-backup=true +``` diff --git a/site/docs/v1.4-pre/basic-install.md b/site/docs/v1.4-pre/basic-install.md new file mode 100644 index 0000000000..fde47086ff --- /dev/null +++ b/site/docs/v1.4-pre/basic-install.md @@ -0,0 +1,64 @@ +# Basic Install + +- [Basic Install](#basic-install) + - [Prerequisites](#prerequisites) + - [Install the CLI](#install-the-cli) + - [Option 1: macOS - Homebrew](#option-1-macos---homebrew) + - [Option 2: GitHub release](#option-2-github-release) + - [Install and configure the server components](#install-and-configure-the-server-components) + - [Command line Autocompletion](#command-line-autocompletion) + +Use this doc to get a basic installation of Velero. +Refer [this document](customize-installation.md) to customize your installation. + +## Prerequisites + +- Access to a Kubernetes cluster, v1.10 or later, with DNS and container networking enabled. +- `kubectl` installed locally + +Velero uses object storage to store backups and associated artifacts. It also optionally integrates with supported block storage systems to snapshot your persistent volumes. Before beginning the installation process, you should identify the object storage provider and optional block storage provider(s) you'll be using from the list of [compatible providers][0]. + +There are supported storage providers for both cloud-provider environments and on-premises environments. For more details on on-premises scenarios, see the [on-premises documentation][2]. + +## Install the CLI + +### Option 1: macOS - Homebrew + +On macOS, you can use [Homebrew](https://brew.sh) to install the `velero` client: + +```bash +brew install velero +``` + +### Option 2: GitHub release + +1. Download the [latest release][1]'s tarball for your client platform. +1. Extract the tarball: + + ```bash + tar -xvf .tar.gz + ``` + +1. Move the extracted `velero` binary to somewhere in your `$PATH` (e.g. `/usr/local/bin` for most users). + +## Install and configure the server components + +There are two supported methods for installing the Velero server components: + +- the `velero install` CLI command +- the [Helm chart](https://vmware-tanzu.github.io/helm-charts/) + +Velero uses storage provider plugins to integrate with a variety of storage systems to support backup and snapshot operations. The steps to install and configure the Velero server components along with the appropriate plugins are specific to your chosen storage provider. To find installation instructions for your chosen storage provider, follow the documentation link for your provider at our [supported storage providers][0] page + +_Note: if your object storage provider is different than your volume snapshot provider, follow the installation instructions for your object storage provider first, then return here and follow the instructions to [add your volume snapshot provider][4]._ + +## Command line Autocompletion + +Please refer to [this part of the documentation][5]. + +[0]: supported-providers.md +[1]: https://github.com/vmware-tanzu/velero/releases/latest +[2]: on-premises.md +[3]: overview-plugins.md +[4]: customize-installation.md#install-an-additional-volume-snapshot-provider +[5]: customize-installation.md#optional-velero-cli-configurations diff --git a/site/docs/v1.4-pre/build-from-source.md b/site/docs/v1.4-pre/build-from-source.md new file mode 100644 index 0000000000..57918632da --- /dev/null +++ b/site/docs/v1.4-pre/build-from-source.md @@ -0,0 +1,158 @@ +# Build from source + +## Prerequisites + +* Access to a Kubernetes cluster, version 1.7 or later. +* A DNS server on the cluster +* `kubectl` installed +* [Go][5] installed (minimum version 1.8) + +## Get the source + +### Option 1) Get latest (recommended) + +```bash +mkdir $HOME/go +export GOPATH=$HOME/go +go get github.com/vmware-tanzu/velero +``` + +Where `go` is your [import path][4] for Go. + +For Go development, it is recommended to add the Go import path (`$HOME/go` in this example) to your path. + +### Option 2) Release archive + +Download the archive named `Source code` from the [release page][22] and extract it in your Go import path as `src/github.com/vmware-tanzu/velero`. + +Note that the Makefile targets assume building from a git repository. When building from an archive, you will be limited to the `go build` commands described below. + +## Build + +There are a number of different ways to build `velero` depending on your needs. This section outlines the main possibilities. + +When building by using `make`, it will place the binaries under `_output/bin/$GOOS/$GOARCH`. For example, you will find the binary for darwin here: `_output/bin/darwin/amd64/velero`, and the binary for linux here: `_output/bin/linux/amd64/velero`. `make` will also splice version and git commit information in so that `velero version` displays proper output. + +Note: `velero install` will also use the version information to determine which tagged image to deploy. If you would like to overwrite what image gets deployed, use the `image` flag (see below for instructions on how to build images). + +### Build the binary + +To build the `velero` binary on your local machine, compiled for your OS and architecture, run one of these two commands: + +```bash +go build ./cmd/velero +``` + +```bash +make local +``` + +### Cross compiling + +To build the velero binary targeting linux/amd64 within a build container on your local machine, run: + +```bash +make build +``` + +For any specific platform, run `make build--`. + +For example, to build for the Mac, run `make build-darwin-amd64`. + +Velero's `Makefile` has a convenience target, `all-build`, that builds the following platforms: + +* linux-amd64 +* linux-arm +* linux-arm64 +* linux-ppc64le +* darwin-amd64 +* windows-amd64 + +## Making images and updating Velero + +If after installing Velero you would like to change the image used by its deployment to one that contains your code changes, you may do so by updating the image: + +```bash +kubectl -n velero set image deploy/velero velero=myimagerepo/velero:$VERSION +``` + +To build a Velero container image, first set the `$REGISTRY` environment variable. For example, if you want to build the `gcr.io/my-registry/velero-amd64:master` image, set `$REGISTRY` to `gcr.io/my-registry`. If this variable is not set, the default is `velero`. + +Optionally, set the `$VERSION` environment variable to change the image tag. Then, run: + +```bash +make container +``` + +For any specific platform, run `ARCH=- make container` + +For example, to build an image for the Power (ppc64le), run: + +```bash +ARCH=linux-ppc64le make container +``` +_Note: By default, ARCH is set to linux-amd64_ + +To push your image to the registry. For example, if you want to push the `gcr.io/my-registry/velero-amd64:master` image, run: + +```bash +make push +``` + +For any specific platform, run `ARCH=- make push` + +For example, to push image for the Power (ppc64le), run: + +```bash +ARCH=linux-ppc64le make push +``` +_Note: By default, ARCH is set to linux-amd64_ + +To create and push your manifest to the registry. For example, if you want to create and push the `gcr.io/my-registry/velero:master` manifest, run: + +```bash +make manifest +``` + +For any specific platform, run `MANIFEST_PLATFORMS= make manifest` + +For example, to create and push manifest only for amd64, run: + +```bash +MANIFEST_PLATFORMS=amd64 make manifest +``` +_Note: By default, MANIFEST_PLATFORMS is set to amd64, ppc64le_ + +To run the entire workflow, run: + +`REGISTRY=<$REGISTRY> VERSION=<$VERSION> ARCH=- MANIFEST_PLATFORMS= make container push manifest` + +For example, to run the workflow only for amd64 + +```bash +REGISTRY=myrepo VERSION=foo MANIFEST_PLATFORMS=amd64 make container push manifest +``` + +_Note: By default, ARCH is set to linux-amd64_ + +For example, to run the workflow only for ppc64le + +```bash +REGISTRY=myrepo VERSION=foo ARCH=linux-ppc64le MANIFEST_PLATFORMS=ppc64le make container push manifest +``` + +For example, to run the workflow for all supported platforms + +```bash +REGISTRY=myrepo VERSION=foo make all-containers all-push all-manifests +``` + +Note: if you want to update the image but not change its name, you will have to trigger Kubernetes to pick up the new image. One way of doing so is by deleting the Velero deployment pod: + +```bash +kubectl -n velero delete pods -l deploy=velero +``` + +[4]: https://blog.golang.org/organizing-go-code +[5]: https://golang.org/doc/install +[22]: https://github.com/vmware-tanzu/velero/releases diff --git a/site/docs/v1.4-pre/code-standards.md b/site/docs/v1.4-pre/code-standards.md new file mode 100644 index 0000000000..cd5255cee7 --- /dev/null +++ b/site/docs/v1.4-pre/code-standards.md @@ -0,0 +1,133 @@ +# Code Standards + +## Adding a changelog + +Authors are expected to include a changelog file with their pull requests. The changelog file +should be a new file created in the `changelogs/unreleased` folder. The file should follow the +naming convention of `pr-username` and the contents of the file should be your text for the +changelog. + + velero/changelogs/unreleased <- folder + 000-username <- file + +Add that to the PR. + +## Code + +- Log messages are capitalized. + +- Error messages are kept lower-cased. + +- Wrap/add a stack only to errors that are being directly returned from non-velero code, such as an API call to the Kubernetes server. + + ```bash + errors.WithStack(err) + ``` + +- Prefer to use the utilities in the Kubernetes package [`sets`](https://godoc.org/github.com/kubernetes/apimachinery/pkg/util/sets). + + ```bash + k8s.io/apimachinery/pkg/util/sets + ``` + +## Imports + +For imports, we use the following convention: + + + +Example: + + import ( + corev1api "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + corev1client "k8s.io/client-go/kubernetes/typed/core/v1" + corev1listers "k8s.io/client-go/listers/core/v1" + + velerov1api ""github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov1client "github.com/vmware-tanzu/velero/pkg/generated/clientset/versioned/typed/velero/v1" + ) + +## Mocks + +We use a package to generate mocks for our interfaces. + +Example: if you want to change this mock: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/pkg/restic/mocks/restorer.go + +Run: + +```bash +go get github.com/vektra/mockery/.../ +cd pkg/restic +mockery -name=Restorer +``` + +Might need to run `make update` to update the imports. + +## Kubernetes Labels + +When generating label values, be sure to pass them through the `label.GetValidName()` helper function. + +This will help ensure that the values are the proper length and format to be stored and queried. + +In general, UIDs are safe to persist as label values. + +This function is not relevant to annotation values, which do not have restrictions. + +## DCO Sign off + +All authors to the project retain copyright to their work. However, to ensure +that they are only submitting work that they have rights to, we are requiring +everyone to acknowledge this by signing their work. + +Any copyright notices in this repo should specify the authors as "the Velero contributors". + +To sign your work, just add a line like this at the end of your commit message: + +``` +Signed-off-by: Joe Beda +``` + +This can easily be done with the `--signoff` option to `git commit`. + +By doing this you state that you can certify the following (from https://developercertificate.org/): + +``` +Developer Certificate of Origin +Version 1.1 + +Copyright (C) 2004, 2006 The Linux Foundation and its contributors. +1 Letterman Drive +Suite D4700 +San Francisco, CA, 94129 + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + + +Developer's Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +(a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + +(b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + +(c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + +(d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. +``` diff --git a/site/docs/v1.4-pre/contributions/ibm-config.md b/site/docs/v1.4-pre/contributions/ibm-config.md new file mode 100644 index 0000000000..69d34420c3 --- /dev/null +++ b/site/docs/v1.4-pre/contributions/ibm-config.md @@ -0,0 +1,96 @@ +# Use IBM Cloud Object Storage as Velero's storage destination. +You can deploy Velero on IBM [Public][5] or [Private][4] clouds, or even on any other Kubernetes cluster, but anyway you can use IBM Cloud Object Store as a destination for Velero's backups. + +To set up IBM Cloud Object Storage (COS) as Velero's destination, you: + +* Download an official release of Velero +* Create your COS instance +* Create an S3 bucket +* Define a service that can store data in the bucket +* Configure and start the Velero server + +## Download Velero + +1. Download the [latest official release's](https://github.com/vmware-tanzu/velero/releases) tarball for your client platform. + + _We strongly recommend that you use an [official release](https://github.com/vmware-tanzu/velero/releases) of +Velero. The tarballs for each release contain the `velero` command-line client. The code in the master branch +of the Velero repository is under active development and is not guaranteed to be stable!_ + +1. Extract the tarball: + + ```bash + tar -xvf .tar.gz -C /dir/to/extract/to + ``` + + We'll refer to the directory you extracted to as the "Velero directory" in subsequent steps. + +1. Move the `velero` binary from the Velero directory to somewhere in your PATH. + +## Create COS instance +If you don’t have a COS instance, you can create a new one, according to the detailed instructions in [Creating a new resource instance][1]. + +## Create an S3 bucket +Velero requires an object storage bucket to store backups in. See instructions in [Create some buckets to store your data][2]. + +## Define a service that can store data in the bucket. +The process of creating service credentials is described in [Service credentials][3]. +Several comments: + +1. The Velero service will write its backup into the bucket, so it requires the “Writer” access role. + +2. Velero uses an AWS S3 compatible API. Which means it authenticates using a signature created from a pair of access and secret keys — a set of HMAC credentials. You can create these HMAC credentials by specifying `{“HMAC”:true}` as an optional inline parameter. See step 3 in the [Service credentials][3] guide. + +3. After successfully creating a Service credential, you can view the JSON definition of the credential. Under the `cos_hmac_keys` entry there are `access_key_id` and `secret_access_key`. We will use them in the next step. + +4. Create a Velero-specific credentials file (`credentials-velero`) in your local directory: + + ``` + [default] + aws_access_key_id= + aws_secret_access_key= + ``` + + where the access key id and secret are the values that we got above. + +## Install and start Velero + +Install Velero, including all prerequisites, into the cluster and start the deployment. This will create a namespace called `velero`, and place a deployment named `velero` in it. + +```bash +velero install \ + --provider aws \ + --bucket \ + --secret-file ./credentials-velero \ + --use-volume-snapshots=false \ + --backup-location-config region=,s3ForcePathStyle="true",s3Url= +``` + +Velero does not currently have a volume snapshot plugin for IBM Cloud, so creating volume snapshots is disabled. + +Additionally, you can specify `--use-restic` to enable restic support, and `--wait` to wait for the deployment to be ready. + +(Optional) Specify [CPU and memory resource requests and limits][15] for the Velero/restic pods. + +Once the installation is complete, remove the default `VolumeSnapshotLocation` that was created by `velero install`, since it's specific to AWS and won't work for IBM Cloud: + +```bash +kubectl -n velero delete volumesnapshotlocation.velero.io default +``` + +For more complex installation needs, use either the Helm chart, or add `--dry-run -o yaml` options for generating the YAML representation for the installation. + +## Installing the nginx example (optional) + +If you run the nginx example, in file `examples/nginx-app/with-pv.yaml`: + +Uncomment `storageClassName: ` and replace with your `StorageClass` name. + +[0]: namespace.md +[1]: https://console.bluemix.net/docs/services/cloud-object-storage/basics/order-storage.html#creating-a-new-resource-instance +[2]: https://console.bluemix.net/docs/services/cloud-object-storage/getting-started.html#create-buckets +[3]: https://console.bluemix.net/docs/services/cloud-object-storage/iam/service-credentials.html#service-credentials +[4]: https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0/kc_welcome_containers.html +[5]: https://console.bluemix.net/docs/containers/container_index.html#container_index +[14]: http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html +[15]: customize-installation.md#customize-resource-requests-and-limits diff --git a/site/docs/v1.4-pre/contributions/minio.md b/site/docs/v1.4-pre/contributions/minio.md new file mode 100644 index 0000000000..12daff4a15 --- /dev/null +++ b/site/docs/v1.4-pre/contributions/minio.md @@ -0,0 +1,267 @@ +## Quick start evaluation install with Minio + +The following example sets up the Velero server and client, then backs up and restores a sample application. + +For simplicity, the example uses Minio, an S3-compatible storage service that runs locally on your cluster. +For additional functionality with this setup, see the section below on how to [expose Minio outside your cluster][1]. + +**NOTE** The example lets you explore basic Velero functionality. Configuring Minio for production is out of scope. + +See [Set up Velero on your platform][3] for how to configure Velero for a production environment. + +If you encounter issues with installing or configuring, see [Debugging Installation Issues](debugging-install.md). + +### Prerequisites + +* Access to a Kubernetes cluster, version 1.7 or later. **Note:** restic support requires Kubernetes version 1.10 or later, or an earlier version with the mount propagation feature enabled. Restic support is not required for this example, but may be of interest later. See [Restic Integration][17]. +* A DNS server on the cluster +* `kubectl` installed + +### Download Velero + +1. Download the [latest official release's](https://github.com/vmware-tanzu/velero/releases) tarball for your client platform. + + _We strongly recommend that you use an [official release](https://github.com/vmware-tanzu/velero/releases) of +Velero. The tarballs for each release contain the `velero` command-line client. The code in the master branch +of the Velero repository is under active development and is not guaranteed to be stable!_ + +1. Extract the tarball: + + ```bash + tar -xvf .tar.gz -C /dir/to/extract/to + ``` + + We'll refer to the directory you extracted to as the "Velero directory" in subsequent steps. + +1. Move the `velero` binary from the Velero directory to somewhere in your PATH. + +#### MacOS Installation + +On Mac, you can use [HomeBrew](https://brew.sh) to install the `velero` client: + +```bash +brew install velero +``` + +### Set up server + +These instructions start the Velero server and a Minio instance that is accessible from within the cluster only. See [Expose Minio outside your cluster][31] for information about configuring your cluster for outside access to Minio. Outside access is required to access logs and run `velero describe` commands. + +1. Create a Velero-specific credentials file (`credentials-velero`) in your local directory: + + ``` + [default] + aws_access_key_id = minio + aws_secret_access_key = minio123 + ``` + +1. Start the server and the local storage service. In the Velero directory, run: + + ``` + kubectl apply -f examples/minio/00-minio-deployment.yaml + ``` + ``` + velero install \ + --provider aws \ + --plugins velero/velero-plugin-for-aws:v1.0.0 \ + --bucket velero \ + --secret-file ./credentials-velero \ + --use-volume-snapshots=false \ + --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000 + ``` + + This example assumes that it is running within a local cluster without a volume provider capable of snapshots, so no `VolumeSnapshotLocation` is created (`--use-volume-snapshots=false`). + + Additionally, you can specify `--use-restic` to enable restic support, and `--wait` to wait for the deployment to be ready. + + +1. Deploy the example nginx application: + + ```bash + kubectl apply -f examples/nginx-app/base.yaml + ``` + +1. Check to see that both the Velero and nginx deployments are successfully created: + + ``` + kubectl get deployments -l component=velero --namespace=velero + kubectl get deployments --namespace=nginx-example + ``` + +### Back up + +1. Create a backup for any object that matches the `app=nginx` label selector: + + ``` + velero backup create nginx-backup --selector app=nginx + ``` + + Alternatively if you want to backup all objects *except* those matching the label `backup=ignore`: + + ``` + velero backup create nginx-backup --selector 'backup notin (ignore)' + ``` + +1. (Optional) Create regularly scheduled backups based on a cron expression using the `app=nginx` label selector: + + ``` + velero schedule create nginx-daily --schedule="0 1 * * *" --selector app=nginx + ``` + + Alternatively, you can use some non-standard shorthand cron expressions: + + ``` + velero schedule create nginx-daily --schedule="@daily" --selector app=nginx + ``` + + See the [cron package's documentation][30] for more usage examples. + +1. Simulate a disaster: + + ``` + kubectl delete namespace nginx-example + ``` + +1. To check that the nginx deployment and service are gone, run: + + ``` + kubectl get deployments --namespace=nginx-example + kubectl get services --namespace=nginx-example + kubectl get namespace/nginx-example + ``` + + You should get no results. + + NOTE: You might need to wait for a few minutes for the namespace to be fully cleaned up. + +### Restore + +1. Run: + + ``` + velero restore create --from-backup nginx-backup + ``` + +1. Run: + + ``` + velero restore get + ``` + + After the restore finishes, the output looks like the following: + + ``` + NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR + nginx-backup-20170727200524 nginx-backup Completed 0 0 2017-07-27 20:05:24 +0000 UTC + ``` + +NOTE: The restore can take a few moments to finish. During this time, the `STATUS` column reads `InProgress`. + +After a successful restore, the `STATUS` column is `Completed`, and `WARNINGS` and `ERRORS` are 0. All objects in the `nginx-example` namespace should be just as they were before you deleted them. + +If there are errors or warnings, you can look at them in detail: + +``` +velero restore describe +``` + +For more information, see [the debugging information][18]. + +### Clean up + +If you want to delete any backups you created, including data in object storage and persistent +volume snapshots, you can run: + +``` +velero backup delete BACKUP_NAME +``` + +This asks the Velero server to delete all backup data associated with `BACKUP_NAME`. You need to do +this for each backup you want to permanently delete. A future version of Velero will allow you to +delete multiple backups by name or label selector. + +Once fully removed, the backup is no longer visible when you run: + +``` +velero backup get BACKUP_NAME +``` + +To completely uninstall Velero, minio, and the nginx example app from your Kubernetes cluster: + +``` +kubectl delete namespace/velero clusterrolebinding/velero +kubectl delete crds -l component=velero +kubectl delete -f examples/nginx-app/base.yaml +``` + +## Expose Minio outside your cluster with a Service + +When you run commands to get logs or describe a backup, the Velero server generates a pre-signed URL to download the requested items. To access these URLs from outside the cluster -- that is, from your Velero client -- you need to make Minio available outside the cluster. You can: + +- Change the Minio Service type from `ClusterIP` to `NodePort`. +- Set up Ingress for your cluster, keeping Minio Service type `ClusterIP`. + +You can also specify a `publicUrl` config field for the pre-signed URL in your backup storage location config. + +### Expose Minio with Service of type NodePort + +The Minio deployment by default specifies a Service of type `ClusterIP`. You can change this to `NodePort` to easily expose a cluster service externally if you can reach the node from your Velero client. + +You must also get the Minio URL, which you can then specify as the value of the `publicUrl` field in your backup storage location config. + +1. In `examples/minio/00-minio-deployment.yaml`, change the value of Service `spec.type` from `ClusterIP` to `NodePort`. + +1. Get the Minio URL: + + - if you're running Minikube: + + ```shell + minikube service minio --namespace=velero --url + ``` + + - in any other environment: + 1. Get the value of an external IP address or DNS name of any node in your cluster. You must be able to reach this address from the Velero client. + 1. Append the value of the NodePort to get a complete URL. You can get this value by running: + + ```shell + kubectl -n velero get svc/minio -o jsonpath='{.spec.ports[0].nodePort}' + ``` + +1. Edit your `BackupStorageLocation` YAML, adding `publicUrl: ` as a field under `spec.config`. You must include the `http://` or `https://` prefix. + +## Expose Minio outside your cluster with Kubernetes in Docker (KinD): + +Kubernetes in Docker currently does not have support for NodePort services (see [this issue](https://github.com/kubernetes-sigs/kind/issues/99)). In this case, you can use a port forward to access the Minio bucket. + +In a terminal, run the following: + +```shell +MINIO_POD=$(kubectl get pods -n velero -l component=minio -o jsonpath='{.items[0].metadata.name}') + +kubectl port-forward $MINIO_POD -n velero 9000:9000 +``` + +Then, in another terminal: + +```shell +kubectl edit backupstoragelocation default -n velero +``` + +Add `publicUrl: http://localhost:9000` under the `spec.config` section. + +### Work with Ingress + +Configuring Ingress for your cluster is out of scope for the Velero documentation. If you have already set up Ingress, however, it makes sense to continue with it while you run the example Velero configuration with Minio. + +In this case: + +1. Keep the Service type as `ClusterIP`. + +1. Edit your `BackupStorageLocation` YAML, adding `publicUrl: ` as a field under `spec.config`. + +[1]: #expose-minio-with-service-of-type-nodeport +[3]: ../customize-installation.md +[17]: ../restic.md +[18]: ../debugging-restores.md +[26]: https://github.com/vmware-tanzu/velero/releases +[30]: https://godoc.org/github.com/robfig/cron diff --git a/site/docs/v1.4-pre/contributions/oracle-config.md b/site/docs/v1.4-pre/contributions/oracle-config.md new file mode 100644 index 0000000000..a7ccaed47a --- /dev/null +++ b/site/docs/v1.4-pre/contributions/oracle-config.md @@ -0,0 +1,245 @@ +# Use Oracle Cloud as a Backup Storage Provider for Velero + +## Introduction + +[Velero](https://velero.io/) is a tool used to backup and migrate Kubernetes applications. Here are the steps to use [Oracle Cloud Object Storage](https://docs.cloud.oracle.com/iaas/Content/Object/Concepts/objectstorageoverview.htm) as a destination for Velero backups. + +1. [Download Velero](#download-velero) +2. [Create A Customer Secret Key](#create-a-customer-secret-key) +3. [Create An Oracle Object Storage Bucket](#create-an-oracle-object-storage-bucket) +4. [Install Velero](#install-velero) +5. [Clean Up](#clean-up) +6. [Examples](#examples) +7. [Additional Reading](#additional-reading) + +## Download Velero + +1. Download the [latest release](https://github.com/vmware-tanzu/velero/releases/) of Velero to your development environment. This includes the `velero` CLI utility and example Kubernetes manifest files. For example: + + ``` + wget https://github.com/vmware-tanzu/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz + ``` + + *We strongly recommend that you use an official release of Velero. The tarballs for each release contain the velero command-line client. The code in the master branch of the Velero repository is under active development and is not guaranteed to be stable!* + +2. Untar the release in your `/usr/bin` directory: `tar -xzvf .tar.gz` + + You may choose to rename the directory `velero` for the sake of simplicty: `mv velero-v1.0.0-linux-amd64 velero` + +3. Add it to your PATH: `export PATH=/usr/local/bin/velero:$PATH` + +4. Run `velero` to confirm the CLI has been installed correctly. You should see an output like this: + +``` +$ velero +Velero is a tool for managing disaster recovery, specifically for Kubernetes +cluster resources. It provides a simple, configurable, and operationally robust +way to back up your application state and associated data. + +If you're familiar with kubectl, Velero supports a similar model, allowing you to +execute commands such as 'velero get backup' and 'velero create schedule'. The same +operations can also be performed as 'velero backup get' and 'velero schedule create'. + +Usage: + velero [command] +``` + + + +## Create A Customer Secret Key + +1. Oracle Object Storage provides an API to enable interoperability with Amazon S3. To use this Amazon S3 Compatibility API, you need to generate the signing key required to authenticate with Amazon S3. This special signing key is an Access Key/Secret Key pair. Follow these steps to [create a Customer Secret Key](https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#To4). Refer to this link for more information about [Working with Customer Secret Keys](https://docs.cloud.oracle.com/iaas/Content/Identity/Tasks/managingcredentials.htm#s3). + +2. Create a Velero credentials file with your Customer Secret Key: + + ``` + $ vi credentials-velero + + [default] + aws_access_key_id=bae031188893d1eb83719648790ac850b76c9441 + aws_secret_access_key=MmY9heKrWiNVCSZQ2Mf5XTJ6Ys93Bw2d2D6NMSTXZlk= + ``` + + + +## Create An Oracle Object Storage Bucket + +Create an Oracle Cloud Object Storage bucket called `velero` in the root compartment of your Oracle Cloud tenancy. Refer to this page for [more information about creating a bucket with Object Storage](https://docs.cloud.oracle.com/iaas/Content/Object/Tasks/managingbuckets.htm#usingconsole). + + + +## Install Velero + +You will need the following information to install Velero into your Kubernetes cluster with Oracle Object Storage as the Backup Storage provider: + +``` +velero install \ + --provider [provider name] \ + --bucket [bucket name] \ + --prefix [tenancy name] \ + --use-volume-snapshots=false \ + --secret-file [secret file location] \ + --backup-location-config region=[region],s3ForcePathStyle="true",s3Url=[storage API endpoint] +``` + +- `--provider` Because we are using the S3-compatible API, we will use `aws` as our provider. +- `--bucket` The name of the bucket created in Oracle Object Storage - in our case this is named `velero`. +- ` --prefix` The name of your Oracle Cloud tenancy - in our case this is named `oracle-cloudnative`. +- `--use-volume-snapshots=false` Velero does not currently have a volume snapshot plugin for Oracle Cloud creating volume snapshots is disabled. +- `--secret-file` The path to your `credentials-velero` file. +- `--backup-location-config` The path to your Oracle Object Storage bucket. This consists of your `region` which corresponds to your Oracle Cloud region name ([List of Oracle Cloud Regions](https://docs.cloud.oracle.com/iaas/Content/General/Concepts/regions.htm?Highlight=regions)) and the `s3Url`, the S3-compatible API endpoint for Oracle Object Storage based on your region: `https://oracle-cloudnative.compat.objectstorage.[region name].oraclecloud.com` + +For example: + +``` +velero install \ + --provider aws \ + --bucket velero \ + --prefix oracle-cloudnative \ + --use-volume-snapshots=false \ + --secret-file /Users/mboxell/bin/velero/credentials-velero \ + --backup-location-config region=us-phoenix-1,s3ForcePathStyle="true",s3Url=https://oracle-cloudnative.compat.objectstorage.us-phoenix-1.oraclecloud.com +``` + +This will create a `velero` namespace in your cluster along with a number of CRDs, a ClusterRoleBinding, ServiceAccount, Secret, and Deployment for Velero. If your pod fails to successfully provision, you can troubleshoot your installation by running: `kubectl logs [velero pod name]`. + + + +## Clean Up + +To remove Velero from your environment, delete the namespace, ClusterRoleBinding, ServiceAccount, Secret, and Deployment and delete the CRDs, run: + +``` +kubectl delete namespace/velero clusterrolebinding/velero +kubectl delete crds -l component=velero +``` + +This will remove all resources created by `velero install`. + + + +## Examples + +After creating the Velero server in your cluster, try this example: + +### Basic example (without PersistentVolumes) + +1. Start the sample nginx app: `kubectl apply -f examples/nginx-app/base.yaml` + + This will create an `nginx-example` namespace with a `nginx-deployment` deployment, and `my-nginx` service. + + ``` + $ kubectl apply -f examples/nginx-app/base.yaml + namespace/nginx-example created + deployment.apps/nginx-deployment created + service/my-nginx created + ``` + + You can see the created resources by running `kubectl get all` + + ``` + $ kubectl get all + NAME READY STATUS RESTARTS AGE + pod/nginx-deployment-67594d6bf6-4296p 1/1 Running 0 20s + pod/nginx-deployment-67594d6bf6-f9r5s 1/1 Running 0 20s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/my-nginx LoadBalancer 10.96.69.166 80:31859/TCP 21s + + NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE + deployment.apps/nginx-deployment 2 2 2 2 21s + + NAME DESIRED CURRENT READY AGE + replicaset.apps/nginx-deployment-67594d6bf6 2 2 2 21s + ``` + +2. Create a backup: `velero backup create nginx-backup --include-namespaces nginx-example` + + ``` + $ velero backup create nginx-backup --include-namespaces nginx-example + Backup request "nginx-backup" submitted successfully. + Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details. + ``` + + At this point you can navigate to appropriate bucket, which we called `velero`, in the Oracle Cloud Object Storage console to see the resources backed up using Velero. + +3. Simulate a disaster by deleting the `nginx-example` namespace: `kubectl delete namespaces nginx-example` + + ``` + $ kubectl delete namespaces nginx-example + namespace "nginx-example" deleted + ``` + + Wait for the namespace to be deleted. To check that the nginx deployment, service, and namespace are gone, run: + + ``` + kubectl get deployments --namespace=nginx-example + kubectl get services --namespace=nginx-example + kubectl get namespace/nginx-example + ``` + + This should return: `No resources found.` + +4. Restore your lost resources: `velero restore create --from-backup nginx-backup` + + ``` + $ velero restore create --from-backup nginx-backup + Restore request "nginx-backup-20190604102710" submitted successfully. + Run `velero restore describe nginx-backup-20190604102710` or `velero restore logs nginx-backup-20190604102710` for more details. + ``` + + Running `kubectl get namespaces` will show that the `nginx-example` namespace has been restored along with its contents. + +5. Run: `velero restore get` to view the list of restored resources. After the restore finishes, the output looks like the following: + + ``` + $ velero restore get + NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR + nginx-backup-20190604104249 nginx-backup Completed 0 0 2019-06-04 10:42:39 -0700 PDT + ``` + + NOTE: The restore can take a few moments to finish. During this time, the `STATUS` column reads `InProgress`. + + After a successful restore, the `STATUS` column shows `Completed`, and `WARNINGS` and `ERRORS` will show `0`. All objects in the `nginx-example` namespace should be just as they were before you deleted them. + + If there are errors or warnings, for instance if the `STATUS` column displays `FAILED` instead of `InProgress`, you can look at them in detail with `velero restore describe ` + + +6. Clean up the environment with `kubectl delete -f examples/nginx-app/base.yaml` + + ``` + $ kubectl delete -f examples/nginx-app/base.yaml + namespace "nginx-example" deleted + deployment.apps "nginx-deployment" deleted + service "my-nginx" deleted + ``` + + If you want to delete any backups you created, including data in object storage, you can run: `velero backup delete BACKUP_NAME` + + ``` + $ velero backup delete nginx-backup + Are you sure you want to continue (Y/N)? Y + Request to delete backup "nginx-backup" submitted successfully. + The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed. + ``` + + This asks the Velero server to delete all backup data associated with `BACKUP_NAME`. You need to do this for each backup you want to permanently delete. A future version of Velero will allow you to delete multiple backups by name or label selector. + + Once fully removed, the backup is no longer visible when you run: `velero backup get BACKUP_NAME` or more generally `velero backup get`: + + ``` + $ velero backup get nginx-backup + An error occurred: backups.velero.io "nginx-backup" not found + ``` + + ``` + $ velero backup get + NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR + ``` + + + +## Additional Reading + +* [Official Velero Documentation](https://velero.io/docs/v1.4.0-beta.1/) +* [Oracle Cloud Infrastructure Documentation](https://docs.cloud.oracle.com/) diff --git a/site/docs/v1.4-pre/csi.md b/site/docs/v1.4-pre/csi.md new file mode 100644 index 0000000000..01669d8209 --- /dev/null +++ b/site/docs/v1.4-pre/csi.md @@ -0,0 +1,32 @@ +# Container Storage Interface Snapshot Support in Velero + +_This feature is under development. Documentation may not be up-to-date and features may not work as expected._ + +# Prerequisites + +The following are the prerequisites for using Velero to take Container Storage Interface (CSI) snapshots: + + 1. The cluster is Kubernetes version 1.17 or greater. + 1. The cluster is running a CSI driver capable of support volume snapshots at the [v1beta1 API level](https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-feature-cis-volume-snapshot-beta/). + 1. The Velero server is running with the `--features EnableCSI` feature flag to enable CSI logic in Velero's core. See [Enabling Features][1] for more information. + 1. The Velero [CSI plugin](https://github.com/vmware-tanzu/velero-plugin-for-csi/) is installed to integrate with the CSI volume snapshot APIs. + 1. When restoring CSI volumesnapshots across clusters, the name of the CSI driver in the destination cluster is the same as that on the source cluster to ensure cross cluster portability of CSI volumesnapshots + +# Implementation Choices + +This section documents some of the choices made during implementation of the Velero [CSI plugin](https://github.com/vmware-tanzu/velero-plugin-for-csi/): + +1. Volumesnapshots created by the plugin will be retained only for the lifetime of the backup even if the `DeletionPolicy` on the volumesnapshotclass is set to `Retain`. To accomplish this, during deletion of the backup the prior to deleting the volumesnapshot, volumesnapshotcontent object will be patched to set its `DeletionPolicy` to `Delete`. Thus deleting volumesnapshot object will result in cascade delete of the volumesnapshotcontent and the snapshot in the storage provider. +1. Volumesnapshotcontent objects created during a velero backup that are dangling, unbound to a volumesnapshot object, will also be discovered, through labels, and deleted on backup deletion. + +# Known Limitations + +1. The Velero [CSI plugin](https://github.com/vmware-tanzu/velero-plugin-for-csi/), to backup CSI backed PVCs, will choose the first VolumeSnapshotClass in the cluster that has the same driver name. _[Issue #17](https://github.com/vmware-tanzu/velero-plugin-for-csi/issues/17)_ + +# Roadmap + +Velero's support level for CSI volume snapshotting will follow upstream Kubernetes support for it, and will reach general availability sometime +after volume snapshotting is GA in upstream Kubernetes. Beta support is expected to launch in Velero v1.4. + + +[1]: customize-installation.md#enable-server-side-features diff --git a/site/docs/v1.4-pre/custom-plugins.md b/site/docs/v1.4-pre/custom-plugins.md new file mode 100644 index 0000000000..3ccd9ea2ab --- /dev/null +++ b/site/docs/v1.4-pre/custom-plugins.md @@ -0,0 +1,91 @@ +# Plugins + +Velero has a plugin architecture that allows users to add their own custom functionality to Velero backups & restores without having to modify/recompile the core Velero binary. To add custom functionality, users simply create their own binary containing implementations of Velero's plugin kinds (described below), plus a small amount of boilerplate code to expose the plugin implementations to Velero. This binary is added to a container image that serves as an init container for the Velero server pod and copies the binary into a shared emptyDir volume for the Velero server to access. + +Multiple plugins, of any type, can be implemented in this binary. + +A fully-functional [sample plugin repository][1] is provided to serve as a convenient starting point for plugin authors. + +## Plugin Naming + +When naming your plugin, keep in mind that the name needs to conform to these rules: +- have two parts separated by '/' +- none of the above parts can be empty +- the prefix is a valid DNS subdomain name +- a plugin with the same name cannot not already exist + +### Some examples: + +``` +- example.io/azure +- 1.2.3.4/5678 +- example-with-dash.io/azure +``` + +You will need to give your plugin(s) a name when registering them by calling the appropriate `RegisterX` function: + +## Plugin Kinds + +Velero currently supports the following kinds of plugins: + +- **Object Store** - persists and retrieves backups, backup logs and restore logs +- **Volume Snapshotter** - creates volume snapshots (during backup) and restores volumes from snapshots (during restore) +- **Backup Item Action** - executes arbitrary logic for individual items prior to storing them in a backup file +- **Restore Item Action** - executes arbitrary logic for individual items prior to restoring them into a cluster + +## Plugin Logging + +Velero provides a [logger][2] that can be used by plugins to log structured information to the main Velero server log or +per-backup/restore logs. It also passes a `--log-level` flag to each plugin binary, whose value is the value of the same +flag from the main Velero process. This means that if you turn on debug logging for the Velero server via `--log-level=debug`, +plugins will also emit debug-level logs. See the [sample repository][1] for an example of how to use the logger within your plugin. + +## Plugin Configuration + +Velero uses a ConfigMap-based convention for providing configuration to plugins. If your plugin needs to be configured at runtime, +define a ConfigMap like the following: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + # any name can be used; Velero uses the labels (below) + # to identify it rather than the name + name: my-plugin-config + + # must be in the namespace where the velero deployment + # is running + namespace: velero + + labels: + # this value-less label identifies the ConfigMap as + # config for a plugin (i.e. the built-in change storageclass + # restore item action plugin) + velero.io/plugin-config: "" + + # add a label whose key corresponds to the fully-qualified + # plugin name (e.g. mydomain.io/my-plugin-name), and whose + # value is the plugin type (BackupItemAction, RestoreItemAction, + # ObjectStore, or VolumeSnapshotter) + : + +data: + # add your configuration data here as key-value pairs +``` + +Then, in your plugin's implementation, you can read this ConfigMap to fetch the necessary configuration. See the [restic restore action][3] +for an example of this -- in particular, the `getPluginConfig(...)` function. + +## Feature Flags + +Velero will pass any known features flags as a comma-separated list of strings to the `--features` argument. + +Once parsed into a `[]string`, the features can then be registered using the `NewFeatureFlagSet` function and queried with `features.Enabled()`. + +## Environment Variables + +Velero adds the `LD_LIBRARY_PATH` into the list of environment variables to provide the convenience for plugins that requires C libraries/extensions in the runtime. + +[1]: https://github.com/vmware-tanzu/velero-plugin-example +[2]: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/pkg/plugin/logger.go +[3]: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/pkg/restore/restic_restore_action.go diff --git a/site/docs/v1.4-pre/customize-installation.md b/site/docs/v1.4-pre/customize-installation.md new file mode 100644 index 0000000000..c064ffd3ff --- /dev/null +++ b/site/docs/v1.4-pre/customize-installation.md @@ -0,0 +1,298 @@ +# Customize Velero Install + +- [Customize Velero Install](#customize-velero-install) + - [Plugins](#plugins) + - [Install in any namespace](#install-in-any-namespace) + - [Use non-file-based identity mechanisms](#use-non-file-based-identity-mechanisms) + - [Enable restic integration](#enable-restic-integration) + - [Enable features](#enable-features) + - [Enable server side features](#enable-server-side-features) + - [Enable client side features](#enable-client-side-features) + - [Customize resource requests and limits](#customize-resource-requests-and-limits) + - [Configure more than one storage location for backups or volume snapshots](#configure-more-than-one-storage-location-for-backups-or-volume-snapshots) + - [Do not configure a backup storage location during install](#do-not-configure-a-backup-storage-location-during-install) + - [Install an additional volume snapshot provider](#install-an-additional-volume-snapshot-provider) + - [Generate YAML only](#generate-yaml-only) + - [Use a storage provider secured by a self-signed certificate](#use-a-storage-provider-secured-by-a-self-signed-certificate) + - [Additional options](#additional-options) + - [Optional Velero CLI configurations](#optional-velero-cli-configurations) + - [Enabling shell autocompletion](#enabling-shell-autocompletion) + - [Bash on Linux](#bash-on-linux) + - [Install bash-completion](#install-bash-completion) + - [Enable Velero CLI autocompletion for Bash on Linux](#enable-velero-cli-autocompletion-for-bash-on-linux) + - [Bash on macOS](#bash-on-macos) + - [Install bash-completion](#install-bash-completion-1) + - [Enable Velero CLI autocompletion for Bash on macOS](#enable-velero-cli-autocompletion-for-bash-on-macos) + - [Autocompletion on Zsh](#autocompletion-on-zsh) + +## Plugins + +During install, Velero requires that at least one plugin is added (with the `--plugins` flag). Please see the documentation under [Plugins](overview-plugins.md) + +## Install in any namespace + +Velero is installed in the `velero` namespace by default. However, you can install Velero in any namespace. See [run in custom namespace][2] for details. + +## Use non-file-based identity mechanisms + +By default, `velero install` expects a credentials file for your `velero` IAM account to be provided via the `--secret-file` flag. + +If you are using an alternate identity mechanism, such as kube2iam/kiam on AWS, Workload Identity on GKE, etc., that does not require a credentials file, you can specify the `--no-secret` flag instead of `--secret-file`. + +## Enable restic integration + +By default, `velero install` does not install Velero's [restic integration][3]. To enable it, specify the `--use-restic` flag. + +If you've already run `velero install` without the `--use-restic` flag, you can run the same command again, including the `--use-restic` flag, to add the restic integration to your existing install. + +## Enable features + +New features in Velero will be released as beta features behind feature flags which are not enabled by default. A full listing of Velero feature flags can be found [here][11]. + +### Enable server side features + +Features on the Velero server can be enabled using the `--features` flag to the `velero install` command. This flag takes as value a comma separated list of feature flags to enable. As an example [CSI snapshotting of PVCs][10] can be enabled using `EnableCSI` feature flag in the `velero install` command as shown below: + +```bash +velero install --features=EnableCSI +``` + +Feature flags, passed to `velero install` will be passed to the Velero deployment and also to the `restic` daemon set, if `--use-restic` flag is used. + +Similarly, features may be disabled by removing the corresponding feature flags from the `--features` flag. + +Enabling and disabling feature flags will require modifying the Velero deployment and also the restic daemonset. This may be done from the CLI by uninstalling and re-installing Velero, or by editing the `deploy/velero` and `daemonset/restic` resources in-cluster. + +```bash +$ kubectl -n velero edit deploy/velero +$ kubectl -n velero edit daemonset/restic +``` + +### Enable client side features + +For some features it may be necessary to use the `--features` flag to the Velero client. This may be done by passing the `--features` on every command run using the Velero CLI or the by setting the features in the velero client config file using the `velero client config set` command as shown below: + +```bash +velero client config set features=EnableCSI +``` + +This stores the config in a file at `$HOME/.config/velero/config.json`. + +All client side feature flags may be disabled using the below command + +```bash +velero client config set features= +``` + +## Customize resource requests and limits + +By default, the Velero deployment requests 500m CPU, 128Mi memory and sets a limit of 1000m CPU, 256Mi. +Default requests and limits are not set for the restic pods as CPU/Memory usage can depend heavily on the size of volumes being backed up. + +Customization of these resource requests and limits may be performed using the [velero install][6] CLI command. + +## Configure more than one storage location for backups or volume snapshots + +Velero supports any number of backup storage locations and volume snapshot locations. For more details, see [about locations](locations.md). + +However, `velero install` only supports configuring at most one backup storage location and one volume snapshot location. + +To configure additional locations after running `velero install`, use the `velero backup-location create` and/or `velero snapshot-location create` commands along with provider-specific configuration. Use the `--help` flag on each of these commands for more details. + +## Do not configure a backup storage location during install + +If you need to install Velero without a default backup storage location (without specifying `--bucket` or `--provider`), the `--no-default-backup-location` flag is required for confirmation. + +## Install an additional volume snapshot provider + +Velero supports using different providers for volume snapshots than for object storage -- for example, you can use AWS S3 for object storage, and Portworx for block volume snapshots. + +However, `velero install` only supports configuring a single matching provider for both object storage and volume snapshots. + +To use a different volume snapshot provider: + +1. Install the Velero server components by following the instructions for your **object storage** provider + +1. Add your volume snapshot provider's plugin to Velero (look in [your provider][0]'s documentation for the image name): + + ```bash + velero plugin add + ``` + +1. Add a volume snapshot location for your provider, following [your provider][0]'s documentation for configuration: + + ```bash + velero snapshot-location create \ + --provider \ + [--config ] + ``` + +## Generate YAML only + +By default, `velero install` generates and applies a customized set of Kubernetes configuration (YAML) to your cluster. + +To generate the YAML without applying it to your cluster, use the `--dry-run -o yaml` flags. + +This is useful for applying bespoke customizations, integrating with a GitOps workflow, etc. + +If you are installing Velero in Kubernetes 1.14.x or earlier, you need to use `kubectl apply`'s `--validate=false` option when applying the generated configuration to your cluster. See [issue 2077][7] and [issue 2311][8] for more context. + +## Use a storage provider secured by a self-signed certificate + +If you intend to use Velero with a storage provider that is secured by a self-signed certificate, +you may need to instruct Velero to trust that certificate. See [use Velero with a storage provider secured by a self-signed certificate][9] for details. + +## Additional options + +Run `velero install --help` or see the [Helm chart documentation](https://vmware-tanzu.github.io/helm-charts/) for the full set of installation options. + +## Optional Velero CLI configurations + +### Enabling shell autocompletion + +**Velero CLI** provides autocompletion support for `Bash` and `Zsh`, which can save you a lot of typing. + +Below are the procedures to set up autocompletion for `Bash` (including the difference between `Linux` and `macOS`) and `Zsh`. + +#### Bash on Linux + +The **Velero CLI** completion script for `Bash` can be generated with the command `velero completion bash`. Sourcing the completion script in your shell enables velero autocompletion. + +However, the completion script depends on [**bash-completion**](https://github.com/scop/bash-completion), which means that you have to install this software first (you can test if you have bash-completion already installed by running `type _init_completion`). + +##### Install bash-completion + +`bash-completion` is provided by many package managers (see [here](https://github.com/scop/bash-completion#installation)). You can install it with `apt-get install bash-completion` or `yum install bash-completion`, etc. + +The above commands create `/usr/share/bash-completion/bash_completion`, which is the main script of bash-completion. Depending on your package manager, you have to manually source this file in your `~/.bashrc` file. + +To find out, reload your shell and run `type _init_completion`. If the command succeeds, you're already set, otherwise add the following to your `~/.bashrc` file: + +```shell +source /usr/share/bash-completion/bash_completion +``` + +Reload your shell and verify that bash-completion is correctly installed by typing `type _init_completion`. + +##### Enable Velero CLI autocompletion for Bash on Linux + +You now need to ensure that the **Velero CLI** completion script gets sourced in all your shell sessions. There are two ways in which you can do this: + +- Source the completion script in your `~/.bashrc` file: + + ```shell + echo 'source <(velero completion bash)' >>~/.bashrc + ``` + +- Add the completion script to the `/etc/bash_completion.d` directory: + + ```shell + velero completion bash >/etc/bash_completion.d/velero + ``` + +- If you have an alias for velero, you can extend shell completion to work with that alias: + + ```shell + echo 'alias v=velero' >>~/.bashrc + echo 'complete -F __start_velero v' >>~/.bashrc + ``` + +> `bash-completion` sources all completion scripts in `/etc/bash_completion.d`. + +Both approaches are equivalent. After reloading your shell, velero autocompletion should be working. + +#### Bash on macOS + +The **Velero CLI** completion script for Bash can be generated with `velero completion bash`. Sourcing this script in your shell enables velero completion. + +However, the velero completion script depends on [**bash-completion**](https://github.com/scop/bash-completion) which you thus have to previously install. + + +> There are two versions of bash-completion, v1 and v2. V1 is for Bash 3.2 (which is the default on macOS), and v2 is for Bash 4.1+. The velero completion script **doesn't work** correctly with bash-completion v1 and Bash 3.2. It requires **bash-completion v2** and **Bash 4.1+**. Thus, to be able to correctly use velero completion on macOS, you have to install and use Bash 4.1+ ([*instructions*](https://itnext.io/upgrading-bash-on-macos-7138bd1066ba)). The following instructions assume that you use Bash 4.1+ (that is, any Bash version of 4.1 or newer). + + +##### Install bash-completion + +> As mentioned, these instructions assume you use Bash 4.1+, which means you will install bash-completion v2 (in contrast to Bash 3.2 and bash-completion v1, in which case kubectl completion won't work). + +You can test if you have bash-completion v2 already installed with `type _init_completion`. If not, you can install it with Homebrew: + + ```shell + brew install bash-completion@2 + ``` + +As stated in the output of this command, add the following to your `~/.bashrc` file: + + ```shell + export BASH_COMPLETION_COMPAT_DIR="/usr/local/etc/bash_completion.d" + [[ -r "/usr/local/etc/profile.d/bash_completion.sh" ]] && . "/usr/local/etc/profile.d/bash_completion.sh" + ``` + +Reload your shell and verify that bash-completion v2 is correctly installed with `type _init_completion`. + +##### Enable Velero CLI autocompletion for Bash on macOS + +You now have to ensure that the velero completion script gets sourced in all your shell sessions. There are multiple ways to achieve this: + +- Source the completion script in your `~/.bashrc` file: + + ```shell + echo 'source <(velero completion bash)' >>~/.bashrc + + ``` + +- Add the completion script to the `/usr/local/etc/bash_completion.d` directory: + + ```shell + velero completion bash >/usr/local/etc/bash_completion.d/velero + ``` + +- If you have an alias for velero, you can extend shell completion to work with that alias: + + ```shell + echo 'alias v=velero' >>~/.bashrc + echo 'complete -F __start_velero v' >>~/.bashrc + ``` + +- If you installed velero with Homebrew (as explained [above](#install-with-homebrew-on-macos)), then the velero completion script should already be in `/usr/local/etc/bash_completion.d/velero`. In that case, you don't need to do anything. + +> The Homebrew installation of bash-completion v2 sources all the files in the `BASH_COMPLETION_COMPAT_DIR` directory, that's why the latter two methods work. + +In any case, after reloading your shell, velero completion should be working. + +#### Autocompletion on Zsh + +The velero completion script for Zsh can be generated with the command `velero completion zsh`. Sourcing the completion script in your shell enables velero autocompletion. + +To do so in all your shell sessions, add the following to your `~/.zshrc` file: + + ```shell + source <(velero completion zsh) + ``` + +If you have an alias for kubectl, you can extend shell completion to work with that alias: + + ```shell + echo 'alias v=velero' >>~/.zshrc + echo 'complete -F __start_velero v' >>~/.zshrc + ``` + +After reloading your shell, kubectl autocompletion should be working. + +If you get an error like `complete:13: command not found: compdef`, then add the following to the beginning of your `~/.zshrc` file: + + ```shell + autoload -Uz compinit + compinit + ``` + +[1]: https://github.com/vmware-tanzu/velero/releases/latest +[2]: namespace.md +[3]: restic.md +[4]: on-premises.md +[6]: velero-install.md#usage +[7]: https://github.com/vmware-tanzu/velero/issues/2077 +[8]: https://github.com/vmware-tanzu/velero/issues/2311 +[9]: self-signed-certificates.md +[10]: csi.md +[11]: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/pkg/apis/velero/v1/constants.go diff --git a/site/docs/v1.4-pre/debugging-install.md b/site/docs/v1.4-pre/debugging-install.md new file mode 100644 index 0000000000..4abc5a3e1b --- /dev/null +++ b/site/docs/v1.4-pre/debugging-install.md @@ -0,0 +1,71 @@ +# Debugging Installation Issues + +## General + +### `invalid configuration: no configuration has been provided` +This typically means that no `kubeconfig` file can be found for the Velero client to use. Velero looks for a kubeconfig in the +following locations: +* the path specified by the `--kubeconfig` flag, if any +* the path specified by the `$KUBECONFIG` environment variable, if any +* `~/.kube/config` + +### Backups or restores stuck in `New` phase +This means that the Velero controllers are not processing the backups/restores, which usually happens because the Velero server is not running. Check the pod description and logs for errors: +``` +kubectl -n velero describe pods +kubectl -n velero logs deployment/velero +``` + + +## AWS + +### `NoCredentialProviders: no valid providers in chain` + +#### Using credentials +This means that the secret containing the AWS IAM user credentials for Velero has not been created/mounted properly +into the Velero server pod. Ensure the following: + +* The `cloud-credentials` secret exists in the Velero server's namespace +* The `cloud-credentials` secret has a single key, `cloud`, whose value is the contents of the `credentials-velero` file +* The `credentials-velero` file is formatted properly and has the correct values: + + ``` + [default] + aws_access_key_id= + aws_secret_access_key= + ``` + +* The `cloud-credentials` secret is defined as a volume for the Velero deployment +* The `cloud-credentials` secret is being mounted into the Velero server pod at `/credentials` + +#### Using kube2iam +This means that Velero can't read the content of the S3 bucket. Ensure the following: + +* There is a Trust Policy document allowing the role used by kube2iam to assume Velero's role, as stated in the AWS config documentation. +* The new Velero role has all the permissions listed in the documentation regarding S3. + + +## Azure + +### `Failed to refresh the Token` or `adal: Refresh request failed` +This means that the secrets containing the Azure service principal credentials for Velero has not been created/mounted +properly into the Velero server pod. Ensure the following: + +* The `cloud-credentials` secret exists in the Velero server's namespace +* The `cloud-credentials` secret has all of the expected keys and each one has the correct value (see [setup instructions][0]) +* The `cloud-credentials` secret is defined as a volume for the Velero deployment +* The `cloud-credentials` secret is being mounted into the Velero server pod at `/credentials` + + +## GCE/GKE + +### `open credentials/cloud: no such file or directory` +This means that the secret containing the GCE service account credentials for Velero has not been created/mounted properly +into the Velero server pod. Ensure the following: + +* The `cloud-credentials` secret exists in the Velero server's namespace +* The `cloud-credentials` secret has a single key, `cloud`, whose value is the contents of the `credentials-velero` file +* The `cloud-credentials` secret is defined as a volume for the Velero deployment +* The `cloud-credentials` secret is being mounted into the Velero server pod at `/credentials` + +[0]: azure-config.md#create-service-principal diff --git a/site/docs/v1.4-pre/debugging-restores.md b/site/docs/v1.4-pre/debugging-restores.md new file mode 100644 index 0000000000..d50d4aacbd --- /dev/null +++ b/site/docs/v1.4-pre/debugging-restores.md @@ -0,0 +1,106 @@ +# Debugging Restores + +* [Example][0] +* [Structure][1] + +## Example + +When Velero finishes a Restore, its status changes to "Completed" regardless of whether or not there are issues during the process. The number of warnings and errors are indicated in the output columns from `velero restore get`: + +``` +NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR +backup-test-20170726180512 backup-test Completed 155 76 2017-07-26 11:41:14 -0400 EDT +backup-test-20170726180513 backup-test Completed 121 14 2017-07-26 11:48:24 -0400 EDT +backup-test-2-20170726180514 backup-test-2 Completed 0 0 2017-07-26 13:31:21 -0400 EDT +backup-test-2-20170726180515 backup-test-2 Completed 0 1 2017-07-26 13:32:59 -0400 EDT +``` + +To delve into the warnings and errors into more detail, you can use `velero restore describe`: + +```bash +velero restore describe backup-test-20170726180512 +``` + +The output looks like this: + +``` +Name: backup-test-20170726180512 +Namespace: velero +Labels: +Annotations: + +Backup: backup-test + +Namespaces: + Included: * + Excluded: + +Resources: + Included: serviceaccounts + Excluded: nodes, events, events.events.k8s.io + Cluster-scoped: auto + +Namespace mappings: + +Label selector: + +Restore PVs: auto + +Phase: Completed + +Validation errors: + +Warnings: + Velero: + Cluster: + Namespaces: + velero: serviceaccounts "velero" already exists + serviceaccounts "default" already exists + kube-public: serviceaccounts "default" already exists + kube-system: serviceaccounts "attachdetach-controller" already exists + serviceaccounts "certificate-controller" already exists + serviceaccounts "cronjob-controller" already exists + serviceaccounts "daemon-set-controller" already exists + serviceaccounts "default" already exists + serviceaccounts "deployment-controller" already exists + serviceaccounts "disruption-controller" already exists + serviceaccounts "endpoint-controller" already exists + serviceaccounts "generic-garbage-collector" already exists + serviceaccounts "horizontal-pod-autoscaler" already exists + serviceaccounts "job-controller" already exists + serviceaccounts "kube-dns" already exists + serviceaccounts "namespace-controller" already exists + serviceaccounts "node-controller" already exists + serviceaccounts "persistent-volume-binder" already exists + serviceaccounts "pod-garbage-collector" already exists + serviceaccounts "replicaset-controller" already exists + serviceaccounts "replication-controller" already exists + serviceaccounts "resourcequota-controller" already exists + serviceaccounts "service-account-controller" already exists + serviceaccounts "service-controller" already exists + serviceaccounts "statefulset-controller" already exists + serviceaccounts "ttl-controller" already exists + default: serviceaccounts "default" already exists + +Errors: + Velero: + Cluster: + Namespaces: +``` + +## Structure + +Errors appear for incomplete or partial restores. Warnings appear for non-blocking issues (e.g. the +restore looks "normal" and all resources referenced in the backup exist in some form, although some +of them may have been pre-existing). + +Both errors and warnings are structured in the same way: + +* `Velero`: A list of system-related issues encountered by the Velero server (e.g. couldn't read directory). + +* `Cluster`: A list of issues related to the restore of cluster-scoped resources. + +* `Namespaces`: A map of namespaces to the list of issues related to the restore of their respective resources. + +[0]: #example +[1]: #structure diff --git a/site/docs/v1.4-pre/development.md b/site/docs/v1.4-pre/development.md new file mode 100644 index 0000000000..cf3b3c4fc0 --- /dev/null +++ b/site/docs/v1.4-pre/development.md @@ -0,0 +1,30 @@ +# Development + +## Update generated files + +Run `make update` to regenerate files if you make the following changes: + +* Add/edit/remove command line flags and/or their help text +* Add/edit/remove commands or subcommands +* Add new API types +* Add/edit/remove plugin protobuf message or service definitions + +The following files are automatically generated from the source code: + +* The clientset +* Listers +* Shared informers +* Documentation +* Protobuf/gRPC types + +You can run `make verify` to ensure that all generated files (clientset, listers, shared informers, docs) are up to date. + +## Test + +To run unit tests, use `make test`. + +## Vendor dependencies + +If you need to add or update the vendored dependencies, see [Vendoring dependencies][11]. + +[11]: vendoring-dependencies.md diff --git a/site/docs/v1.4-pre/disaster-case.md b/site/docs/v1.4-pre/disaster-case.md new file mode 100644 index 0000000000..c9f126a8b7 --- /dev/null +++ b/site/docs/v1.4-pre/disaster-case.md @@ -0,0 +1,41 @@ +# Disaster recovery + +*Using Schedules and Read-Only Backup Storage Locations* + +If you periodically back up your cluster's resources, you are able to return to a previous state in case of some unexpected mishap, such as a service outage. Doing so with Velero looks like the following: + +1. After you first run the Velero server on your cluster, set up a daily backup (replacing `` in the command as desired): + + ``` + velero schedule create --schedule "0 7 * * *" + ``` + + This creates a Backup object with the name `-`. The default backup retention period, expressed as TTL (time to live), is 30 days (720 hours); you can use the `--ttl ` flag to change this as necessary. See [how velero works][1] for more information about backup expiry. + +1. A disaster happens and you need to recreate your resources. + +1. Update your backup storage location to read-only mode (this prevents backup objects from being created or deleted in the backup storage location during the restore process): + + ```bash + kubectl patch backupstoragelocation \ + --namespace velero \ + --type merge \ + --patch '{"spec":{"accessMode":"ReadOnly"}}' + ``` + +1. Create a restore with your most recent Velero Backup: + + ``` + velero restore create --from-backup - + ``` + +1. When ready, revert your backup storage location to read-write mode: + + ```bash + kubectl patch backupstoragelocation \ + --namespace velero \ + --type merge \ + --patch '{"spec":{"accessMode":"ReadWrite"}}' + ``` + +[1]: how-velero-works.md#set-a-backup-to-expire diff --git a/site/docs/v1.4-pre/examples.md b/site/docs/v1.4-pre/examples.md new file mode 100644 index 0000000000..40b284ec35 --- /dev/null +++ b/site/docs/v1.4-pre/examples.md @@ -0,0 +1,63 @@ +## Examples + +After you set up the Velero server, try these examples: + +### Basic example (without PersistentVolumes) + +1. Start the sample nginx app: + + ```bash + kubectl apply -f examples/nginx-app/base.yaml + ``` + +1. Create a backup: + + ```bash + velero backup create nginx-backup --include-namespaces nginx-example + ``` + +1. Simulate a disaster: + + ```bash + kubectl delete namespaces nginx-example + ``` + + Wait for the namespace to be deleted. + +1. Restore your lost resources: + + ```bash + velero restore create --from-backup nginx-backup + ``` + +### Snapshot example (with PersistentVolumes) + +> NOTE: For Azure, you must run Kubernetes version 1.7.2 or later to support PV snapshotting of managed disks. + +1. Start the sample nginx app: + + ```bash + kubectl apply -f examples/nginx-app/with-pv.yaml + ``` + +1. Create a backup with PV snapshotting: + + ```bash + velero backup create nginx-backup --include-namespaces nginx-example + ``` + +1. Simulate a disaster: + + ```bash + kubectl delete namespaces nginx-example + ``` + + Because the default [reclaim policy][1] for dynamically-provisioned PVs is "Delete", these commands should trigger your cloud provider to delete the disk that backs the PV. Deletion is asynchronous, so this may take some time. **Before continuing to the next step, check your cloud provider to confirm that the disk no longer exists.** + +1. Restore your lost resources: + + ```bash + velero restore create --from-backup nginx-backup + ``` + +[1]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming diff --git a/site/docs/v1.4-pre/faq.md b/site/docs/v1.4-pre/faq.md new file mode 100644 index 0000000000..a778568cb9 --- /dev/null +++ b/site/docs/v1.4-pre/faq.md @@ -0,0 +1,44 @@ +# FAQ + +## When is it appropriate to use Velero instead of etcd's built in backup/restore? + +Etcd's backup/restore tooling is good for recovering from data loss in a single etcd cluster. For +example, it is a good idea to take a backup of etcd prior to upgrading etcd itself. For more +sophisticated management of your Kubernetes cluster backups and restores, we feel that Velero is +generally a better approach. It gives you the ability to throw away an unstable cluster and restore +your Kubernetes resources and data into a new cluster, which you can't do easily just by backing up +and restoring etcd. + +Examples of cases where Velero is useful: + +* you don't have access to etcd (e.g. you're running on GKE) +* backing up both Kubernetes resources and persistent volume state +* cluster migrations +* backing up a subset of your Kubernetes resources +* backing up Kubernetes resources that are stored across multiple etcd clusters (for example if you + run a custom apiserver) + +## Will Velero restore my Kubernetes resources exactly the way they were before? + +Yes, with some exceptions. For example, when Velero restores pods it deletes the `nodeName` from the +pod so that it can be scheduled onto a new node. You can see some more examples of the differences +in [pod_action.go](https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/pkg/restore/pod_action.go) + +## I'm using Velero in multiple clusters. Should I use the same bucket to store all of my backups? + +We **strongly** recommend that each Velero instance use a distinct bucket/prefix combination to store backups. +Having multiple Velero instances write backups to the same bucket/prefix combination can lead to numerous +problems - failed backups, overwritten backups, inadvertently deleted backups, etc., all of which can be +avoided by using a separate bucket + prefix per Velero instance. + +It's fine to have multiple Velero instances back up to the same bucket if each instance uses its own +prefix within the bucket. This can be configured in your `BackupStorageLocation`, by setting the +`spec.objectStorage.prefix` field. It's also fine to use a distinct bucket for each Velero instance, +and not to use prefixes at all. + +Related to this, if you need to restore a backup that was created in cluster A into cluster B, you may +configure cluster B with a backup storage location that points to cluster A's bucket/prefix. If you do +this, you should configure the storage location pointing to cluster A's bucket/prefix in `ReadOnly` mode +via the `--access-mode=ReadOnly` flag on the `velero backup-location create` command. This will ensure no +new backups are created from Cluster B in Cluster A's bucket/prefix, and no existing backups are deleted +or overwritten. diff --git a/site/docs/v1.4-pre/hooks.md b/site/docs/v1.4-pre/hooks.md new file mode 100644 index 0000000000..7255feee73 --- /dev/null +++ b/site/docs/v1.4-pre/hooks.md @@ -0,0 +1,89 @@ +# Hooks + +Velero currently supports executing commands in containers in pods during a backup. + +## Backup Hooks + +When performing a backup, you can specify one or more commands to execute in a container in a pod +when that pod is being backed up. The commands can be configured to run *before* any custom action +processing ("pre" hooks), or after all custom actions have been completed and any additional items +specified by custom action have been backed up ("post" hooks). Note that hooks are _not_ executed within a shell +on the containers. + +There are two ways to specify hooks: annotations on the pod itself, and in the Backup spec. + +### Specifying Hooks As Pod Annotations + +You can use the following annotations on a pod to make Velero execute a hook when backing up the pod: + +#### Pre hooks + +* `pre.hook.backup.velero.io/container` + * The container where the command should be executed. Defaults to the first container in the pod. Optional. +* `pre.hook.backup.velero.io/command` + * The command to execute. If you need multiple arguments, specify the command as a JSON array, such as `["/usr/bin/uname", "-a"]` +* `pre.hook.backup.velero.io/on-error` + * What to do if the command returns a non-zero exit code. Defaults to Fail. Valid values are Fail and Continue. Optional. +* `pre.hook.backup.velero.io/timeout` + * How long to wait for the command to execute. The hook is considered in error if the command exceeds the timeout. Defaults to 30s. Optional. + + +#### Post hooks + +* `post.hook.backup.velero.io/container` + * The container where the command should be executed. Defaults to the first container in the pod. Optional. +* `post.hook.backup.velero.io/command` + * The command to execute. If you need multiple arguments, specify the command as a JSON array, such as `["/usr/bin/uname", "-a"]` +* `post.hook.backup.velero.io/on-error` + * What to do if the command returns a non-zero exit code. Defaults to Fail. Valid values are Fail and Continue. Optional. +* `post.hook.backup.velero.io/timeout` + * How long to wait for the command to execute. The hook is considered in error if the command exceeds the timeout. Defaults to 30s. Optional. + +### Specifying Hooks in the Backup Spec + +Please see the documentation on the [Backup API Type][1] for how to specify hooks in the Backup +spec. + +## Hook Example with fsfreeze + +We are going to walk through using both pre and post hooks for freezing a file system. Freezing the +file system is useful to ensure that all pending disk I/O operations have completed prior to taking a snapshot. + +We will be using [examples/nginx-app/with-pv.yaml][2] for this example. Follow the [steps for your provider][3] to +setup this example. + +### Annotations + +The Velero [example/nginx-app/with-pv.yaml][2] serves as an example of adding the pre and post hook annotations directly +to your declarative deployment. Below is an example of what updating an object in place might look like. + +```shell +kubectl annotate pod -n nginx-example -l app=nginx \ + pre.hook.backup.velero.io/command='["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]' \ + pre.hook.backup.velero.io/container=fsfreeze \ + post.hook.backup.velero.io/command='["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]' \ + post.hook.backup.velero.io/container=fsfreeze +``` + +Now test the pre and post hooks by creating a backup. You can use the Velero logs to verify that the pre and post +hooks are running and exiting without error. + +```shell +velero backup create nginx-hook-test + +velero backup get nginx-hook-test +velero backup logs nginx-hook-test | grep hookCommand +``` + +## Using Multiple Commands + +To use multiple commands, wrap your target command in a shell and separate them with `;`, `&&`, or other shell conditional constructs. + +```shell + pre.hook.backup.velero.io/command='["/bin/bash", "-c", "echo hello > hello.txt && echo goodbye > goodbye.txt"]' +``` + + +[1]: api-types/backup.md +[2]: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/examples/nginx-app/with-pv.yaml +[3]: cloud-common.md diff --git a/site/docs/v1.4-pre/how-velero-works.md b/site/docs/v1.4-pre/how-velero-works.md new file mode 100644 index 0000000000..543ab2f94b --- /dev/null +++ b/site/docs/v1.4-pre/how-velero-works.md @@ -0,0 +1,82 @@ +# How Velero Works + +Each Velero operation -- on-demand backup, scheduled backup, restore -- is a custom resource, defined with a Kubernetes [Custom Resource Definition (CRD)][20] and stored in [etcd][22]. Velero also includes controllers that process the custom resources to perform backups, restores, and all related operations. + +You can back up or restore all objects in your cluster, or you can filter objects by type, namespace, and/or label. + +Velero is ideal for the disaster recovery use case, as well as for snapshotting your application state, prior to performing system operations on your cluster (e.g. upgrades). + +## On-demand backups + +The **backup** operation: + +1. Uploads a tarball of copied Kubernetes objects into cloud object storage. + +1. Calls the cloud provider API to make disk snapshots of persistent volumes, if specified. + +You can optionally specify hooks to be executed during the backup. For example, you might +need to tell a database to flush its in-memory buffers to disk before taking a snapshot. [More about hooks][10]. + +Note that cluster backups are not strictly atomic. If Kubernetes objects are being created or edited at the time of backup, they might not be included in the backup. The odds of capturing inconsistent information are low, but it is possible. + +## Scheduled backups + +The **schedule** operation allows you to back up your data at recurring intervals. The first backup is performed when the schedule is first created, and subsequent backups happen at the schedule's specified interval. These intervals are specified by a Cron expression. + +Scheduled backups are saved with the name `-`, where `` is formatted as *YYYYMMDDhhmmss*. + +## Restores + +The **restore** operation allows you to restore all of the objects and persistent volumes from a previously created backup. You can also restore only a filtered subset of objects and persistent volumes. Velero supports multiple namespace remapping--for example, in a single restore, objects in namespace "abc" can be recreated under namespace "def", and the objects in namespace "123" under "456". + +The default name of a restore is `-`, where `` is formatted as *YYYYMMDDhhmmss*. You can also specify a custom name. A restored object also includes a label with key `velero.io/restore-name` and value ``. + +By default, backup storage locations are created in read-write mode. However, during a restore, you can configure a backup storage location to be in read-only mode, which disables backup creation and deletion for the storage location. This is useful to ensure that no backups are inadvertently created or deleted during a restore scenario. + +## Backup workflow + +When you run `velero backup create test-backup`: + +1. The Velero client makes a call to the Kubernetes API server to create a `Backup` object. + +1. The `BackupController` notices the new `Backup` object and performs validation. + +1. The `BackupController` begins the backup process. It collects the data to back up by querying the API server for resources. + +1. The `BackupController` makes a call to the object storage service -- for example, AWS S3 -- to upload the backup file. + +By default, `velero backup create` makes disk snapshots of any persistent volumes. You can adjust the snapshots by specifying additional flags. Run `velero backup create --help` to see available flags. Snapshots can be disabled with the option `--snapshot-volumes=false`. + +![19] + +## Backed-up API versions + +Velero backs up resources using the Kubernetes API server's *preferred version* for each group/resource. When restoring a resource, this same API group/version must exist in the target cluster in order for the restore to be successful. + +For example, if the cluster being backed up has a `gizmos` resource in the `things` API group, with group/versions `things/v1alpha1`, `things/v1beta1`, and `things/v1`, and the server's preferred group/version is `things/v1`, then all `gizmos` will be backed up from the `things/v1` API endpoint. When backups from this cluster are restored, the target cluster **must** have the `things/v1` endpoint in order for `gizmos` to be restored. Note that `things/v1` **does not** need to be the preferred version in the target cluster; it just needs to exist. + +## Set a backup to expire + +When you create a backup, you can specify a TTL (time to live) by adding the flag `--ttl `. If Velero sees that an existing backup resource is expired, it removes: + +* The backup resource +* The backup file from cloud object storage +* All PersistentVolume snapshots +* All associated Restores + +The TTL flag allows the user to specify the backup retention period with the value specified in hours, minutes and seconds in the form `--ttl 24h0m0s`. If not specified, a default TTL value of 30 days will be applied. + +## Object storage sync + +Velero treats object storage as the source of truth. It continuously checks to see that the correct backup resources are always present. If there is a properly formatted backup file in the storage bucket, but no corresponding backup resource in the Kubernetes API, Velero synchronizes the information from object storage to Kubernetes. + +This allows restore functionality to work in a cluster migration scenario, where the original backup objects do not exist in the new cluster. + +Likewise, if a backup object exists in Kubernetes but not in object storage, it will be deleted from Kubernetes since the backup tarball no longer exists. + +[10]: hooks.md +[19]: img/backup-process.png +[20]: https://kubernetes.io/docs/concepts/api-extension/custom-resources/#customresourcedefinitions +[21]: https://kubernetes.io/docs/concepts/api-extension/custom-resources/#custom-controllers +[22]: https://github.com/coreos/etcd + diff --git a/site/docs/v1.4-pre/image-tagging.md b/site/docs/v1.4-pre/image-tagging.md new file mode 100644 index 0000000000..d86e49456a --- /dev/null +++ b/site/docs/v1.4-pre/image-tagging.md @@ -0,0 +1,21 @@ +# Image tagging policy + +This document describes Velero's image tagging policy. + +## Released versions + +`velero/velero:` + +Velero follows the [Semantic Versioning](http://semver.org/) standard for releases. Each tag in the `github.com/vmware-tanzu/velero` repository has a matching image, e.g. `velero/velero:v1.0.0`. + +### Latest + +`velero/velero:latest` + +The `latest` tag follows the most recently released version of Velero. + +## Development + +`velero/velero:master` + +The `master` tag follows the latest commit to land on the `master` branch. diff --git a/site/docs/v1.4-pre/img/README.md b/site/docs/v1.4-pre/img/README.md new file mode 100644 index 0000000000..85c071c63b --- /dev/null +++ b/site/docs/v1.4-pre/img/README.md @@ -0,0 +1 @@ +Some of these diagrams (for instance backup-process.png), have been created on [draw.io](https://www.draw.io), using the "Include a copy of my diagram" option. If you want to make changes to these diagrams, try importing them into draw.io, and you should have access to the original shapes/text that went into the originals. diff --git a/site/docs/v1.4-pre/img/backup-process.png b/site/docs/v1.4-pre/img/backup-process.png new file mode 100644 index 0000000000000000000000000000000000000000..7d4f10d5602a3a9798f449e4f9a7be0b74923a9e GIT binary patch literal 33630 zcmeFZWmuG5+cpft07DPmFr-KhN_Q)rA|TzJQqoAbAOZr?B`w`uN_R?wf^_$La^2T+ z-`8`0f4-mZkC$ze8O~X=dL74p?8m+gS5bO_g-(W!fPjD{`%+2`0RagKcz+5(0e<)C zT~h-uh%Ra`BoNAmDSiX*wXHO?-fAf-2$(q7u^X8>7@M(s*f|2F5DmF6Ts-aH8hNnUyU_is zlK-hk%FM;Y*~;;)m4iL(v0fu%2iLd4w6u>6{rl%%{d{X>{-2iYUH%>xFhGvSPdGT) z;T-?28~Ccw<5K};XDc(H<;VI(IEDTy`M;k1-Hs5)W8?qm%)dtZ*Hd7qBIrUK|DHAx zbcN0*+6V|@2(nV*8Xky$(oyR*rq1fzKWgQWRHibWhY&KLN6c36ibZFOOEa7kX>grO zsNi}b6VUiGFw-(KX9l)-GoLHbL$iD1Em^~F-W(ja^|@>h52lM)3~TRtZVjiayIt+6 zYZ{22NDTKw;b2Te$@c%esD-wB7o}Dz&i?1o-vtN|@lpOr{C~TLr7?t% zfI|HL{nF{LhxCs>A3qe6!-X5c@-Q4Gc>b$d1(FJzfAr$7CSN`UF=FPvUA3G4uWEo6 z-LU>wTR`PcrQ3r7KC(9}F8p`TjoQop*ARiqVR?{bc-(k!ofY{%y9`vrh&jvsUju>U zJ`P>2@sP*vpELE?#~{Qel)tCve_jGU(i(P($r1m|xBK7o0kmj1^PlPY*N1UxU;Jyh$ZUXKZlyXbzS}L` z*7v>I{EFE;zegxo_{P4WN6%w5xD?xA(nAM}K7f3l!BL5a;h&pDEffLsvX>yy>%&KW2#gu0~ zdhScE`-?6ku3mpCLvL0?sh|1GiZ(pjOK^=TS$ep?XdBcr`Y7nWUlMf{O6~ir^=4mirpD&=#qz`b%-u=G!&|?*U4!R+AD9qXf{&{q*AXJT`Uz>ABqi3}G?dby~aZO`Uy=$k;C& zDxq0QzuTkzd7E-q_+3NqSgVg$sLD}4U99EA;O}og*j-r4>a!&?J+{-`b+e9OrnaqD zTlN!*ystU}e9wP+IiyqH#0cy#@w#j&@H(!D@h*G!?#Rq=%sgx{e#A%wEqw7g*S<(< zyX{fT5Ls%uJ6~W9cl$NkQQENe*}k~6cAWg1d=fjdPKBP84D2TBvpMVcbU_YUy1#bH zcm#sN<3Qan3PkS)*lK5uzenZVg=Ar${>q7MM8f84H^TKVYuF;I>m{=><=UUaeAc!- zn35xbq;T7b!BH~(9*bwTee>PcI582j2ek;W9!X6P_cwYYUxZy1;A=lvtD>5&wo+YU zkj$w=?$TU_okFSJ6?&~l^M1TqHjZSgF0Ji-f9@B~@X4_t%{jtmrYxOY;B6r~9<|Tu zhhd<)nOD2uo2Yr2_U^n|&T+c>tLxL>>*vdUes!Y~7{3+6;O&Z?yu1txb0ba^4yKR| z34fbjkxipG4CMv$zYP0lX#3t4Z`)P0^@%{v!NOrR4+j*QmyEBGYa3}nZSl& za(2;3MuGLHYvA4}(skdPg=HCK4n8W~sKefs8(`t+jXH4|FfSI{JHIm+{n-8xe{vh&ZiDO_kxBrm^@ZpI2aFlJh9s z_^pLMnPfmE&ah?X!VU`V!V%TIyAC&NUt*FA?tM=_D$mi9D@4SJjnAl{(m@MhS{0oC za+g7k=#u|EHhP~1vK8xho3mh%;WN+oZPPxc&prACt(x;c77bA{Qw{8R(Q$|J;0;Mi zMy2*vA$_~}&N02?&FZAOvMYROl_b!0t`4wIg>P^X`yq(MWsP6+>GS}TyR_uKTpCcH zR<&2Vup=@&AibU8*Se0hTBWSe*>uoa-1ffQH_OQprcZFHyp`>!kAg|q6$*X zRY)@GPm+2j8dLBH){>Rqywr-t+8eVm4^P9+x7^;Bju*2gv77wadqXdKLMh#n1%G9} zpBJeQkA$vW^M4Sl;6l8>nO zC49Fnt(hL5)E}gb2IF~!ohm=0;^CFdWCMRYNb%Zl=rb~2wyRrOjOP7W$f4C_F)uN8 zP7eAlgFx-awQ>bU92HlA)$^pPGE(`wRMoj)>zsQBBHZB8J5qPngq|+^Ucl> z5t1p-CfQKg)!5wlJ;I=j3dwHfw}BB&iVR3VA2TOvS_T*uU%x%-@&&v3GzKwqlGzL_(<-e~&UJJV6vz zyA1dVvOt9IqF>%z?vOJe0YL4zgznxQSX-|zEqQkNl=JemxZZE4UcGKGiQ$*wen?aS&I%_?7k6*j zLAr?%s}BVanq3%?t-N1Y)3Fdewgo#ools^`PqlEDO&!RS_mZWrPJupcg&EVeKHPhm zUAszd;3R^w=-B`nGRhQ?pR z{LS#cBQfeULW485`nI zCKNPxq0k>9{>5H5NZ)glP_gDkRP@sG;J9j^zoFxl7_b~dOv9&AmqaM2)tS;EgbdYmtA84IU(M8_~e5`=20W6y?#xB8FLBSvKqog%Z}JxLI<9PZp^^nKBP? z-U{A&DMtT!BvDhRXw;~71p1N-sBh3LNiY9|!GPzt)IfI);JO6&iJL6&e4fs3`~}s+ zkU@Z-wIF30<$Q!h-^3^im@sTN+{5+Sdb07A#i?j5 z-nJJ?U2c+r?;K4KSe$;a#iTDL28Ufyp0!eH)^<5WYz#ubde8Rmn_DK$s1yXid?Uto z0MhupUp}5~!lgVf2JG^V+x4xA0mjtAw&x9@F1W${`AX?K09+dg!2mV%S}b4&yrY2@ zc%f|hbq1lup!dg$To^%h$J&&_6a+|k$ z!t1owE#xJ?paAR{jd_pt=vS%r*E_}A>9kMr3_Mnm>w1XZYzX(dk$|kkGsiA4qeNfD z`d$4nXnc<&GV60bXYqZ~p?XZljvcWNhTwhJf%=Z!Zj{K5>~C+$MGY*JhT%{l_}wu7 zw#Ei?P$bk&=wi_Y`Q10YGnq`)tsR-OHxAx5QyHtTNxx}4h#Ym}CpsB;ZYCzCydY_( zb?m1JCD5a!_<^wRTaWc<`S(A)v3II6A5ydFs4(i%<&=;%dtO_A6I7!e6O=L@X|&99{LSfz^mD%MKMkufs9`#Zq94fEGVnXku#&~Len z%RZ@bGTmLe9LMKM>uz72uCgm*iyl|u{<^%SlYX1SeS2(@;D8e)U@xSF93y(~VfHfz zv0V#ACYmg-pzF{2zGU19(Ah#OrJTLa6?_)4IL__|EeNQRdUYj&rA%eF{n6V^zpq$s zorv~dYCF<;P!_$LYymLbe#^yAmq%DO{a{3%GF0f^d543$EVbIzLu{a{hn}QuPFfnF z^v>~Hc4_0z*Q4aP=xpQ1_!3FNKDkVzrG8L1TKbT4(yt} z47XXoO0i5;jy#z&a*WYpIHaqb3yO;{x{|ZhmyQY?G6?$E=>YtE+2V;xce*7JJg`FQE>#E)lm-&9Ue7nW_ zSs|CRs#>2#JX4S1Nx`Zv$yL*Sqdal|0QdXSeXoDpDct}8fm(vVb_#5iQP=H}wwOc+ zD1`afiV{l>%L~iF&zQtOuDjE0x2ac*N`||ob#Af3=gIat8Ma0US?}*>W8lUhaTJl8 zK4>8Z2!0RLT+np!(`)JWWaOY0h^(+~+YUQ1mdI>MR1-N>2yo31NBjX4@>9=chv1rj?Jq)4x8LsfFn==w>FPH1_zH(Lt-wD(4iO*r7aXmHE+brf9)Q` z;xzkCz-vB8a+b1rS$JL&X8#8Y1O=7|AcwZQ+Mi9Y^;{Fzf3nN0IJCom?BGQr%+IPp zkb8lMN%S!mnYyfTwa=PGm;l!Wf7#;&e+@;rxN)7)+m?x;C?dN9ES7R&3b}2C_?bX> z^=YKRt;9tB3-tnZe}YYcon8tl1RgnJh{`BY)J{f$z zpZHqeNekuYyIrSx+w4 z*JY0^gL})b0n5Y&-dP-9QK>w{FhFZhezZjK-`?LGxL;YDeH7j4O((5BA51}5w&kTE z@l<<{BJn=A7)T#d=x*pERZwuwF9k8-9DWopF#yQZy`H z5~|33Pmf*86~hz`xm`}9@;(&j(D5^ywmHMg(F;Pn&JxY(rBT_Dx5^48I)+CB9EXmy zgm~p$#B{g(qAUumjPhyV_KdEJA_u#5%HG^V*h3TX^Gf23gFkxQ4{T*j`Xxr(dj(pz z=vWfEcg-7}7shkyy`LP$(r&u21qUNP$<-YV9ps`x;0JiO#V!QVEzj=Y*SZGT10(S3 z72RL>V8mf*$4s`Eg1th26-!E6gKKDE?_H|Mq|S&|U|>t`)v0l?O-t24?aR?h1D~_0 zGtyfiRNRO}kh&D;&*(HFd@%^WT_f7)34U2l31>fD)pGJY^$uOP zES{@QNI|ai+X)flK3HzV5-_Jcy8Y12a1Zb)k#qgbac>!h9u;1zNl zI`YM#?6ykGOwRZyJ^{_A$=hQ;QQqAi^}Q-Iq=4|nznFIsvA5NYtpDL?w?@QJ^Q_t- zCUH$Q3+-{o6+}taY2TF!w zzuK)Zy_VaU^dvsompF0*2&Ye6FW?P;Ic&*wyLdt26toN&$*IrmPM`JHgr*4bHgU?L zez4W0CDC)}<24tmZ2-*6^gW|O@(P2hg6tjR=86U?6#q}{{q=4WU_f+E4!*ztT1HR& zMQ|(Ggey2hHt&#TCK~D z_4o1e>|?ZiHwSHPC};6j*=)>2ZME~^`x!vYvS1WS$z{~e-t-4=>D8g1(XA^LJ$jfg zB<#n%V>B1ikpev|InzW$b?x)2q@z>ONU^2hx!*h7!k)WSA})vKBbmPE>bg<*8DT*e zQ<>uLDYQqWm>l&Q-&$1;Qv2OjPV?hR$D4a?T*))LKjS^VU*|t7Hn@{ZXgQ3b7Y~Av z#FWJ5DsH9A67hd!Pe4y(9?WibZnp5mi3senYy6#m-AXTrFN@zzm)S#tcR}}5O_Bcu z8Y0V`y_xo=)Md2)8FtP!EsWQBfO2}`nSXCOXGko3EagZJU118_h}-|Kow@WP0n12H z>_!PHIFARiL>go4%bt@i=8(>POT2eP&H|EpLlK$>vO#ph5JzGU^pC-1L7-ktaa3h~ zOX1QrWICM2Kkr-N|0|Z~{$rKB9am9Pao!U9bRcxPAh~jmGek@cS~zu=O5lAwWOBSQ zurr=q>yfrw-lnhjb?p2En@$8RNvWAx%4+lBv_RA|F!ueF5VdcfecwDAI=kPrwnfoy zu7;_mJkt6H+qZc%q1SLm7-r3TRiLM(8$(%`MPA1{8bDO?ro)Kmm zESdFnLJ~L~l}!9CayF@+n1zFr#dM+`*=L&F~f0UwpQuMP$%eF=9^Bwj5Au7kG{l*cTnQ4GH_Np|tp~bC?xCzI9 zGVMcA^T$!7fnQygPA1Cn1(j#+8}FkYasj_<3yP>drdG&HBAO@nXV^((3LoF2;s6wO zsZ11_db0ZyGPKgLCV&CtACQ{L9;I9l2wI&XBQ|197Rd>cB`{vB$6Ec_$zOl*l2uHW zow~EyMwPtvsdOyqYp&-yhOr{_KQIl-O4qyF^jX~1HzF4=Hrkb$S80R1h%T-ad6zcy zf#Bu|D6k6|)16jfvEDU2vuIz305dy+sRvt|<&lVe$pKmuUUAI(GgbVWSsW_;ktnoV zkC3+r;8JS$`Bu!OxVI}!`gWR#>c%lRnpIesPZX3vd732c~eXy=t_1#L)5Qs=ECtTiw5V{oB`Z37 z=y>fKkAY<7Td>M>m`xnMd-Aw)?Q`qEcG^hRweHPnSu%}iixtZLn;2v3A%(rqo`;(; z>Hgv*xForJP+)IGMKTA<97+1Gl)l3YP*SD!M;8wY9knTM$>%^#cKoW7lh>aQV8dNT zE;7q~G?Dy9b{$J*OUX5_-}J)i=qTXVgy_y#H$}`|b7o0JGC@4qd-xg=HU5SyfvGN( zeeXZ6xL-3uNR6q$#|7zu{{DnCP>>N9pXcS_>b0PtSjuwnG)J|?@)a&D4!TR)$r>=Z zMwU7AT$78HAMy<{4DK+l?27O5k><7myOsH|dVo9SYvXSz=cptzUxIVtNsMMp$Bu#} z!;r)MO-VDeY4lC<1FJ86Lo!8!jNs?A!KsUJG?Kjr)W=84)_+s4D0#o z0LQX=%F|XHZjhSZbLwx4NCUxOgc0b&9VO#>ze-O-iy~L)_sVW|4IXyPN4}IQGjxTX zt;&7wE61vvxm|mgv|WsG0rt z$CPhrZJrv5p;K;!lq}hR4x5-IadbRE$7tsFilJ7UP_Z0+x#*obx0lm(W?khT%@d5U z_EU9rq`Law69vVLR7oty!6w{UBA^-N9JJST*8GNJMuXr8lOkA#XDpTQqT7O9frQyo^yB?5@UjkLx#CVg-z}U`f8s_zN?hp+UjoQmW84}gK#m0y8ZODc!Ah1 zd_GG@B%!cb^;RHO4Ub48XGEqN(XWI1>M;f7#G33Kqy|&2Qqp21mG9JneiEq9 z75eVB*w^fg92Sdw8)AeVVir8B-OzK_YjS%PLi!qA*Ee3Go@iAdBJ%{mUjld_s3moQ zyClEXldF3M-whyxFg56}|DZ-b|}eErOhjdkW!nF zD{)HWA~Ys7D3W5B0l1+T7~*>Xm8O5I)^bTo@-4TII4KEdz=Q*~W(<~dClcifAg7eth zQht%hVmB)u=tir&1lYqiTC*8L4yw&weN>G`JR*&set zlmKn5zi4~zpq&x)vMZB=QPpET$FF9%nvW&$PvC zrjZh_vjju9=B*0SC%2(&0v%44ANt+Ii9lTDof}qhi~CIn`{u)Gzahq5X{`V$XL%T} zkvSznm0Trqdnj4giOy1=pe&+2v4Kk(%e7GFU@)X4DfeHh$kT?LR^z!uj&Wv?MX8tS8Jf8LlUVO} zLB>n->)E0 zt$;*iqQw!2H6v7ye=>Fv1mjm~?KNzFi6)IZBYOj0=?Fs$ei6?0id~uOHHO;Kg#7TO z7WQW+&Q`UT$#zq@IohE?Stn{mn%rA!slS48XZG+i^FIq8nN2POekA_u<>EV8@!w-5 z@+g>7AbDwFjt*OWk{7WIVi6Lap0h>f9z)zSr)d~Q!f_aHV#lx^E?%IUdV0dlx%%<= zHuIb_i4i3hvZ=#3$9UQ*sa_6y$0j1-Jue<+n1AU1KzE0TN|AYU)m2|fYq% z0gn6G6xX3Ntv)}pXamZEc>=wu?}4xsjZ>X2JlvUW*BoMqZhNQ~Ui0UGc;|i48&ys~D~YJ% z!LLma}Zy(m(t`v0{O_ zxFq}L>bvq z#I4G6P@4rVfle+ey(_F+B!?OorbpS+$K;@lZ2awwRHUwIs((%IPUpjhV(+-dP<3Q2 z(SSHi3z|Ixy&xaSJA9g_Lgsrs>9^`jJR{X|9{*m`)q*yHt}k>pJWDcEz$xMFWO1d& z$EhCvc`|N#DW}bD?w6$qvMhKJJ(Thy%T6$-$O+~%Z(c#w%mlL6TJt@d9Bvb`>2QSi z_!dU~M~M;|2|Z?^Nl%cS;CgB$8|_Fqwqux{<(#}G-*c0(`|vivT~WV1Cks_~s5%ux zz50RJ=Gk@v8e+Bz!S4gdv(;eb)Q@t+d%jy*_Z_doL(O0jw{a)rA&9HA5TcD&*mf=@ zm#|`J7cU2wu7Baq04D^y`bQTZ^p1`db_n+yuS8y}QBBW(H?fZ1j}Gk za9SLrb#o&q@XaQWbE_0gED}{9wfdKg!omO zHmm0Y$)oA17ayHYFha6xuJ_~Pq~4Wena7qu>Xxow_Pycv_CFG2c;HkJc{&V`@BGn7 zV)7#o@JaaMWq8-y1iWZeltE!o`>(N_bg)v*I4v|*Y#O6_M2mLxkWNZPE0hGJe+!|( z3;U(1?`cMe&4+?0rki%khz63W_OquVnbmK3S7azAgg_|K<0{)nm|lLjOor(EnNC5P z`YXX3iNmh!px|IWoHdjR<^W;-iYhko(l@niA!0`0V>C!waBzgz)uV=?kHSS^HN@6I z9}LaBG#pwd6*6o`n3C8M?r?#nUaZL-F(W9_n&3RfaVw^MpfJIRGMZ&v8RDjD!{O;Xa2&N%>by)%9Rj2X^yl__qVenO}Ld z73S`A3(2ZyPz2>~69F82eZd~JyyW3$laUOsswTJ>W+N%f;q%w}gqymR1(SPxz1^-| zBne_`ftx)o{BG*9dg%dxutKk9-D&V95{1g@gRM{=o{nC1gQZ7K>*d_u3P#LnY_ZB@ zDGD3>2;KIv+m!qXV_Qa?hSdAQwxRt}l7jUI1k9fUf5ht^vFQfzQD4Ar?`#ELzYjF)Agc@n``fDz4i1 zYC8>(z1df<_3{`OWO|@n=xw?lv$e;#?V#XdWZ(UI4ALnS#E5T3zom}qd_FEpPm7Cq2!>X_Y!tH|->X~JUwWA{KxIAS@=WKGt@!5%{-QvHh)m10_CIq~ zlo>MkN2J{%Yh+S`2n2OnA#!k9X#t63G^E5#t4ZSL0ztvEwYhjMK?5@CLk|H*_NU&SBAKj)3Y7S9vP|U4#*Db#Yf^h_T*>dE zkS{*1?5oAz%l&@u#3}MGK~sDXf{qAiqA!Znq=~5S2c9{lT7z?##iEZ8Y z#TUHWn#Qt@uL1!4dRO&g*yG-)yQW}OsgBCp})4KZQdpGUe2xbi}l4=|9J&o zkJ%m)cKr}MpQ|3K7W&j))>e%HGpPA$buDt8H^pV}8MH64k`u_2iamt{eSQLvz6P|4 zcX@`mbtX{+k_ka?v_!E~-e-0Ct~1V@vk_rse3`3we-jsySd-)4Uc{Sa;(~W2Dd1k_B^s> z)181d4c(}tIHixA;zm)|bVh#tix-2>N9x(?7N2Q8mrsQIpDti_inSasCzNM=ba`$u z{1yon`>jUd-p`h%gg_%DbhRSo-$1ZmeGM;~GMx2Pk;5e)x{B0(jUH+xH?Cb}ZpFR} zq(8sh*qDosJ(Cuq{ofbg-PV{5E3?l`$b1R%Q#8>sou+_J4|bJkyW zK6K!SvcC$+Nev00f`}X zb?!M#_>kIH@(H-lHkmTb@_05v?Ry~WpJ@(JX~xE4_uPwKi$&P&ephLKmJeik6`MBh zkdF4dj-y_Y7K?r4S$eB}cLiiLqxjy8hpzPWbz41waDppd)e~3 zC=)}go(dFLwgH;64`D>hD>8GyHhkqg%uV%`j{NZ`+(KatE5|=L%I|+3k*7y9QG+hG ztL#e*gwg8Meu{7Cg%+Mh+Azsw9_phlgLT)GG3SYv_%zOKuSZe^+@Y%RTDmsBAK}g z0ncU*bpIP`G7;Bb~6!0_2!tj>OEG2{O9i7PQf zMe-ig=a*WQ!vedeiz+tUO5Tk*bJ5mQ%JK8ksxPx~jqkLSb3Q>F4OXcXIcxaTn)tB4 zYMSU+W(7a{yrXkArm)v}$kDJ7pF%_J+)wkBa4iqvFC~l_C|X4IK3N&bX`;NM6WxBT zJKP?yl{l$OfX+DY2nqhD8~S#CQTt-mn1HFtKw&aD2Pp$ql^)>0SJpn}{WSp5KmJTdWN=q|_eKQ8~Y3 z`A#*$qsm;fqx7xJBmBvzJ4CR@M9C%nd`B%eXf_*E;h-RqiBpX!Ng*y5qs?HVRf zrqO^l5NxIa25Dhr6TNptU#LJaGzuRF80GARqwPV)>N`O2XL-Jr*2d5Cn*6Fap$d1w z5}0yV=@lS!SD%S#^|_g@G1x8PfmFoQ?^Kl4@`Ga)KAMx&F=^_A_?0o+2C0v3pULxO7KayqMX8=H#f>Jdn8%XV$K#dAczvVw741QgxnnSUB2~blt zvl;o+xr%*??=pPO!_TS}Ny5&KXIK8EbK(WjDElqrpS)qq4G*~i=r zgod<9Yq9;;IO3ea=7Y(rfCBo9GeJDR&vfFMBA%*Q%FcP-IM0G@PRicPa78x?HOGR) zLSCusBTKiA+KTpXk~*{1xb_mJQi}k8Rc*;KRTJ(PPL~3Q_PzQGVc|t16N*vQ?yn!xcMq~C-C#5;Skf7Qif#`${ zWDi``y!vE?K~JtWD6O$fq~m!~l!S~ZO~uaF;zl6dufQe`V5SRYL9)vX zG$btWaw89TEvQ|ZSZ@P@y)GzA5QM3H40waHX1&or8%@H@Xa~YD7!bB{Ppn_M7Cpu($8*x zN(XOO$1V?Hz0@r=ua=llKV>}-gS}9Cgc;3o^I8~4q*5a1_m{>(&q76Svp)TpsHkmn z-QCJHt{y%DF4c_Q(9Re_J#gb2-~L|17P(x}4bAl>mn+&#GSDwik$G>_DVFoP{hAkS z?2n+(3n&yncB}}(DLF}W<|O-xK&^M@Z(MP-fJ;J@`yiP_8vjp39q4=34b|gvELNKF zfE+^`>4Q=3)=+A}P^B+??X(KG+f&CB7%Y#{a5#tB_2EZdI>0vF>}`uY+`5z^kd;}L zAu$IYllP|WQkDwZW%UI84(X73qp-3Z{^ZLq2?p2RP%%=jYc9i@vE%HE@W zJI>w*k9k4uqd}VUe3(!2WO_oha|E#0#h3t6yXahp1v$o}2$SrOoQ@>OK%U_*`w)QRdoK>*`<I*q{m^wD4MXu6(` zy-1o+m`HmL&@rXr9X`jSk~qJG$xudWBG1ry6{?rJ)pcE`UK!Q-Y>WAy<}iJ&sOO50 zcP-$>ligi_S{j2ZP}&j1y-!k!O`qeN`&Adrk!e&n!*yg5>BK^yhpaO_f&TI!Tq8rH zUq)X5t$`CI$-}Z!A7vwDKDnr^F5J)?7oipqQ<;4?;sKyf`oMa0K5@uKoFulYdvq|7 z&dZZ-)cnu8UO>+NyXm?bSn{P&5g>PyTrgs-hjYg0zVg7uhE<;c;>esB6X4>LfaY!X z06WsUa?@q}bJ1kFAbhUAf(+lr)aMp4)(U7PLOX>y)dspxE27}U-=f=LFmG73q4UQ_5F_*IXf^eyc-}dktKbx$kBG$!$A?eQE^KHBdv*U55uK9)pqVc zCf;b7S+LLhMyEhhh$$)v%;Gz+Aky#afSD|@Qv&c8-l%TN%Lm&`TU zg=@}oZmhQJqk4bD-YH?;A-OniNSWWmJu@tZ^tlW&7KvwJoH>O#c?jvUV~Jv#WC$r3 zD<~I0{jd9E&dHKJgAl8GEk8J6WyTS8BJ@R5i|SV`s;oqC!Y;*fD|74BMdXPTz=OVr zf>a{j^*X{N7({W8$@T*iyOM6vdWXb#`PS|&a_7s(G&@m-T!u}FU?B0#YX)``EX1sk z^yy-2SFD23iu?w5+-%3L&TKUYYk-oyfRu;=e!`Loq*^RS^0ERuXnE*z2!nbT<-Q62 z!ZOMU$3Lt4FJn5b?`1a?^=Sgn9wnC{h-RycJJg&ai864OMn3|+0RbPhJ+#_OT3!x0 z_$CaziNV0xMMD~{1ayv-))U&>wGbN6mXKD&s5MGwb~{N``2A6$FK}PzbaTiRc?@M= zDT-2tg4!7zAl9LX7X9poW~+Jd*$>N~ZrASsnY~_tQhMjp7*1F>O0kT2-@~2cvIbwC zgUEt?Q|Yy1XhWUDvdGQHY(JByMu0B`wya4HI`(v+tw1hG3vkyb9vnNA7XXQC!RR3@ zi-?BdqmTtOo>@V<0pS6Gcg~SCHantsS9gUl9L`B^c6mR_wBKQix0*VOx9443OiHr?r11Cp zBQf0z@xv8fOl}aJLlL z6saQ;0hy67AR9znMhXD$x?VohI)fdDY3}GU4!hN$AH@784ZTs;u?XTfRy3@ynRuZ( zyNZmMljKgjBhWBh5S#~oU#|GpwKP%g$eV$ifAB;~$ffRGHq$!;IL?n0jM zZSPZ=V7FE_lGi=tUlZ`yoqWjb)j!4|yOiH;{Y8J1~-kJ5=gTSpTo0XF<( zSnBp1^>03jovB@MkdA6F5=3Cl8zVg+C+#3StcYv~# za-R);3g3b^T#l)ipo-cBX?#nDs<%JD+u(cRUf3>*=UG62ug!)Lg=2JKDC+<@$w>Ev z{6O-(p&8}dkP4j(rD1%7N#@0BuFE?~1q(~zLsMgYhTR(`k3UFD8vm{8zT%qtBlLg% zF96Z{Kd_P&sveAty$)Jb-zRB~fz~LvVUAuBFj_ZMv8cobD1FPL9y&45ZZMh; zG#^b>{v`1*^bE{9-Tmtc*l`^u(^cpTfgR?Dt1vopBv_SvIek6`yR}>|u_&KTq8T*= zduY$+Ex*IXOqxy(VzYD}t3J*gmbTj)aaF>r?P%|0oe7kcAjdJGB(Zw7*-kH6kJrI! zqPg8@7ioB{-8M?pmv9Hh00p8Fh77DEED?1|SA5=_AK|k^LZl~iQLz-#Wf^3Fz~}m& z5q(jvrzMnx@2hB3Fl{w^_d!G3(`1!(6s@x$d{ zC?wHvd&;_(fH9VF{*lwbfFb|6@KMtYL_nnjAB|3P2ECx*Ur|wF@V2=@uhsxV+=$v9 zr_1@Seo650mNf>J;LY)#1x2n3t0pwFd}X}*p&_g_f)LC{P4SfhGD@{O(5NW~v6(9Q zq`P-6mqKl&3+Snf)s!kten`OtJ0t;7g&G|MfpkW@|C=Ub5JSeaMkm5mG8#G^u}w}* zRgb{IC(&OF241U?K9PEV^PO)=i4(jNg3u9|Tiabg`QYl@fnOrqOQ_53 z$Uw8rBVKly{CcgJGMxYu!u}K=mu40L&m4<2;EWz3lU4D;W%3AbS!ss<$b4Uaab49b-VZ^wxF z{=ssEY=~8T4lkq3ItHnVGyfdGs&!xYrq3dnzdh$F+F;^HJYxFhDL4o%Bm=>3FWvD6 z!y`ulWzI)!+BQBbMx!*t!3IB1k<}`_=uF8~qp$8cK zOTLk+PbPHcVf3K0-0_TS!YFLUFSL|G{!b~MJ8!(aizbkALk>gj1Olcq^Ve!kAFODh zo?84o#JH2eR>&@Fh*lb98;D(z34ic&K{?8xafHtZ?B^$V!dkbZ9-QMMWPbXe?QlPH z>t_p9k7mZ{(UM7pev0-*dzK?-#-y&?WJ;H*#;|3kFLg=N=_p2-7eUVp4Wef8$ig)( zA^4QVg!Pah{NBfudxbaH3LCghG3178-}*<4-;S#bZ?&TN@LB}SvDy%xlTx#otOZ8N%q<9bNMM9`UoqL-dC(8AlAmQm zqWG9lM|EgdB&L|8x}OJSYJ)NNt@b0C%%1Pv%A|4uiXk8N!?ewN1}5IzhoHIb`>t;R zGTy+QXr0i2x1YdW&tjd!wsA)EgD`Wpr2WrK01n4i;tlz$ZfUL(QlIr&3MLboE=Fuj8o*F^FauNTVHba+qB_Vnf_btM7CkaF!9;3_p@1 z1|cMSiPS57>xBwoi???(ff+^Xcb%|7pg^Q%F(~@V``JF%IGN5y@_6J)yZcW6sYI#u zl})1+>88v(xp^)QjIuU{vs7ZX)Fk51NvJ_d-(&Qj3H*96iaQoNayIfb`akWRqSMa<+iUNbN>Bchl83@}AL%iAOPnVZS2$7FM&CRuwiB zzAvl>6U!Pm&<1EqV=Z8yGw8E%inyrWe^ z=>gHTVKtz!5_b_S;MS~>AiFCP|#WmYAF2cWAO{Fr3=DW z(uY#vH)|IOdd2TFoVa#Pb|{B8|6bF#7BrS+50a-J-($jSyhZ$K$a;=_H1YNu`&K<@ z3=&t96egPZvAFJvWShv!xz5W+5A7#>b3CKi18laAL=1YCcT@CWhUU)8gEu-z+79i-&YN_mBEAUo1E7QJ>0 z6e(l8S-gnZ=)f>5Ub?LLB`=(2Eh<`BYK;rgibD%qgiY`{?H3t(KHXSPDm$uFpU`jn zxn;z!By+L|x?cMx$OPEpLdh7fvn7>0Cu>|@3+Bb&jVX_V+~o`}F0Lgr3!>-p<0TC2 z_9{1qw!R4`i$5%nhH@(JC9jAKM7=&m1?ZvCBls9n; z=lDOgjQ)09m>VeSLMJTMK9EoNM>C!$>=2Z&xnPe?JiYUW&U=oh8!!q8k>?$bd=6(JR&@U_pVhmyqobYCi* zh)Xy(SphdZKjjpi>>6P>{S~@*v-IhXleEHb&&8>CUc?0Lk^|n81-yDL>6T071Gvj2 zyURMWw1U=&Pkk2#=_lZ{NCJbzAE7TC>6QXW(bS0cp^A@2A3T2W>a(GQ?^0{HWKQd{ z>B)ZiCm`Ez3}n9{Hc8=$tnYR#SzLvV_NcM&WSrNoNtk%05lfNSXBny;Ar%VG*v;~D zcP+_aUeo?%=0{V;w=bMy8YRp+oj8E3N;EPg#B#V&jFmtx6idbJ4qhOW&IG1;AB-tQ zS0#j;#pDF}Eui3Amz=bP5Vm*7VfOEi$dKjlK9>#3q&k?gZbu@ ze3ufJpz(2#jt&RL3jwo`9fA73y&g6t$w;Bo`@bz{{o@D*E1kQ7a51@U1FbGPD!JgN zME38i@0Vw+)13NhS?}p|;K!{K@?xArshpp-`DwS1lmAo=BNvnU6~&&ayD>HDCH1!h zJb)k=<}>$<1uBv`Nw`>!|AXDHmYvK#N~9)7Ep?H9lm8~#eEdtUK(VcR6=jp=%?@p= z%Ng-38k(D#!HPlJJmbIfQ)!(w9sV=pE4~66m)O}ER=(6b+tOP?h)9T=2-YvONtJYO zW9beUWP1A~Rs;ftl*9AH4Coj9{i}Dv@K-v&h%b^cr@nGUZ;fJ(m53d~RHl%f52V!Q zGTU9H8fv87~vn<=nbtIQu%&g?N`3-M7O%r9P$I4kEB8t!CUO z?2ofwpANAhT1Ltb$BU*PD#p;3%m#eaI`UAZ%Z?ph#~UoShF>q(`-^93UlE#qGUT2p zfiHe*I}Q{l*yS`1Zgm}$Y;_IA<=Uky<(Lb0zjd^8-rb8{@rp(=ht%#Yf9jI#s2n(E z-xqmz?-E5oxm8=J-&uZ|ALYnRM&-bHvZ&KlpManXwJ~V&xhB!%!KX9pliV3?o%*XJ zaJ_kF3=G?Sy!Yx3em?Ldgjho`U#LA$*?M2=KIwUbzZe>d<+n)M^9|FbyI1)eCOf?T zPkZ2>juK@$E}k%+qxcy$?=6q!P)gqArcgg*Y_rE%wpyw*_fy_ma8zyiJiBEQi$J|L zBZGvGKnMZ!ug@H%6ZX+h908(3x>?IbHk zy%KHQHv9zqITJ7nt`c4uWlVC~5jY+W9{TLrULSU=L?r2K=xiZb7D-D?s;2}DqoUtM zPoVI_Yo!Wl5^4q$wpz(c-yH>$o9wWkzx0P4zM2owzrQj~`g`sHGwVd5feVu#q`#*yM43us(ffV)QgVzo`&5%-XtZCF;DKr1f5O zO}So%6maF+qQAJGB1vq<-hS&y-1Dm4vBl@vz%!up5y? zF)wzWxnf3A5_C zLdLdP^`b~$8STo=VK9`knR_{I5U#PVdzWFoKgW7YuM@Sj9wGiBFEEUbT)_jq(7~

Oa*AXK>A{SCkTGAeMuLb#W122eJe?g{aj^h2pED z7A(SE4DPTu$416 z`jlHRi5DxuMy49pjA94NK_>#6Thv~_I@#)U|Cz8Qp?~n<(Q6~DZ#{%!e zr_gkobk%Z({n0X0UuJG_bACjK!k9JW^zYuO8Q?W`&N81Z3PLN@gMLvZJ^dEkVo zy!htkBabLL5mR|yg1!fJ6&}fBDI6qMsVbsamUZ9nApd)8Hwgq?3Vvttz{fSO^3Is; zNp+ewW5rLuto=E|71uFw~kvmo5aI_ z?vbehjzI;rfTh+nhs@zIlgFUo->{W;T%wXTjY0_(l7&=7s*~N0g(NcKO@>fb&GaqD z@6D6%_7+-Fb-qoxY*)9!V%7amai%}-*(9f;1!)$H{t}RQZhE4%r(!06Kg23hF=OTV zx59WrgA9^e#d6JAXrFfzVbI5_AYt{{={)W!&)xKhpSycVowlFaD(PF%I-fbdRJvcQ ze|m`s7cI~e0&aZHPu@E<>((`gUmwD-bFtbZqXGx_W$Fe*S6(XOegzItaSo~M%kEK5 z1!cV@rj*e5x!j9~ofnAGmmEC#>&p9#lV2W}ku5KpH4ZCMi*g9$6TG?Vg_xzT$mSSe z%w(iXdWt#6$>r|j`bU#ojo`6I#?@>As>9uupxZqEq8&N^RpAzIXbRz!EZ0j8+qG^I zGnAqMIC=^FWL7;`F)5Cp%yz3&ji)`%nT06*dbWVm=tKymic)L12*GZr2|UWN2sfk2 zbuY?iC^@0fvOS1?FRLFt)}}bFgb2^N&#fT#db3V>@o5QnIWXfs$YAbxHIG6l9VaRi zh*^)ZD@EaPM6L%D(3LugIgvw21kKQN74+z&w!XGi$d<3Ho)R4`=beR{Bl0w{H}{;G zY+N-}TJm4c6e4g`TIjgXKOEnp9RFXqb&}$&Gvd`|brnv5!f0G*)xTR$X@dCRDyyh~ z1p6>sa_D}P-YTgKlMC02%o)aOV<0Y*(&HuTk1?QX1bBpclJx_i)E)ZSmla;7{Pn5F z6C7|?FzK^}fDb~ifx&7#ywJHXq@3UWA~XK}avXxcBwID??2r3_dd`MWZ^!R9FG{=o z@RQ$V+Im|sHS<48B)KqNI=2Cktp!EWcaF9eLBC{~G?x!CZ)6`nH%SaNnX62^g$u{w z{nmz+ZF$$%iB>&t@-Yej?s1>0yjLlmT)>`b{B^bzWOyeyU@gc=YcK9&F9Y} z8kTd_PIpUcc}6I=MdOFTR#4#ggVdu;{jMt94DPFjspw05Dxg04ZLA@!r-m13kg+^P z&e9j8(`Y2PYbHM9WmZx7lTKfOmzNW`#%rm+i;~@W2v<%f$x4y(t|g;t#bIEXp`k)A zCb*)zf0zG;>sf6#FNmx<*`Y|3Ehj&q@ z1N>_9z=|SCZdU=$#+!)=|KMhjh1g0b4Hh`!ZsY7eS~H;5^y=+}^_W27`Q$3PZq~9Q=Y=|Y= zSP-m=h8C7(D?t#h^+U3vt&Kr7k`fuB3f5_TobXZ=claeC4M1(z(k_T-U;mG>%7-Q?&fHWOK&a+9cv9oC{%&9M zu=)yixXU6QTg%8>3Y3l>9@y@Lk1F&#xi^-_`a|%+9^nOLf^zldMZ44wv(hWz`8pEu zIDa)4*y`@6lC3$H)!w?Wir!He#!z!c%%7~v%BcRnHd*DavmNU~tAhun#|es~h_F|PYQ@KXyW8XomtwV+R8I-+O1%HH#`%I6iHHB?|f!lHSj^&MYaXqi+K z+cl|^=7C0{YP9l0G!tgKKg)Pke#TPnf%f#pk2l5A8J*|EPLsuEk6cWq{Lb zaUWzPM>~F~hhf#|Zyw)MkUm)EUQ=jlhTxt>(})+rTYTc&8=x#Uza=F0 zq~-s}P&DuAK_c~Gv3^#XX_$~l@!ck>(xv&(PaaMzhvumhHfYAxWM}}Gz0NT-Z|wW{T13Q8i9cTQ?ceX#C<@_yiH?o}DH_&X^8Mz$ zrFVI%(}C)Rf2?x(ULk0eRdYjHNkX1`3#n+8wml2JMza6<_9r_+%xee5@DaYfIt+Qn z*(iOv#V@#9Ic3;R&?!flZu|J&@P$oNO((~LZDl(JFGOLWC{ad-WnBtqp^nO0E<8fZ zx~|_*f+FBk+$c$4q8yvuQ+h&L8rS=wOaO_r6zFI8K+p)c@xVftI4Hc9Oi-aSnh{Bo z7s=!!TRgF8^k}OckRrZpWJlOCz7Lspsq*RdkBD%`{KGaIRPB>v8J!-ADG6{{_jV=k zt0cE+3U;X<#_)}|B76)->OwLtXz80S=Xb|637>b^E;3{PnNr*MtS=8E{gO;PS=FcL zKOKNbo3^~AfbhzqnWbkx<>`aLfYJcCS1#r@ft1(CJdDL$V5U`0X7TWBIY!)xSYM$J zoK+V^O8ejSSy1t~GwV{d=c={c5dTt(I0J61ZrZ4A0}a(+6e3v++pUPl#;|&JE^(Nz z%7Q)!KacOgQt!id?gm`Y>H zRs8(ma_>b{o|Jg}DKu7FU=O>m)kr_Yv&M{ircFh#_t2Dttw3|jF-kt?9RNCY8~f0T z0cByYOdU}Na9@vOU5U4z9F@jb1+LA-5YF=Q=lx-5eFTw0vXBnuo*7w1vam888RU{o zHCYJ|fTT|FX3V}8{hrK>|H!#jyoa|;U!{i(qfJx0o9FJgj+P^)uUd|L_!finoh5vf zwF6R@K+k^vOEnNCmFVLfbw(98N*=1y_cBt_s!)fM2gbPo7PNk>Ds(Gv?A>(vT|62w zy-H=W2SZeJvm&&Me0vL>S9FnfVEYYtg2p8!Ar8lb!{H=w z3i!QH?bu?6`+CaxKcGzdH)C5hMpAC$QVFg3@v_DNFIA`*HQ0mA!nGH8>sk(#v7|PkKpOt<5nf-PqAAq(iDBb0-EEy zgx!ol74ThIjXS7t(0*okFF^Lo2KtDfyAddReM#Q{bN3S+aVznuK;-=rkr6gwVtodW zg(O+yinEGzfT?21FjAEseo@GNg@T1_b{Fb?W_`p2|725on{c4higg}DZsM5u=(Dk8 z)3DCrc z{GLMcWE^3ob8t|~e;=h&lcQI9+wr;W4{BAMsXL?_ECOCo%se62Y7?Ceq{fz@iby2#Z)QI4eK#l6T7;JNAKpnUjdF7h`D zjelPWL86tOozySie=%1AeWCq>Lz+>Qg8RKLw`!Ye>X+V{y*nhe0bOQ&BxXsq{5R3! zJKZE)CA|peHxNyR9`rX}@qGFbfi1TvBea-QIo^s$uTW(W!o^vh?+Z$SuJ$#D@8i;) zEb_mAF5=o~{E!HipG0LQTbPdWJ zAq~GtodLC(R!ma;gh}q8`bQYXoPeg}apJlnQAsfYISYZ^eW(k%?zQ2FA;~FL_)Wh zW^(nzkD1SzkZ;6%p-?F%rRby2>FC1dpj984uG$p!2>vJsQ1n zSpJOeJS95%{R)$Jh)=fyvua^BaW+u(LN4rqVmW^JvPFI5$9;Z+wHK((*eVmffp}9l zV}$!J-+tNt6VTL;%M**TH6MP}s~RJafZ!y&Vk$%~a@YQ0B)zM3vQPL0nVgtsQqQ2* zj1&$(AWs?~OqbQ!d$*sam5#m2di9}N*Hgu_)N2$dy7p&yQVnlIyYzTl*J( z%-WBdmKd+c#!#LH0ABEE^iQX=hKM`QpOJ{r>M}PDQZq4&7hiP{xj~)o8M2lZBa3QxyP!uIT8@hLcbOcO*)Q(e1@BtEll&TLVEQUUd13PvBh~Ygi!k3` zk(B%-&u2RHS3oske9cjt!JqD-a+cD@7cFDP(!71GAqo@7cr6sWvdfP9iF%! zo&Z=Q-F2vA;VVnNr+a4hbz9%m>e)){HAV?o+YXgpTpHu?H4 z4}qSg2QTuiOII`vf78cRI(IH9q$3BTAlr`Uo@Tb~c%QMwDq}QWBSw{8`ZaX2Ed+_V zQ?N(a>TZMZag^)8OoEcNVCs|9%_Ol)2$E??;8*xM=G_n?FBQiX=-;=Z;jxlak*jkN zl~i1m12NmLFoWl~)v?mTB&mz{xT>4G4`^mut5Q|H+Cm5>NgVV(^}hPC|K%wI4VHL( zQ`-G8Sg=DT?`rw)wYz!X> zpyP-y`Kdfw!ril5p>Mfyp!e-D z!FQdDZ+RLGFM8_VTgaZZF!{x3Cos^bZsapNiSEZQX<-=b%doOPrVSS(caq#Qk3Imr z`g%4lKl;c1Z)v@cw4+F4v$?4pBL}aCmFP-jI@@6|j4$O_PdtNJ6x+$+UxBS4Ik>a8m%e#&fP=@@;c)lq#13&!`>HA4Sq@L#| zRq=z!`a4Ese+h2DMsU-37gT=KJtOsT{<=H$4|~8X@Dv&HA-|>8Sk-IEaf3Ul9E)N2 zX+`B|j355GnG-ZWDRRHCpRctthTi;cHw$%}jKEYn#T_kb^lqnz+seoCdI!5l$o|dy zG;tm?m+!h$Ua`|2pWULB_h~#$i8!&-3$+;sydrsz)ecqTi!gib~5Q4%9UH@7S6F|~h4m}26n2wn7qTsKBKuwRr=9c+15uPdX) z=G)5$M3WffWVM*ppW}=y2#0tQ`ncx7MOOhkb3E;Hs8-fl<|=p4khz~t;fpMSuP-H2 z)G}2kA5`d;I20>6%NcyBDXMBat2S}$P!(Hj?5reOKUTh=nz~9QM}Ih1{rX@o?~ZaB z2kJ7-mFhE?@rR91b4^%(MTa+$T!Qu8QgE}E6SY?Pa?4;O7A_ZW9uDLvP7O}K=2ySb zerJI;u|I8LQZYlqU5ym_hd7umgTC2kP5bWQCrx;!hRI#<@wAXk{Q*M}a@!n#wAuU5 zxM!59Rf?wd)I~WD)jP~*nEytjg%<{}I7kZGM=l}J$|K{qdD_m1VQK@ObBqARD8e{Y z@gDiqkE2wOs#<*(M%i&THEDit0-#qz>w?4g32lhJ8O23$STUErzANK*23QFexW83C zmh}VHDJi6#j8Hu?No0WJblqO?E7QguN-QjVs>B{s!zi~Be7Vf`D2}Pxo09D*M)+y6 z4h0?Zw?%mCh-@>b%pal*A|vBjs+9IvOSsgBK*s<^$ex>sDRCj{+hy)H*2U1>k^Tv z_Vs^L!87O3aQvPug-L|wx8BHP2|(XaW%x_;)_e@3VJP9;XKHVRw;@l$(O#O%!N>+# zon>Df+zm2=LD)jm>xYjN9L20Gu}yPD==Oz3KACwPDHIuv^f49`15j;h_WS+SY}vzp z*ZO}e<_VZ`r|?^V+qXvcOn6c^vGkmZc2sG-$vmQWt%+k8QU@(Qr2ge4=;jb7UydYx zsc+ialtjVz`X%thC>2eo%;pOBC~%sarxJLy`q4Zh>MKnH>7j=Y(CC6#S-g23QFs*yK zo9iNln4_UPxF=7_F7IJt4SwG$b~AId+a~BDz2rQ`5ODp&l5DKA#o?C507vEBgpNB; znQjIl8cbAI2iSI&AG|d=A5fe1I$SiJ#Yd_V(z`8yD$5&`OD9i5|N83P+`6HBxy~;T z{1IKvtP>Pcv0y0IbJ~Q$;Go!uoP|jSy-AFDJ3wXc3ZT6XQ&Y1t@K62E+uf4zcda{YQE+mscqN6tl6POxMf!Tukj& zbN@{GairimWP0XjW#C;pDdYKXz%E62>F=P8x8*BGPmRpy6nig82;>j{6V5Pj4Pqh2 z`ee_(eGdEOzKs7CFzXB9@@b>Mk9bdoB=iF#ZTKKqW(hd80tVxnVkCw356pkKR~RiY zND0Z%Ymh&YUAJyU$dXUwzx>Y^$3B9>Ud;ns-`F#D4VYFJl+hl-h=fcJbesR!`qo>E zGAkvpf8{1|X2#Vj3>-(M-u1mX%y_mw@JUT_J{s6lLO>`|xe*a{>_2ZAkeq5!LdNb< zfEAo4=!o}i3c?10Rl1120v+f^0SX0vy)*s&bG`^IpGBxRuzHNiA}&?|ia|dpNGD$e zR3`r=(*T#%?`=>b%}cZHqJos8BCV#lqVR!Ak^nOH@NDb*pz5YA;FQ|~*78s!4a+A> zED&IYcm_XP<@VjULqNOrNOU!^_J};>(O0NEcIPcfop_@s;ky#2GSz?z_53pLNs zH^Po~e?Ompfda9f-TfBMWb|$$uTn6@Eu57R4Ju5(nWuFNMfa19kZ9_br~p=0ON}J+ z;NdTixlxetn(qT*UsBq(DdaM(P^nH_o-3l<*l(hi=&d144ly1Z4xO?rN`uKM`F24WRh0 zmkbeb1!9yF?s$7$RMC`|OBbwKYsfTnQ^oe7Bq|Ss-FFtBz5#?iQkQDf9g{+~Q>`n6 z48ixY<065{Yr=K+*F4B%cvcF$tk#=1K$>~rTonc71yb0%qd$x8EeFoq zvsGoOQ@n+{;2JlDjV4UDm{Au-4Otl{pmZ|L`sw)GM@sU!$749 zhksCNE+o?&;^%~$eIaJ03??3CMpM^2@zq5_pM?k)ULsO!s&y*6gn{qJm7Kl+?mx{| zNjoikS9Aq_{NR^mPShB=8wl7K^1g=xUWe@R{ubY^NWj6jWsA5Y`s1_u2OYO)`x$J# z?ek!_ZPaFwq#RDVC<2fOSxO zF3g>5NhmP}4t3598gnp`$T=nmYzA&~tkaFp8NnJCvUy@`+|%5RXN=!ze2q~s4NER4 zu&8d-x@{L(2|0aGVYI8?@7w|AGXqws-&cFrY}oP?#p^K@b?2UL`v4ZN_; zVTfgh_VLB$j#t!BKs?}6AiGMR0s(90a|EaMC^NZX|0>TV@Z+W5ncIg*9CR}Yj&=f4 z>~8m96n*mvD*Oc#6B+@EI*<8XRyGENP_Hc>A^i;&r|`aX7UD5p$dmn8WrkXiNwi`Y zA`%WlL{y0$em0cVfOq3&Z7WkiUQsGf`Omz;LmI>2p391zEAOHV?=@9 z{!L=@64>HR#JRE?;@!D>?@ui--VOagdGhQF^WKjf5=fnyo`N&8Kes1cpg*Kyq5I$7 zo>VniLD0;7KzXTVjnXShNtH_HKJE({4PEbGsy6F>VRQ%?Vh$Q%Ay+H~sl03P2GOCE%`D_o8Y?6^NH$1v8lPsP(KdKdk*cubqBbsu3?%|ANi#u!P8V zB9cLi5P~v@J%zA6N;6EZWpHW$Y6&-xCRK<_%9R8N3p4PqDaeYF>i#M#IDM}s6+2ky zwLP7=_YoHf%+=k%dS{R-l=3FciU|_rgp0x|SF(S^hFH}>Y#Y%*TnU5-h#lKuNGW5Fxfa>M3A_^S6+xRd!dT?8Uv7DYX-6u}KLt%MBJL025 z_w7KTo-iW)U(-u=ni0Tt89#LTN0gHJ~p{fNkf zeLj1gR6c^6*Cwnmfi|2OiqSx{W5ogv*+niX5c=h!tC=xY2#%jVpoWWiK>M}#N9dO2 z<2M?BG>=S!?x%I-ikWdh#^C1x4ttLkH1=jev11N^083>Y^?Fx28oCG=?1Hu zi$?|_6Nfdu-{YU+02+9jfx2*Vra`Y6H69WV%!d}4Uwuac5t7eUE89ssFHz-R3v>_A zc1eUNcHEAEk)2=_5W&^`^Q)|VKG*|wm~F8)e=^V8$&GAzT`^FG)ygRf%+6+J_^*K9 zzPAlI6w_cqL8pY%h)QDH0rURb44;d_OK?mx52V%=@f*Nhy7H<&HLqnrk6(d)W2`U% zW!6TmK#t#9PBx@Q$EfIM2#%!2#<4UQAa}S)M~E2Ri$Si2ewljO zhs5g_;5YE){M6-OkhA&Q&Bsm7D3cRYaG0kQS37?UMMy7qlko=~5)yVqyt+h&8bhK; z{R6Fg3Q5w_cF_EcQ^@Tubh{LK1L-e#svb?LXc1}uU0V7S6R*E&#ieWQ0$?U z^pd|7|7HKI$RbGaOpHV7*F8|?$ni$KM3CO1k0@}kioO2&^+P| z82qW9CD|WKo0p8Aplaq;z>dB%x_r(wvDD`lHh7Q?J1}qQk5mb)F&Ok1-nst9a*BB9i5Ts}rz5hmYYXCuRjhqhC{e@p3n z%|FStZu^JJa#zgbNp=53r#zya0qlUSXsHX!x7(GI;C|~K=~l+e%cNLvZgp1oOv#t? z&smFvT%#))OQT%suDw4e36`xv++8~VyCX~d16B@{kjkYjYR8`*lO7IumULZ1{>|H+` ze*YT2I}4LV9TAWQcy+l%YkR?YgtN|ygQD8O(D+Z5V2J}Yy{9tV^BUMkc7S(s^4?80 zuN%sI%-6_#x4HkT4x@dT>8s)5IsgNz^SxT8dn&L6cqC3nc9r^h{mEWZT<_ub0$<8* zZfC_m!l{1?`TZ_y6~JMlW_I9v!9YgEifa}4(9rOMJmGmbz7YF4MP5pjygAv_*?nSM zzCvNTgqY&5d)n3pLS$r9jq7j6UBuQt@s}0)4tlJm{nk%gPh0m%zx45Wt1)?O{QS^d z*Nbo6eY`&a}n9`;@Ugc$Xzcb_VDZ*cbAIx2?8kDzhpc zfBd}Cp&<~O^BA{NONf9W>!XlexTh0mE+$8_j=wZKagFeF!5II~q z%p8ms9(uZ*ni^4A<+|EhM){;ePzO|}UC1hjNJF#kFFDC3>4V>#(YDBJ! zm9jk#yCh^9y#5e!ZfyqpYT}&BWn2wcB&g8`{WU7>;XqcdE&{zLo!}*1kOy#4*OzH% zXk=X_1O$yl%6xTPY!z7(BXJ9az?u5xvS5K8-M6w?pC3LXw6!Xh=Qo@u$NtWCaW~`pIWt3Vs6~32d1mGT1R2RHeJzW%;qod!P!` zEVOOrxgOuDPLd|IwIG8kta~hTutwOTgj|pdowN^L4l^z}jTr0YBTObd0;vt+fK!YY zufZ5Zcp~4*OS8)H5Q!&Ds~N2G5c2L$cqLt_B7`UwT)q}LRO>)oOl*nMT~kw2g18gi zjGw)Uy{=wi?cYOaY;0_ImwjFBldCFJiKeicou`(p=&^c!xE>O+1fXN9!k29&VMTWA zyNS>9iwGS0d42Y97M{`nMtY%l4v!*JJQFX52fmwkgda2*cCGd%w~Y=>OppbWXGu|D zCHX&I0Jt;kfS_x7_0T(ZNv*q{CtE6yOF>2n#u`H+Y>~2BTJe#CgTqYu3nh10QB+xd zSyu;r=Icy~%Z2->w*S|uFrb!c-Lno76dz&-d5rflTkW(?w9|RteY+P9Mjy7DI17mv zosqDF`}*|Zvx_tgN{pVpQmG!@xS8bgs|~8-;pX@hbSvU-+}v)4h5gR4`)9tI$@Mik zul`gPcr_3E1+6CqOU2N8nDz9^g^W6Z!rtcxEV7AbQC^s}eF>(ZB znqMFrDC~Cw`})(O;?R*MDG1h54Oavjlc|7gweCx%ias?&pCpcu|6|jFNH^M22P-Se zAL6_=1!Qgt*(bNekubc-rljHM+u2L8oeY*t-o3$tBl~~d0@bn$W!PG@iNb%vp&&Pt zC+lh4+h3_f1;TG#HO$0nPXq2F!w7rEhyc*GUxIvObtG zI-H|4TBQFa8&}}8PmC(&jql(%0eTejVp=Uj{y+Vm9?7fdr;~w2K&u`@C(TuM1%j=H zQzD77FfcHNr*K))^Pf*sdtZ>~sY$%r>WCm81!}GlP$3!t{k8XW%}-VFNG)^p5G<{- zi#GrSl31fBv6Cs2cgWpyhgc1abpKia4jJ3}oXh@GeBO(&9(`>8^W$c)wqoiFSu+E& zU3G4gdSw6{j!`rdcKCJggae?N$Y_zAMnB}0OjG^v@dcND9g#NU4uKKe1nHYx3wscy z)I5?MRkbcv{=c6$CL=;=8C=w7@I^{;U%~r&1TM${~V!c$re{&HI`>AwMfoe1x(9L@hu5V6e^EIjse z)mmc?bJcO3fNB_hPm~Df-Rg3hnCtyS_j;^1ENh-*2702=IP!XscjpH$Q~6%1GSdNW zv-BrPPa(VcTEXCWeuVN6_{|60#{kN1i6`)<^Kz zOx(AxMcc0i(?t0zD=Gr97N^Qzgbk%SW&>MZ7@UiesYc+3*|NM#{bTbZ()X(SkhQ>^mnRL|9b|hEj_^d!mHySE%$2o9Y z?|h){&@b^wtb~mt{&&Zz=#Jk)Lqq?lA}{mrlg`YUys=ESpX2}gx}o|oA`WKx mfBzNgJOBSD|DUwd75U5W&YjW@xoK$NpNhgG`BGU^|NjNw`k$Nt literal 0 HcmV?d00001 diff --git a/site/docs/v1.4-pre/img/velero.png b/site/docs/v1.4-pre/img/velero.png new file mode 100644 index 0000000000000000000000000000000000000000..a1937a1dcaca91fa844ae9d45b30ebadb636513a GIT binary patch literal 45564 zcmeEugLh?1^JttC+qP}n#>6O;t_lJIhV`es4S@RdTQ@7-`{xhBLPS9X1f)Iz z?#&4DPo3CAT2%oA#G4WXBrp^N%xBKEYi zwRhq56d?T@g7;7PuVyAv;=e&$Z3IZQ6qJd@9GuOFIT+a)nMnmHeoCKx*mg>d4E)t z9C0%j6K5+&S1Sj5;=jf!y|Iq1bW&S^y>|Oq0>korWe_df>VPt0d zckDl1`TuI=RdTj6`-Azf@da7<|Azd(+Wx`A&-54a|3%DyIQ_fz537Q({7nB&n;@)w zKb1BJh%ktZxQMzZ=(!(spuWW7=lbpGL*}{pJeqWY2L)^a88sjr>6uQo2p}m6ea{vH zSzM${|CRbBN)!whjJE61AQ|Muu5rw~@qJPEqi3trgUdm;K_H7cW!>+`L=MmIEIya1 zY#tX$00uf3ENEDus4z)BvNC`g;{R9w-<}O$6qK3n$goTO41I;x6D|j2n)MQT*>}c_ z9f1?xlLse@QnmRef}SrIS)bI$9zS89hs3) zs7u%$RH&m%sLhI~hZM{*17_wKVB}YLc}++EqU#Q&CY*jPT1_?Fcs?}>mbpu#^a{i7 z_mE_Q+k##aOqhDpC_3|LcM@CX=s8DM5ZR4?1T?51vX@hm8@(KlIg?nPUau&If+Dkw ze-1MdmegQ}Lmrm;xIrHPr_Qjs#S1~Q>U`J(8jXXM09+9yp5;fu0<3#Z@gi);O zo9_&6k?Wy_I)E-ofJNL)EJEvWZHFxR08wPjPqOMOZ>n$Kr1t^%QoK2^f(HEemH*X| zpq$uVEV))k67^1E_#4s^LRJnmk1jM!?;Fbk?FeB@Zg@9m0Ja~EKU}C#Hbt}F;K{j z_H-2&`CJv>n(N5~WlN%{=F3=dy@`RrYz)gvc2g@}N{*OM;#5Fo&*= zeUyKv{jZe*CL*uAhr&&B7C7@RAOZsPG)U0;?PW?BHJgM7P+@-;8(DX$y#1 zTtmPoVKkGD`?0o3#L><1vPGRXq*%WE*4%mQA)6wAqk##Ur))HmOD-=@LRujDSHj`> z!Nf8O+K21jdojGhHZ6!{>oqEQ+hrj+uQe`8d7o&^R37pBvMrfWfb; z2sOq@ONAv(Ys}1X{m$AqX~H{XGd}dFSn{Eh;qc6QVGn!oZMPCreMMK5{|gs3SVy;G z-$?>nlbsq7T&hw8$^Jq5FK*G$-_B}kFqnCTRmh^@E#co^F%+uBt;1wxEdil!R&1zt zI!JvdPUtS(+yVF{%E-r!Vju1eof?A50sk8GnE=3fdw-b4=;#-sY$$Aw7j@>faYy-- zN_`&UgUCWBFCk^~!BvXU|&w_JvC%^F7irtJ5^0)uRp=liCJs{G@B z_*R2(N7xDG1aCB|kmd`4%u9vad1aB@!!lWnAW5E4Z)MaBFT$6*M-SQ%60~Dw#sEvu zcZHJ7f+w*@4R@;5)1NayYkJ))kS+U1;K5^iXRsJ)^)2a|78>Rt$C&~Z4iQMd>Zck+ z!<0k?AoD?Cmu`?ee@!$>57&#vv-5@wYghCQ=&x*W%+M*=C}^sH|A*JQd1UN%`HXQn zoU-axk7tPN4cJe_(iw9VEtqp${vM3GsuRnAz?@zy=gUg9O&d&&)%0uV0^2_tvw{dFCe=F1zS`;Y3uQ(5+yH=y zCSFC++~lnJyQGa^CSscmV4q2YAZRlAIqaPkGvtM~B=bqKd74ck-9#$$4;kT{UPR>TtM5Jj_E9&AIUpWAHdc9E99by^iaYo0y`nPjeVH)V7B z@oSt zIE~zdZ@e@KxpQQv+bD(3s{fhTAbqT$oXWO*VHSdM`71pg^575Xf>a^g2?d3a^Sdh; zLPZV$Zi!)}>7C{b@J&5=A10VDM^NIw5f!`{gnO>z4ogKxYTStGD3v&IYiXoZd9^Yf zM))umI0+Kt;JZjGasnf|%A@T~VU*K(!>W5>Yk^4RchUb&1u__)g^qZ8&hT;i+IPH{ zc~az6w9E1~TyYpov7^1-AE2eAQDxwzW)_i&p3-l{ymE}1#eoIE|GM{fVc<0f*5)Wz zGslYwHjaf6P7x%T;)-S&GX#n>jyKe{wE&@?)XE{(po9r$#W!-Xx)OAmzeN-nNzB~Z zANE6KNnku77;RLlo|GITr##qlEt>TNQeKVW=4SA>@)r=`>0>KvxF`u1ii5RF$A6(-h z&HaA@ZQ*4AeBxwSr4c@y86@^pFouWm@}yinXoe2PoSVRTg_0=)95`o=#%;%La8v+B z?((93vv{=qKV%=Xfq}ElRr&TM219a@{D!2_N`#@=n-tVsPLvDmFBdb^O(eg<^ayaT zb3Llbwnb=*R)wJX%XbF#iRXWEVCiL!+O=a<;dmx4g`E{iwuWf4ep?2T0!_Rj z6kCjm-~yc6pEmb(tO7(Pzrp@vwywj(@&o+Pi&DvTc5@RVV7S^qbl~(Ec-}>Q)xy~q z5RT3nzALOOYGl*cw+01QqB3?PFV@1A|1A`&q%#Tgq;-wb+B(8wgqcjDZ6*{0D)S_E zC|iED@&HWcYxwtsR*NAdakFAF0nf|sgD(_n|7ap3^0Cm4F-cmpD1!ghZOk_1i##)~ z?OOE%+zHs@#-lrHd_$YdAW<4Y;_AL_K|}K|KO5a|@_%cQYqap^ir!3+Dg{5g^Q)`A zk1n=kkfW`c#VpN!$NS_0A$k2_+TbG(wBw{boyT0MsDlkJ6(>nKnnbh~e=jU^8Q@N#L%2H3 z^=osBUgV&z=wlpm3natv_9a`AcBHCdAga-( zo| z6I=^A#h2`j5#jzG)@k+LXRjjUQP5-lAo`A4tMQZ6Vp#p?zpgHeMEGO2WBbCr9XQ0! zE${8boNbmg`)x<-(z0Jx*f-yS@z&D*F-~pl{@4cGW4~<+IR##GVOJX=?36|!dbA$!6S$=CxHX2WHhyVhh5wYTr!Qs*Ccy3G`>O+YX=Zr&y~UdW7&WfB*Ol4 zzY`6-Uy6HbcwX*{USpy7;I2^-=UxS3#VgLJhs6n;cCUqZPyK%1I~5lU9{UnOl2wxI zNdDl|h580cPRa-5E!O6>AbM+7;j1sAMfKdCFB!bj_z@bWgjVRN9;BxBZEGtC7!1Jz zPsMJrE-NzXu!2OG?x4hg6f-su?1WAgxfaKvAOkM1lMKXd*G;vtR@T}l*GJLsWI^@z zv6H*)PssA^caFUcq%6hf*55rKF+3j;%C`Q_+!VC>E%a=Yn}W(|;mLMu&}c{6NEs=g zJZW;W1x;opbLtrny_tIL=YJ=y({*TW&m<+~m236F;1oVYna7-#(m=q zz#tv3GsxCR=C`$tw=$-GTUw2LAG|F&zt=Sx^DwEoXUX6Oe=-KezJW0kBg#2L!sno@ zd>)12;dSR?5h{)bQA;2dfG6b~BqgL@@LY^252& zJofBKB{<^vsH}tCB$Gld)WB>(iXJ0l8RD;jkW)QrOEL4CW%q@?tDZfS8Y?x$W(S$} zP0!v=A5UAzoxg)8o^QAURSvIJTZ-hBY~8=44i;Z6n?ZuL0|npBV){Bq0^`l{pT{Kx zAG-^G8l%a9e<)QJMj7^K+kBXbEqo1zfQbk#GC3vAZYLp4j4qmHVQfkwO{Vm3k#gUC z1BcyEf%M;^tJEJC|7+Lp_hyu}?$0ir z#|9-_a8--}E9M#bswkXh*6atXLmy|S#8?ViXL%2luwE8z?=L2E-5+5JtAV0)sO z9V%>1WpGx(Fx9~lfTDJMl zwq*zS8sS#wJux4$!>l5CDs#Pw14q~roz1qx8^G5o?d!7zX0f;=?tJ)r@emEqSP+F% z9A!5#gG~n?hvNFIYq~9&nDI%Dax};tt(mB;dr)^?&KEjiLbkJ|LYwa`LxKH`eT1w4 zqpJvKY+M)*ay86WSTHSqyv3f!J$;AmZskpr^R@P)E=V%do931z4j#Q<8q13fJen-u zF^Nt_i_*ZsxUaYUil2jGj1egH1|{( zTK_|BJT_%?qx{(q2U`~ZCxb4Sxa3pVf5C+1fA>mpTmPLkIz0)a9#Yyy ziKFZRmO+!TVVO-o;u^%XR%amOtaaO)w6l#Shx?tXryhvd*9TK)td}4@Cft$58AP^m@YEQOk9GFtl8;#^ zx+#RNWiaJP=!ZaA5>rn+TcX zd`#A^kp4P0x`@6>_GIL`er5)b)l{F49!vw`Qne5^Ui^mV0<*yWa@M`-hCT^Q49@pX+bP zJx(JRw$`}XbC_p7QW1gw>`4E}=Xvm~f`)Yr7E|hxTB+jwrB_J^Zf;5#?4Z+)UW38< zUM>GoV;^Gj=%rb^LR^l=Rdg6a&^gZcdX|rX2#oeEp8Odt&9o-}CttT~!?VK7T(d|O zAS-A3sRlYjF#Kv-qC6vBMLgK#B6m!E(1ehA3{mWKi$9N_Rs*vsc8|YR%JW_VZnIIH z)HMktqszIScj`1T*`DXYl1O*Q$L zWcgHELNyZ`NM4;y-(~P`eB+`hzC)e!?4Kgm1Q+yCaR8pI(8|2FU7_sgw21E2MTl4P zkQk+N{eVeo8|m=)?^kgT$<05LmOS;JV+$x5~U?B63|aLGr6T1A>1Y)tZZLz6DrW?-{Fx z_|<^J>Y`nq4lF`D(Jx=VX&4yT?|C_7nP%NV-5L+ah7B*2%gq>j#LeWi40wwWV3&n3 zp4Ifoow4ZdPx3=EY;}JHUj{?$6RxY!wppO!36vEAEZGZg<5*o`!5J1`EQ$5^v{{Aw<9Q(HC86+qD zO6GV!M%n+>H75d*NjA4zDX67*$o;6NMf-Iz8WRn8+HVmFJCC6qDW`_cWOF=Ov=!CN zNjNx5_3CCCMWK0E(0O`;Wyym>fIi2ELK)j-w{y&@5+OPh)+$5=3T4;4gM#U#m1{owkSXJ zj);hsGvv62l6%(S!#8XpDNM*l_CN=OUvgKNwy-i(P9HA}lZx}WX}AcR^|sXVI9p4* zRc(OlM)JLhS;3G|cj;!2ksJQ>bM#V6GRxr#A@4%3n3glf+_hEqP6oOg=ROs@gaIui zIN_9#o9&5>5vm}WrHLq?Mp0XWj-JH6r<-B3Z~69&)$zYHd)kM5zw=th2PuqFjuO~) zezDKXwijT4Kv?iu6i$ErD06&kXTspyivd4eZ8zP$Q92X5vWu3&vXy}V(t*ZFLOB$X zn2**eO+HGLB0X9V2(U1*xOpnq#%VnWI5=?B7_I}qGd97BpT(G$Rf@(&D7TRcDBMmo z8J`_T@vJX0>a8IQ*=zY5{?oH`A!{FVH>JP-yPQKOD4spX&x zfwdE9hW;0u@hRD%X2uOEifNEr>4c*7n;jP|SweXIZuDmd(R4K#@jCQS7aHdJ$ZSY4 zQ`h6Zsb0A_X^&nx8TKBRW+g{M7QAo6t{ZwB4x5u#+p2TZuLjq3-RBLJ4^cz;)to#7O)B_)d(MG&c z`EFs}0^Sb>d*o>c#it0f6V)!IzTg5OxoKJQb@~7OYoNu_H>_tuE4V@P&_$yXAMxF7h zx1MkO=vrj5$ezlN)CuChtd54f^t0^p`@L^_v@hc|OkqeHB?7u{knkMu3HENa8UV!x zb%qK0^g%uL5XIkCddg;@##P0z^>91o;iCoj zM2LurpMyM)C`SXj~kc+mE|p3CGkhd#YKBPlv;RC{~7 z#o=Vf-im96sg?OedXNVXq^TkUBcmrVYE=H-sY-xQS()d7amGw}!+>8!!!1-{gLBq6 zZbhKlVmkRnRTMpXx9{{o09nc|x3o z2|rSeq){>{!{sNAew&1M?y#)C47XfWAT*=eG<9VR4Ha)W*xI7+zky>sRVTur@r^f} z)}x6m9;6v|;f6#gDPJT^<@n6o^lCluc;T>yi@WsJu{}gCI za58k`ciakBcqnjKfDQkgYy z6-?8;1;QXAJU?QhU;e|}WF+*SKm6_cHEt>WsJhfZx{E^pect^kX41P+myc}0mWR4T z>p|~MLFO&Kp@#d+=wU9!`TXg3#f_h<>90pfq+U##QWK?TBrHa_W~8?A6}{qIo>7n4 zRrXyXJ@|i43i4;iLd5251g%(Wo||O&%6lyo>EI}ORFl4N@oqUBM%;m2!>Ge_1`MB< zR6x<)u1A+-A8ps}_EjeelExQAAQZxnTQ8ZzOsqyZ>Amd`G4+X<5IUq;u zafNhclT${j|rI*K1RTkV$b<-!&0m=B*-$paUnN=hh` zwFuqWkO|q_P6>XG+I45dp1m9%Qo?G+9?8g4TbN+h6o`hq{HUnlyt>^~1~j2a&)4JJ zvtpi;Lz(k9X1892x%Uo&=O-a1LgH|&>Sn29iK$EnawEjWDLg6ep#XZT?U?SHwmwm% z(C9gdg5eWrfbE7Mz0ZZ(jWs&)ps(-Y1p=OD)jw)TyUaQGD|$>V-)+4`o9f;PrqRY%8xwX zRaLFq)U@tatR-z-iAa5If)B+UIE&^dT4z=QrByB`h#zcS0x+6QQGW#!@zB&IzRHU#P4=7 zaP2t*Tb+<&TUs-p#r0MBRkE(55^!bLW@D}AoQ2ttqTOxn55)nFctIRtB%6fvr{z(! zNXLv^{66EXw_$ev(7(3GbES@W{MGIt=w1$%<#js5^s0oP{ccB zPZ6#D5D19!w$jkrZpZ1A{l9*}1E9${zlzj{pv1&a?Jx4U*dN}ZMaz5qwgWfEVYk$C zVjG+4R>-o)b2%~y^4A7=s~Z9S;tQpaQ-w-Ajz+LHBG#*u&vgVPk-N3@(%017w8AAo z-H_o@hLY#N#&2$b7t?8#wwl;p>WN{-KqH~?0;do)%)_MDstH1TvmMpbCmfBnBaSq$ z$CAA3Q6%4829gJ0rJrA9bCRS`h&+As8v;QoRp|)f=VqQEKGsW>Z`Xtd`*5D24g*C+ z;FW<3E0kx|g1XxF%k>FD=^EYSnzrCy&Y%nxuBvxLxPvQHv2aP;Xv4OBhi`gAlj?!f z$`DUXj5pdL9R!8$@_06Q5>OS#yD4;gCUN5LEeGXop$#-LWm&jnccdRU?&kNq?nHl# zzNRN?!HVrby_&H(1VS(M(nghJZdv~oWSkML$_>DkmA zUkC1A6XulgjYoFJZ4LiZSh)`oJVMZ7D%&IJ15SPX`;Iv#Q7ktMR1tyS5QZkcB<{*E zba3j2bP56byvT?v3@=>pFRO^i+qcvk?$U9?Kf8ECVu+{7#+*b$6~|!^FiCo3c493@ z3P}-W>~_U@-&!E~XO1h;j&e*;D>#ZNUQzhIlu)}h6Z69niTVVaLLojlAznZ8E16;n z^XJ6F&z8yAkQjft2={^O*LNs(6!TL+dgmc97*wKYyFW6F>2=0Wmjd1@)inZ0aRJQtQj?#kF2{)V-(%2a@qUg-j|2%!#3UkUBS~oio0U??fZ}Mwd2vnF8xj$ze}wdf;puLBbI9rLbN;$y zo^&^HTmT=Yhq5#_YX&l4mjxX_%21WCuqCK6czfrQA&m7*1M@q**~2a|!-W^!&I~{I zlc)(WZ_DS8gOK;?lS#f|Zl)XDvM;*tL1+8LnOI#Ja1BA6T>^F^zswQ}gGLzq#%m{? zKeh_e2FmuLL*{riA;=*@uL~OQi%$_GoR2ITBD}h=>2;Cjq$*ql()WOf8HQZq$GA%f zNw(`Ea`fAP|Lvh%(4AgSy@cVJ;6gsBw<YK=N z2G_U&wP+i&F@zz_RZrtjiCe;j0`wcbI#RUhd*)JJ>k}E8iXv|m?;B(1k;C?a78c@r zHCF{D*b;73FELB3*EoFSEgVj~Q8 z@w<#^;f8I*i1Vv^^U*4I?C6A@foyYh?OM4()MzJU0YgmbtxXr!c2rh0;9DsSIF8il zod8N%A3e{VFsd!T$Bm@!rmTsIF?9Mr#`q^6!hi?OZRk=`H|S~xR-p3CP#@QLD` zN-b5&5?t>Q!ZC%Qlo#i87MFvKIbo68L^=zFCns2Ne{P}xk_-NYOrbyE`|cFd;z!YQ#85bGoSLV1=T#|rw3h5;0@NG?xY~Zk zSdh4dUDsMl#UQro4#mhRc>g1GtM{R9=^-8IjL zp1W|VklF&&IfTvNB~rNep^VnUJ4H&b^K^Lt{vxyJr_^BxVfOeA;heq5kvrmM;~aq# zQZjaWSw76f7vI4JIy#F&mm2)^*l_cTH%{#zJ>ZSc&`(aFr(uQDIhZry>z3@WZXNZR z;`!B{WC!-MgD;^Vl?5B`1mh8(p$@#YX=c05xBTo!PG2bkC#lnQ<-t&ZH9CZjaHL4M zj4oCVB}wEbU-{#7;CqCa36>KI{rm`qa~U#%LCGH26HmwxUX$f$8$A8BZe*|tykyyNOa&dXk%=M0T+XW(%UuYbEg`X}5EmtJUetBQ3 z(r)yJI+*A~W-X)Pf%JN1CJ<-IaofLfawnvOfb*9cT?@d1h!q83_kjAH<2~~?CJdTA z_CJKdIVeP)WRttH0Z{6E$}6;Bg9u3SclH#e3qMbcFJ6oyyG8u_?Vk4IQBM|Ko>2-n z`0*2{YD!GE6nzQ`KSla4JuId=Kit_@aq-25>or#0!F~^4YR;k%f>b)AX8nqvtv|&t z=MS2lbe>T(igxlwpQ(4;PtFz!m=Y84c2XvL^jGI7Gkq{LUywG6GCQwcbI5zO{iIl{ zrMZtpNebG@ukA@VoXYv_+M^mgSkv{e+9}@8gKkG48sfu`eg1(zw0Di|Yo1`@jIIO< z-h(8<%#C9g;ST7s{ycds0-^xenQmW0sPZpYFxLQFmKKA(#mb;NRPmixLEjIC<+t$B zRbgRaU3o#tB&FBF+$G{4yyA%cJd|Dbhs5v|vykUYcTDz`i+lu3)6^oleonhm13eE) z^Y@4YR$bq4)i8^v}MI*g0+2mgTY|E2j$+c0dtM zGludd&EN*3c|*{Mc$@KA;JIhFVCTklPu{T)dN3y0>>-R;?oG#U!eX9%e3lw0KH+j= z_!8>Nj1G33`UEs}Cy+EJAHxfhzMXNX3kfdiYYXA38;s2<^j(S(s~ zv>a%mbklm6X=4u&Qe3^5&^P)xlph?MvZZ?XLt!!zL16@jOfvQZa&Pz8Nw`~;xUcrt z%*-1II#n)kHup$%?|TR#LX!ycOl_9XC#L5zy7N8M8}PbKPccDPweC0=+nS00&5Zvi z{YhVBZzI>uW;ydFKX}g%;vSV_s#fn%cX@+PDg4&>hbzX=As9)8%o@z@BMsK#3wybG>u8V6pc7D zf#Iq4MN?O^iHs6!KPAXal1y(_wk>^vUfEcTwdxo?0+X8u21O#y>*_KO#+5w?b1!W= zX?I@dmcVj`d$z6a#Pk}sj@7;x;S>yLlkC5TnH}?0rHjOaOn$9`t626(^Bz-V5~~^3TMss9nX3WE;8Tx!rvH<%#vL|zDsTs z1o`^I#SO_=#Yy=1)0K*(3u#om8xMpx_iRVU-F372=C!DIeYDTn8MXvi3#LE(5iiEU zuhVeUTlzCI*|JhDfQ$nrIP3*rdQbOmbY-Q;h~MU`012gFqa62`FF9p1 zWh$k(JnRm-RKgFPLs*6hXoIj*!N;TW6I{L7(M*thmLy&^=EYu+zTQkHx&(!&_|ixR zg4T+4BZ)>^JXBAh#U)#dtzSuZROwx@-BXT8N$|{UDZd;5J|~8rE2c* zM9S$00yC1jc)tMPc_8-9L$gJNeP+hUZ*2NXw$t+O*jq?ovAG@WKDjMAf*w%$B&~KO zvl|(j5d}TFx_bFXS6G@Iao91joPW{n{5p~X1;o2F3Hql@dWNq zVy=aQt-`q)+UR87`6^}$R+{>GEMbgSGGr|17mU`GV90v9OoAJ)5DE=o3OnJr6Y$Za zzH44pUd$P5#XA+bVReS~ka7 z=9uRLM}@11E~P*^-2>h+vO!DLN{nw_Vd=w3nN3(P76vHmxjbcz%3Go3;}%#*Xq2$9bo`%uoV%hxFk@TrdEpVILirSjYZ71CGpqA2C=gb>++>0g~h= zV2z7(#xq;AR~Rz2ub<>q5TGR_^(#R9SM-5YNd_3gwIg}XG1=8Np*XsCpr(0VqdxDN zoLY6#1{_jTQ%va7kb#$)kZsGLnA<2;BO91vNq7+a>N6LD>hh36ht_wU-tNqJ>g|J@ zSmS`b+2U?x17yTdUl8bN8N>&0l={|`w<5t`tPX1Y9W5|;UPj>!Ad0nGTDIYW)Iq@t zh>@%CD-}BoUAS%y0R=cbF6mo1-|Wm&%qOzsbdGbrjQA5*W}TPx*vI3=*q|Am#) zu2JgK)N=RT8vOJChLU2~EvrF;9o;L6_i6Y60hUtHkja+;nSq`!$tl4+mD`-gx z#dk?4N=gaKNVUMQlddpIFC=7eoShEL5*wP0hR0 zg?l8D)NAn*y!slGXw?|7J!}%`j0I2jek{gIk5-ze3XizWs4oLn8q%>EGy1n=9S#r--bDF>^?tz?p)q==~VAR6F;mTQ>UvUA+@cENU`-kR?kyse?x zp*;{DH5Lbq=UO=Axl)(N3}|Yi=`5#-V(4~b!EM0Yu<(d-e_i)>yeO98c$4i%Jmjn9 zPE7CeFfbOiqVAv`XLorc1Q4JuA#ZS13Klq&#f;d0wu<=V^HRp;zqsh$JhXYUtEq|f z*`RR>qiL7CMs&I8#^Yvv%Sk(6lWk<~nQ>#DvTK=OP=Gnh_{K1+Er;jweI?13yVD_g z(&;xVQ$kJ3@dlbTH7)UZRcyg~uf)~FtoH1c;?LRegXeb)NPo1&nJ+U_vaN@c2u@fX z+nk2aMt+{!CUeJ2k9~{Mcl>GJj?ffzcR(?BetZ0gNQj+N3RvkY!fSRHlBA&`({F+1 z#P|QaTOBM8_z)+3uv_=RAc#Phb`)F*I+cfp@@;$CttUxaXi~mEolSDjG^>sqDGbV1!4$J8>e z_x^G1cFaZT0usoXIc8FPLuqd8`FY;v0&Q4#eH!r&^UNQ|;p9*CQwTPAVG%tXnJUG+ zCf_LH$Kr_#VR;h!U^75V1prFeVJbuh*!0R&IXx5^eM^(stdi|UQQjqF=-WM+``EYeL z-7>(|QaY1m{62y9I)*839OmO5597l_33t}l+^Saxa#9LsZqoiD%}090{W}`8B7(Zf z!WJ0hoIW%rc}p$}t{hT_lJ`&)fzSe3O|qv%JwP zVDvHWN;rR9L$LXYAb(?alTcr?3;1ppI=$BB1Ry8JgPNbrimM1}RE=nZTQXE^;85cU zmog2?xtpDnDhX=_+~UFzThLWqWA}l}=3-4#O0=-9gL3)xTp3{#2NLH|nn6x>ksm|7 zl3B0EE3b3(5s{i*u2Fs!v;B*L#@pnV$<;Hw!dYETBh5$2@vli;t}933@J`-ILNC=S zJKKGQje}3$1HFVLs9%)^=&JTMw{QinHrk21suCci^p{8K%cNW|Z;3$rc3l8W7Vy8R z0=zw~oUe97rs5+*FNO_uW=(w?XfU%n1P^tW6XAEvPv`HC5wVTsYnc4MNQ?>q#iax5 z*H%^q?ov($a!yQicbB`nhdhe~tu8^U+S;=Aqc=PJo{ED`aBto7pPrJ6lh7!S!t!-V zHpY-DNLQ2NRE~n0XID&`o$?}^3UP1|lk8T62nnMV@Nn4wNDUejBF+xZ>%Ij6D{c1M zfNt}YgP;RqeE$dKN|NNI_}R{)EbGyWC4H4h>z~xxo!#~N#gKY(jvOI8ED8@iH6rHV z1^;DP&C}3TxT@akgBAB%u=N)j-q1UZg9~HCZHnU*Yk7JrGK358Ecl~{9jC4C2&68W zya$xU=07Q#Gnk+b5Qv7r&+xU+IOMr;;*Lgz`by*c5T+0Oc%jEahFk3j5Co7B3+B0n znnD2yJ_NvCiik_az0(S>V|361hTp!Re9N>5tkz&(OWf~mr2Q|6LabEae_s0@y*#7JphXck>Ee&MmSVoMw7FkwiU90%v`_5pC?i>|JAJ=LZUa0sr%+_CS z56PSGXNlb*1Rl3ZKl!M>jg++*LR*+EbTZ$Cn|DiPTA_&+&Q6on_zdi4KN0m9>=Z%5 zoeg13-dugSm37P*S(~yjraO7y&A=pn8>9LJjxo7$b^AxH<0qOkJ-P=+YK?&pY$|{1co6 zcRePryle#QIhnn@M_~Q3DU9FF4Sf@g#PVkq`&l-RK?ZD0%JOFB`~4i4>yr~^mxW}+ zv%>4jc!AJC+|mBk(4yCxmu>^6nXbP#3GlKH0uuLPGb&LtIpSdcfq5<=-Rq~kW-pyU z4QTu;16r61C7(-C+z;ge*|$*awU0PJmA4#my44l^VxfOwOvEx^OWsD9b*(JKJ`CO) zE|o%W$`v_pny={Fl!Oymfehq>+xXb9}*jsi0Brhp~d=QUI5h%~{H1dL&Y|pK#Li`i)tAH8Wx1OK!i2 zLkYfR>*zrS`wBnrh`g`!&&a2-6qjE%r&kV++!<+>;dv0M=!mVbXjeAneN-8j+PfRWao|H(0i54 z7ox)_IB~mPKfs$Y=&v*!6|2%6G5w3VZ&S!3o*Uo1fohFr?{B{pqR8^95a|cqkI@Q{ zunCs*&XXB63V*8JdAoKPGhBw%-@mrfZxGDS)S)-wt z?nI=Hd!7v5iZ|A1FU}+?r<>?oKp=FOYLU_jDOuWwbqblOtYPCML2s*DGhMdsEps3f ze~8`_Erex6lhDgGTh7-(Losg|eH>%+NBM({TBVSGsZPdZKXqYFfV(RSA&Ek|EPx74 zu#aI`@lJFoky7UW^4Fs_$53FFCbpXJS4s83r49OCL924vHZ)nxSC=h@c02tzRtTX^ zED-O^>;wytK32&j^FuuLZ9N%WS6sQv>jnAEmJDG()UoC{9{XfY=8vI&jhOBV?*!|t zk41&by|RO@x2!4s0Gr8#Q_@MRYcnwtBwh+mq7Mr*71Yp+1@(BlSS<^<6IzBeFjyq5 z)-OqzMy%m)CiU+d1v7_Swe2@2D_CXWAKoV9fz0f6c9YD^h##MJ{hvYym=@8@wM6+j%O55^mCOY0(eO4oG zeY{{^TIbY=7s-3qHt}Rv#~mu8vOnn_cIT)d<%O$~5T?=~MZ7uWBdgaj zJ@j$Y-tsXHi7EfejcH>k^x_=#IkXqf1-HUp6%mRu$UY+uOl}s)PvZPaVT_-85HF0w zwbvEbYrdlyZQR4}-l~nx>6=t!@_yx#mF>`_m+~dQjTZt+-lc7qEl?YYzQY=(3f#VM zigyg0vQSwnpeh=YlD|RldZIy+Rq~Z?ui>pyi2#SMgFdl{d8FMu2kCb=lA{X%+p^~p zdl0LCElSe>{w;4Prd0Y)c|9zh&aPJ;!3V5%)E8<;lookPG~{+~`vZbcxbw=j6pv|X z+|UJCm5)OpA&X{(JDto-^Xbm+$D|kJ%>la7mWu`rHAJ^3ARljaP}|L2+~A~GFR9(K z>quaaX^lDX&x1BXS!H6icgTyia%6}@=uJl^XG7Jv478o&kh`J`wEHcwt(ANFz1r&Ps^ zsYn;C2y?tNC;22K4I8He8h^>7S|ak@<9btiAH*)ArD@DPVf4_VMacZ`{BI7^7T@y$xO_v3O}l|{?uUTSv7 zor0mAA4;?_0Uhj#PyD;BXU2>kJxNpFB~54K+O4X1Fni)!CrxkTdkhWkGSq03 zeNv%95@80_@;D?*1T_wH=aUzb^7ItcVqSCnGe%1rdzl>I474Il@Xs*_f8jqSyeva) z>7J1;bVk<3k584qn1|`NS-8W;Zs{z6fvY}^5ZjTh8R;jL@ZeqbQH8Zo9&)F}AyQxl z6q53qHS_Vm&oneXIiZuI8gnqz<00{Z=k17v8C7_@Th20oS))i@ApXeHQ>ryP%Fa zM~9T7TS|LPVR6no873U73POPbUIw(LAEkcA+kkzlV90f9m4mXRGI-xnBiR0QeT#jwu_Lz*y$gl@F{r59d?GeK58@)$y9jTPS3=#@k)6J7Ne3Ljd z67d;EWP$A#&P*0S8+5RAtYo(353z>0V8J;?W@RrtA*8k;Mkb9HZ}}pG zmP{M?F14OcjUQW?;(?jT3ZqEfrKZ=G(uVG@5x3+VU}(MkR|aUmyOcW&*6)jVSCaM` z-=X8+p+D3Kqw{*ckNKPUk-yBl&2Fcf$GkYz9+w-&PFYnt$tU$3yh4XhfJ<4}phbhR z2h89ZyWc)=Sr1L8vYTQ+xiNrDaGRxw)~uPjN4BC`se>HdZ`(zDbkLq5y&jBA@sy2f zrtYy>5OJ&v_&G0ln<%*Kde|86zE|Hcr=7PLPfkR!rT-Y^MEh}tqe#fgO6PcNS$+f* zt0Od=+xCBtTMd+MKZLN<{yzYNKzzSGM}EBxUfopVgx^OoB)W6nt_!`nIUc+2%O&#r zI1}H;v3umk-(1-w{<|g`y<=lE8-}8=yTCUbbH>QLcTVGzB}WHyH%1xXM$Khv2uPMj zl_^w%7vuaXG%BySlH>vSgFZ-eXU+O?nWiPq!+b5XXl=-7=yEqnXevLBM4BGs!}L5T zPgPI|C47;% zc}yG}vHT57X1~Qw2?b=dQ%B)~LIpebdzX*EZ+a)x zjw;}KWXJjTFu&uQXRxM^# zs9twrt6`Ssm<4{Hp_k_5W_{F-|KpQ-=lqWwf?#&qD5Sl*Q?e1e7l9(!a->&^ zIfBn)wr~2=)#l9`a${#7P^e)EXz=Jga$Ow3^P7?w5xN3PT5DsrYk@e8JbGA`o|hB$ z1+oH0AP)irk~6Rp#Cm0QOYFvXxB8#mk}$`mwH2PY*RxNx^FmQ$0~t#c!Y$IpVAiix zMUz(_THLZ%SttvK4~KQyKDLO%fmm&e)m#&?n*F+_L!;#82DDIVjxX0|>Sk%SbXOtk zd_`?MdFR`;&iqx8fFIK@*_aC=HvJX6?_(?kjNh+`1pi*M-nn=G{K$!u^8!V%ANktU zd~x)oyx_IXNiC6dY%F-4W#V^1-XFl@Iq=j-X?Lg>}bDlR}nl3|!wA2lS zfI?u$2%tHar1&WL9lLj2wF(=pC#?ygy>S}SfuZ+a7#r=)xamH@i$g|Ct>^u7=g!+Y zko|?1VL$vzF3{3qJeQSaeE@CDe(hGiu+yW6Fdb#T2?s|#1gk?o*ipDhFGGa_HWxET zU^yM$X+uAm|7b9fHAzQ5^`$M@`+QiRoCE(36H9`@^87%uJkJN->@&xe2vl5aN9}c# z7JN1(*VV?2x7W7B&its+FCvx^0JAGZ&Q!7YcOVLzQKWpZ3>aQ&*DP)bU9-H|{r!~O zrX$B?2g~yO_V65Ob)my~Q~{#%6&|F<12M6_ArV*&^NpK6sMj9f42=OaGvgr)ahd2q z1lnrQCSyXDA>hT+_t(cZ_y|ZTaS6vb)EzZZbkH>rf!Z7I`}o!?Tf$#j(r8ugo8L07IFL2EC}@qy zlj?qBY;iCL`SaLm0S$!)l;{K8-^1;Hd*Ol|2RjQ`eOniR$RAvE(M4=OU77y$x*X|F z$eKB`5*w;D&tPLl$gCk)pPQJ+O|Hcb$xrq=BlV&}Kp`N|=F`2ldq7J2kZN#*COvq- z^nkXk6R6n7Vh?5atC%v)f}k%MZ#o{^E0>5v+v$3mMaK8Wo#|*p>xWoy<;dN4-_!u7 zSV+?Js=pl_1Ce}K%sb#;;Pc<>qCk^N?Cr1VKg;li`jH?>iS6PAW{?cZv5O)s5! zfGq&^{YMeockI(OJ`vEYStCOB{)Ij_5HjK6S=TFa%m!@oYE(5_*#`Q`hNh%=x6ZQu zP!-ig!U+kC`iCKnW>m|@%&xRueto(0lI6&{1vCtU0X_^DwI94+pCdkKa8E4>#7`*o zY55$=L$gv8&|~>QJ-VtsQNVYTP-T2xm$ZMr5$kUo?Esf+6oY0y8z{jdv%EA6ErCq~ zztG#)MMAf)iiom&UvySkz^TYJg%L0{c;FTldHz~BxXNq7bcbd;qPj8ZJia+0p4l7^ zK#WnNNS7^ZTG7ZWx5U4iBWqKf5_w_zmaskwJT3Eb9q{eZs&h4ZpC* z@Z-e?y^|ZD{Y)$t{{*%ddqHDBTBuoLkaLcp@nksF`pujo{93CI4A8mO-+qpRZKCSO~k#=Je!sFIeI{}zYmV1W6x<$26TdD$Jnh7v}jr} z3RC*9)idX>i{^;sutDSu0);^z_7Vdlmrf5zs)LFM8oBiG;9zcq>VIk#_-HT!hn!x1 zG*ZRar9HAWk^M*&Ej&O4mn9h*IG<~vKq-cx$->l18P?p=Sd_+>gBh3~v&@B% z!6}~QF*|har@)z-b5;QH_GvkoS~8@~g|xfS!&qS=_Q3oKwG>RFr^c~;DI<+N=|~Gx zW1<4SNZ@-`pSKqF7R&G*V-<_w0u95-J#y3gh_Q2mC$-O?KffMRBY(lH!_5#b2(=fu z94i?Phbp4c=oz?q3l4lFQmf^~>FX~?0djL8IL$hjiJ?7brpYz)RJGP&Q1s{2gGHIo z`}G`isxos5QJ#TmzhXj8X%A){^(HTrN!BaIc&}ru*8sDTA{dn;sT3?;cLe*i{c-bg?72jb!5y+<$Jj2+ zNL_0#r!fM8iFSOi-8Fq8=lErnKt2Chaa zhjNHCj}PM;0o`TLvAj@I+Q~PWnZm%HZB$r4Ut9a zK?`Y3r_&RpC)290Z$1=xABE8Y^Y%0+@|{#?C!W0HjyX%HwHQD{PM7u^%oKF71EJb? zP!Hyd?7dH3mj8?VhkZDlzk79(80`AU%x_n(2MT6nj4TI)gDatE;Ay4+!0*mLKM?Wx z^w;>%3zd~O$adP?+&mJW=eV@SBeU<~K!lr9e#mzU3JNlxdC3gbUzS^gQ@*YEU7A`! zlPO3-?0_~LWt+?b@SEUo`_NHyQIhT`U3Y{CKBd>JuKWkI z1ec~e4li4F8t^aSgGw;;b=>?r-tk7)UYtJvf{9#@M%!6Bbm|4n^xOE54tzR~*4eAw zzn9X23iGnduAF8VE}V2p1Lp20&<`b=+2b*9$1y z2Po6D_LNhTfwGMP9$iigcN8vKQ-O5ssX}N{&2AfeO#(OAo zUFChZgF`{T$h+KfnN#-osJc=J>;M7H7zz@;| zKf>le3)@Ux?4XogJ(lee%={cUu*pT|#I2dV&Mxzvzw(0fxS2Nx;+*;KtbqPUN8aqD zyY8n3qR>M>&7D;x{1#=M@D~iZvu|@Kqeq5GX2`(+_5!*31f2fUF!V6mcl(`&DcExP z`OEX?&$%D(@ZHQlK$a6zmm$RCd27FwlFQ{|3z{C9n$aug;7jL{gueX&E)6L;pv~w) z3KuA60aEX>ZFc#fZzQR=6#~090_jc&jLrQcIv8pQ(vr3;7aNxq2n2oTn7+q5Q>RKx z@LqSe$dBqp>$1zP9fjo1;?xI1X8ZKC7VleD{NH@Y(GY)k;{cdBZ8jj`5{Rl;P6wE&e;gy0xW?N&~xX~B~5S2I=cqg;{^Og*dN%)C& z2&wKuL@}jO#fQ7uapb$_9v_k-$i5#WC74B15Ir0Lu z7e~Ulk{n;Q8T2SH?2J>RaXR)DxCQY)^eoTYCDo%#UAInv&@uv@Qfpr61k$4g?E%bl zH*zgH^4yhXzj}m&iaazCAfrw*&VD5jOiZ${+S*zJ?Z{(ZJ;;L#5f-%y+-Mt2L#C5v z>{42X9o1=A-7yo|h9fy>=tQt!o@wV-iCf@Q?>VH^rCk9dw-3xO4N`(ODLs9^eR@0d z+I4+fQuR$JTcwQFlkL&grEpP>zCkOzhk{=E%NR<5VCREodfND#uvK=HbCD4yumB#h|qW1=(spZ(T zgUAeMQ~t)r#;KTD`v$t(6JZE_C|8BhMgsBJ*R=0%l4#RD2=~vlv^4&NY4y~miUJr8 zYCQtm+8C7L)}Vwex;j$&MrX1W96?rQkyS)ichY8IOMmY%&h zaIw)TNCqB>WId3T)%3qG(i($#W^YtNRj2g82hz{J^wLXzgX>yQgM##|_@2fCBdXJo z$KIKFq)R6q^1vhK-*3J3md&86M)57jKO zQYm0^o}c&(cHJ;f$ho4WrKK#9*d}cE7A&ME88P0U3`6Dv!N~F{wD%-EU3ZWMZMEE7 z;4v09OXz;wQ82ikfHnoAQ@t2Y=X|6aLW7G%A?$eB)3y&5G?Wx9Uc;>C46Y<$TSGfY zchv2pqp_#Vf;v1%*A2#yfnj*KI(5C18d3-9x_`sAjS(r6k~Z&n1!*sswTaY0xL7zv zxbzP6v+;RVBeRX#B@I4v-sX6AQ*+Y5;2QO$i!rcY;c`tbf9!Ej+9vz$UwCPk;Dnlr zbyP`6Awn0E@_dQ2O8rfv;cBuxThGq+8~$k0wl_4}joc9AvFdpE+1f;EZxrxAdbOn4 z4VOQ9>25#wzCC4L+>vw@*i#E>#($Azx--lD^?MZroDo@iupne)@uS-swOguM?AUu- zlm5G_W7y~oZHs~uJWh>A6IjV+(>HRu{Wf=g-GKb+u?4+c$Lcr5X~n+gua*R2Zy?BKouEFzaGgjfWNv02_~v`vR`H!-X|>Hw_|ApE zbT0eDRIkw+>gC=Z@`gmH?=oz|Z3Vcvb$2q?Ac2Vv%PVML&v`chHl1H`6P!6zwYcr;?x-3%%;bKPG zht`j$mqr7I@W*Wt_*uf%8zOe|14|q9C)c-xp5F?NiSbN1s!94PXqR(;U6Sgt$YT{% z50>kk4&UG>kIk+>etcFqD`@(|em%_1(D2mji231!@Sb~K!Z9JCx^JzIHvV;KL+<_6 zG2GVl2^f|yAE4&wG15Yu$X!t*wm2l;=bTg#j?LIJ*T!BYIX*)R1x?8~GqI;)Lyy3t z=FJd9*4HH(@B6qe{HG03eMO6{kH!2HeS>g+k%0=NeFD&1Mwon>b7@7`Iek)Y(|%>a z0(hDZU>_Fg`xDU^m4TRtsmQn@VH?dcJO0G7=J>%3t z$=&Y~7uNWY3eW&QjOmXbV>DERPODXW0U~+M4&@A}K1A!Ow+k-;QI{Lhz44D?-E2bi0v)IirQ)IeTm>`eI2CZ|(}g z6BtpXJ<{~H(hrQ&_`0i#ii*1It1}r~n=(VNphiA~Uo!x`cYesozIIxP_+g?b{_jug z-80^A@Kr@^170A-W-u5zVb&&vzFlw%qzK2Vh$D_I30T)1P+)y!T#gPe;avXWh4wxX z2mrR?!W0TzUDMt5347+`;YeP(CSv{JgRRD2HpKj8h{5R++BYz6x3~b}btjBOw(AHt zCT#9LxFGS(y$ZDAut7SR6$6zp?E@7ZzA&^aVC*8VESP&sS($U`)@1yVC0mWl7dQDi zvM&R(9*8|xh9O(-wX!VYx>rpIC(hfi(3w0eNP)-0TGH2!XbKkw>O#>w3YMb&G`dg z1g>oF+x^}=zZForG+=a3qsIyyOc%8T9X~Xh(8TyGE4eqWA5VMe+-{nPHw1%w5Wl(1 zz;fb1Ikk1bsmq~y2jSv|H1H38RJVCVZeY!XqCiP$wh;zzBQw!3f(^{rr_kCkCda+` zwVL>CD_R1@*pP}wid`#6GD++DoVCJYusf;7NfVrd-o9Uc)UBp03Khd=anLfzkt~LBOEJc@@Fl)(0%`H|gXQv8+MjUiEjxP$D zCD@AIz*w7cA=Cv=->1Mnd~A+0_l=tPEz4U%MG!E!O>5WR2O1?bt$2MCMB@G`8glHn zz)lbA;Pv~!=)Z@~J-ls{LLpwW(2qc+8b1#so`=!kXsbaywp$i7TE6SdoO$(lB!3p> zdP#6*o0LwZV0`xKym_}imhSYJ_Mim{7U}W~TAnJS&JpnXcnG5M8=zo8yMWGPtGn$G zAQL2W_n9U~adN68hJnEX$gMZcojaF|QtqF_Hrejo4ws&hr(sy@(Z#RAv`z)>Iq`Utc3ac5u&tnL8q^+dr6;+cJGzcAzNBWU6L)u20N6q1-uUNqzL|FKr3Y zK43VWP$=ka;H#;TKhEIV?1kfpCyy8}_doYWa`uLXr1h_*4ao=Av}CQ0*tu)D^#a_S zOZxz-)eSAz@6+A=0=n_T$t*>lO{_3+9b5ux&<)AB#RC|jx zrJgeI^W{-n9A6r=7aTi0F(S`LnpwXB(;e2Q)zQQ~AJ^;e)Wm|X$DIOb7IXoh@{(m@ z?-I~lFd5s{4+|OEc@uM5j+v0-E6ari9%vqpJwt9fw#+_!M1Azsm$w8_FIpK=?p*W2 znwO2Tfq4(LK;}v{>AZP2zlCL~&!Zzb2@JoLXwXX+!Q4>n_z@#UAA;-G@y-sXFSsce z;<`ScdzQ}^$VErTPEF#EUp87D2c#DvM^v86VLnqqA+S3kfbX1)d527WJ7s2DYHd61 z;?XWjhq0y}i9|5_Vts}&%#+COE(jH$GcMYz$##x|mpcrg)mu###q!o=X!~DD)9(!? zX-k4OtnGX0rPmiB-C7)jVu-y%;&%`MbjrR9D{Ro0ImM|yNcPX^H)#u{qhHeB;|+K; zChi@yG982qIdZIYbumlZRJ344@-t{?KdXyr&%L}QTfA3iJ-mNj@@rEHEZ7amMGz-X z$+zFm@i_-SSL0SkErV)sZm8AWDF@lRf-M=s&wW;(_{sYX<~!JJl~W!pOdbp(voQ^TecdFVhP-Rk;hlG&sd;B(RQ!ES zB9VNM^9+T(WSJsrx@XtNsct$|I* zulLH0{qV371DXqB!{9F26l=Nr{jI*+mbZjGnqkE0D~5;{1l|MuqlKeIS0~HvM*)wtT^)?SWzhDD)X3d(F5BcFjba>tf zARQa_?}mZ7XnSQ|UiL%0Ie;6KbxxtWs}L9@1i<_?eD5A(?s2XhU84=5SLx6$(;5Wp zO>Fh?A7~xl&CShv3T=;z+ZfmmdGxd!O})(o{(pPl0cTfLt-nvXeQKs;l1zGrl7O_( zVh{oF4AK`xhq2_NP2rKr7l zee&eV-Np=DA94vAF+K-I2RoHZJD5n3Gg5%y7)8Ot#IDfocx0P-t+&)&xM0p%0BAO> z{e`;?+oF!+dUz>vwpfmN?xF_sf^{9qpPp2nm{1z<0ZC)(=!n?zr9k}d*J`c1*LMVH z%vS}Y#r5#oqd2pZiLmf=@~)9{PDE3dGhhN9?`?tj0hOJyqD^CXCGV|D(!~WpW!7syXlyNf*g-u&E!dX!HGOV^sWgT1ouO zg9`|puOw_551%#4I`y4~=%w#AhXAEE8H|ss%CtZ5=tJJl#A{6WSxmq-;Zd}2ZOjp8 zln3m4PZ=JsDhp6Pk_jFlv0GoR3CvyH5fnjP8%v{15X^hqK`dj!l4d3ySNY_yx`cW) zF%nzWFKSMjNB?=7e%q9Sc4v|%1Np?`vHMX@e0ODe zcc`O6;6Ouw%!>Au=u3Zufbe_B2neZ@c16Wco&5n_gIVQwo_p;CtHZp~<&b(Rg!`FX zuEcMh_oegktESDJd&kY*eHnvI_FrlsI?MBZ@8eO932=Q~jKQlNOg9hh@Rml{o$1;N zt!DBpl=nL7l==Hu_I}AmjFGq$>c=>D8qTFa!6OrnG%7aXvGitMUY^wYELgyuWxGI* zygdsSnZX@$;A#Xuo<^2JtA9>yC*>L9$=Zbf$v)jLpEe&gfOX)%yoS`_Ff6<&LhVTOI~nfu?}HK zx*Q>k_;5$OedfZNAZ#S`L$KvMc88T32L9GlBE9@jbY^|DsVD~>VR&Z$U|CD>t@?y@ z=ZuPuvBf^P1qtn_QK85SUm9gSvZNt-)2eo8avVJftS#J3nzJ7&@t`0^iK-SsC~5)v zO;wS^x2Begi6#EdWst_06Px>5t+;hnd#Dlu3;Qpu=$d$qf}^Jk<8m15I2g+z z|7c7ECOm`pFLKZ!|_@1ci!j-luc zW<9q>(^h5wKxEc}QI3Fme+OpzGyYx^nRj$Dap|!o^co-^GS8k;s7)IeabDZd-f`E8 zR{tw4DIYhU?Z%-M6Yd;fhDk!p)VjbiA@3w^At53#fpHya#F7opQ)DIsK3f%v>QOWOYjV6D$vellm zB^H8hflI+>7rNO(w6h@WvmTk6pE!0*BsgvuM3kK~f9q)&a-6dmHU_|A4C-6SX@5VG zmzzsFCSao*>QnVe^Qot+#Y-obCr_T3M-7HHENuF2JbAc#&S6FNzt*4L95{7oBeh zKHanzbU(#9r49RxGcXu3ta!Fy!FJ0{lVBhAGzP=9C?h9>Aq!SXAuvP{kmC{Fbsqc6 z_}RxS?4ySLLQ?;RU_J~Q40c41DQf6r_^Mrn&B|UtUVd`Yr%;aU-x$H3K)tt9SYk!^ zV8FvPADRsG@kv-&_$n?vmeH87zYocT_DI?=e18K18H0{>j!je@angfp@GI2*!UpVa zEBW0E7mirXb&0Nts&Z;E3Im!p^X;bKYc(J>=kXnQZ7~C;%3Qby3^9v zD=dTFEW`1XnjF_W)Pi`|q^`-mFjBw@fdKj~|wPKg#n=?sKD%>|4 zdkf6&so?#54IR&+q<+U8ckIAS<@Zp-{z&Q@cG6{Fc_W3m6jjhV{0A;S*ptSAf(37Z zv~?-QpPhH7&p1Wb0lrCHt5Dw+0z(S{yzAZ|PuK7MzV9-0KJHrsE#|D<>AD}V074>+Yb`s#7H0al0?D%!syNAm= zo$Vbm4b(0}1eFb9>Y3q~`*cND6VF#94ial3e1BMi=48fP#h(HMw@r(TOd^5ifH6`Dvj=GO5Z89F% z7!4P~*dvHHp*qv0J*@grPOn)wPoVt@g}L8sj~T!D;$i?*GDp{Q0)qIyvOlj zkczC79$26zoq~&>PtT}21#9M$9T)@EKhS~S!~Jq*lvRK_RR|~qx*&jlVrTlwZxN5? zvxr}D5&9N2ARdpS;Ln|mCt*~*3)e64+)29>)vvFplf$JpXU=Wwu`Dwm6N?KOs|MFR zJ2n|044&$T<>eJ#j_VK1o_#ZxSNpA<=@(slliBn7fM)h)Xj}Z*hMbx5FwRj0tv}G0 z^1-rtNZ#J_rqd)GdjnW5C9Q7KBoW*MXbUhq9%Z=ZHx@UV=R8yGyLWMYa!q}*H3#3}9v0LX!7W(4h^40K&)4YRL*4V2Hg_~c?I`VeL6=9bhqXLH z@gsxZw#M!FGs|0BuYZ1h0NvT>VU5?OfN&HH zqCUf>aM1Ib#DOJ!Zd`>9Ou?|nw;X$n-0R(=jtT*Vz#b9k?1Q%LyB>o7dT1)ZMRw_5 z(q$Zd^I`}BUt_;T-(;SFB$grjG@>W`8iED;vyV#%nL;_~$eg=3Aa+DK^3&zDb4(aW z<+-|-IM5sI+G}q-8xy84L$F}c-h_$ga(^R)o&PW%kNlAraF5qUe(jce2G%Qb{G=wa zZrxJh3yc>w7Z!R>bUmf_0}U=0wOi(*_nO_SK5Q{d5ZUuf#s0*!;h}cyOM$1HE`lBr z>spf9I}ORm)Afl^Q`{~DXM{hfF6g1KuflU@Q%{aYTMTMP+OR?CEo<7vE$gFsfq<@` zTIjPrTOMesD)vJ&Ko?gRp(qfEw8PfkZ%p|AQJ*Myzde;NSvfKsjCOAZ2j3SAI-6#m zbKj3*yvn6v-FR?YTs*in9u{vkici2URGtpdeH2J zrkH_UDj#0+p6~^~DB|s8L)n#eE0YG7WPD*TTzC&?4RK36vegbfQVj!1{I>J?;+$hI zNyR76r|F#CQb-vL!GeU78YVK=5WbB9qk~O`4tHDed|ljpzBW-J;Z0{t|L5x|X-4C~ zxkq5$N%j9K*|zMA`}UkMB3dBuRA89|r&wtU@ci#(8A$sc~|RHIy<0 z66Xf2TR^k-W^~Q&-ImKE>sSM<3xHofux*b2clZmnm1Q=fZjFvuBD z)6l6TjR>ms?seDQ_$Le|XJ@NmF?hj{OQJg+Gl3b%zmT0VQz>Ls?JEfNJrZl0%;a5{ zWpd!dgzDFiKR%o8r+e(c18Ru6St0NdLx8hy3JZ7NeaBzn9deuBAN&E_Cdnwhex^gH z$MQ&^6#Z{5u75t4@nbNln}xA}8cVOf48j_hOR3Sk36Zcb~{Adrl{x;$8nh#5nInWoZvh?|P3%GCnRlaHQnS z8J>xvBA~g5D(Q|6Y4XGKa$L*9Zp2BfVT+@dS&( zrkxlQYr6D826e-TkX{imc6_=t@OGM(MSFZ{2b<;|>T_Y7g~dC$9P+!227>USRp=@F z0aJirFa)C=`%B*a7)-32Zn~)u!o}C2@sL@)-*Dv4kX#pDYgzuMGuICUEV7!QP89>X41r{sBj~dw0bl{L!mX`!!`{ytPvbn zMDZT)5blP5UNmvDU}V@nytG3BcUXPPfup*5<5Lg_-o`-1zSQ|zphIGPH;#^}U3$$m zKbU~dDD%T5?X#E!9FBXssvR@UaUr)zQ4^3bJL?!=4sXnoc6^e$Y?=F$iT`)?ky z3TSPBvnDGgFH=b&kShdW(1O0_h}Uve;2@SmLoR6q1qI!2(COrW<<+0jUfAfEd9Yvp zhIbwLuA)5*j=S_d4t^Ny+Mx4zh-&&Y3f(5&fG&4YinDl+Nax1{gK8CgvDK_I&!P zt8N;TsdqqvO&=iX%!2x;t)OthfPh~FjlY9Ag#nt0PxiP(LC8cQr{H<144ik3PZe*F5~6_6*bs5vvab}RLb{TC?e|6 zLD@gD2NFGPb*+7e`L4R^y3=u(Mtcl&XkPLXjlFpO6nAIDv1>VAK6O+GC^pG;^->&$GA734Yt5hn`NA#op#Vr<$q9G@|W@>zi+zvtsRm0--5(u_8gYp(zd+lSlzP??JJqUGg#&@p2R?cWeshY!HeTqTH9#ff|hvx4eK9 z&!ttka`)4&GoAeg0(MBm6BdF45(m7Y@EHggc*SIUDk>WGTU&_bK-$1(!21!w5rlBuxWp*S@$ny*x4>nuOaCq|dz$YJO@ByN)w-vqbgoFByko!GiOp$H@Tbp@1Nt zv27>MhfuQ^!Miw&b`vnTeE{knhE<8ju|)CERaad#9QRtJ5yxDx11^7<_;AU_p&=>a z@4O0u=UPsD_yJuU=Do!4F2*Dykw`uW4b0uRkAg|R1Pd0{x0n2UEeX-@`3ipkyJ~9t zZQ$dso+I$#v=70oL_30XHKkmw278uo?r^jnaaZH_wsi0=nXe*qPv06!NOGGBQ2%ID zjx9UnMs0CKx@y8J)`0zBsu5^nD`@69b}bhO$1U?v_~2IVdyHwCTT(kqV=owm`iJAg zfME!PCP*K#-G<%Zh;=P6Z^r8u1J-;Nd^;&DmEhB-#K{Af1~jo)=BMlP0hl;TVdtAO z=jR`S{AZzJ<>1w+y}-HLUjQZ*!*N=H9;8Pw$gF&USk} zC@cUg^wGQi5Ai0R0AnRP$Yz!dd?%xK!inp47T|fDbStcn7x;{+HQ{^aIL22ITV<^8NG9o9#y#%qwvv{h1{V($4Y28$)m^lN2<&ATT^0VYs%RYcM9HbNd=d6q@3;C8Wkh^Bt*|_LUHt*@XFm&{R92| zAk6SrV<&Bd2*>nab0!81eR6@%IkM2#obS`@qsl`XNE9m@6Ha}?3BJ>sD%jYTa@Qvu zx6u}Q3F1LOpmH~X=7G|(r~5rjC{*z6txn_S4EGCdLd36qvNX_^kKOJM2T|CkM67K} zx|`Z9ZDqS%ys9N(EQ>o%HG~g@KGhCJ7vdfzd|Y0+UFvrbn9YQX^Hx0ISLJWBPI1KS zF1ZejukPf@lL;A;_6R&g&PRXsXF96)+}QEG4(eUw4Kma>2DW$R%(?UTEFd$W@VEV% zNG(Vn!;=qs*X8zy3gJ>ItUGCZSwJq+)${lF1gExms3 z+}k$pG4p=P;O+4PS?oFp^E05q7>#~MAurQ+1--i*T)+70tFL_-{qQk-IOE@IeA8N! z^Nf4$`T1t_qcfn{d=iZNlQ?24PZyr}os69{xUm1g{wKyryyN#&R<>UXA!9xiDv!eJ zr51FMR)I2k5ctk}D69x*7hP^kfZ~`(zDr)dtFF57U|n}EMNF>?8C!=F83~GEcwK9)$~U6RA^*1Mh!knArGwoqybO+v09P9(MXd*KXh-)UPW0 zo|z*DH5J<+W32>({_2Q8>hy7W?u?1~(GmG(5dvBHv1-qW{w@i#ZHCwob!}GrOY7Ur zSGIKcez`fJBd~%o1}$cMioJegcB!fR7t+EdgnZPo8r!l@E;3V>ROM+WPsnfk#E4M7 zX=n^d!L33F8?(D##6LV)Fd%3zQcm*U)zR4EP3`$V+8DJP60UzNZ2cIYB9~#sQROa) zZ)qinS}&I$HTwtSL*kfG;lyc^@i%HX?{Equ6T=a~Xcf;Tu@h)N~J57V9i}5PoOOT1Q zUrB=?#$N4Oh)jGYh*{+AF55>)eArBmS!ZI|{c^O4W%cpa++~BgO)3!Zqd&Pc?)^KS zH)96$bvW|c@L!l=--uCTCws~riTy1JOA1;5@9-12@nW=v{eX;WNF_3Qye|g&YCWQN zej9>?7rnESPB45<0yr1F4Z^@nXvcbL3wUp5rhcCAUX}q5lC6l1^I3!ud;$#oTnHE} zi8x`Dl!@a5JxLsVsKq!Ev>b_XYNG&Bb;rq1!W7G%39)F3l^vR9{fFPCO0>YkF8Lm$m;Cv)GZ8tC9K-X?f(wo)7Uv#b9KUEw)cM7Gjj3O6i3jtb z#h_gVgGqVyb>2LH6Vh9O%B!(od9$fI|93)R^0b4Bv{3~naV}=jpqZ(w)F|m7En&YN zICWwkKh{^E&Uki3OZ@r|TLX+ZH5#ROn~TX(2KdiwJB5lZcvaUUvekc8MG{{PW0t|{RkqceLb_2Krx$pRYX>}YUm z>%A5ZjvV}OL{G5g;JAGGa=ya-P3U`?6jS%Pe{GK@NkGQW%o$d7Qm8BCw<=d7Rag7-eiBefW~MonEQJ{LefxB$X&%I~+AR#uL! zfqg~;q7G$R001xbNkl+@Rs?rOnrtca@ldO*89>AJr-qA>p4e~dOxtI9ViSfn+Z=|VVC@1tjCQB2Oj-=rS|N}Wr=p7+1s&o+61(IuiqFJo|R~fNgIV~NMGL^8H%s_ zQl)dz)KU|IMZk7k10R{2wBe6msFL;QPVriXZec<7g6t-CY7Jg@Qn|70i=&dK7x_}v zF-J`7>l2f;?fxH$s6u&LAi$0F>#2ojR3v`-*%2lL3y6%uPpnx2Xv4tHGpX?LXyy@9 zCwX0N@6P;%0X_VM$%W?fb1I#i56(|+#WQ3CRtk$UQMxkZJHA~%t6 zU1{*X%X%`uXP>v+*K+V(Q+%@N~+w9VqZci(;Itr$_y0z+R9LB@pez_EftMt0(O zz@S?$g#!o{{N0CfdP-fQ>6_2^=kqp&|q zKz4tu2@Sfz)JNGiG{i{Vn-L>Mr>x3OiK#E_Xw&3wM2N~?bASv9?q;_rF(8NsK>(J{ z`5x%&ewQp>2S? zez_BxViu@ai#}cA;~D68`{`DpbIN;yz_{d&;9U{j#~FuCD*t&e-?C?lvhrJKV%~=P zVt9O-znj@}KHt$Pk%%QKdw!kEhOi(*l5i>k9k`+_71hqLa$>t(c41-EUeCTShz z_yD1k(r|lduqbYekE>morpamyw6q-bSF>(a473aorqrvY)?%KR@wVL%a29;w_k13` zHGlI{<)$&DDZxvqLr%!hwi6CqOg;~_?9XPdENe_1bYyaF0bArAPW}6Cs5AgSnRM90 zk?+tL{&cxambGfuw@$MXGu|`5+tas{_7Un{eDz%szg}K>NS}M<#^F;7cvJ645P0h; z*x%lMGQ8tCUj?|a40oK7JbK@GDsOb8BvhO)-J0iGq;AaKcI$#CcCsx#=I7{NpD>0E zd4`T=@(*DiY-zL4<4xlc98kOj7LVR+&pzX&_UFB_Q6b@sBF`C|zk$ z1jQB;FxL&bi>W%(1!s3SD#jY*ap+9_?EbNF;ETn|SzMn#Co2-VQ)ei~iXZPeL;<_C zxA=I|7t*x9qA2T(g}=euBeB1bN{uY_(OTrnaTUSby|+|lyIMWWM4%TBhL8~No(pBU z@5|cV#+Q~Obh{VZ>R>&=`pMu8)aN!uZgv;YpLCZIYj}Zh(S3IleRDjprNLe$` zNX-wG^j25Z7~iKOH-7;XoUkq2GtcP_7X;EaZtlUcw>_?(r%6N7Bc``Xa4~#&CH-d} zeo_sOsWBvf)LrAVuwv0hoX^P48eKg*{%r z|5V=zxb=)8@-fu%`jA74YV(w|IB?0%y#^sM%WF&Q_Jk>Q_iI+p6#<1}CJT3y+be?F zKzH`2yS801S3PM;60j~n5;rpSm^KrU0gxfLNPcQ+_z1Vgh0YKSTsAY~Lzmd}!-1 zn?GIMKged{kPzcA8myLZtcCZuc3)a?ckMZ6$)8OOz>5Z*aNeP0fpr^@cCc2IfW&^z zWr?5^@?~E5W~}csHc0oV9%T?_^=mU|JKay_vw8Q{_@AGV0ter=pNT!3didu-Bq=aR zrlv%Gf$64*sHQ{&iJpD;dZZoCJKi5{QLWX};az$9iFW-w>c$G|7& zgQOq%EgxDC(+Mxy-bxD|Le%9Og?^wl`Dm90Y?bJcH6=RZEm(M*+8hgSl=D$yLb&eofm$HuYi(Zx<&Fl6>?0R(O7KsqPahS5A`unmG& zphx~7QkM!cIP-(u5L*dO5o@|OC8b~5?ar;3$WC5;o(F+E;J#CT!|nJZ0=?2x!**i% zO+1$lZpaMUbF!9Bab7K^O~E7%4vrr>tHD$jWbFhs>d;j0o~E1se*Hb8%RXTzHlK=$ z3VFe_B?5LvZ$l)w}3yIX-q`1(l-u{piNitnImu2O*!KZB9ox)XMiq+kP3j(whi z^WqNj;7&lI1sKQzS76W`*EbPDyJyM#%?FP^`|SRSF#>ZKvGbaEzQMq>4dL61wFJVY z<-KK6qj7q$JX!5&L27sx>7G!g+;8{SZM+tH6X+>gfx2Ff0~3|p5aa+Q1|)d9VV+5Q z@9mD^6a%iZ0dmo6RfGU-A%|lNY3lEBR&^6TUBJdcqs{(%crKk~6+`S#K1=C>JsZ{z zWZdY+*vvE~xKBIIv1H~-x$!OFv0V3OZ*^|3-(VidR^g0Aqt7sP!hkNi6!$&~v#9Kl z9p>-@QCxY_DQwF)yBJu~rybn;npt_kgizxqMV+WYV`VX(K5v40-jpzK%9GVXn^ zS@Ja%f>1_y!2SuH+)#V^XCrOQogH58<2vlf(;8v{1@v_7;67Xdwy(e38g=S&yL6e( z)&hV;+;OPKPIOb>63us4Zq2sb5f>0-*j)Pc3Eh2iU%g`kQ|I_Q$E27x4Zv}z4` zzQ&U5H#eyt>BzKa;@cgqqCiCX!&5NC9(kZ%H96er$}$3OK2_ts7dw7^J_feC7ZZ6O zlSr$k^)fS%1C^S;&kv5rZ+t{lwhstX)tFN zi4x)ZDO9x%$#(a2Jn;UN9s(71g01Q2QICd51!dc5WkKE~f8~b~ z_BOUUeVcy9Oy6SRL_j{iTlGGYs14sP!Ntp$XRR6ka1P7CfHec){*_B+M2igQzH`Cc zgkNNGwhCD9y$53^8B+&GPa0|aaeACt)fSibVD7+)!T?{vKFY`+^cQabu-BHTh?`9O zQVHw~%zPz4Nb`U1g|U*M2{lBV9Y`LV^>m&zCz^N2DYMf-#tQfpjm+lb=e2afg*I^j zo@zr8W6XT-+Qr7CeKNez2%9QtSHXj*7_{e6VBJnpVom~uI$RX4{I+fs!h!GsIOZm+ z?50&_@MG4};O1i?nSucstgDFoivr$M%iEZOiTFSTmL0yepOAp4>v$t4S)XYSD(y>& z!vhe4>9}f~r0i+k{;o!>GrBn1?VFSPhweNeXlNjVl+y=KVI(R?w0J}#hYj^UPiu3` zpxc8t<*Qn*zGfj`v(9UT;S^B-hC$d3Q=LV-ELSgK+=3S>f@sx1ZkD@t2uGb{Xu@e@ zgcz=ci3I263VOz1Zh0Yy�fu>MSxrxgi#n;Lqh=@0@!ay&F0yspK)28*_577-8-b z3r6UEwH0zS3Xzjm6e;P;>2Ie|SYWS8>Fo)f{91$2DxzRS)3~7HtG!~!V^TFhzAxdo zVe4~(oKLW?@9I<47Jel^E~=-XH&iL#vUkV2I>FNuA-soAyK5bE8LA%p$!*tpRn^k{ zJ&m&4t8UhP!-W1yDzf&&DuwCcl(}gz@tL&CEdD?lepJE(|MN5A92exXHDxTZd|B_# zgPb&+3<`?bfE>v4QXj>Pgg$aKDlK%Ga6c8s&~y}m7_>Go1ZVr+<6V2$($n(^AV zN+nh2qJ&=39O~ORx0K#o?E3w=;LiNFQxKx2T;9kC%i#UpZ>LCoIP)$G!JB5|`6vlvNh)KDhi%?D~SMf6{I-kqfCM_)0c)&!Z zbpgD=V1_aaQCp>)j#l8rv+%-nSN>9m1+nEi!X&PyTjYW=i*gQoEHbB+9pyze)2I5%Q;g_+Vy*ZrPoH$uaiwLyf&KP^J8nTN{iGMX)23X-V|O<+ zZB3i?&-~-b;yfJC8y?eaPkJ6`_L~8r*=WA&?H*`R>lRYuxuQ%>fc$W1`V_qA{Kb-O zV}grty%2zd8#c4h)CJK7NwAgG{s`oA7`Ui~Cuf`-=NDlP*6QjNstN@H{ap0K~T-O|sLc+B@K%tx1bU5?dOj6$lj)9i268bBQCv z-Xpu8HJa>YC(+S**yL5$y>=Bd?Db~8J36Q{%ojf9C5(b24!P!aG@$l0CV|xBKdT^y zB%KFqW7{=z{6d~eAJ3LFyR>djbEd8PeWR&bn4K6!FoNuED6^kQ^hqlj6jnJ)f$|jI_i+#f0j;5vVYn zN7q@@05|;Qd3P&+bnjJT!hqbI`2nFd=PtOWz2xti_fFvusVsu&m=gkm_h!NCD^E@` zUfP&bZ%`a7kf~0CR5oN;_!F6J)y3nzI9hLmMzACYB>KekV6HYdigYq$i&eOEp|po@ z1VXIvJ>)`~zfchm>h3dEbru@o1@_Fj$HxuJJFD2jOPGGY{Lny)A#8D;fQacCo6djW z8U7YoN8`sY!zryDC2sezC0JgQ=@tg*)z(O$*VE4|>gD!c=b3$geFzAyuUmMOX~BB( z0gYu;zYk~h{ggl1?riP2y;$c-VM#^rK0T4#jH(uus7MYftZl@$HX2r|zFwa%bv;gm zyMAmBw#N^qATpcH=GBOhspoG~z=Dt1%5zf>d8-f=?b^QAOj8U?p8RZ9PW;|iPK@H> zQX(KU-l;FV|#j1-TgjxpS5{7TONKOG+uJ3#iNM;M ze;^k)r%Lf(*4eQ~jXak$UIq zdg-7jFW%uVm3e!RAt^J10$Gj5J!{}ZMpMCw!_guM(8t{HvVaqxSycI495VYZUB)l( z&45+j3GO>KC}EDtZq;1^cai!tR?LYX^U=-PhKp*6o7wi138x zFJ+Si*ql&=%y=_IhpJQ%emS*L@_kBOtHwI^+{LxZ%~SGQ3F0B}sbYt^{P+4_xv;z2Mse3N8jK1u5h{38qnSh6MQz*A7 z8^7qQvzS%Z7(Z>>s3Um;TaDP_S75S|z1GxD9cu44Y2Nl>}`S*T^xFMJ}I@RMt!jHJoo~9GEVy{=#u9N>sH-Qp~Ut z5%*egC_3TB!Ag~v;abSyG<8qh^F64C*OMnmGmXa@ke4xzU1N1IDSy}x{>%=OGSn0F z=|}m;|I`5+YFAi5ChbhL{mc_{3_#MM=Y*zcqV!E<9YM?nGk<mcYCN2BI*Kl!e(i~@32)I7S=Z| zP2XpFoulAl0AHN2dBfl5l{N>HqWe{!#Snj&5*{-$=kE=g`F7(*<3-1D_(cnIz(~!4C z;C&xKWN(kp`D2_z@ljGCm+X>D(*t4*b-IoQT>(VUEX8uLVAo6|Xlv_L62(}G(i{Yw z0hraQ(I6CR*eD9(hH$Ycx0Fbt_(MFWJ_S15vnTdj<~>szr+8GXDRnPcH;n$lA1IWe z9F7GRrR&qJ9@>$7rq3&ba1?!|GEVcvROh33aIkNql*F$9&M9u^4Aiyv%e@qTFpjT! z3f_ppwK}`m2Uo0r=3s$4R#d?Zn~#U>uo&#!D3C#ywXKX( z=L2RiA7@r4u*g4U>0!#~s(m{k)4~0ksY~jBNfnftbDKsUNS(4kZCqhf2wU^s!P|L3 zI%qfggW(DaaZ}B<5QfSSI1gi1$zUZf=a1Rv9A^iq5q=-Z)+Qlic5@B;){d3slak^U z!7Ghl16dUsO(uL=3mPatG|ylRI?zK>su(&+6`9Sjlim9k zVwDWfrMPIP+QnI&FBDyo87$qr@Ui7MYDOzz00k^U}7;}c~M<-`zHKDMo<2Pq38;IZ|Yz!G+Vc@9h)v{6spE?ta;^6(T z^!A2Gfo26EX^|rGxDItlz#E4cd04^+GVE5o zx9bzGidN|yoNsjLb^ zO>A|uI(L+1}L3YqAA4(BrJ=v>!Rd8RsvsAt;{pS|;#zk?*6PH*LkDh#4Ln6)c4^(89R zz#7=<5;QqG8#*{4(uq$STPtGdNzm8!;^U9iqDlueoKLJS-J|t;atrbuTp8cOLXgs> z&1VUfmM{R{UCr`0L-5%J*bVYvGoYZh$oIjeiq14=@|%{g>rLZ~m)0#tPL}1l@+>10 zK%b2=GW=z1@0I7nxRCGj(^5NMq)D}|4Ja`lt(Jv9G;ES`qcv8MTdcsL&x40UlSI=w zRvF^bKqzm?3WEK4B!X;>k}StDEOkR>u66ai$S5(!kgq6}8N_NQHKJAu)GGRh`;Vw? zsUcedqt|{{wu{tG$ML&*HNP@MJ$^{Bp(lynO_qRq^65Pa5$ZCA;xPNYOokkeTc?fA zpqSogtU+>`UoD?Ht^|0L= zH_iQ+Z?yi-Qd#H*RFG6W4hWG(uDPX&>0M{bYzwN|6O1F#$Z9`4#6pGm)8*CarJ6zl z-^^20Hm||!TJ)u*NRHU+Fi$*wKh$+Cor8A9;pT8;L0sf?Fa#MdgUfs@Z$g&lZ;zXY z@qkYy*4=oXXyM;WNUHUOfg<#S21~V_;BZrkFb0GEQ+$(rq2hy$2uD-vPGDguEICe^ zsoQT)zyrY*s7mW?eJaQ2Aqe;M(vHKLd-Fu@n{R3n_}1^VGEwaL4;9iX_7GkVP>t&y z5(;pA59qzQfoF+u*014HX%!$W$6n@gLXTa)18jLgT9v>6?`H%Na5ZAl2kq0ooW-(U zg>Xh9VWi-ir`h@X_Wijo|8jiv%g(4Rf#}TSi6&~OYW5gt>u4Ody zMIGVzte;9$?+s8<$n~^9`+&@Kv_}ReAA*%L2R`PL=Q__NE9;6w4c?4_;*7(dI}RtPL%%lZ0c=sVPr5!=Zo@zkn-hpzA&o=xN{9 zovz-bTQwh#|Lt$BdygINkXs$Le9lSzrS>LB+%7@ zS}heu{%p~Ku|F64F-jY@B;aaub}lc?r765lf_aAQTj&(zxI|j3l_Y&otOYe;P7pO| zCQKcTp-pTZ7C!yh5^_{a=yO;p;VF>fn_J?Eog^J@`6$nT>ca)Ohh zVBi>;aeJleLU|Z$O7^m}mmF|;b@FlCjvkeAfcPN7nsi7%H_O;ampyR4ZZ~J%yuDt^2f+=lZ`s2Fv}YqbdjvcR~oG z*zEgu$Z;2g`io~UN?0UQhPwoJABz1>-|Kqrw8C&K*_Rc(KLz-jjXzf{@VRQvzJ88U zthL<$0tYl98cw2^7?3j|Vp{{jxS{5vp1KjbS60-@Z1gLqb5^mg z`%{4LVYhv~_f)-U?_?NoLKyQ`6;*{2{~0R$p3a-ITDlhgz#lX}YQ*(~C5<^+!JI#N z4A1Awl2{9hLuFxre}{s+yq}@E(F8_n;qmd6t{AL%Vbub;%a!RDC(%oc4*a zzgon`%i3?Moo`az(d7Nh7h6AE>NB^Mp2AiNVMk#;;ix=u4`o|+E2BU7W>|^$bNXr4 zcGXW&PpB;=ef2)z3@<&d1(^TPHCg{bL5PnxN8Qyi^eeuZ38UID59_#9;=MpBhMtLK zguZ2V%tQtLU&lud1z4IT-{c_6fjQ_;g6NFvt;y-P5EOBOg^5-xJT@7!n=7+wQ4{qn z{?cLZ8|Kgb>H2na1I3Rg{u3;P32K3vUl%1tuB%#GDI;>cmik3}4kk6V33rAb>L9@p zW^2ZDI(=n)+coJj!$tCf-@U&2Q6C?XBD3fr6$Of2;VMS}wVo6~G_oVtt4J63YkCk8 zr#v9Bq0n?L>-AV0*$piz5Jo#&DKQyESjaR5m__r8X#6C>KxINgSF zRwe1r?Ls(*;(1%*4C$p9R0<J_w55X!?!kTZpbhp?2?XNcbC#LI_D^o(`RYmFxRO6^^^{Y z#3Q*#u3xiS_ItOG{8Dq;-aDDRwlCD)Ip|?7-KP=pEtWTVI*0^W$eaALOfMx5cHp3d zhv9FpIW-wLH`l)s6@OuZYJ{5*3fC?=x($j?hOm@fv2SraRqm5Im>v6Nc2&9o4rpk< z{}5Ut!If7nV&^-h2df3_EE@zE{nj$MURKVpvccw!AJ2U*cWl9|CFJXruBu4Dd7*yac;^|9FmNAMe+R{O@3-3WV^S_mf|O3T?ID zYmx8Yhc=oG^JQ(ti#T{}wI3ydgdqonSGlkhbNBC|U(9hKjUTeB$Hk)lhU+uMk2>zj zG19-3lrL9t(Ttu=&>!i&LK0T9Y$VudDRN86LC*ji9@OQq)}b^5Gcbj0eJDCLDb10y zbQ+aY;BP-G%DU^UI?mIcbo%q@fRy6!Mt~-m+ew*CXV#4j6+NHBJwUY)Y1BFxM7KoS z_2p8(Nt7buv6uzPf(2Fo-6~Sd#hN}?hQI(IVcx)0R$$nC)_S>>a=2KjE_K}-e*uF_ z`Ia}eC?6gL5yC)0A|Ps7I_bojNk~xDKf^N##I??=CgNwh3SSPt3onYAqC$Z<@=V2} ziw=^FEXvHEeM4ZMFGSWSMidxwhSl3)u#Q zeZ^_O%P?+^>|BCny8ZlbqQeVFKF;C&=%~WR=Ay_U(J&x!jMiuSsEWyug$6gA&N(8+ z90F9ROjw50%xTBr9qDNc5qy;-q6|d)#h#4AD;}BQE^&E)?YX6=y zmXS3%oO+th3-+VsT~%m9nSsJQl5_O@Rbw`rEIlH9=9)Ask$z{KT#{uxu1Y!GkXmw-5BQ6sSa7C&yTZ;`AG2>bdU-H@XDnsc=LUWYyP< zP|TOp|C2na0~%U#T66N1EM3rg+MZgDRi5K^Po;*$pz_%bZKZHM9wR57)8~DnANh*-^9v{ir1@0E|SiF`R ziVlM?)9=saiD3{SNWtpE8rWxDQ=fYLC9%G?^O8^b~~~=T!gI9H-zO5QVNs z%r|kk6PD8HO{VJIrXp3p09%rL5QrdK%7 ziTPz!xj)!CIm$|D`b)@v*h>ipO@a2Lt+JlvosC88I#dhC*67K;!Z)I*4~oei)h&Mz zbz7%FctBOCeFE63W7LQ*mpsF2_(3WTA_vQ}`kUH-cBO8W|6LyrRGY+R4-V_1pEbu< zIHKDV7{5oO90a%tNarrdp~x_^Kp;gfv=#JOiWQK`eFU{C(ROnSi428@0EUTz@B15* zanKlMs^ar^YS+7yX!r?RGYyW$oa};sQiMLc82k<)Q76&~RoidBLi1`*xyTiCL-i;co!-W18ihhSPO9VE*5*pA=B)mAu+#&)-sgxomuzS~zp8>VpS^*qAs zJbz!Z`UioEh;J5B=wc90jL|923&x0eo@yK`2dZ4su&+ATnR0#{8Vx9naH!o$<#rj^ zryxvGLW>^B&6#El(kbS_!J#Lnr>q<>9#QZQth@|Ul2B~Ug&dveXDwmG$DLM@{Z}?{ zh@iMwhXhpo9UgX1d0OR4mSNnzVbJJAaNeb?5&j?sFaDP-H$+|9yf6I$`ay!A{9r^F zBrAY8p)Y45t(#~_Xy8Umfv|so3x~nORcd4uqo4z-U1SRSJkh0ykDaK21+Uj zD5Ovl5efU<%+O!rOoL2l(sy!&`A?{{erWhLQt)tX27VXCl4p} z(e-Q>SHS@azBrC5bRopnv+Laqt%q0BZP%3^rWh0Z;yU;-gs{k&EXKSP#c)$g51K1xFe0;exclaZ>--0EEuD6y=lf-zsDkjy@7Hv$ z1^+X}mk_Z~inE%ddx@?Y@GgymW6=^9J#*2l?E;P&IEa#12&Zp|0Rt))rX?2!O_Z1b ziJTdS(mWVVQChp7!rdlhvv*e-;fVaLc> ztK5Vaz$BTT0M;=URqqW`J{dkVK|Tr-4V43mSUG+iajSvEgZy*1eBw`}&8YF7;AR09 zP-_(dMFGLyX+U%SJG`PTWoV8nm|}E%wfrs7F3<5yndW&*AaCMt|1~VxH6-vq>Bc{c?<>RWy@E|3E=KJUpDJ=Qgsj12khZ{rko5G8B zB7?Ex{!BN7(G?Rm_p6Rq7ccE)c zf*1W7Sj@+lCH;dEBe7OzRZ%7N?rB7ROv{s!0D&xBotj{z*Uo_gnw2;s3w>2POWI=)bY+|8ufx cr?_~7RG)fuILZd?K)gS)Qc9B5;)a3$2S#KsC;$Ke literal 0 HcmV?d00001 diff --git a/site/docs/v1.4-pre/locations.md b/site/docs/v1.4-pre/locations.md new file mode 100644 index 0000000000..d0dc636e7f --- /dev/null +++ b/site/docs/v1.4-pre/locations.md @@ -0,0 +1,164 @@ +# Backup Storage Locations and Volume Snapshot Locations + +## Overview + +Velero has two custom resources, `BackupStorageLocation` and `VolumeSnapshotLocation`, that are used to configure where Velero backups and their associated persistent volume snapshots are stored. + +A `BackupStorageLocation` is defined as a bucket, a prefix within that bucket under which all Velero data should be stored, and a set of additional provider-specific fields (e.g. AWS region, Azure storage account, etc.) The [API documentation][1] captures the configurable parameters for each in-tree provider. + +A `VolumeSnapshotLocation` is defined entirely by provider-specific fields (e.g. AWS region, Azure resource group, Portworx snapshot type, etc.) The [API documentation][2] captures the configurable parameters for each in-tree provider. + +The user can pre-configure one or more possible `BackupStorageLocations` and one or more `VolumeSnapshotLocations`, and can select *at backup creation time* the location in which the backup and associated snapshots should be stored. + +This configuration design enables a number of different use cases, including: + +- Take snapshots of more than one kind of persistent volume in a single Velero backup (e.g. in a cluster with both EBS volumes and Portworx volumes) +- Have some Velero backups go to a bucket in an eastern USA region, and others go to a bucket in a western USA region +- For volume providers that support it (e.g. Portworx), have some snapshots be stored locally on the cluster and have others be stored in the cloud + +## Limitations / Caveats + +- Velero only supports a single set of credentials *per provider*. It's not yet possible to use different credentials for different locations, if they're for the same provider. + +- Volume snapshots are still limited by where your provider allows you to create snapshots. For example, AWS and Azure do not allow you to create a volume snapshot in a different region than where the volume is. If you try to take a Velero backup using a volume snapshot location with a different region than where your cluster's volumes are, the backup will fail. + +- Each Velero backup has one `BackupStorageLocation`, and one `VolumeSnapshotLocation` per volume provider. It is not possible (yet) to send a single Velero backup to multiple backup storage locations simultaneously, or a single volume snapshot to multiple locations simultaneously. However, you can always set up multiple scheduled backups that differ only in the storage locations used if redundancy of backups across locations is important. + +- Cross-provider snapshots are not supported. If you have a cluster with more than one type of volume (e.g. EBS and Portworx), but you only have a `VolumeSnapshotLocation` configured for EBS, then Velero will **only** snapshot the EBS volumes. + +- Restic data is stored under a prefix/subdirectory of the main Velero bucket, and will go into the bucket corresponding to the `BackupStorageLocation` selected by the user at backup creation time. + +## Examples + +Let's look at some examples of how we can use this configuration mechanism to address some common use cases: + +#### Take snapshots of more than one kind of persistent volume in a single Velero backup (e.g. in a cluster with both EBS volumes and Portworx volumes) + +During server configuration: + +```shell +velero snapshot-location create ebs-us-east-1 \ + --provider aws \ + --config region=us-east-1 + +velero snapshot-location create portworx-cloud \ + --provider portworx \ + --config type=cloud +``` + +During backup creation: + +```shell +velero backup create full-cluster-backup \ + --volume-snapshot-locations ebs-us-east-1,portworx-cloud +``` + +Alternately, since in this example there's only one possible volume snapshot location configured for each of our two providers (`ebs-us-east-1` for `aws`, and `portworx-cloud` for `portworx`), Velero doesn't require them to be explicitly specified when creating the backup: + +```shell +velero backup create full-cluster-backup +``` + +#### Have some Velero backups go to a bucket in an eastern USA region, and others go to a bucket in a western USA region + +During server configuration: + +```shell +velero backup-location create default \ + --provider aws \ + --bucket velero-backups \ + --config region=us-east-1 + +velero backup-location create s3-alt-region \ + --provider aws \ + --bucket velero-backups-alt \ + --config region=us-west-1 +``` + +During backup creation: + +```shell +# The Velero server will automatically store backups in the backup storage location named "default" if +# one is not specified when creating the backup. You can alter which backup storage location is used +# by default by setting the --default-backup-storage-location flag on the `velero server` command (run +# by the Velero deployment) to the name of a different backup storage location. +velero backup create full-cluster-backup +``` + +Or: + +```shell +velero backup create full-cluster-alternate-location-backup \ + --storage-location s3-alt-region +``` + +#### For volume providers that support it (e.g. Portworx), have some snapshots be stored locally on the cluster and have others be stored in the cloud + +During server configuration: + +```shell +velero snapshot-location create portworx-local \ + --provider portworx \ + --config type=local + +velero snapshot-location create portworx-cloud \ + --provider portworx \ + --config type=cloud +``` + +During backup creation: + +```shell +# Note that since in this example we have two possible volume snapshot locations for the Portworx +# provider, we need to explicitly specify which one to use when creating a backup. Alternately, +# you can set the --default-volume-snapshot-locations flag on the `velero server` command (run by +# the Velero deployment) to specify which location should be used for each provider by default, in +# which case you don't need to specify it when creating a backup. +velero backup create local-snapshot-backup \ + --volume-snapshot-locations portworx-local +``` + +Or: + +```shell +velero backup create cloud-snapshot-backup \ + --volume-snapshot-locations portworx-cloud +``` + +#### Use a single location + +If you don't have a use case for more than one location, it's still easy to use Velero. Let's assume you're running on AWS, in the `us-west-1` region: + +During server configuration: + +```shell +velero backup-location create default \ + --provider aws \ + --bucket velero-backups \ + --config region=us-west-1 + +velero snapshot-location create ebs-us-west-1 \ + --provider aws \ + --config region=us-west-1 +``` + +During backup creation: + +```shell +# Velero will automatically use your configured backup storage location and volume snapshot location. +# Nothing needs to be specified when creating a backup. +velero backup create full-cluster-backup +``` + +## Additional Use Cases + +1. If you're using Azure's AKS, you may want to store your volume snapshots outside of the "infrastructure" resource group that is automatically created when you create your AKS cluster. This is possible using a `VolumeSnapshotLocation`, by specifying a `resourceGroup` under the `config` section of the snapshot location. See the [Azure volume snapshot location documentation][3] for details. + +1. If you're using Azure, you may want to store your Velero backups across multiple storage accounts and/or resource groups/subscriptions. This is possible using a `BackupStorageLocation`, by specifying a `storageAccount`, `resourceGroup` and/or `subscriptionId`, respectively, under the `config` section of the backup location. See the [Azure backup storage location documentation][4] for details. + + + +[1]: api-types/backupstoragelocation.md +[2]: api-types/volumesnapshotlocation.md +[3]: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/master/volumesnapshotlocation.md +[4]: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/master/backupstoragelocation.md diff --git a/site/docs/v1.4-pre/migration-case.md b/site/docs/v1.4-pre/migration-case.md new file mode 100644 index 0000000000..85ec5edefe --- /dev/null +++ b/site/docs/v1.4-pre/migration-case.md @@ -0,0 +1,50 @@ +# Cluster migration + +*Using Backups and Restores* + +Velero can help you port your resources from one cluster to another, as long as you point each Velero instance to the same cloud object storage location. In this scenario, we are also assuming that your clusters are hosted by the same cloud provider. **Note that Velero does not support the migration of persistent volumes across cloud providers.** + +1. *(Cluster 1)* Assuming you haven't already been checkpointing your data with the Velero `schedule` operation, you need to first back up your entire cluster (replacing `` as desired): + + ``` + velero backup create + ``` + + The default backup retention period, expressed as TTL (time to live), is 30 days (720 hours); you can use the `--ttl ` flag to change this as necessary. See [how velero works][1] for more information about backup expiry. + +1. *(Cluster 2)* Configure `BackupStorageLocations` and `VolumeSnapshotLocations`, pointing to the locations used by *Cluster 1*, using `velero backup-location create` and `velero snapshot-location create`. Make sure to configure the `BackupStorageLocations` as read-only + by using the `--access-mode=ReadOnly` flag for `velero backup-location create`. + +1. *(Cluster 2)* Make sure that the Velero Backup object is created. Velero resources are synchronized with the backup files in cloud storage. + + ``` + velero backup describe + ``` + + **Note:** The default sync interval is 1 minute, so make sure to wait before checking. You can configure this interval with the `--backup-sync-period` flag to the Velero server. + +1. *(Cluster 2)* Once you have confirmed that the right Backup (``) is now present, you can restore everything with: + + ``` + velero restore create --from-backup + ``` + +## Verify both clusters + +Check that the second cluster is behaving as expected: + +1. *(Cluster 2)* Run: + + ``` + velero restore get + ``` + +1. Then run: + + ``` + velero restore describe + ``` + +If you encounter issues, make sure that Velero is running in the same namespace in both clusters. + +[1]: how-velero-works.md#set-a-backup-to-expire diff --git a/site/docs/v1.4-pre/namespace.md b/site/docs/v1.4-pre/namespace.md new file mode 100644 index 0000000000..9a3d628089 --- /dev/null +++ b/site/docs/v1.4-pre/namespace.md @@ -0,0 +1,19 @@ +# Run in a non-default namespace + +The Velero installation and backups by default are run in the `velero` namespace. However, it is possible to use a different namespace. + +### 1) Customize the namespace during install + +Use the `--namespace` flag, in conjunction with the other flags in the `velero install` command (as shown in the [the Velero install instructions][0]). This will inform Velero where to install. + +### 2) Customize the namespace for operational commands + +To have namespace consistency, specify the namespace for all Velero operational commands to be the same as the namespace used to install Velero: + +```bash +velero client config set namespace= +``` + +Alternatively, you may use the global `--namespace` flag with any operational command to tell Velero where to run. + +[0]: basic-install.md#install-the-cli diff --git a/site/docs/v1.4-pre/on-premises.md b/site/docs/v1.4-pre/on-premises.md new file mode 100644 index 0000000000..ec199287c5 --- /dev/null +++ b/site/docs/v1.4-pre/on-premises.md @@ -0,0 +1,24 @@ +# On-Premises Environments + +You can run Velero in an on-premises cluster in different ways depending on your requirements. + +### Selecting an object storage provider + +You must select an object storage backend that Velero can use to store backup data. [Supported providers][0] contains information on various +options that are supported or have been reported to work by users. + +If you do not already have an object storage system, [MinIO][2] is an open-source S3-compatible object storage system that can be installed on-premises and is compatible with Velero. The details of configuring it for production usage are out of scope for Velero's documentation, but an [evaluation install guide][3] using MinIO is provided for convenience. + +### (Optional) Selecting volume snapshot providers + +If you need to back up persistent volume data, you must select a volume backup solution. [Supported providers][0] contains information on the supported options. + +For example, if you use [Portworx][4] for persistent storage, you can install their Velero plugin to get native Portworx snapshots as part of your Velero backups. + +If there is no native snapshot plugin available for your storage platform, you can use Velero's [restic integration][1], which provides a platform-agnostic file-level backup solution for volume data. + +[0]: supported-providers.md +[1]: restic.md +[2]: https://min.io +[3]: contributions/minio.md +[4]: https://portworx.com diff --git a/site/docs/v1.4-pre/output-file-format.md b/site/docs/v1.4-pre/output-file-format.md new file mode 100644 index 0000000000..887743451d --- /dev/null +++ b/site/docs/v1.4-pre/output-file-format.md @@ -0,0 +1,222 @@ +# Output file format + +A backup is a gzip-compressed tar file whose name matches the Backup API resource's `metadata.name` (what is specified during `velero backup create `). + +In cloud object storage, each backup file is stored in its own subdirectory in the bucket specified in the Velero server configuration. This subdirectory includes an additional file called `velero-backup.json`. The JSON file lists all information about your associated Backup resource, including any default values. This gives you a complete historical record of the backup configuration. The JSON file also specifies `status.version`, which corresponds to the output file format. + +The directory structure in your cloud storage looks something like: + +``` +rootBucket/ + backup1234/ + velero-backup.json + backup1234.tar.gz +``` + +## Example backup JSON file + +```json +{ + "kind": "Backup", + "apiVersion": "velero.io/v1", + "metadata": { + "name": "test-backup", + "namespace": "velero", + "selfLink": "/apis/velero.io/v1/namespaces/velero/backups/test-backup", + "uid": "a12345cb-75f5-11e7-b4c2-abcdef123456", + "resourceVersion": "337075", + "creationTimestamp": "2017-07-31T13:39:15Z" + }, + "spec": { + "includedNamespaces": [ + "*" + ], + "excludedNamespaces": null, + "includedResources": [ + "*" + ], + "excludedResources": null, + "labelSelector": null, + "snapshotVolumes": true, + "ttl": "24h0m0s" + }, + "status": { + "version": 1, + "formatVersion": "1.1.0", + "expiration": "2017-08-01T13:39:15Z", + "phase": "Completed", + "volumeBackups": { + "pvc-e1e2d345-7583-11e7-b4c2-abcdef123456": { + "snapshotID": "snap-04b1a8e11dfb33ab0", + "type": "gp2", + "iops": 100 + } + }, + "validationErrors": null + } +} +``` +Note that this file includes detailed info about your volume snapshots in the `status.volumeBackups` field, which can be helpful if you want to manually check them in your cloud provider GUI. + +## Output File Format Versioning + +The Velero output file format is intended to be relatively stable, but may change over time in order to support new features. + +In order to accommodate this, Velero follows [Semantic Versioning](http://semver.org/) for the file format version. + +Minor and patch versions will indicate backwards-compatible changes that previous versions of Velero can restore, including new directories or files. + +A major version would indicate that a version of Velero older than the version that created the backup could not restore it, usually because of moved or renamed directories or files. + +Major versions of the file format will be incremented with major version releases of Velero. +However, a major version release of Velero does not necessarily mean that the backup format version changed - Velero 3.0 could still use backup file format 2.0, as an example. + +## Versions + +### File Format Version: 1.1 (Current) + +In version 1.1, we have added the support of API groups versions as part of the backup (previously, only the preferred version of each API Groups was backed up). Each resource has one or more sub-directories, one sub-directory for each supported version of the API group. The preferred version API Group of each resource has the suffix "-preferredversion" as part of the sub-directory name. For backward compatibility, we kept the classic directory structure without the API Group version, which sits on the same level as the API Group sub-directory versions. +By default, only the preferred API group of each resource is backed up. +In order to take a backup of all API group versions, you need to run the Velero server with `--features=EnableAPIGroupVersions` feature flag. This is an experimental flag and the restore logic to handle multiple API Group Versions will be added in the future. + + +When unzipped, a typical backup directory (e.g. `backup1234.tar.gz`) taken with this file format version looks like the following (with the feature flag): + +``` +resources/ + persistentvolumes/ + cluster/ + pv01.json + ... + v1-preferredversion/ + cluster/ + pv01.json + ... + configmaps/ + namespaces/ + namespace1/ + myconfigmap.json + ... + namespace2/ + ... + v1-preferredversion/ + namespaces/ + namespace1/ + myconfigmap.json + ... + namespace2/ + ... + pods/ + namespaces/ + namespace1/ + mypod.json + ... + namespace2/ + ... + v1-preferredversion/ + namespaces/ + namespace1/ + mypod.json + ... + namespace2/ + ... + jobs.batch/ + namespaces/ + namespace1/ + awesome-job.json + ... + namespace2/ + ... + v1-preferredversion/ + namespaces/ + namespace1/ + awesome-job.json + ... + namespace2/ + ... + deployments/ + namespaces/ + namespace1/ + cool-deployment.json + ... + namespace2/ + ... + v1-preferredversion/ + namespaces/ + namespace1/ + cool-deployment.json + ... + namespace2/ + ... + horizontalpodautoscalers.autoscaling/ + namespaces/ + namespace1/ + hpa-to-the-rescue.json + ... + namespace2/ + ... + v1-preferredversion/ + namespaces/ + namespace1/ + hpa-to-the-rescue.json + ... + namespace2/ + ... + v2beta1/ + namespaces/ + namespace1/ + hpa-to-the-rescue.json + ... + namespace2/ + ... + v2beta2/ + namespaces/ + namespace1/ + hpa-to-the-rescue.json + ... + namespace2/ + ... + + ... +``` + +### File Format Version: 1 + +When unzipped, a typical backup directory (e.g. `backup1234.tar.gz`) looks like the following: + +``` +resources/ + persistentvolumes/ + cluster/ + pv01.json + ... + configmaps/ + namespaces/ + namespace1/ + myconfigmap.json + ... + namespace2/ + ... + pods/ + namespaces/ + namespace1/ + mypod.json + ... + namespace2/ + ... + jobs/ + namespaces/ + namespace1/ + awesome-job.json + ... + namespace2/ + ... + deployments/ + namespaces/ + namespace1/ + cool-deployment.json + ... + namespace2/ + ... + ... +``` diff --git a/site/docs/v1.4-pre/overview-plugins.md b/site/docs/v1.4-pre/overview-plugins.md new file mode 100644 index 0000000000..74eb3b0fb1 --- /dev/null +++ b/site/docs/v1.4-pre/overview-plugins.md @@ -0,0 +1,27 @@ + +# Velero plugin system + +Velero uses storage provider plugins to integrate with a variety of storage systems to support backup and snapshot operations. + +For server installation, Velero requires that at least one plugin is added (with the `--plugins` flag). The plugin will be either of the type object store or volume snapshotter, or a plugin that contains both. An exception to this is that when the user is not configuring a backup storage location or a snapshot storage location at the time of install, this flag is optional. + +Any plugin can be added after Velero has been installed by using the command `velero plugin add `. + +Example with a dockerhub image: `velero plugin add velero/velero-plugin-for-aws:v1.0.0`. + +In the same way, any plugin can be removed by using the command `velero plugin remove `. + +## Creating a new plugin + +Anyone can add integrations for any platform to provide additional backup and volume storage without modifying the Velero codebase. To write a plugin for a new backup or volume storage platform, take a look at our [example repo][1] and at our documentation for [Custom plugins][2]. + +## Adding a new plugin + +After you publish your plugin on your own repository, open a PR that adds a link to it under the appropriate list of [supported providers][3] page in our documentation. + +You can also add the [`velero-plugin` GitHub Topic][4] to your repo, and it will be shown under the aggregated list of repositories automatically. + +[1]: https://github.com/vmware-tanzu/velero-plugin-example/ +[2]: custom-plugins.md +[3]: supported-providers.md +[4]: https://github.com/topics/velero-plugin diff --git a/site/docs/v1.4-pre/rbac.md b/site/docs/v1.4-pre/rbac.md new file mode 100644 index 0000000000..d8686bf6ee --- /dev/null +++ b/site/docs/v1.4-pre/rbac.md @@ -0,0 +1,47 @@ +# Run Velero more securely with restrictive RBAC settings + +By default Velero runs with an RBAC policy of ClusterRole `cluster-admin`. This is to make sure that Velero can back up or restore anything in your cluster. But `cluster-admin` access is wide open -- it gives Velero components access to everything in your cluster. Depending on your environment and your security needs, you should consider whether to configure additional RBAC policies with more restrictive access. + +**Note:** Roles and RoleBindings are associated with a single namespaces, not with an entire cluster. PersistentVolume backups are associated only with an entire cluster. This means that any backups or restores that use a restrictive Role and RoleBinding pair can manage only the resources that belong to the namespace. You do not need a wide open RBAC policy to manage PersistentVolumes, however. You can configure a ClusterRole and ClusterRoleBinding that allow backups and restores only of PersistentVolumes, not of all objects in the cluster. + +For more information about RBAC and access control generally in Kubernetes, see the Kubernetes documentation about [access control][1], [managing service accounts][2], and [RBAC authorization][3]. + +## Set up Roles and RoleBindings + +Here's a sample Role and RoleBinding pair. + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + namespace: YOUR_NAMESPACE_HERE + name: ROLE_NAME_HERE + labels: + component: velero +rules: + - apiGroups: + - velero.io + verbs: + - "*" + resources: + - "*" +``` + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: ROLEBINDING_NAME_HERE +subjects: + - kind: ServiceAccount + name: YOUR_SERVICEACCOUNT_HERE +roleRef: + kind: Role + name: ROLE_NAME_HERE + apiGroup: rbac.authorization.k8s.io +``` + +[1]: https://kubernetes.io/docs/reference/access-authn-authz/controlling-access/ +[2]: https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/ +[3]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/ +[4]: namespace.md diff --git a/site/docs/v1.4-pre/release-instructions.md b/site/docs/v1.4-pre/release-instructions.md new file mode 100644 index 0000000000..b77f6ac184 --- /dev/null +++ b/site/docs/v1.4-pre/release-instructions.md @@ -0,0 +1,83 @@ +# Release Instructions + +## Ahead of Time + +### (GA Only) Release Blog Post PR + +Prepare a PR containing the release blog post. It's usually easiest to make a copy of the most recent existing post, then replace the content as appropriate. + +You also need to update `site/index.html` to have "Latest Release Information" contain a link to the new post. + +### (Pre-Release and GA) Changelog and Docs PR + +1. In a branch, create the file `changelogs/CHANGELOG-..md` (if it doesn't already exist) by copying the most recent one. +1. Run `make changelog` to generate a list of all unreleased changes. Copy/paste the output into `CHANGELOG-..md`, under the "All Changes" section for the release. + - You *may* choose to tweak formatting on the list of changes by adding code blocks, etc. +1. (GA Only) Remove all changelog files from `changelogs/unreleased`. +1. Update the main `CHANGELOG.md` file to properly reference the release-specific changelog file: + - (Pre-Release) List the release under "Development release" + - (GA) List the release under "Current release", remove any pre-releases from "Development release", and move the previous release into "Older releases". +1. If there is an existing set of pre-release versioned docs for the version you are releasing (i.e. `site/docs/v1.4-pre` exists, and you're releasing `v1.4.0-beta.2` or `v1.4.0`): + - Remove the directory containing the pre-release docs, i.e. `site/docs/`. + - Delete the pre-release docs table of contents file, i.e. `site/_data/-toc.yml`. + - Remove the pre-release docs table of contents mapping entry from `site/_data/toc-mapping.yml`. + - Remove all references to the pre-release docs from `site/_config.yml`. +1. Run `NEW_DOCS_VERSION=v VELERO_VERSION=v make gen-docs` (e.g. `NEW_DOCS_VERSION=v1.2 VELERO_VERSION=v1.2.0 make gen-docs` or `NEW_DOCS_VERSION=v1.2-pre VELERO_VERSION=v1.2.0-beta.1 make gen-docs`). + - Note that: + - **NEW_DOCS_VERSION** defines the version that the docs will be tagged with (i.e. what's in the URL, what shows up in the version dropdown on the site). This should be formatted as either `v1.4` (for a GA release), or `v1.4-pre` (for an alpha/beta/RC). + - **VELERO_VERSION** defines the tag of Velero that any `https://github.com/vmware-tanzu/velero/...` links in the docs should redirect to. +1. Follow the additional instructions at `site/README-JEKYLL.md` to complete the docs generation process. +1. Do a review of the diffs, and/or run `make serve-docs` and review the site. +1. Submit a PR containing the changelog and the version-tagged docs. + +### (Pre-Release and GA) GitHub Token + +To run the `goreleaser` process to generate a GitHub release, you'll need to have a GitHub token. See https://goreleaser.com/environment/ for more details. + +You may regenerate the token for every release if you prefer. + +#### If you don't already have a token +1. Go to https://github.com/settings/tokens/new. +1. Choose a name for your token. +1. Check the "repo" scope. +1. Click "Generate token". +1. Save the token value somewhere - you'll need it during the release, in the `GITHUB_TOKEN` environment variable. + +#### If you do already have a token, but need to regenerate it +1. Go to https://github.com/settings/tokens. +1. Click on the name of the relevant token. +1. Click "Regenerate token". +1. Save the token value somewhere - you'll need it during the release, in the `GITHUB_TOKEN` environment variable. + +## During Release + +This process is the same for both pre-release and GA, except for the fact that there will not be a blog post PR to merge for pre-release versions. + +1. Merge the changelog + docs PR, so that it's included in the release tag. +1. Make sure your working directory is clean: `git status` should show `nothing to commit, working tree clean`. +1. Run `git fetch upstream master && git checkout upstream/master`. +1. Run `git tag ` (e.g. `git tag v1.2.0` or `git tag v1.2.0-beta.1`). +1. Run `git push upstream ` (e.g. `git push upstream v1.2.0` or `git push upstream v1.2.0-beta.1`). This will trigger the Travis CI job that builds/publishes the Docker images. +1. Generate the GitHub release (it will be created in "Draft" status, which means it's not visible to the outside world until you click "Publish"): + + ```bash + GITHUB_TOKEN=your-github-token \ + RELEASE_NOTES_FILE=changelogs/CHANGELOG-..md \ + PUBLISH=true \ + make release + ``` + +1. Navigate to the draft GitHub release, at https://github.com/vmware-tanzu/velero/releases. +1. If this is a patch release (e.g. `v1.2.1`), note that the full `CHANGELOG-1.2.md` contents will be included in the body of the GitHub release. You need to delete the previous releases' content (e.g. `v1.2.0`'s changelog) so that only the latest patch release's changelog shows. +1. Do a quick review for formatting. **Note:** the `goreleaser` process should detect if it's a pre-release version, and check that box in the GitHub release appropriately, but it's always worth double-checking. +1. Publish the release. +1. By now, the Docker images should have been published. Perform a smoke-test - for example: + - Download the CLI from the GitHub release + - Use it to install Velero into a cluster (or manually update an existing deployment to use the new images) + - Verify that `velero version` shows the expected output + - Run a backup/restore and ensure it works +1. (GA Only) Merge the blog post PR. +1. Announce the release: + - Twitter (mention a few highlights, link to the blog post) + - Slack channel + - Google group (this doesn't get a lot of traffic, and recent releases may not have been posted here) diff --git a/site/docs/v1.4-pre/restic.md b/site/docs/v1.4-pre/restic.md new file mode 100644 index 0000000000..03056030a6 --- /dev/null +++ b/site/docs/v1.4-pre/restic.md @@ -0,0 +1,436 @@ +# Restic Integration + +Velero has support for backing up and restoring Kubernetes volumes using a free open-source backup tool called [restic][1]. This support is considered beta quality. Please see the list of [limitations](#limitations) to understand if it currently fits your use case. + +Velero has always allowed you to take snapshots of persistent volumes as part of your backups if you’re using one of +the supported cloud providers’ block storage offerings (Amazon EBS Volumes, Azure Managed Disks, Google Persistent Disks). +We also provide a plugin model that enables anyone to implement additional object and block storage backends, outside the +main Velero repository. + +We integrated restic with Velero so that users have an out-of-the-box solution for backing up and restoring almost any type of Kubernetes +volume*. This is a new capability for Velero, not a replacement for existing functionality. If you're running on AWS, and +taking EBS snapshots as part of your regular Velero backups, there's no need to switch to using restic. However, if you've +been waiting for a snapshot plugin for your storage platform, or if you're using EFS, AzureFile, NFS, emptyDir, +local, or any other volume type that doesn't have a native snapshot concept, restic might be for you. + +Restic is not tied to a specific storage platform, which means that this integration also paves the way for future work to enable +cross-volume-type data migrations. Stay tuned as this evolves! + +\* hostPath volumes are not supported, but the [new local volume type][4] is supported. + +## Setup + +### Prerequisites + +- Velero's restic integration requires the Kubernetes [MountPropagation feature][6], which is enabled by default in Kubernetes v1.10.0 and later. + +### Instructions + +Ensure you've [downloaded latest release][3]. + +To install restic, use the `--use-restic` flag on the `velero install` command. See the [install overview][2] for more details. When using restic on a storage provider that doesn't currently have Velero support for snapshots, the `--use-volume-snapshots=false` flag prevents an unused `VolumeSnapshotLocation` from being created on installation. + +Please note: For some PaaS/CaaS platforms based on Kubernetes such as RancherOS, OpenShift and Enterprise PKS, some modifications are required to the restic DaemonSet spec. + +**RancherOS** + +The host path for volumes is not `/var/lib/kubelet/pods`, rather it is `/opt/rke/var/lib/kubelet/pods` + +```yaml +hostPath: + path: /var/lib/kubelet/pods +``` + +to + +```yaml +hostPath: + path: /opt/rke/var/lib/kubelet/pods +``` + +**OpenShift** + +The restic containers should be running in a `privileged` mode to be able to mount the correct hostpath to pods volumes. + +1. Add the `velero` ServiceAccount to the `privileged` SCC: + + ``` + $ oc adm policy add-scc-to-user privileged -z velero -n velero + ``` + +2. For OpenShift version >= `4.1`, Modify the DaemonSet yaml to request a privileged mode: + + ```diff + @@ -67,3 +67,5 @@ spec: + value: /credentials/cloud + - name: VELERO_SCRATCH_DIR + value: /scratch + + securityContext: + + privileged: true + ``` + + or + + ```shell + oc patch ds/restic \ + --namespace velero \ + --type json \ + -p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]' + ``` + +3. For OpenShift version < `4.1`, Modify the DaemonSet yaml to request a privileged mode and mount the correct hostpath to pods volumes. + + ```diff + @@ -35,7 +35,7 @@ spec: + secretName: cloud-credentials + - name: host-pods + hostPath: + - path: /var/lib/kubelet/pods + + path: /var/lib/origin/openshift.local.volumes/pods + - name: scratch + emptyDir: {} + containers: + @@ -67,3 +67,5 @@ spec: + value: /credentials/cloud + - name: VELERO_SCRATCH_DIR + value: /scratch + + securityContext: + + privileged: true + ``` + + or + + ```shell + oc patch ds/restic \ + --namespace velero \ + --type json \ + -p '[{"op":"add","path":"/spec/template/spec/containers/0/securityContext","value": { "privileged": true}}]' + + oc patch ds/restic \ + --namespace velero \ + --type json \ + -p '[{"op":"replace","path":"/spec/template/spec/volumes/0/hostPath","value": { "path": "/var/lib/origin/openshift.local.volumes/pods"}}]' + ``` + + +If restic is not running in a privileged mode, it will not be able to access pods volumes within the mounted hostpath directory because of the default enforced SELinux mode configured in the host system level. You can [create a custom SCC](https://docs.openshift.com/container-platform/3.11/admin_guide/manage_scc.html) in order to relax the security in your cluster so that restic pods are allowed to use the hostPath volume plug-in without granting them access to the `privileged` SCC. + +By default a userland openshift namespace will not schedule pods on all nodes in the cluster. + +To schedule on all nodes the namespace needs an annotation: + +``` +oc annotate namespace openshift.io/node-selector="" +``` + +This should be done before velero installation. + +Or the ds needs to be deleted and recreated: + +``` +oc get ds restic -o yaml -n > ds.yaml +oc annotate namespace openshift.io/node-selector="" +oc create -n -f ds.yaml +``` + +**Enterprise PKS** + +You need to enable the `Allow Privileged` option in your plan configuration so that restic is able to mount the hostpath. + +The hostPath should be changed from `/var/lib/kubelet/pods` to `/var/vcap/data/kubelet/pods` + +```yaml +hostPath: + path: /var/vcap/data/kubelet/pods +``` + +**Microsoft Azure** + +If you are using [Azure Files][8], you need to add `nouser_xattr` to your storage class's `mountOptions`. See [this restic issue][9] for more details. + +You can use the following command to patch the storage class: + +```bash +kubectl patch storageclass/ \ + --type json \ + --patch '[{"op":"add","path":"/mountOptions/-","value":"nouser_xattr"}]' +``` + +You're now ready to use Velero with restic. + +## Back up + +1. Run the following for each pod that contains a volume to back up: + + ```bash + kubectl -n YOUR_POD_NAMESPACE annotate pod/YOUR_POD_NAME backup.velero.io/backup-volumes=YOUR_VOLUME_NAME_1,YOUR_VOLUME_NAME_2,... + ``` + + where the volume names are the names of the volumes in the pod spec. + + For example, for the following pod: + + ```yaml + apiVersion: v1 + kind: Pod + metadata: + name: sample + namespace: foo + spec: + containers: + - image: k8s.gcr.io/test-webserver + name: test-webserver + volumeMounts: + - name: pvc-volume + mountPath: /volume-1 + - name: emptydir-volume + mountPath: /volume-2 + volumes: + - name: pvc-volume + persistentVolumeClaim: + claimName: test-volume-claim + - name: emptydir-volume + emptyDir: {} + ``` + + You'd run: + + ```bash + kubectl -n foo annotate pod/sample backup.velero.io/backup-volumes=pvc-volume,emptydir-volume + ``` + + This annotation can also be provided in a pod template spec if you use a controller to manage your pods. + +1. Take a Velero backup: + + ```bash + velero backup create NAME OPTIONS... + ``` + +1. When the backup completes, view information about the backups: + + ```bash + velero backup describe YOUR_BACKUP_NAME + ``` + ```bash + kubectl -n velero get podvolumebackups -l velero.io/backup-name=YOUR_BACKUP_NAME -o yaml + ``` + +## Restore + +1. Restore from your Velero backup: + + ```bash + velero restore create --from-backup BACKUP_NAME OPTIONS... + ``` + +1. When the restore completes, view information about your pod volume restores: + + ```bash + velero restore describe YOUR_RESTORE_NAME + ``` + ```bash + kubectl -n velero get podvolumerestores -l velero.io/restore-name=YOUR_RESTORE_NAME -o yaml + ``` + +## Limitations + +- `hostPath` volumes are not supported. [Local persistent volumes][4] are supported. +- Those of you familiar with [restic][1] may know that it encrypts all of its data. We've decided to use a static, +common encryption key for all restic repositories created by Velero. **This means that anyone who has access to your +bucket can decrypt your restic backup data**. Make sure that you limit access to the restic bucket +appropriately. We plan to implement full Velero backup encryption, including securing the restic encryption keys, in +a future release. +- An incremental backup chain will be maintained across pod reschedules for PVCs. However, for pod volumes that are *not* +PVCs, such as `emptyDir` volumes, when a pod is deleted/recreated (e.g. by a ReplicaSet/Deployment), the next backup of those +volumes will be full rather than incremental, because the pod volume's lifecycle is assumed to be defined by its pod. +- Restic scans each file in a single thread. This means that large files (such as ones storing a database) will take a long time to scan for data deduplication, even if the actual +difference is small. + +## Customize Restore Helper Container + +Velero uses a helper init container when performing a restic restore. By default, the image for this container is `velero/velero-restic-restore-helper:`, +where `VERSION` matches the version/tag of the main Velero image. You can customize the image that is used for this helper by creating a ConfigMap in the Velero namespace with +the alternate image. + +In addition, you can customize the resource requirements for the init container, should you need. + +The ConfigMap must look like the following: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + # any name can be used; Velero uses the labels (below) + # to identify it rather than the name + name: restic-restore-action-config + # must be in the velero namespace + namespace: velero + # the below labels should be used verbatim in your + # ConfigMap. + labels: + # this value-less label identifies the ConfigMap as + # config for a plugin (i.e. the built-in restic restore + # item action plugin) + velero.io/plugin-config: "" + # this label identifies the name and kind of plugin + # that this ConfigMap is for. + velero.io/restic: RestoreItemAction +data: + # The value for "image" can either include a tag or not; + # if the tag is *not* included, the tag from the main Velero + # image will automatically be used. + image: myregistry.io/my-custom-helper-image[:OPTIONAL_TAG] + + # "cpuRequest" sets the request.cpu value on the restic init containers during restore. + # If not set, it will default to "100m". A value of "0" is treated as unbounded. + cpuRequest: 200m + + # "memRequest" sets the request.memory value on the restic init containers during restore. + # If not set, it will default to "128Mi". A value of "0" is treated as unbounded. + memRequest: 128Mi + + # "cpuLimit" sets the request.cpu value on the restic init containers during restore. + # If not set, it will default to "100m". A value of "0" is treated as unbounded. + cpuLimit: 200m + + # "memLimit" sets the request.memory value on the restic init containers during restore. + # If not set, it will default to "128Mi". A value of "0" is treated as unbounded. + memLimit: 128Mi + + +``` + +## Troubleshooting + +Run the following checks: + +Are your Velero server and daemonset pods running? + +```bash +kubectl get pods -n velero +``` + +Does your restic repository exist, and is it ready? + +```bash +velero restic repo get + +velero restic repo get REPO_NAME -o yaml +``` + +Are there any errors in your Velero backup/restore? + +```bash +velero backup describe BACKUP_NAME +velero backup logs BACKUP_NAME + +velero restore describe RESTORE_NAME +velero restore logs RESTORE_NAME +``` + +What is the status of your pod volume backups/restores? + +```bash +kubectl -n velero get podvolumebackups -l velero.io/backup-name=BACKUP_NAME -o yaml + +kubectl -n velero get podvolumerestores -l velero.io/restore-name=RESTORE_NAME -o yaml +``` + +Is there any useful information in the Velero server or daemon pod logs? + +```bash +kubectl -n velero logs deploy/velero +kubectl -n velero logs DAEMON_POD_NAME +``` + +**NOTE**: You can increase the verbosity of the pod logs by adding `--log-level=debug` as an argument +to the container command in the deployment/daemonset pod template spec. + +## How backup and restore work with restic + +We introduced three custom resource definitions and associated controllers: + +- `ResticRepository` - represents/manages the lifecycle of Velero's [restic repositories][5]. Velero creates +a restic repository per namespace when the first restic backup for a namespace is requested. The controller +for this custom resource executes restic repository lifecycle commands -- `restic init`, `restic check`, +and `restic prune`. + + You can see information about your Velero restic repositories by running `velero restic repo get`. + +- `PodVolumeBackup` - represents a restic backup of a volume in a pod. The main Velero backup process creates +one or more of these when it finds an annotated pod. Each node in the cluster runs a controller for this +resource (in a daemonset) that handles the `PodVolumeBackups` for pods on that node. The controller executes +`restic backup` commands to backup pod volume data. + +- `PodVolumeRestore` - represents a restic restore of a pod volume. The main Velero restore process creates one +or more of these when it encounters a pod that has associated restic backups. Each node in the cluster runs a +controller for this resource (in the same daemonset as above) that handles the `PodVolumeRestores` for pods +on that node. The controller executes `restic restore` commands to restore pod volume data. + +### Backup + +1. The main Velero backup process checks each pod that it's backing up for the annotation specifying a restic backup +should be taken (`backup.velero.io/backup-volumes`) +1. When found, Velero first ensures a restic repository exists for the pod's namespace, by: + - checking if a `ResticRepository` custom resource already exists + - if not, creating a new one, and waiting for the `ResticRepository` controller to init/check it +1. Velero then creates a `PodVolumeBackup` custom resource per volume listed in the pod annotation +1. The main Velero process now waits for the `PodVolumeBackup` resources to complete or fail +1. Meanwhile, each `PodVolumeBackup` is handled by the controller on the appropriate node, which: + - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data + - finds the pod volume's subdirectory within the above volume + - runs `restic backup` + - updates the status of the custom resource to `Completed` or `Failed` +1. As each `PodVolumeBackup` finishes, the main Velero process adds it to the Velero backup in a file named `-podvolumebackups.json.gz`. This file gets uploaded to object storage alongside the backup tarball. It will be used for restores, as seen in the next section. + +### Restore + +1. The main Velero restore process checks each existing `PodVolumeBackup` custom resource in the cluster to backup from. +1. For each `PodVolumeBackup` found, Velero first ensures a restic repository exists for the pod's namespace, by: + - checking if a `ResticRepository` custom resource already exists + - if not, creating a new one, and waiting for the `ResticRepository` controller to init/check it (note that + in this case, the actual repository should already exist in object storage, so the Velero controller will simply + check it for integrity) +1. Velero adds an init container to the pod, whose job is to wait for all restic restores for the pod to complete (more +on this shortly) +1. Velero creates the pod, with the added init container, by submitting it to the Kubernetes API +1. Velero creates a `PodVolumeRestore` custom resource for each volume to be restored in the pod +1. The main Velero process now waits for each `PodVolumeRestore` resource to complete or fail +1. Meanwhile, each `PodVolumeRestore` is handled by the controller on the appropriate node, which: + - has a hostPath volume mount of `/var/lib/kubelet/pods` to access the pod volume data + - waits for the pod to be running the init container + - finds the pod volume's subdirectory within the above volume + - runs `restic restore` + - on success, writes a file into the pod volume, in a `.velero` subdirectory, whose name is the UID of the Velero restore + that this pod volume restore is for + - updates the status of the custom resource to `Completed` or `Failed` +1. The init container that was added to the pod is running a process that waits until it finds a file +within each restored volume, under `.velero`, whose name is the UID of the Velero restore being run +1. Once all such files are found, the init container's process terminates successfully and the pod moves +on to running other init containers/the main containers. + +## 3rd party controllers + +### Monitor backup annotation + +Velero does not currently provide a mechanism to detect persistent volume claims that are missing the restic backup annotation. + +To solve this, a controller was written by Thomann Bits&Beats: [velero-pvc-watcher][7] + +### Add backup annotation + +Velero does not currently provide a single command or automatic way to backup all volume resources in the cluster without annotating pods or pod templates. + +The [velero-volume-controller][10] written by duyanghao helps to solve this problem by adding backup annotation to pods with volumes automatically. + +[1]: https://github.com/restic/restic +[2]: customize-installation.md#enable-restic-integration +[3]: https://github.com/vmware-tanzu/velero/releases/ +[4]: https://kubernetes.io/docs/concepts/storage/volumes/#local +[5]: http://restic.readthedocs.io/en/latest/100_references.html#terminology +[6]: https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation +[7]: https://github.com/bitsbeats/velero-pvc-watcher +[8]: https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv +[9]: https://github.com/restic/restic/issues/1800 +[10]: https://github.com/duyanghao/velero-volume-controller diff --git a/site/docs/v1.4-pre/restore-reference.md b/site/docs/v1.4-pre/restore-reference.md new file mode 100644 index 0000000000..9ea1920745 --- /dev/null +++ b/site/docs/v1.4-pre/restore-reference.md @@ -0,0 +1,90 @@ +# Restore Reference + +## Restoring Into a Different Namespace + +Velero can restore resources into a different namespace than the one they were backed up from. To do this, use the `--namespace-mappings` flag: + +```bash +velero restore create RESTORE_NAME \ + --from-backup BACKUP_NAME \ + --namespace-mappings old-ns-1:new-ns-1,old-ns-2:new-ns-2 +``` +## What happens when user removes restore objects +A **restore** object represents the restore operation. There are two types of deletion for restore objects: +### 1. Deleting with **`velero restore delete`** +This command will delete the custom resource representing it, along with its individual log and results files. But, it will not delete any objects that were created by it from your cluster. +### 2. Deleting with **`kubectl -n velero delete restore`** +This command will delete the custom resource representing the restore, but will not delete log/results files from object storage, or any objects that were created during the restore in your cluster. + +## Restore command-line options +To see all commands for restores, run : `velero restore --help` +To see all options associated with a specific command, provide the --help flag to that command. For example, **`velero restore create --help`** shows all options associated with the **create** command. + +### To List all options of restore use : **`velero restore --help`** + +```Usage: + velero restore [command] + +Available Commands: + create Create a restore + delete Delete restores + describe Describe restores + get Get restores + logs Get restore logs +``` + +## Changing PV/PVC Storage Classes + +Velero can change the storage class of persistent volumes and persistent volume claims during restores. To configure a storage class mapping, create a config map in the Velero namespace like the following: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + # any name can be used; Velero uses the labels (below) + # to identify it rather than the name + name: change-storage-class-config + # must be in the velero namespace + namespace: velero + # the below labels should be used verbatim in your + # ConfigMap. + labels: + # this value-less label identifies the ConfigMap as + # config for a plugin (i.e. the built-in restore item action plugin) + velero.io/plugin-config: "" + # this label identifies the name and kind of plugin + # that this ConfigMap is for. + velero.io/change-storage-class: RestoreItemAction +data: + # add 1+ key-value pairs here, where the key is the old + # storage class name and the value is the new storage + # class name. + : +``` + +## Changing PVC selected-node + +Velero can update the selected-node annotation of persistent volume claim during restores, if selected-node doesn't exist in the cluster then it will remove the selected-node annotation from PersistentVolumeClaim. To configure a node mapping, create a config map in the Velero namespace like the following: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + # any name can be used; Velero uses the labels (below) + # to identify it rather than the name + name: change-pvc-node-selector-config + # must be in the velero namespace + namespace: velero + # the below labels should be used verbatim in your + # ConfigMap. + labels: + # this value-less label identifies the ConfigMap as + # config for a plugin (i.e. the built-in restore item action plugin) + velero.io/plugin-config: "" + # this label identifies the name and kind of plugin + # that this ConfigMap is for. + velero.io/change-pvc-node-selector: RestoreItemAction +data: + # add 1+ key-value pairs here, where the key is the old + # node name and the value is the new node name. + : diff --git a/site/docs/v1.4-pre/run-locally.md b/site/docs/v1.4-pre/run-locally.md new file mode 100644 index 0000000000..913cbc122b --- /dev/null +++ b/site/docs/v1.4-pre/run-locally.md @@ -0,0 +1,49 @@ +# Run Velero locally in development + +Running the Velero server locally can speed up iterative development. This eliminates the need to rebuild the Velero server +image and redeploy it to the cluster with each change. + +## Run Velero locally with a remote cluster + +Velero runs against the Kubernetes API server as the endpoint (as per the `kubeconfig` configuration), so both the Velero server and client use the same `client-go` to communicate with Kubernetes. This means the Velero server can be run locally just as functionally as if it was running in the remote cluster. + +### Prerequisites + +When running Velero, you will need to ensure that you set up all of the following: + +* Appropriate RBAC permissions in the cluster + * Read access for all data from the source cluster and namespaces + * Write access to the target cluster and namespaces +* Cloud provider credentials + * Read/write access to volumes + * Read/write access to object storage for backup data +* A [BackupStorageLocation][20] object definition for the Velero server +* (Optional) A [VolumeSnapshotLocation][21] object definition for the Velero server, to take PV snapshots + +### 1. Install Velero + +See documentation on how to install Velero in some specific providers: [Install overview][22] + +### 2. Scale deployment down to zero + +After you use the `velero install` command to install Velero into your cluster, you scale the Velero deployment down to 0 so it is not simultaneously being run on the remote cluster and potentially causing things to get out of sync: + +`kubectl scale --replicas=0 deployment velero -n velero` + +#### 3. Start the Velero server locally + +* To run the server locally, use the full path according to the binary you need. Example, if you are on a Mac, and using `AWS` as a provider, this is how to run the binary you built from source using the full path: `AWS_SHARED_CREDENTIALS_FILE= ./_output/bin/darwin/amd64/velero`. Alternatively, you may add the `velero` binary to your `PATH`. + +* Start the server: `velero server [CLI flags]`. The following CLI flags may be useful to customize, but see `velero server --help` for full details: + * `--log-level`: set the Velero server's log level (default `info`, use `debug` for the most logging) + * `--kubeconfig`: set the path to the kubeconfig file the Velero server uses to talk to the Kubernetes apiserver (default `$KUBECONFIG`) + * `--namespace`: the set namespace where the Velero server should look for backups, schedules, restores (default `velero`) + * `--plugin-dir`: set the directory where the Velero server looks for plugins (default `/plugins`) + * `--metrics-address`: set the bind address and port where Prometheus metrics are exposed (default `:8085`) + +[15]: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#the-shared-credentials-file +[16]: https://cloud.google.com/docs/authentication/getting-started#setting_the_environment_variable +[18]: https://eksctl.io/ +[20]: api-types/backupstoragelocation.md +[21]: api-types/volumesnapshotlocation.md +[22]: basic-install.md diff --git a/site/docs/v1.4-pre/self-signed-certificates.md b/site/docs/v1.4-pre/self-signed-certificates.md new file mode 100644 index 0000000000..aca33d9c1d --- /dev/null +++ b/site/docs/v1.4-pre/self-signed-certificates.md @@ -0,0 +1,31 @@ +# Use Velero with a storage provider secured by a self-signed certificate + +If you are using an S3-Compatible storage provider that is secured with a self-signed certificate, connections to the object store may fail with a `certificate signed by unknown authority` message. +In order to proceed, a certificate bundle may be provided when adding the storage provider. + +## Trusting a self-signed certificate during installation + +When using the `velero install` command, you can use the `--cacert` flag to provide a path +to a PEM-encoded certificate bundle to trust. + +```bash +velero install \ + --plugins + --provider \ + --bucket \ + --secret-file \ + --cacert +``` + +Velero will then automatically use the provided CA bundle to verify TLS connections to +that storage provider when backing up and restoring. + +## Trusting a self-signed certificate with the Velero client + +To use the describe, download, or logs commands to access a backup or restore contained +in storage secured by a self-signed certificate as in the above example, you must use +the `--cacert` flag to provide a path to the certificate to be trusted. + +```bash +velero backup describe my-backup --cacert +``` diff --git a/site/docs/v1.4-pre/start-contributing.md b/site/docs/v1.4-pre/start-contributing.md new file mode 100644 index 0000000000..f8071682bb --- /dev/null +++ b/site/docs/v1.4-pre/start-contributing.md @@ -0,0 +1,21 @@ +## Start contributing + +### Before you start + +* Please familiarize yourself with the [Code of Conduct][1] before contributing. +* Also, see [CONTRIBUTING.md][2] for instructions on the developer certificate of origin that we require. + +### Finding your way around + +You may join the Velero community and contribute in many different ways, including helping us design or test new features. For any significant feature we consider adding, we start with a design document. You may find a list of currently in progress new designs here: https://github.com/vmware-tanzu/velero/pulls?q=is%3Aopen+is%3Apr+label%3ADesign. Feel free to review and help us with your input. + +For information on how to connect with our maintainers and community, join our online meetings, or find good first issues, start on our [Velero community](https://velero.io/community/) page. + +Please browse our list of resources, including a playlist of past online community meetings, blog posts, and other resources to help you get familiar with our project: [Velero resources](https://velero.io/resources/). + +### Contributing + +If you are ready to jump in and test, add code, or help with documentation, please use the navigation on the left under `Contribute`. + +[1]: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/CODE_OF_CONDUCT.md +[2]: https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/CONTRIBUTING.md diff --git a/site/docs/v1.4-pre/support-process.md b/site/docs/v1.4-pre/support-process.md new file mode 100644 index 0000000000..c4bdf82a00 --- /dev/null +++ b/site/docs/v1.4-pre/support-process.md @@ -0,0 +1,41 @@ +# Support Process + +## Weekly Rotation + +The Velero maintainers use a weekly rotation to manage community support. Each week, a different maintainer is the point person for responding to incoming support issues via Slack, GitHub, and the Google group. The point person is *not* expected to be on-call 24x7. Instead, they choose one or more hour(s) per day to be available/responding to incoming issues. They will communicate to the community what that time slot will be each week. + +## Start of Week + +We will update the public Slack channel's topic to indicate that you are the point person for the week, and what hours you'll be available. + +## During the Week + +### Where we will monitor +- `#velero` public Slack channel in Kubernetes org +- [all Velero-related repos][0] in GitHub (`velero`, `velero-plugin-for-[aws|gcp|microsoft-azure|csi]`, `helm-charts`) +- [Project Velero Google Group][1] + +### GitHub issue flow + +Generally speaking, new GitHub issues will fall into one of several categories. We use the following process for each: + +1. **Feature request** + - Label the issue with `Enhancement/User` or `Enhancement/Dev` + - Leave the issue in the `New Issues` swimlane for triage by product mgmt +1. **Bug** + - Label the issue with `Bug` + - Leave the issue in the `New Issues` swimlane for triage by product mgmt +1. **User question/problem** that does not clearly fall into one of the previous categories + - When you start investigating/responding, label the issue with `Investigating` + - Add comments as you go, so both the user and future support people have as much context as possible + - Use the `Needs Info` label to indicate an issue is waiting for information from the user. Remove/re-add the label as needed. + - If you resolve the issue with the user, close it out + - If the issue ends up being a feature request or a bug, update the title and follow the appropriate process for it + - If the reporter becomes unresponsive after multiple pings, close out the issue due to inactivity and comment that the user can always reach out again as needed + +## End of Week + +We ensure all GitHub issues worked on during the week on are labeled with `Investigating` and `Needs Info` (if appropriate), and have updated comments so the next person can pick them up. + +[0]: https://app.zenhub.com/workspaces/velero-5c59c15e39d47b774b5864e3/board?repos=99143276,112385197,213946861,190224441,214524700,214524630 +[1]: https://groups.google.com/forum/#!forum/projectvelero diff --git a/site/docs/v1.4-pre/supported-providers.md b/site/docs/v1.4-pre/supported-providers.md new file mode 100644 index 0000000000..ecda493541 --- /dev/null +++ b/site/docs/v1.4-pre/supported-providers.md @@ -0,0 +1,85 @@ +# Providers + +Velero supports a variety of storage providers for different backup and snapshot operations. Velero has a plugin system which allows anyone to add compatibility for additional backup and volume storage platforms without modifying the Velero codebase. + +## Velero supported providers + +| Provider | Object Store | Volume Snapshotter | Plugin Provider Repo | Setup Instructions | +|-----------------------------------|---------------------|------------------------------|-----------------------------------------|-------------------------------| +| [Amazon Web Services (AWS)][7] | AWS S3 | AWS EBS | [Velero plugin for AWS][8] | [AWS Plugin Setup][35] | +| [Google Cloud Platform (GCP)][11] | Google Cloud Storage| Google Compute Engine Disks | [Velero plugin for GCP][12] | [GCP Plugin Setup][36] | +| [Microsoft Azure][9] | Azure Blob Storage | Azure Managed Disks | [Velero plugin for Microsoft Azure][10] | [Azure Plugin Setup][37] | + +Contact: [#Velero Slack][28], [GitHub Issues][29] + +## Community supported providers + +| Provider | Object Store | Volume Snapshotter | Plugin Documentation | Contact | +|---------------------------|------------------------------|------------------------------------|------------------------|---------------------------------| +| [AlibabaCloud][21] | Alibaba Cloud OSS | Alibaba Cloud | [AlibabaCloud][22] | [GitHub Issue][23] | +| [DigitalOcean][15] | DigitalOcean Object Storage | DigitalOcean Volumes Block Storage | [StackPointCloud][16] | | +| [Hewlett Packard][24] | 🚫 | HPE Storage | [Hewlett Packard][25] | [Slack][26], [GitHub Issue][27] | +| [OpenEBS][17] | 🚫 | OpenEBS CStor Volume | [OpenEBS][18] | [Slack][19], [GitHub Issue][20] | +| [Portworx][31] | 🚫 | Portworx Volume | [Portworx][32] | [Slack][33], [GitHub Issue][34] | +| [VMware vSphere][39] | 🚫 | vSphere Volumes | [VMware vSphere][39] | [GitHub Issue][40] | + +## S3-Compatible object store providers + +Velero's AWS Object Store plugin uses [Amazon's Go SDK][0] to connect to the AWS S3 API. Some third-party storage providers also support the S3 API, and users have reported the following providers work with Velero: + +_Note that these storage providers are not regularly tested by the Velero team._ + + * [IBM Cloud][1] + * [Oracle Cloud][2] + * [Minio][3] + * [DigitalOcean][4] + * [NooBaa][5] + * Ceph RADOS v12.2.7 + * Quobyte + * [Cloudian HyperStore][38] + +_Some storage providers, like Quobyte, may need a different [signature algorithm version][6]._ + +## Non-supported volume snapshots + +In the case you want to take volume snapshots but didn't find a plugin for your provider, Velero has support for snapshotting using restic. Please see the [restic integration][30] documentation. + +[0]: https://github.com/aws/aws-sdk-go/aws +[1]: contributions/ibm-config.md +[2]: contributions/oracle-config.md +[3]: contributions/minio.md +[4]: https://github.com/StackPointCloud/ark-plugin-digitalocean +[5]: http://www.noobaa.com/ +[6]: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/master/backupstoragelocation.md +[7]: https://aws.amazon.com +[8]: https://github.com/vmware-tanzu/velero-plugin-for-aws +[9]: https://azure.com +[10]: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure +[11]: https://cloud.google.com +[12]: https://github.com/vmware-tanzu/velero-plugin-for-gcp +[15]: https://www.digitalocean.com/ +[16]: https://github.com/StackPointCloud/ark-plugin-digitalocean +[17]: https://openebs.io/ +[18]: https://github.com/openebs/velero-plugin +[19]: https://openebs-community.slack.com/ +[20]: https://github.com/openebs/velero-plugin/issues +[21]: https://www.alibabacloud.com/ +[22]: https://github.com/AliyunContainerService/velero-plugin +[23]: https://github.com/AliyunContainerService/velero-plugin/issues +[24]: https://www.hpe.com/us/en/storage.html +[25]: https://github.com/hpe-storage/velero-plugin +[26]: https://slack.hpedev.io/ +[27]: https://github.com/hpe-storage/velero-plugin/issues +[28]: https://kubernetes.slack.com/messages/velero +[29]: https://github.com/vmware-tanzu/velero/issues +[30]: restic.md +[31]: https://portworx.com/ +[32]: https://docs.portworx.com/scheduler/kubernetes/ark.html +[33]: https://portworx.slack.com/messages/px-k8s +[34]: https://github.com/portworx/ark-plugin/issues +[35]: https://github.com/vmware-tanzu/velero-plugin-for-aws#setup +[36]: https://github.com/vmware-tanzu/velero-plugin-for-gcp#setup +[37]: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#setup +[38]: https://www.cloudian.com/ +[39]: https://github.com/vmware-tanzu/velero-plugin-for-vsphere +[40]: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/issues/new diff --git a/site/docs/v1.4-pre/troubleshooting.md b/site/docs/v1.4-pre/troubleshooting.md new file mode 100644 index 0000000000..c2ded30d2b --- /dev/null +++ b/site/docs/v1.4-pre/troubleshooting.md @@ -0,0 +1,123 @@ +# Troubleshooting + +These tips can help you troubleshoot known issues. If they don't help, you can [file an issue][4], or talk to us on the [#velero channel][25] on the Kubernetes Slack server. + +- [Troubleshooting](#troubleshooting) + - [Debug installation/ setup issues](#debug-installation-setup-issues) + - [Debug restores](#debug-restores) + - [General troubleshooting information](#general-troubleshooting-information) + - [Getting velero debug logs](#getting-velero-debug-logs) + - [Known issue with restoring LoadBalancer Service](#known-issue-with-restoring-loadbalancer-service) + - [Miscellaneous issues](#miscellaneous-issues) + - [Velero reports `custom resource not found` errors when starting up.](#velero-reports-custom-resource-not-found-errors-when-starting-up) + - [`velero backup logs` returns a `SignatureDoesNotMatch` error](#velero-backup-logs-returns-a-signaturedoesnotmatch-error) + - [Velero (or a pod it was backing up) restarted during a backup and the backup is stuck InProgress](#velero-or-a-pod-it-was-backing-up-restarted-during-a-backup-and-the-backup-is-stuck-inprogress) + - [Velero is not publishing prometheus metrics](#velero-is-not-publishing-prometheus-metrics) + +## Debug installation/ setup issues + +- [Debug installation/setup issues][2] + +## Debug restores + +- [Debug restores][1] + +## General troubleshooting information + +You can use the `velero bug` command to open a [Github issue][4] by launching a browser window with some prepopulated values. Values included are OS, CPU architecture, `kubectl` client and server versions (if available) and the `velero` client version. This information isn't submitted to Github until you click the `Submit new issue` button in the Github UI, so feel free to add, remove or update whatever information you like. + +Some general commands for troubleshooting that may be helpful: + +* `velero backup describe ` - describe the details of a backup +* `velero backup logs ` - fetch the logs for this specific backup. Useful for viewing failures and warnings, including resources that could not be backed up. +* `velero restore describe ` - describe the details of a restore +* `velero restore logs ` - fetch the logs for this specific restore. Useful for viewing failures and warnings, including resources that could not be restored. +* `kubectl logs deployment/velero -n velero` - fetch the logs of the Velero server pod. This provides the output of the Velero server processes. + +### Getting velero debug logs + +You can increase the verbosity of the Velero server by editing your Velero deployment to look like this: + + +``` +kubectl edit deployment/velero -n velero +... + containers: + - name: velero + image: velero/velero:latest + command: + - /velero + args: + - server + - --log-level # Add this line + - debug # Add this line +... +``` + +## Known issue with restoring LoadBalancer Service + +Because of how Kubernetes handles Service objects of `type=LoadBalancer`, when you restore these objects you might encounter an issue with changed values for Service UIDs. Kubernetes automatically generates the name of the cloud resource based on the Service UID, which is different when restored, resulting in a different name for the cloud load balancer. If the DNS CNAME for your application points to the DNS name of your cloud load balancer, you'll need to update the CNAME pointer when you perform a Velero restore. + +Alternatively, you might be able to use the Service's `spec.loadBalancerIP` field to keep connections valid, if your cloud provider supports this value. See [the Kubernetes documentation about Services of Type LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer). + +## Miscellaneous issues + +### Velero reports `custom resource not found` errors when starting up. + +Velero's server will not start if the required Custom Resource Definitions are not found in Kubernetes. Run `velero install` again to install any missing custom resource definitions. + +### `velero backup logs` returns a `SignatureDoesNotMatch` error + +Downloading artifacts from object storage utilizes temporary, signed URLs. In the case of S3-compatible +providers, such as Ceph, there may be differences between their implementation and the official S3 +API that cause errors. + +Here are some things to verify if you receive `SignatureDoesNotMatch` errors: + + * Make sure your S3-compatible layer is using [signature version 4][5] (such as Ceph RADOS v12.2.7) + * For Ceph, try using a native Ceph account for credentials instead of external providers such as OpenStack Keystone + +## Velero (or a pod it was backing up) restarted during a backup and the backup is stuck InProgress + +Velero cannot currently resume backups that were interrupted. Backups stuck in the `InProgress` phase can be deleted with `kubectl delete backup -n `. +Backups in the `InProgress` phase have not uploaded any files to object storage. + +## Velero is not publishing prometheus metrics + +Steps to troubleshoot: + +- Confirm that your velero deployment has metrics publishing enabled. The [latest Velero helm charts][6] have been setup with [metrics enabled by default][7]. +- Confirm that the Velero server pod exposes the port on which the metrics server listens on. By default, this value is 8085. + +```yaml + ports: + - containerPort: 8085 + name: metrics + protocol: TCP +``` + +- Confirm that the metric server is listening for and responding to connections on this port. This can be done using [port-forwarding][9] as shown below + +```bash +$ kubectl -n port-forward 8085:8085 +Forwarding from 127.0.0.1:8085 -> 8085 +Forwarding from [::1]:8085 -> 8085 +. +. +. +``` + +Now, visiting http://localhost:8085/metrics on a browser should show the metrics that are being scraped from Velero. + +- Confirm that the Velero server pod has the nessary [annotations][8] for prometheus to scrape metrics. +- Confirm, from the Prometheus UI, that the Velero pod is one of the targets being scraped from Prometheus. + +[1]: debugging-restores.md +[2]: debugging-install.md +[4]: https://github.com/vmware-tanzu/velero/issues +[5]: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html +[6]: https://github.com/vmware-tanzu/helm-charts/blob/master/charts/velero +[7]: https://github.com/vmware-tanzu/helm-charts/blob/master/charts/velero/values.yaml#L44 +[8]: https://github.com/vmware-tanzu/helm-charts/blob/master/charts/velero/values.yaml#L49-L52 +[9]: https://kubectl.docs.kubernetes.io/pages/container_debugging/port_forward_to_pods.html +[25]: https://kubernetes.slack.com/messages/velero diff --git a/site/docs/v1.4-pre/uninstalling.md b/site/docs/v1.4-pre/uninstalling.md new file mode 100644 index 0000000000..727bc1aa0f --- /dev/null +++ b/site/docs/v1.4-pre/uninstalling.md @@ -0,0 +1,8 @@ +# Uninstalling Velero + +If you would like to completely uninstall Velero from your cluster, the following commands will remove all resources created by `velero install`: + +```bash +kubectl delete namespace/velero clusterrolebinding/velero +kubectl delete crds -l component=velero +``` diff --git a/site/docs/v1.4-pre/upgrade-to-1.4.md b/site/docs/v1.4-pre/upgrade-to-1.4.md new file mode 100644 index 0000000000..869185c6cb --- /dev/null +++ b/site/docs/v1.4-pre/upgrade-to-1.4.md @@ -0,0 +1,71 @@ +# Upgrading to Velero 1.4 + +## Prerequisites + +- Velero [v1.3.x][4] installed. + +If you're not yet running at least Velero v1.3, see the following: + +- [Upgrading to v1.1][1] +- [Upgrading to v1.2][2] +- [Upgrading to v1.3][3] + +## Instructions + +1. Install the Velero v1.4 command-line interface (CLI) by following the [instructions here][0]. + + Verify that you've properly installed it by running: + + ```bash + velero version --client-only + ``` + + You should see the following output: + + ```bash + Client: + Version: v1.4.0-beta.1 + Git commit: + ``` + +1. Update the container image used by the Velero deployment and, optionally, the restic daemon set: + + ```bash + kubectl set image deployment/velero \ + velero=velero/velero:v1.4.0-beta.1 \ + --namespace velero + + # optional, if using the restic daemon set + kubectl set image daemonset/restic \ + restic=velero/velero:v1.4.0-beta.1 \ + --namespace velero + ``` + +1. Update the Velero custom resource definitions (CRDs) to include the new backup progress fields: + + ```bash + velero install --crds-only --dry-run -o yaml | kubectl apply -f - + ``` + +1. Confirm that the deployment is up and running with the correct version by running: + + ```bash + velero version + ``` + + You should see the following output: + + ```bash + Client: + Version: v1.4.0-beta.1 + Git commit: + + Server: + Version: v1.4.0-beta.1 + ``` + +[0]: basic-install.md#install-the-cli +[1]: https://velero.io/docs/v1.1.0/upgrade-to-1.1/ +[2]: https://velero.io/docs/v1.2.0/upgrade-to-1.2/ +[3]: https://velero.io/docs/v1.3.2/upgrade-to-1.3/ +[4]: https://github.com/vmware-tanzu/velero/releases/tag/v1.3.2 diff --git a/site/docs/v1.4-pre/velero-install.md b/site/docs/v1.4-pre/velero-install.md new file mode 100644 index 0000000000..58718d16cf --- /dev/null +++ b/site/docs/v1.4-pre/velero-install.md @@ -0,0 +1,45 @@ +# Velero Install CLI + +This document serves as a guide to using the `velero install` CLI command to install `velero` server components into your kubernetes cluster. + +_NOTE_: `velero install` will, by default, use the CLI's version information to determine the version of the server compoents to deploy. This behavior may be overridden by using the `--image` flag. Refer to [Building Server Component Container Images][1]. + +## Usage + +This section explains some of the basic flags supported by the `velero install` CLI command. For a complete explanation of the flags, please run `velero install --help` + +```bash +velero install \ + --plugins + --provider \ + --bucket \ + --secret-file \ + --velero-pod-cpu-request \ + --velero-pod-mem-request \ + --velero-pod-cpu-limit \ + --velero-pod-mem-limit \ + [--use-restic] \ + [--restic-pod-cpu-request ] \ + [--restic-pod-mem-request ] \ + [--restic-pod-cpu-limit ] \ + [--restic-pod-mem-limit ] +``` + +The values for the resource requests and limits flags follow the same format as [Kubernetes resource requirements][3] +For plugin container images, please refer to our [supported providers][2] page. + +## Examples + +This section provides examples that serve as a starting point for more customized installations. + +```bash +velero install --provider gcp --plugins velero/velero-plugin-for-gcp:v1.0.0 --bucket mybucket --secret-file ./gcp-service-account.json + +velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.0.0 --bucket backups --provider aws --secret-file ./aws-iam-creds --backup-location-config region=us-east-2 --snapshot-location-config region=us-east-2 --use-restic + +velero install --provider azure --plugins velero/velero-plugin-for-microsoft-azure:v1.0.0 --bucket $BLOB_CONTAINER --secret-file ./credentials-velero --backup-location-config resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,storageAccount=$AZURE_STORAGE_ACCOUNT_ID[,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] --snapshot-location-config apiTimeout=[,resourceGroup=$AZURE_BACKUP_RESOURCE_GROUP,subscriptionId=$AZURE_BACKUP_SUBSCRIPTION_ID] +``` + +[1]: build-from-source.md#making-images-and-updating-velero +[2]: supported-providers.md +[3]: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ diff --git a/site/docs/v1.4-pre/vendoring-dependencies.md b/site/docs/v1.4-pre/vendoring-dependencies.md new file mode 100644 index 0000000000..1729f21ece --- /dev/null +++ b/site/docs/v1.4-pre/vendoring-dependencies.md @@ -0,0 +1,18 @@ +# Vendoring dependencies + +## Overview + +We are using [dep][0] to manage dependencies. You can install it by following [these +instructions][1]. + +## Adding a new dependency + +Run `dep ensure`. If you want to see verbose output, you can append `-v` as in +`dep ensure -v`. + +## Updating an existing dependency + +Run `dep ensure -update [ ...]` to update one or more dependencies. + +[0]: https://github.com/golang/dep +[1]: https://golang.github.io/dep/docs/installation.html diff --git a/site/docs/v1.4-pre/website-guidelines.md b/site/docs/v1.4-pre/website-guidelines.md new file mode 100644 index 0000000000..e47a762d57 --- /dev/null +++ b/site/docs/v1.4-pre/website-guidelines.md @@ -0,0 +1,46 @@ +# Website Guidelines + +## Running the website locally + +When making changes to the website, please run the site locally before submitting a PR and manually verify your changes. + +At the root of the project, run: + +```bash +make serve-docs +``` + +This runs all the Ruby dependencies in a container. + +Alternatively, for quickly loading the website, under the `velero/site/` directory run: + +```bash +bundle exec jekyll serve --livereload --future +``` + +For more information on how to run the website locally, please see our [jekyll documentation](https://github.com/vmware-tanzu/velero/blob/v1.4.0-beta.1/site/README-JEKYLL.md). + +## Adding a blog post + +The `author_name` value must also be included in the tags field so the page that lists the author's posts will work properly (Ex: https://velero.io/tags/carlisia%20campos/). + +Note that the tags field can have multiple values. + +Example: + +```text +author_name: Carlisia Campos +tags: ['Carlisia Campos', "release", "how-to"] +``` + +### Please add an image + +If there is no image added to the header of the post, the default Velero logo will be used. This is fine, but not ideal. + +If there's an image that can be used as the blog post icon, the image field must be set to: + +```text +image: /img/posts/ +``` + +This image file must be added to the `/site/img/posts` folder. diff --git a/site/docs/v1.4-pre/zenhub.md b/site/docs/v1.4-pre/zenhub.md new file mode 100644 index 0000000000..45298b3a25 --- /dev/null +++ b/site/docs/v1.4-pre/zenhub.md @@ -0,0 +1,15 @@ +# ZenHub + +As an Open Source community, it is necessary for our work, communication, and collaboration to be done in the open. +GitHub provides a central repository for code, pull requests, issues, and documentation. When applicable, we will use Google Docs for design reviews, proposals, and other working documents. + +While GitHub issues, milestones, and labels generally work pretty well, the Velero team has found that product planning requires some additional tooling that GitHub projects do not offer. + +In our effort to minimize tooling while enabling product management insights, we have decided to use [ZenHub Open-Source](https://www.zenhub.com/blog/open-source/) to overlay product and project tracking on top of GitHub. +ZenHub is a GitHub application that provides Kanban visualization, Epic tracking, fine-grained prioritization, and more. It's primary backing storage system is existing GitHub issues along with additional metadata stored in ZenHub's database. + +If you are a Velero user or Velero Developer, you do not _need_ to use ZenHub for your regular workflow (e.g to see open bug reports or feature requests, work on pull requests). However, if you'd like to be able to visualize the high-level project goals and roadmap, you will need to use the free version of ZenHub. + +## Using ZenHub + +ZenHub can be integrated within the GitHub interface using their [Chrome or FireFox extensions](https://www.zenhub.com/extension). In addition, you can use their dedicated [web application](https://app.zenhub.com/workspace/o/vmware-tanzu/velero/boards?filterLogic=all&repos=99143276).