Skip to content

Commit

Permalink
Update retention policies docs (#2401)
Browse files Browse the repository at this point in the history
* Update retention policy tentacle page

* Update retention policies index
  • Loading branch information
N-lson authored Aug 5, 2024
1 parent 3cf4213 commit 075d2ab
Show file tree
Hide file tree
Showing 11 changed files with 37 additions and 38 deletions.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 27 additions & 25 deletions src/pages/docs/administration/retention-policies/index.mdx
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
layout: src/layouts/Default.astro
pubDate: 2023-01-01
modDate: 2023-10-04
modDate: 2024-08-02
title: Retention policies
icon: fa-broom
description: Retention policies allow you to specify the releases, packages and files you want to keep as well as the ones you want cleaned up.
navOrder: 70
hideInThisSectionHeader: true
Expand Down Expand Up @@ -46,7 +47,7 @@ We talk about Tentacles here, but the same process and logic applies to [SSH Tar
A retention policy can be applied to packages in the built-in Octopus package repository. By default, the policy is set to keep all packages indefinitely. This policy is *separate* from the [release retention policy](#releases-whats-deleted) described above.

:::figure
![](/docs/administration/retention-policies/images/3278059.png "width=500")
![](/docs/administration/retention-policies/images/built-in-repository.png "width=500")
:::

When a package retention policy is applied, Octopus will delete packages that meet *both* of the following criteria:
Expand All @@ -66,29 +67,25 @@ When configuring the repository retention policy, it's also worth making note of

## What isn't deleted \{#what-is-not-deleted}

Some items in Octopus are not affected by Retention policies, and are never deleted. One example of this is [Audit logs](/docs/security/users-and-teams/auditing). Octopus actively [prevents modifying or deleting audit logs](/docs/security/users-and-teams/auditing/#modifying-and-deleting-audit-logs-is-prevented).

:::div{.hint}
From version **Octopus 2023.1** the [Audit Retention functionality](/docs/security/users-and-teams/auditing/#archived-audit-events) will start being rolled out. This **does not** delete audit records. It just moves them from the database to the file system.
:::
Some items in Octopus are not affected by retention policies, and are never deleted. One example of this is [audit logs](/docs/security/users-and-teams/auditing). Octopus actively [prevents modifying or deleting audit logs](/docs/security/users-and-teams/auditing/#modifying-and-deleting-audit-logs-is-prevented).

## When the retention policies are applied \{#when-retention-policies-applied}

Both the Octopus Server and built-in repository retention policies are run under a scheduled task from the Octopus Server every 4 hours. This task does not apply retention policies to Tentacles.
Both release and built-in repository retention policies are run under a scheduled task from the Octopus Server every 4 hours. This task does not apply retention policies to Tentacles.

Tentacle retention policies are run **during a deployment**, specifically **after all package acquisition steps have completed**. So if you have a retention policy of 3 days and do not deploy to a Tentacle for 5 days, the files that are over 3 days old will not be deleted until after a deployment is run to that Tentacle. It will also only delete any packages or files that are associated with the **current project** being deployed. If it's a development server, and you have multiple projects deploying there, only the active deployed project files will be deleted. It does not have any information about other project's retention policies tagged with the deployment.

## How retention policies work with lifecycle phases \{#retention-policies-and-lifecycle-phases}

You can set individual retention policies to phases. This gives you much more control over environments, such as never deleting from Production but having a strict rigorous deletion from your Development environment.
You can set individual retention policies to phases. This gives you much more control over environments, such as never deleting from production but having a strict rigorous deletion from your development environment.

But how does it work? For a release we determine what phase it is currently in. So if a release is created and no deployments have been done, the default Lifecycle retention policy is the active setting for that release. When it is then deployed to the first phase, that retention policy becomes the active setting for the release, and so on. So when the release hits the production phase, and is set to keep all, that release will be kept, as will any deployments done to any environment of that release.
But how does it work? For a release we determine what phase it is currently in. So if a release is created and no deployments have been done, the default lifecycle retention policy is the active setting for that release. When it is then deployed to the first phase, that retention policy becomes the active setting for the release, and so on. So when the release hits the production phase, and is set to keep all, that release will be kept, as will any deployments done to any environment of that release.

If you have an Octopus Server retention policy for a project that has a final phase of keep all releases, once the release enters that phase it will never be deleted. But if you have a release that has not yet deployed to any environments in the final phase, and is set to only keep the last 3 releases, then the release will be deleted when it becomes the 4th release of the project that has not yet been deployed to any final phase environment. (Unless it is still on the dashboard!).
If you have a release retention policy for a project that has a final phase of keep all releases, once the release enters that phase it will never be deleted. But if you have a release that has not yet deployed to any environments in the final phase, and is set to only keep the last 3 releases, then the release will be deleted when it becomes the 4th release of the project that has not yet been deployed to any final phase environment (unless it is still on the dashboard).

### Example lifecycle retention policy task

In this section we walk through what's deleted for an example project when the lifecycle retention policy task runs. Consider a project that has the following lifecycle defined:
In this section we walk through what's deleted for an example project when the retention policy task runs. Consider a project that has the following lifecycle defined:

:::figure
![Release retention task example lifecycle](/docs/administration/retention-policies/images/retention-lifecycle-example.png "width=500")
Expand All @@ -104,7 +101,7 @@ The project has 15 releases, of which:

When the retention policy runs for this lifecycle, it keeps a count of both **undeployed releases** it finds, as well as a separate count for **releases per phase**.

For a release to be deleted, the count must be greater than the required number for each phase the release has been deployed to. In this example, Release `0.0.2` is the fourth release kept for Staging, but only the third for Production, so it will be kept.
For a release to be deleted, the count must be greater than the required number for each phase the release has been deployed to. In this example, release `0.0.2` is the fourth release kept for Staging, but only the third for Production, so it will be kept.

Each phase defined in this lifecycle will keep a maximum of **5 releases**:
- 1 release for the current dashboard version.
Expand Down Expand Up @@ -144,19 +141,19 @@ Undeployed releases will simply keep the number we have selected and no more. Wh

## Set release retention policies \{#set-release-retention-policies}

Under **Library ➜ Lifecycles** you select the Lifecycle you want to define or edit your retention policy for:
Under **Deploy ➜ Lifecycles** you select the Lifecycle you want to define or edit your retention policy for:

:::figure
![](/docs/administration/retention-policies/images/3278063.png "width=500")
![](/docs/administration/retention-policies/images/lifecycles.png "width=500")
:::

Each phase will inherit the retention policy from the above phase, but this is something you can change by expanding the Retention Policy panel.

:::figure
![](/docs/administration/retention-policies/images/3278062.png "width=500")
![](/docs/administration/retention-policies/images/lifecycle-retention-policy.png "width=500")
:::

Releases determines what is kept on the Octopus Server, and Files on Tentacle determines what files are kept on the Tentacle.
*Releases* determines what is kept on the Octopus Server, and *Files on Tentacle* determines what files are kept on the Tentacle.

In Octopus Server, you can keep all, or select a number of releases to keep.

Expand All @@ -181,26 +178,27 @@ Note how Dev and Test inherit their policies from the lifecycle retention policy

Phases prioritize retention policies from the latest previous phase over the lifecycle retention policy as later phases tend to be treated more seriously. The more serious a phase is, the more the retention policy matters. The final phases are more likely to be production environments, and therefore once a release reaches this point you will want to keep more of them.

## Set Built-in feed retention policy \{#built-in-feed}
## Set built-in feed retention policy \{#built-in-feed}

You can find the built-in repository retention policy settings under **Library ➜ Packages**.
You can find the built-in repository retention policy settings under **Deploy ➜ Packages**.

:::figure
![](/docs/administration/retention-policies/images/3278060.png "width=500")
![](/docs/administration/retention-policies/images/built-in-feed-retention-policy.png "width=500")
:::

In Octopus Server, this can be set to keep packages forever, or for a set number of days.

<OctopusCloudStorageLimits />

:::figure
![](/docs/administration/retention-policies/images/3278059.png "width=500")
![](/docs/administration/retention-policies/images/built-in-repository.png "width=500")
:::

Choosing the *A limited time* option will allow you to select the number of days to keep a package in the repository. The default value is 30, but you can choose something shorter or longer based on your needs.
Choosing *A limited time* will allow you to select the number of days to keep a package in the repository. The default value is 30, but you can choose something shorter or longer based on your needs.

:::div{.hint}
**Note on package clean-up**

Only packages that are not associated with releases will be cleaned up. That means even if a package is older than the value you choose, if it's attached to an existing release, it won't be cleaned up until that release is also cleaned up.
:::

Expand All @@ -210,7 +208,7 @@ Octopus does not apply any retention policies to external feeds. However the pac

## Recommendations

Whether you have an existing Octopus server or are setting it up for the first time, we have some recommendations when setting retention policies.
Whether you have an existing Octopus Server or are setting it up for the first time, we have some recommendations when setting retention policies.

### Change the defaults

Expand All @@ -228,8 +226,12 @@ For the built-in repository, even if you don't plan to use it, it's good to upda

If you have a large number of existing releases, we recommend starting with a large retention policy and adjusting it down to what you need.

For example, if you have 12 months worth of releases now, perhaps set the retention policy to keep 11 months worth of releases. The Octopus server will apply these retention policies periodically. After it has cleaned up the oldest releases, you can change the policy to keep ten months of releases, and so on. You can also apply this method with the number of releases instead of the time-based setting.
For example, if you have 12 months worth of releases now, perhaps set the retention policy to keep 11 months worth of releases. Octopus will apply these retention policies periodically. After it has cleaned up the oldest releases, you can change the policy to keep ten months of releases, and so on. You can also apply this method with the number of releases instead of the time-based setting.

## Older versions

From 2023.1, the [audit retention functionality](/docs/security/users-and-teams/auditing/#archived-audit-events) has been rolled out. This **does not** delete audit records. It just moves them from the database to the file system.

## Learn more

- [Retention policy knowledge base articles](https://help.octopus.com/tag/retention).
- [Retention policy knowledge base articles](https://help.octopus.com/tag/retention)
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
---
layout: src/layouts/Default.astro
pubDate: 2023-01-01
modDate: 2023-01-01
modDate: 2024-08-02
title: Retention policy Tentacle cleanup and troubleshooting
icon: fa-bug
description: Reviewing and troubleshooting why some files aren't cleaned up by Octopus retention policies.
---

Expand Down Expand Up @@ -73,23 +74,23 @@ Defining retention policies is done within lifecycles. Each phase can have a dif
![](/docs/administration/retention-policies/images/default-lifecycle-retention-policy.png)
:::

In this example the default for the Lifecycle is to Keep 3 releases on both Octopus Server and Tentacle.
In this example the default for the lifecycle is to keep 3 releases on both Octopus Server and Tentacle.

Learn more about [Lifecycles](/docs/releases/lifecycles/) and [Retention Policies](/docs/administration/retention-policies) on their own detailed pages.
Learn more about [lifecycles](/docs/releases/lifecycles/) and [retention policies](/docs/administration/retention-policies) on their own detailed pages.

## Retention policies with channels {#retention-policy-channels}

[Channels](/docs/releases/channels) can be used in Octopus to handle many different deployment scenarios. In some cases you may have a hotfix channel in which deployments, as they are promoted through their environments, should be considered as overriding deployments from the default channel for the given environment. Alternatively you may be using channels to deploy feature branches which involve having several concurrent releases active at any one time across different channels for the same environment. When using the feature branch type scenario, you will likely want retention policies to recognize that since both channels should be accessible at the same time, the retention policy rules should apply to each independently. This behavior can be enabled for each project via the `Discrete Channel Releases` flag under `Deployment settings` on the **Project Settings** page.
[Channels](/docs/releases/channels) can be used in Octopus to handle many different deployment scenarios. In some cases you may have a hotfix channel in which deployments, as they are promoted through their environments, should be considered as overriding deployments from the default channel for the given environment. Alternatively you may be using channels to deploy feature branches which involve having several concurrent releases active at any one time across different channels for the same environment. When using the feature branch type scenario, you will likely want retention policies to recognize that since both channels should be accessible at the same time, the retention policy rules should apply to each independently. This behavior can be enabled for each project via the "Discrete Channel Releases" option on the **Project Settings** page.

:::figure
![Discrete Channel Release](/docs/administration/retention-policies/images/discrete-channel-release.png)
:::

## When the retention policy is run {#retention-policies-run}

For a Tentacle the retention policy is run at the end of a deployment, for that project only. So for this example the deployment looks for the project (project-1) and finds all releases within the deployment journal. It finds 4 in total (current is never counted) leaving 3, knowing it just deployed one, it deletes one copy of each package.
For a Tentacle the retention policy is run at the end of a deployment, for that project only. So for this example the deployment looks for the project (Project-1) and finds all releases within the deployment journal. It finds 4 in total (current is never counted) leaving 3, knowing it just deployed one, it deletes one copy of each package.

So for Project-1 we have 8 packages and directories still remaining on the server after the deployment. The current, and then the 3 defined by the policy. This is for each package. Any packages for other projects (project-2) were not evaluated and not removed, even if it was the same package version. Project 2 is considered it's own trigger for that retention policy, and is assumed to have different variables and transformations, thus a unique set of extracted files.
So for Project-1 we have 8 packages and directories still remaining on the server after the deployment. The current, and then the 3 defined by the policy. This is for each package. Any packages for other projects (Project-2) were not evaluated and not removed, even if it was the same package version. Project 2 is considered its own trigger for that retention policy, and is assumed to have different variables and transformations, thus a unique set of extracted files.

See below the messages you will have in your raw deployment logs at the end of a deployment to that environment for the specific project:

Expand All @@ -115,26 +116,22 @@ See below the messages you will have in your raw deployment logs at the end of a
You can find your packages in the `<Tentacle Home>\Files` directory.
or `<Tentacle Home>\[Instance Name])\Files` if you have multiple
Tentacle instances on one machine.
- For Windows Tentacles, the default directory is: `C:\Octopus\Files`
- For Linux Tentacles, the default directory is: `/etc/octopus/files`

Learn more about where Tentacle [files are stored](/docs/administration/managing-infrastructure/tentacle-configuration-and-file-storage/#Tentacleconfigurationandfilestorage-Filestorage).

:::figure
![](/docs/administration/retention-policies/images/3278387.png)
:::

By default your extracted package files can be found under `<Tentacle Home>\Applications\[environment name]\[package name]\`

So if you have multiple packages you will have multiple directories.
If you have multiple packages, you will have multiple directories.

:::figure
![](/docs/administration/retention-policies/images/3278389.png)
:::

![](/docs/administration/retention-policies/images/3278388.png)

If you have more directories than you think you should, check if they have a value in the deployment journal, if they do not they will have to be manually deleted.
If you have more directories than you think you should, check if they have a value in the deployment journal. If they don't, you'll need to manually delete them.

You can have multiple directories for the same version of each package like the following example:

Expand All @@ -146,6 +143,6 @@ This occurs when you have the same package in two different steps inside a singl

## Troubleshooting {#retention-policy-troubleshooting}

If you upgraded from Tentacle 2.x to a modern version of Tentacle the deployment journal location moved. Your choices are to clean up any old deployments manually, merge your deployment journals to the new location or run this [PowerShell Script](https://gist.github.com/vanessalove/dbc656b01df40939dcf8) on your Tentacles.
If you upgraded from Tentacle 2.x to a modern version of Tentacle, the deployment journal location moved. Your choices are to clean up any old deployments manually, merge your deployment journals to the new location, or run this [PowerShell script](https://gist.github.com/vanessalove/dbc656b01df40939dcf8) on your Tentacles.

If your deployment journal is deleted for any reason, you will need to manually remove any remaining packages and package extraction directories that are not in the new deployment journal (which is automatically created on the next deployment).

0 comments on commit 075d2ab

Please sign in to comment.