Skip to content

Commit

Permalink
Merge pull request #1869 from OctopusDeploy/sast/EventRetentionDocsUp…
Browse files Browse the repository at this point in the history
…date

EventExports Directory Documentation
  • Loading branch information
evolutionise authored Jul 30, 2023
2 parents 43742cb + 07ad8e5 commit 0f552c1
Show file tree
Hide file tree
Showing 29 changed files with 133 additions and 34 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ After the Octopus Server has been configured, from Octopus Manager copy the Mast
![](/docs/administration/high-availability/configure/images/wizard-master-key.png "width=500")
:::

Finally, you need to tell Octopus to store artifacts, packages, task logs, and imports in the shared storage that you provisioned, that way each Octopus node can see the same files. To do this, you need to use the command-line.
Finally, you need to tell Octopus to store artifacts, packages, task logs, imports, and event exports in the shared storage that you provisioned, that way each Octopus node can see the same files. To do this, you need to use the command-line.

### Configure shared storage

Expand All @@ -87,6 +87,7 @@ Octopus.Server.exe path --artifacts \\Octoshared\OctopusData\Artifacts
Octopus.Server.exe path --taskLogs \\Octoshared\OctopusData\TaskLogs
Octopus.Server.exe path --nugetRepository \\Octoshared\OctopusData\Packages
Octopus.Server.exe path --imports \\Octoshared\OctopusData\Imports
Octopus.Server.exe path --eventExports \\Octoshared\OctopusData\EventExports
Octopus.Server.exe path --telemetry \\Octoshared\OctopusData\Telemetry
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ After the Octopus Server has been configured, from Octopus Manager copy the Mast
![](/docs/administration/high-availability/configure/images/wizard-master-key.png "width=500")
:::

Finally, you need to tell Octopus to store artifacts, packages, task logs, and imports in the shared storage that you provisioned, that way each Octopus node can see the same files. To do this, you need to use the command-line:
Finally, you need to tell Octopus to store artifacts, packages, task logs, imports, and event exports in the shared storage that you provisioned, that way each Octopus node can see the same files. To do this, you need to use the command-line:

### Configure shared storage

Expand All @@ -86,6 +86,7 @@ Octopus.Server.exe path --artifacts \\Octoshared\OctopusData\Artifacts
Octopus.Server.exe path --taskLogs \\Octoshared\OctopusData\TaskLogs
Octopus.Server.exe path --nugetRepository \\Octoshared\OctopusData\Packages
Octopus.Server.exe path --imports \\Octoshared\OctopusData\Imports
Octopus.Server.exe path --eventExports \\Octoshared\OctopusData\EventExports
Octopus.Server.exe path --telemetry \\Octoshared\OctopusData\Telemetry
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ If your Octopus Server is running in Microsoft Azure, you can use [Azure Files](

If your Octopus Server is running in Microsoft Azure, there is only one solution unless you have a [DFS Replica](https://docs.microsoft.com/windows-server/storage/dfs-replication/dfsr-overview) in Azure. That solution is [Azure Files](https://docs.microsoft.com/azure/storage/files/storage-files-introduction) which presents a file share over SMB 3.0 that can be shared across all of your Octopus servers.

After you have created your File Share, the best option is to add the Azure File Share as a [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) pointing at a local folder, for example `C:\Octopus\` for the Artifacts, Packages, TaskLogs and Imports which need to be available to all nodes.
After you have created your File Share, the best option is to add the Azure File Share as a [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) pointing at a local folder, for example `C:\Octopus\` for the Artifacts, Packages, TaskLogs, Imports and EventExports which need to be available to all nodes.

Run the PowerShell below before installing Octopus, substituting the placeholders with your own values:

Expand All @@ -95,6 +95,7 @@ New-Item -Path C:\Octopus\TaskLogs -ItemType SymbolicLink -Value \\octostorage.f
New-Item -Path C:\Octopus\Artifacts -ItemType SymbolicLink -Value \\octostorage.file.core.windows.net\octoha\Artifacts
New-Item -Path C:\Octopus\Packages -ItemType SymbolicLink -Value \\octostorage.file.core.windows.net\octoha\Packages
New-Item -Path C:\Octopus\Imports -ItemType SymbolicLink -Value \\octostorage.file.core.windows.net\octoha\Imports
New-Item -Path C:\Octopus\EventExports -ItemType SymbolicLink -Value \\octostorage.file.core.windows.net\octoha\EventExports
```
:::div{.hint}
Expand All @@ -109,6 +110,7 @@ It's worth noting that you need to have created the folders within the Azure Fil
& 'C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe' path --taskLogs "C:\Octopus\TaskLogs"
& 'C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe' path --nugetRepository "C:\Octopus\Packages"
& 'C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe' path --imports "C:\Octopus\Imports"
& 'C:\Program Files\Octopus Deploy\Octopus\Octopus.Server.exe' path --eventExports "C:\Octopus\EventExports"
```

### Load balancing in Azure
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ You can see the different file server options Google Cloud has in their [File St
To successfully create a NetApp Cloud SMB Volume in Google Cloud, you must have an Active Directory service that can be used to connect to the SMB volume. Please see the [creating and managing SMB volumes](https://cloud.google.com/architecture/partners/netapp-cloud-volumes/creating-smb-volumes) for further information. It's also worth reviewing the [security considerations for SMB access](https://cloud.google.com/architecture/partners/netapp-cloud-volumes/security-considerations) too.
:::

Once you have configured your NetApp Cloud SMB Volume, the best option is to mount the SMB share and then create a [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) pointing at a local folder, for example `C:\OctopusShared\` for the Artifacts, Packages, TaskLogs, and Imports folders which need to be available to all nodes.
Once you have configured your NetApp Cloud SMB Volume, the best option is to mount the SMB share and then create a [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) pointing at a local folder, for example `C:\OctopusShared\` for the Artifacts, Packages, TaskLogs, Imports, and EventExports folders which need to be available to all nodes.

Before installing Octopus, follow the steps below *on each* Compute engine instance to mount your SMB share.

Expand All @@ -102,11 +102,11 @@ Before installing Octopus, follow the steps below *on each* Compute engine insta

4. Follow the instructions in the **Mount Instructions for SMB window** that appears.

5. Create folders in your **SMB share** for the Artifacts, Packages, TaskLogs, and Imports.
5. Create folders in your **SMB share** for the Artifacts, Packages, TaskLogs, Imports, and EventExports.

![Create folders in your SMB share](/docs/administration/high-availability/design/images/smb-create-folders.png "width=500")

6. Create the symbolic links for the Artifacts, Packages, TaskLogs, and Imports folders.
6. Create the symbolic links for the Artifacts, Packages, TaskLogs, Imports, and EventExports folders.

Run the following PowerShell script, substituting the placeholder values with your own:

Expand Down Expand Up @@ -144,7 +144,13 @@ Before installing Octopus, follow the steps below *on each* Compute engine insta
if (-not (Test-Path -Path $ImportsFolder)) {
New-Item -Path $ImportsFolder -ItemType SymbolicLink -Value "$SmbShare\Imports"
}
$EventExportsFolder = Join-Path -Path $LocalFolder -ChildPath "EventExports"
if (-not (Test-Path -Path $EventExportsFolder)) {
New-Item -Path $EventExportsFolder -ItemType SymbolicLink -Value "$SmbShare\EventExports"
}
```

:::div{.hint}
Remember to create the folders in the SMB share before trying to create the symbolic links.
:::
Expand All @@ -156,7 +162,8 @@ Once you've completed those steps, [install Octopus](/docs/installation/) and th
--artifacts "C:\OctopusShared\Artifacts" `
--nugetRepository "C:\OctopusShared\Packages" `
--taskLogs "C:\OctopusShared\TaskLogs" `
--imports "C:\OctopusShared\Imports"
--imports "C:\OctopusShared\Imports" `
--eventExports "C:\OctopusShared\EventExports"
```

:::div{.hint}
Expand All @@ -165,7 +172,7 @@ Changing the path only needs to be done once, and not on each node as the values

#### Filestore using NFS

Once you have [created a Filestore instance](https://cloud.google.com/filestore/docs/creating-instances), the best option is to mount the NFS share using the `LocalSystem` account, and then create a [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) pointing at a local folder, for example `C:\OctopusShared\` for the Artifacts, Packages, TaskLogs, and Imports folders which need to be available to all nodes.
Once you have [created a Filestore instance](https://cloud.google.com/filestore/docs/creating-instances), the best option is to mount the NFS share using the `LocalSystem` account, and then create a [symbolic link](https://en.wikipedia.org/wiki/Symbolic_link) pointing at a local folder, for example `C:\OctopusShared\` for the Artifacts, Packages, TaskLogs, Imports, and EventExports folders which need to be available to all nodes.

Before installing Octopus, follow the steps below *on each* Compute engine instance to mount your NFS share.

Expand Down Expand Up @@ -276,9 +283,9 @@ Before installing Octopus, follow the steps below *on each* Compute engine insta
This is in effect the same when using the [watchdog](/docs/octopus-rest-api/octopus.server.exe-command-line/watchdog) command to configure a scheduled task to monitor the Octopus Server service.
:::

6. Create folders in your **NFS share** for the Artifacts, Packages, TaskLogs, and Imports.
6. Create folders in your **NFS share** for the Artifacts, Packages, TaskLogs, Imports, and EventExports.

7. Create the symbolic links for the Artifacts, Packages, TaskLogs, and Imports folders.
7. Create the symbolic links for the Artifacts, Packages, TaskLogs, Imports, and EventExports folders.

Run the following PowerShell script, substituting the placeholder values with your own:

Expand Down Expand Up @@ -312,7 +319,13 @@ Before installing Octopus, follow the steps below *on each* Compute engine insta
if (-not (Test-Path -Path $ImportsFolder)) {
New-Item -Path $ImportsFolder -ItemType SymbolicLink -Value "$NfsShare\Imports"
}

$EventExportsFolder = Join-Path -Path $LocalFolder -ChildPath "EventExports"
if (-not (Test-Path -Path $EventExportsFolder)) {
New-Item -Path $EventExportsFolder -ItemType SymbolicLink -Value "$NfsShare\EventExports"
}
```

:::div{.hint}
Remember to create the folders in the NFS share before trying to create the symbolic links.
:::
Expand All @@ -324,7 +337,8 @@ Once you've completed those steps, [install Octopus](/docs/installation/) and th
--artifacts "C:\OctopusShared\Artifacts" `
--nugetRepository "C:\OctopusShared\Packages" `
--taskLogs "C:\OctopusShared\TaskLogs" `
--imports "C:\OctopusShared\Imports"
--imports "C:\OctopusShared\Imports" `
--eventExports "C:\OctopusShared\EventExports"
```

:::div{.hint}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import LoadBalancerEndpointInfo from 'src/shared-content/administration/load-bal
You may already have an existing Octopus Server that you wish to make highly available. The process to migrate to Octopus High Availability is the same as the process detailed in [Configuring High Availability for Octopus](/docs/administration/high-availability/configure), except your existing server will be the **first node** in the cluster. Migrating to HA will involve:

1. Moving the SQL Server Database to a dedicated SQL Server.
1. Moving all the task logs, packages, artifacts, imports etc., to a shared storage folder (BLOB data).
1. Moving all the task logs, packages, artifacts, imports, event exports etc., to a shared storage folder (BLOB data).
1. Configuring a load balancer.

This guide is generic and purposely avoids mentioning specific technologies such as Azure File Storage, AWS RDS SQL Server, etc. Please refer to the guide matching your hosting solution for specifics.
Expand Down Expand Up @@ -67,7 +67,7 @@ You can run that script using the Octopus Deploy [script console](/docs/administ

### Moving BLOB data

Most of the BLOB data (task logs, artifacts, packages, imports etc) stored on the file system can be copied to the new location prior to the outage window. Doing so will reduce the amount of copying you have to do during the outage windows. In addition, you can make sure your Octopus Deploy instance can use that shared location by running a test script to create and delete a file.
Most of the BLOB data (task logs, artifacts, packages, imports, event exports etc) stored on the file system can be copied to the new location prior to the outage window. Doing so will reduce the amount of copying you have to do during the outage windows. In addition, you can make sure your Octopus Deploy instance can use that shared location by running a test script to create and delete a file.

- Provision the shared storage folder.
- If you are going to create a symbolic link to that shared folder, do that now.
Expand All @@ -88,6 +88,7 @@ robocopy C:\Octopus\TaskLogs \\YOURFILESHARE\OctopusHA\TaskLogs /mir /r:5
robocopy C:\Octopus\Artifacts \\YOURFILESHARE\OctopusHA\Artifacts /mir /r:5
robocopy C:\Octopus\Packages \\YOURFILESHARE\OctopusHA\Packages /mir /r:5
robocopy C:\Octopus\Imports \\YOURFILESHARE\OctopusHA\Imports /mir /r:5
robocopy C:\Octopus\EventExports \\YOURFILESHARE\OctopusHA\EventExports /mir /r:5
```

### Configure load balancer
Expand Down Expand Up @@ -137,6 +138,7 @@ $filePath = "YOUR ROOT DIRECTORY"
& .\Octopus.Server.exe path --taskLogs "$filePath\TaskLogs"
& .\Octopus.Server.exe path --nugetRepository "$filePath\Packages"
& .\Octopus.Server.exe path --imports "$filePath\Imports"
& .\Octopus.Server.exe path --eventExports "$filePath\EventExports"
& .\Octopus.Server.exe path --telemetry "$filePath\Telemetry"
```

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ You may need to move your Octopus installation of Server and database. The follo

## Before you move your Octopus instance
- You will need your Master Key in order for your new Octopus installation to connect to your existing database. You can retrieve and save a copy of the [Master Key](/docs/security/data-encryption) in the Octopus Manager.
- Data that is stored in the file system needs to be moved over to the new server. These are your packages stored in the built-in package repository, your artifacts (includes project logos), and your Task Logs.
- Data that is stored in the file system needs to be moved over to the new server. These are your packages stored in the built-in package repository, your artifacts (includes project logos), your Task Logs, and your Event Exports.
- Tentacle thumbprints are stored in the database. If you're using the same database, you won't need to re-configure your Tentacles.

:::div{.warning}
Expand All @@ -34,6 +34,7 @@ Below are instructions on how to move your Octopus Server and SQL Database.
- Task Logs
- Packages
- This folder only needs to be moved if using the built-in package repository. External feed details are stored in the database, and they will connect automatically.
- Event Exports

:::div{.warning}
The database stores the locations for these directories. After you connect to the database, your settings will be the same as they were in your original server. You can change the locations for these directories, but we recommend first moving the directories to there original location and then pointing to the new location. This process is outlined in the [moving the home directory](/docs/administration/managing-infrastructure/moving-your-octopus/move-the-home-directory) page.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ You may want to move only the Octopus Server itself, and continue using your exi

## Before you move your Octopus instance
- You will need your Master Key in order for your new Octopus installation to connect to your existing database. You can retrieve and save a copy of the [Master Key](/docs/security/data-encryption) in the Octopus Manager.
- Data that is stored in the file system needs to be moved over to the new server. These are your packages stored in the built-in package repository, your artifacts (includes project logos), and your Task Logs.
- Data that is stored in the file system needs to be moved over to the new server. These are your packages stored in the built-in package repository, your artifacts (includes project logos), your archived events, and your Task Logs.
- Tentacle thumbprints are stored in the database. If you're using the same database, you won't need to re-configure your Tentacles.

:::div{.warning}
Expand All @@ -30,6 +30,7 @@ You may want to move only the Octopus Server itself, and continue using your exi
- Task Logs
- Packages
- This folder only needs to be moved if using the built-in package repository. External feed details are stored in the database, and they will connect automatically.
- Event Exports
6. Finally, restart your new Octopus instance to index the packages. You can restart either in your Octopus Manager, or via the command line with the following command.
```
Octopus.Server.exe service --stop
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,11 @@ The Octopus Server stores files in the following folders by default:
- This is where all performance and other temporal telemetry files are stored.
- See this [page](/docs/administration/managing-infrastructure/server-configuration-and-file-storage/moving-octopus-server-folders/#MovingOctopusServerfolders-Telemetry) on how to move the Octopus Server telemetry folder
- `C:\Octopus\Imports`
- This folder was added in **Octopus 2021.1**
- This is where imported zip files are stored when using the [Export/Import Projects feature](/docs/projects/export-import).
- See this [page](/docs/administration/managing-infrastructure/server-configuration-and-file-storage/moving-octopus-server-folders/#MovingOctopusServerfolders-Imports) on how to move the Octopus Server imports folder
- `C:\Octopus\EventExports`
- This is where audit log archive zip files are stored when using the [Archived audit logs feature](/docs/security/users-and-teams/auditing/#archived-audit-events).
- See this [page](/docs/administration/managing-infrastructure/server-configuration-and-file-storage/moving-octopus-server-folders/#MovingOctopusServerfolders-EventExports) on how to move the Octopus Server event exports folder

## Clean up post-2.6 migration {#ServerconfigurationandFilestorage-CleanUpCleanuppost-2.6migration}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ A complete migration involves:
1. Copying task logs to the cloud based file storage.
1. Copying built-in feed packages to the cloud based file storage.
1. Copying artifacts to the cloud based file storage.
1. Copying archived events to the cloud based file storage.
1. Installing Octopus on your chosen hosting platform (e.g. a virtual machine or container orchestration platform).
1. Pointing the cloud Octopus instance to the cloud based database.
1. Reindexing the packages in the built-in feed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,12 @@ if ($versionSplit[0] -ne $upgradeSplit[0])
{
Throw "Unable to copy files to $fileBackupLocation\Packages"
}
$msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.EventExportsDirectory) $fileBackupLocation\EventExports /mir" -Wait -Passthru).ExitCode
if ($msiExitCode -ge 8)
{
Throw "Unable to copy files to $fileBackupLocation\EventExports"
}
}
# Finish any remaining tasks and stop the service
Expand Down Expand Up @@ -310,6 +316,12 @@ if ($versionSplit[0] -ne $upgradeSplit[1])
{
Throw "Unable to copy files to $filebackUpFolder\Packages"
}
$msiExitCode = (Start-Process -FilePath "robocopy" -ArgumentList "$($serverFolders.EventExportsDirectory) $filebackUpFolder\EventExports /mir" -Wait -Passthru).ExitCode
if ($msiExitCode -ge 8)
{
Throw "Unable to copy files to $filebackUpFolder\EventExports"
}
}
```

Expand Down
Loading

0 comments on commit 0f552c1

Please sign in to comment.