Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Azure Backups #968

Merged
merged 34 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
26b88a0
Add AzureCredentials and dependent structs
ianstanton Sep 23, 2024
c96bde0
Add to backup and restore spec
ianstanton Sep 23, 2024
98eb636
Add generate_azure_backup_credentials
ianstanton Sep 23, 2024
c4c0254
Add generate_azure_restore_credentials
ianstanton Sep 23, 2024
e804b0f
Fix conductor test
ianstanton Sep 23, 2024
1a3f3af
Merge branch 'main' into slf-113
ianstanton Sep 23, 2024
f3e81fd
Bump tembo-operator chart
ianstanton Sep 23, 2024
14f9b82
Pin google-cloud-storage crate to 0.22.0
ianstanton Sep 23, 2024
6620346
Add init_azure_storage
ianstanton Sep 24, 2024
3dd2b9f
fmt
ianstanton Sep 24, 2024
4efe354
Add info
ianstanton Sep 24, 2024
485a3f1
Fix logic in generate_azure_creds
ianstanton Sep 24, 2024
c0a6934
Check if aws for restore
ianstanton Sep 24, 2024
62125f0
Hack in logic for azure restore
ianstanton Sep 24, 2024
04de7eb
Fix tests
ianstanton Sep 24, 2024
a6ce309
Fix backups_path
ianstanton Sep 24, 2024
5f42e0a
Add duplicate name to path
ianstanton Sep 24, 2024
5de7269
Try to fix duplicate dir
ianstanton Sep 25, 2024
4227c09
Revert restore hack
ianstanton Sep 25, 2024
8300816
fmt
ianstanton Sep 25, 2024
c3fdf84
Remove info
ianstanton Sep 25, 2024
c8a8318
Merge branch 'main' of github.com:CoreDB-io/coredb into slf-113
ianstanton Sep 25, 2024
9892653
Pull out conductor changes
ianstanton Sep 25, 2024
9265eb0
Fill in backup values
ianstanton Sep 26, 2024
5fc5258
Remove debug
ianstanton Sep 26, 2024
b965fd9
Add tests
ianstanton Sep 26, 2024
f665857
Add inheritedMetadata if azure and workload identity
ianstanton Sep 26, 2024
5eb0654
Merge branch 'main' into slf-113
ianstanton Oct 4, 2024
bb06428
Merge branch 'main' into slf-113
ianstanton Oct 16, 2024
5a1ce5e
Merge branch 'main' into slf-113
ianstanton Oct 17, 2024
bec192b
Merge branch 'main' into slf-113
ianstanton Oct 17, 2024
34a3442
`conductor`: Add support for Azure Backup and Restore (#980)
ianstanton Oct 21, 2024
63f679a
Merge branch 'main' into slf-113
ianstanton Oct 21, 2024
fb2f431
Bump controller crate
ianstanton Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/tembo-operator/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: tembo-operator
description: "Helm chart to deploy the tembo-operator"
type: application
icon: https://cloud.tembo.io/images/TemboElephant.png
version: 0.7.3
version: 0.8.0
home: https://tembo.io
sources:
- https://github.com/tembo-io/tembo
Expand Down
139 changes: 135 additions & 4 deletions charts/tembo-operator/templates/crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -1932,13 +1932,79 @@ spec:
endpointURL: null
s3Credentials: null
googleCredentials: null
azureCredentials: null
volumeSnapshot:
enabled: false
description: |-
The backup configuration for the CoreDB instance to facilitate database backups and WAL archive uploads to an S3 compatible object store.

**Default**: disabled
properties:
azureCredentials:
description: The Azure credentials to use for backups
nullable: true
properties:
connectionString:
description: The connection string to be used
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
inheritFromAzureAD:
description: Use the Azure AD based authentication without providing explicitly the keys.
nullable: true
type: boolean
storageAccount:
description: The storage account where to upload data
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
storageKey:
description: The storage account key to be used in conjunction with the storage account name
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
storageSasToken:
description: A shared-access-signature to be used in conjunction with the storage account name
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
type: object
destinationPath:
default: s3://
description: The S3 bucket path to store backups in
Expand All @@ -1954,7 +2020,7 @@ spec:
nullable: true
type: string
googleCredentials:
description: 'GoogleCredentials is the type for the credentials to be used to upload files to Google Cloud Storage. It can be provided in two alternative ways: * The secret containing the Google Cloud Storage JSON file with the credentials (applicationCredentials) * inheriting the role from the pod (GKE) environment by setting gkeEnvironment to true'
description: The Google Cloud credentials to use for backups
nullable: true
properties:
applicationCredentials:
Expand Down Expand Up @@ -2391,6 +2457,71 @@ spec:
**Default**: disabled
nullable: true
properties:
azureCredentials:
description: azureCredentials is the Azure credentials to use for restores.
nullable: true
properties:
connectionString:
description: The connection string to be used
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
inheritFromAzureAD:
description: Use the Azure AD based authentication without providing explicitly the keys.
nullable: true
type: boolean
storageAccount:
description: The storage account where to upload data
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
storageKey:
description: The storage account key to be used in conjunction with the storage account name
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
storageSasToken:
description: A shared-access-signature to be used in conjunction with the storage account name
nullable: true
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
type: object
backupsPath:
description: |-
The object storage path and bucket name of the instance you wish to restore from. This maps to the `Backup` `destinationPath` field for the original instance.
Expand All @@ -2399,11 +2530,11 @@ spec:
nullable: true
type: string
endpointURL:
description: endpointURL is the S3 compatable endpoint URL
description: endpointURL is the S3 compatible endpoint URL
nullable: true
type: string
googleCredentials:
description: s3Credentials is the S3 credentials to use for backups.
description: googleCredentials is the Google Cloud credentials to use for restores.
nullable: true
properties:
applicationCredentials:
Expand All @@ -2428,7 +2559,7 @@ spec:
nullable: true
type: string
s3Credentials:
description: s3Credentials is the S3 credentials to use for backups.
description: s3Credentials is the S3 credentials to use for restores.
nullable: true
properties:
accessKeyId:
Expand Down
93 changes: 90 additions & 3 deletions tembo-operator/src/apis/coredb_types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,84 @@ pub struct GoogleCredentialsApplicationCredentials {
pub name: String,
}

/// AzureCredentials is the type for the credentials to be used to upload files to Azure Blob Storage.
#[derive(Serialize, Deserialize, Clone, Debug, Default, JsonSchema)]
pub struct AzureCredentials {
/// The connection string to be used
#[serde(
default,
skip_serializing_if = "Option::is_none",
rename = "connectionString"
)]
pub connection_string: Option<AzureCredentialsConnectionString>,
/// Use the Azure AD based authentication without providing explicitly the keys.
#[serde(
default,
skip_serializing_if = "Option::is_none",
rename = "inheritFromAzureAD"
)]
pub inherit_from_azure_ad: Option<bool>,
/// The storage account where to upload data
#[serde(
default,
skip_serializing_if = "Option::is_none",
rename = "storageAccount"
)]
pub storage_account: Option<AzureCredentialsStorageAccount>,
/// The storage account key to be used in conjunction with the storage account name
#[serde(
default,
skip_serializing_if = "Option::is_none",
rename = "storageKey"
)]
pub storage_key: Option<AzureCredentialsStorageKey>,
/// A shared-access-signature to be used in conjunction with the storage account name
#[serde(
default,
skip_serializing_if = "Option::is_none",
rename = "storageSasToken"
)]
pub storage_sas_token: Option<AzureCredentialsStorageSasToken>,
}

/// The connection string to be used for Azure Blob Storage backups
#[derive(Serialize, Deserialize, Clone, Debug, Default, JsonSchema)]
pub struct AzureCredentialsConnectionString {
/// The key to select
pub key: String,
/// Name of the referent.
pub name: String,
}

/// The storage account for Azure Blob Storage backups
#[derive(Serialize, Deserialize, Clone, Debug, Default, JsonSchema)]
pub struct AzureCredentialsStorageAccount {
/// The key to select
pub key: String,
/// Name of the referent.
pub name: String,
}

/// The storage account key to be used in conjunction with the storage account name for Azure Blob
/// Storage backups
#[derive(Serialize, Deserialize, Clone, Debug, Default, JsonSchema)]
pub struct AzureCredentialsStorageKey {
/// The key to select
pub key: String,
/// Name of the referent.
pub name: String,
}

/// A shared-access-signature to be used in conjunction with the storage account name for Azure Blob
/// Storage backups
#[derive(Serialize, Deserialize, Clone, Debug, Default, JsonSchema)]
pub struct AzureCredentialsStorageSasToken {
/// The key to select
pub key: String,
/// Name of the referent.
pub name: String,
}

/// VolumeSnapshots is the type for the configuration of the volume snapshots
/// to be used for backups instead of object storage
#[derive(Serialize, Deserialize, Clone, Debug, Default, JsonSchema, PartialEq)]
Expand Down Expand Up @@ -260,9 +338,14 @@ pub struct Backup {
#[serde(rename = "s3Credentials")]
pub s3_credentials: Option<S3Credentials>,

/// The Google Cloud credentials to use for backups
#[serde(rename = "googleCredentials")]
pub google_credentials: Option<GoogleCredentials>,

/// The Azure credentials to use for backups
#[serde(rename = "azureCredentials")]
pub azure_credentials: Option<AzureCredentials>,

/// Enable using Volume Snapshots for backups instead of Object Storage
#[serde(
default = "defaults::default_volume_snapshot",
Expand Down Expand Up @@ -315,18 +398,22 @@ pub struct Restore {
#[serde(rename = "recoveryTargetTime")]
pub recovery_target_time: Option<String>,

/// endpointURL is the S3 compatable endpoint URL
/// endpointURL is the S3 compatible endpoint URL
#[serde(default, rename = "endpointURL")]
pub endpoint_url: Option<String>,

/// s3Credentials is the S3 credentials to use for backups.
/// s3Credentials is the S3 credentials to use for restores.
#[serde(rename = "s3Credentials")]
pub s3_credentials: Option<S3Credentials>,

/// s3Credentials is the S3 credentials to use for backups.
/// googleCredentials is the Google Cloud credentials to use for restores.
#[serde(rename = "googleCredentials")]
pub google_credentials: Option<GoogleCredentials>,

/// azureCredentials is the Azure credentials to use for restores.
#[serde(rename = "azureCredentials")]
pub azure_credentials: Option<AzureCredentials>,

/// volumeSnapshot is a boolean to enable restoring from a Volume Snapshot
#[serde(rename = "volumeSnapshot")]
pub volume_snapshot: Option<bool>,
Expand Down
Loading
Loading