Thanos supports any object stores that can be implemented against Thanos objstore.Bucket interface
All clients are configured using --objstore.config-file
to reference to the configuration file or --objstore.config
to put yaml config directly.
Current object storage client implementations:
Provider | Maturity | Auto-tested on CI | Maintainers |
---|---|---|---|
Google Cloud Storage | Stable (production usage) | yes | @bwplotka |
AWS S3 | Stable (production usage) | yes | @bwplotka |
Azure Storage Account | Stable (production usage) | yes | @vglafirov |
OpenStack Swift | Beta (working PoCs, testing usage) | no | @sudhi-vm |
Tencent COS | Beta (testing usage) | no | @jojohappy |
NOTE: Currently Thanos requires strong consistency (write-read) for object store implementation.
- Create new directory under
pkg/objstore/<provider>
- Implement objstore.Bucket interface
- Add
NewTestBucket
constructor for testing purposes, that creates and deletes temporary bucket. - Use created
NewTestBucket
in ForeachStore method to ensure we can run tests against new provider. (In PR) - RUN the TestObjStoreAcceptanceTest against your provider to ensure it fits. Fix any found error until test passes. (In PR)
- Add client implementation to the factory in factory code. (Using as small amount of flags as possible in every command)
- Add client struct config to bucketcfggen to allow config auto generation.
At that point, anyone can use your provider by spec
Thanos uses minio client to upload Prometheus data into AWS S3.
To configure S3 bucket as an object store you need to set these mandatory S3 variables in yaml format stored in a file:
type: S3
config:
bucket: ""
endpoint: ""
access_key: ""
insecure: false
signature_version2: false
encrypt_sse: false
secret_key: ""
put_user_metadata: {}
http_config:
idle_conn_timeout: 0s
insecure_skip_verify: false
trace:
enable: false
AWS region to endpoint mapping can be found in this link
Make sure you use a correct signature version.
Currently AWS require signature v4, so it needs signature-version2: false
, otherwise, you will get Access Denied error, but several other S3 compatible use signature-version2: true
For debug and testing purposes you can set
insecure: true
to switch to plain insecure HTTP instead of HTTPShttp_config.insecure_skip_verify: true
to disable TLS certificate verification (if your S3 based storage is using a self-signed certificate, for example)trace.enable: true
to enable the minio client's verbose logging. Each request and response will be logged into the debug logger, so debug level logging must be enabled for this functionality.
By default Thanos will try to retrieve credentials from the following sources:
- From config file if BOTH
access_key
andsecret_key
are present. - From the standard AWS environment variable -
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
- From
~/.aws/credentials
- IAM credentials retrieved from an instance profile.
NOTE: Getting access key from config file and secret key from other method (and vice versa) is not supported.
Example working AWS IAM policy for user:
- For deployment (policy for Thanos services):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::<bucket>/*",
"arn:aws:s3:::<bucket>"
]
}
]
}
(No bucket policy)
To test the policy, set env vars for S3 access for empty, not used bucket as well as: THANOS_SKIP_GCS_TESTS=true THANOS_ALLOW_EXISTING_BUCKET_USE=true
And run: GOCACHE=off go test -v -run TestObjStore_AcceptanceTest_e2e ./pkg/...
- For testing (policy to run e2e tests):
We need access to CreateBucket and DeleteBucket and access to all buckets:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:CreateBucket",
"s3:DeleteBucket"
],
"Resource": [
"arn:aws:s3:::<bucket>/*",
"arn:aws:s3:::<bucket>"
]
}
]
}
With this policy you should be able to run set THANOS_SKIP_GCS_TESTS=true
and unset S3_BUCKET
and run all tests using make test
.
Details about AWS policies: https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
To configure Google Cloud Storage bucket as an object store you need to set bucket
with GCS bucket name and configure Google Application credentials.
For example:
type: GCS
config:
bucket: ""
service_account: ""
Application credentials are configured via JSON file and only the bucket needs to be specified, the client looks for:
- A JSON file whose path is specified by the
GOOGLE_APPLICATION_CREDENTIALS
environment variable. - A JSON file in a location known to the gcloud command-line tool.
On Windows, this is
%APPDATA%/gcloud/application_default_credentials.json
. On other systems,$HOME/.config/gcloud/application_default_credentials.json
. - On Google App Engine it uses the
appengine.AccessToken
function. - On Google Compute Engine and Google App Engine Managed VMs, it fetches credentials from the metadata server. (In this final case any provided scopes are ignored.)
You can read more on how to get application credential json file in https://cloud.google.com/docs/authentication/production
Another possibility is to inline the ServiceAccount into the Thanos configuration and only maintain one file. This feature was added, so that the Prometheus Operator only needs to take care of one secret file.
type: GCS
config:
bucket: "thanos"
service_account: |-
{
"type": "service_account",
"project_id": "project",
"private_key_id": "abcdefghijklmnopqrstuvwxyz12345678906666",
"private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n",
"client_email": "[email protected]",
"client_id": "123456789012345678901",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/thanos%40gitpods.iam.gserviceaccount.com"
}
For deployment:
Storage Object Creator
and Storage Object Viewer
For testing:
Storage Object Admin
for ability to create and delete temporary buckets.
To use Azure Storage as Thanos object store, you need to precreate storage account from Azure portal or using Azure CLI. Follow the instructions from Azure Storage Documentation: https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account
To configure Azure Storage account as an object store you need to provide a path to Azure storage config file in flag --objstore.config-file
.
Config file format is the following:
type: AZURE
config:
storage_account: ""
storage_account_key: ""
container: ""
Thanos uses gophercloud client to upload Prometheus data into OpenStack Swift.
Below is an example configuration file for thanos to use OpenStack swift container as an object store.
type: SWIFT
config:
auth_url: ""
username: ""
user_id: ""
password: ""
domain_id: ""
domain_name: ""
tenant_id: ""
tenant_name: ""
region_name: ""
container_name: ""
Minio client used for AWS S3 can be potentially configured against other S3-compatible object storages.
To use Tencent COS as storage store, you should apply a Tencent Account to create an object storage bucket at first. Note that detailed from Tencent Cloud Documents: https://cloud.tencent.com/document/product/436
To configure Tencent Account to use COS as storage store you need to set these parameters in yaml format stored in a file:
type: COS
config:
bucket: ""
region: ""
app_id: ""
secret_key: ""
secret_id: ""
Set the flags --objstore.config-file
to reference to the configuration file.