Skip to content

Commit

Permalink
docs: Update storage docs (v2.9) (#12767)
Browse files Browse the repository at this point in the history
  • Loading branch information
JStickler authored Apr 24, 2024
1 parent d1d7d90 commit 14f126a
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 16 deletions.
5 changes: 2 additions & 3 deletions docs/sources/operations/storage/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ For more information:

The following are supported for the index:

- [Single Store (boltdb-shipper) - Recommended for 2.0 and newer]({{< relref "./boltdb-shipper" >}}) index store which stores boltdb index files in the object store
- [Single Store (TSDB)](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/tsdb/) index store which stores TSDB index files in the object store. Recommended store for Loki 2.8 and newer.
- [Single Store (boltdb-shipper)](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/boltdb-shipper/) index store which stores boltdb index files in the object store. Recommended store for Loki 2.0 through 2.7.x.
- [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
- [Google Bigtable](https://cloud.google.com/bigtable)
- [Apache Cassandra](https://cassandra.apache.org)
Expand Down Expand Up @@ -109,7 +110,6 @@ Resources: `*`

Resources: `arn:aws:iam::<aws_account_id>:role/<role_name>`


### IBM Cloud Object Storage

When using IBM Cloud Object Storage (COS) as object storage, IAM `Writer` role is needed.
Expand Down Expand Up @@ -145,4 +145,3 @@ See the [IBM Cloud Object Storage section]({{< relref "../../storage#ibm-cloud-o
| metasOffset - offset to the point with #blocks |
-------------------------------------------------------------------
```

15 changes: 10 additions & 5 deletions docs/sources/operations/storage/boltdb-shipper.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,17 @@ description: Single Store (boltdb-shipper)
---
# Single Store (boltdb-shipper)

BoltDB is the recommended store for Loki 2.0 through 2.7.x. Starting with Loki v2.8, [TSDB](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/storage/tsdb/) is the Loki recommended store.

BoltDB Shipper lets you run Grafana Loki without any dependency on NoSQL stores for storing index.
It locally stores the index in BoltDB files instead and keeps shipping those files to a shared object store i.e the same object store which is being used for storing chunks.
It also keeps syncing BoltDB files from shared object store to a configured local directory for getting index entries created by other services of same Loki cluster.
This helps run Loki with one less dependency and also saves costs in storage since object stores are likely to be much cheaper compared to cost of a hosted NoSQL store or running a self hosted instance of Cassandra.

**Note:** BoltDB shipper works best with 24h periodic index files. It is a requirement to have index period set to 24h for either active or upcoming usage of boltdb-shipper.
If boltdb-shipper already has created index files with 7 days period, and you want to retain previous data then just add a new schema config using boltdb-shipper with a future date and index files period set to 24h.
{{< admonition type="note" >}}
BoltDB shipper works best with 24h periodic index files. It is a requirement to have index period set to 24h for either active or upcoming usage of boltdb-shipper.
If boltdb-shipper already has created index files with 7 days period, and you want to retain previous data then just add a new schema config using boltdb-shipper with a future date and index files period set to 24h.
{{< /admonition >}}

## Example Configuration

Expand Down Expand Up @@ -68,7 +72,10 @@ they both having shipped files for day `18371` and `18372` with prefix `loki_ind
└── ingester-1-1587254400.gz
...
```
**Note:** We also add a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification.

{{< admonition type="note" >}}
We also add a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification.
{{< /admonition >}}

Let us talk about more in depth about how both Ingesters and Queriers work when running them with BoltDB Shipper.

Expand Down Expand Up @@ -142,5 +149,3 @@ storage_config:
gcs:
bucket_name: GCS_BUCKET_NAME
```


15 changes: 7 additions & 8 deletions docs/sources/storage/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ Single Store refers to using object storage as the storage medium for both Loki'

### TSDB (recommended)

Starting in Loki 2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper".
Starting in Loki 2.8, the [TSDB index store]({{< relref "../operations/storage/tsdb" >}}) improves query performance, reduces TCO and has the same feature parity as "boltdb-shipper". TSDB is the recommended index store for Loki 2.8 and newer.

### BoltDB (deprecated)

Also known as "boltdb-shipper" during development (and is still the schema `store` name). The single store configurations for Loki utilize the chunk store for both chunks and the index, requiring just one store to run Loki.
Also known as "boltdb-shipper" during development (and is still the schema `store` name). The single store configurations for Loki utilize the chunk store for both chunks and the index, requiring just one store to run Loki. BoldDB is the recommended index store for Loki v2.0.0 through v2.7x.

Performance is comparable to a dedicated index type while providing a much less expensive and less complicated deployment.
When using Single Store, no extra [Chunk storage](#chunk-storage) and [Index storage](#index-storage) are necessary.
Expand Down Expand Up @@ -158,7 +158,7 @@ For more information, see the [table manager]({{< relref "../operations/storage/

In the case of AWS DynamoDB, you'll likely want to tune the provisioned throughput for your tables as well. This is to prevent your tables being rate limited on one hand and assuming unnecessary cost on the other. By default Loki uses a [provisioned capacity](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html) strategy for DynamoDB tables like so:

```
```yaml
table_manager:
index_tables_provisioning:
# Read/write throughput requirements for the current table
Expand All @@ -180,6 +180,7 @@ When a new schema is released and you want to gain the advantages it provides, y
First, you'll want to create a new [period_config]({{< relref "../configure#period_config" >}}) entry in your [schema_config]({{< relref "../configure#schema_config" >}}). The important thing to remember here is to set this at some point in the _future_ and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and ensures that existing data isn't queried as if it adheres to the new schema.

As an example, let's say it's 2020-07-14 and we want to start using the `v11` schema on the 20th:

```yaml
schema_config:
configs:
Expand Down Expand Up @@ -209,7 +210,6 @@ We're interested in adding targeted deletion in future Loki releases (think tena

For more information, see the [retention configuration]({{< relref "../operations/storage/retention" >}}) documentation.


## Examples

### Single machine/local development (boltdb+filesystem)
Expand Down Expand Up @@ -304,7 +304,7 @@ The role should have a policy with the following permissions attached.
}
```

**To setup an S3 bucket and an IAM role and policy:**
**To setup an S3 bucket and an IAM role and policy:**

This guide assumes a provisioned EKS cluster.

Expand All @@ -314,14 +314,14 @@ This guide assumes a provisioned EKS cluster.

3. Export the AWS profile and region if not done so:

```
```bash
export AWS_PROFILE=<profile in ~/.aws/config>
export AWS_REGION=<region of EKS cluster>
```

4. Save the OIDC provider in an enviroment variable:

```
```bash
oidc_provider=$(aws eks describe-cluster --name <EKS cluster> --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
```

Expand All @@ -332,7 +332,6 @@ This guide assumes a provisioned EKS cluster.
Note, the bucket name defaults to `loki-data` but can be changed via the
`bucket_name` variable.


### Azure deployment (Azure Blob Storage Single Store)

#### Using account name and key
Expand Down

0 comments on commit 14f126a

Please sign in to comment.