Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Administration/Clustering: Reference articles about scaling clusters #46

Merged
merged 2 commits into from
Feb 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/admin/clustering/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,6 @@ adding more nodes. This section of the documentation covers that topic.

multi-node-setup
multi-zone-setup
scale-up-down
kubernetes
logical-replication-setup
62 changes: 62 additions & 0 deletions docs/admin/clustering/scale-up-down.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
(scaling-clusters)=
# Scaling Clusters Up and Down

A significant feature in CrateDB is that it can scale horizontally, which means
that instead of adding more RAM, CPU, and disk resources to existing nodes, you
will add more individual nodes to your CrateDB cluster.

By running your database cluster on multiple nodes, you will gain two benefits.

- Store data volumes larger than being able to be handled on a single machine.
amotl marked this conversation as resolved.
Show resolved Hide resolved
- Add resiliency to your distributed database cluster, by increasing the number
of replica nodes.


(scaling-expand)=
## Expand Cluster

The article about [how to add new nodes to an existing cluster] walks you
through the process of scaling up your database cluster, and educates you
about the corresponding details to consider.
Comment on lines +15 to +20
Copy link
Member Author

@amotl amotl Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't been able to discover a meaningful/representative code snippet/example on https://community.cratedb.com/t/how-to-add-new-nodes-to-an-existing-cluster/1546, so we just omitted it in this case.

The whole section is a bit thin now, and could be expanded if you have any ideas/suggestions. Otherwise, let's just merge to make progress.



(scaling-ondemand)=
## On-Demand Scaling

The article about [scaling CrateDB clusters up and down to cope with peaks in
demand] shares knowledge about the [shard allocation filtering] feature of
CrateDB.

Along the lines, it demonstrates how this functionality is applied in a real-
world data management scenario, which is about tuning your database cluster to
cope with high-demand situations.

Prepare adding extra nodes to the database cluster.
```sql
/* Apply routing setting to all existing partitions and new partitions. */
ALTER TABLE test SET ("routing.allocation.exclude.storage" = 'temporarynodes');

/* Configure the setting to be excluded / not applied to _new_ partitions. */
ALTER TABLE ONLY test RESET ("routing.allocation.exclude.storage");
```

Before the high-demand event, properly configure table routing accordingly.
```sql
ALTER TABLE ONLY test SET ("routing.allocation.total_shards_per_node" = 2);
```

To decommission extra database nodes, we need to move the data collected during
the days of the event.
```sql
-- Move the collected data off the extra nodes.
ALTER TABLE test SET ("routing.allocation.exclude.storage" = 'temporarynodes');
ALTER TABLE test RESET ("routing.allocation.total_shards_per_node");

-- Invoke the decommissioning.
ALTER CLUSTER DECOMMISSION 'nodename';
```
Comment on lines +23 to +57
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't hesitate to submit suggestions to improve the wording on both guiding comments, and inline comments. Thanks!



[how to add new nodes to an existing cluster]: https://community.cratedb.com/t/how-to-add-new-nodes-to-an-existing-cluster/1546
[scaling CrateDB clusters up and down to cope with peaks in demand]: https://community.cratedb.com/t/scaling-cratedb-clusters-up-and-down-to-cope-with-peaks-in-demand/1314
[shard allocation filtering]: inv:crate-reference#ddl_shard_allocation