Skip to content

Commit

Permalink
[DOCFIX] Update "Configuration Settings" page
Browse files Browse the repository at this point in the history
### What changes are proposed in this pull request?

move information to better fitting pages
update links that were plain text in config properties

### Why are the changes needed?

remove redundant/irrelevant information
links that are plain text are not user friendly

### Does this PR introduce any user facing changes?

Web UI

			pr-link: #17835
			change-id: cid-2722135b7789ab72b7f992e02a338dd11fd4a297
  • Loading branch information
ccmao1130 authored Jul 26, 2023
1 parent 50b67ed commit 915f19a
Show file tree
Hide file tree
Showing 6 changed files with 86 additions and 121 deletions.
2 changes: 2 additions & 0 deletions docs/_data/menu-en.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,5 +166,7 @@
subfiles:
- title: List of Configuration Properties
url: /en/reference/Properties-List.html
- title: List of Environment Variables
url: /en/reference/Environment-List.html
- title: List of Metrics
url: /en/reference/Metrics-List.html
18 changes: 12 additions & 6 deletions docs/en/deploy/Install-Alluxio-Cluster-with-HA.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,18 +41,24 @@ state across service restarts and maintaining consensus among masters about the
on all nodes.

## Basic Setup
### Raft-based Embedded Journal

The minimal configuration to set up a HA cluster is to give the embedded journal addresses to all
nodes inside the cluster.
On each Alluxio node, create the `conf/alluxio-site.properties` configuration file from the
template.
Alluxio admins can create and edit the properties file `conf/alluxio-site.properties` to
configure Alluxio masters or workers.
If this file does not exist, it can be copied from the template file under `${ALLUXIO_HOME}/conf`:

```shell
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
```

Add the following properties to the `conf/alluxio-site.properties` file:
Make sure that this file is distributed to `${ALLUXIO_HOME}/conf` on every Alluxio master
and worker before starting the cluster.
Restarting Alluxio processes is the safest way to ensure any configuration updates are applied.

### Raft-based Embedded Journal

The minimal configuration to set up a HA cluster is to give the embedded journal addresses to all
nodes inside the cluster.
On each Alluxio node, copy the `conf/alluxio-site.properties` configuration file and add the following properties to the file:

```properties
alluxio.master.hostname=<MASTER_HOSTNAME> # Only needed on master node
Expand Down
112 changes: 2 additions & 110 deletions docs/en/operation/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,116 +8,8 @@ An Alluxio cluster can be configured by setting the values of Alluxio
[configuration properties]({{ '/en/reference/Properties-List.html' | relativize_url }}) within
`${ALLUXIO_HOME}/conf/alluxio-site.properties`.

The two major components to configure are:
- [Alluxio servers](#configure-an-alluxio-cluster), consisting of Alluxio processes such as masters and workers
- [Alluxio clients](#configure-applications), which are typically a part of compute applications

Alluxio properties mostly fall into three categories:

- properties prefixed with `alluxio.user` affect Alluxio client operations (e.g. compute applications)
- properties prefixed with `alluxio.master` affect the Alluxio master processes
- properties prefixed with `alluxio.worker` affect the Alluxio worker processes

## Configure Applications

Customizing how an application interacts with Alluxio is specific to each application.
The following are recommendations for some common applications.

### Alluxio Shell Commands

Alluxio shell users can put JVM system properties `-Dproperty=value` after the `fs` command and
before the subcommand to specify Alluxio user properties from the command line.
For example, the following Alluxio shell command sets the write type to `CACHE_THROUGH` when copying
files to Alluxio:

```shell
$ ./bin/alluxio fs -Dalluxio.user.file.writetype.default=CACHE_THROUGH \
copyFromLocal README.md /README.md
```

Note that, as a part of Alluxio deployment, the Alluxio shell will also take the configuration in
`${ALLUXIO_HOME}/conf/alluxio-site.properties` when it is run from Alluxio installation at
`${ALLUXIO_HOME}`.

### Spark

To customize Alluxio client-side properties in Spark applications,
Spark users can use pass Alluxio properties as JVM system properties.
See examples for
[configuring the Spark service]({{ '/en/compute/Spark.html' | relativize_url }}#basic-setup)
or for
[individual Spark jobs]({{ '/en/compute/Spark.html' | relativize_url }}#customize-alluxio-user-properties-for-individual-spark-jobs).

### Hadoop MapReduce

See examples to configure Alluxio properties for
[the MapReduce service]({{ '/en/compute/Hadoop-MapReduce.html' | relativize_url }}#customize-alluxio-user-properties-for-all-mapreduce-jobs)
or for
[individual MapReduce jobs]({{ '/en/compute/Hadoop-MapReduce.html' | relativize_url }}#customize-alluxio-user-properties-for-individual-mapreduce-jobs).

### Hive

Hive can be configured to use customized Alluxio client-side properties for the entire service.
See
[examples]({{ '/en/compute/Hive.html' | relativize_url }}#customize-alluxio-user-properties).

### Presto

Presto can be configured to use customized Alluxio client-side properties for the entire service.
See
[examples]({{ '/en/compute/Presto.html' | relativize_url }}#customize-alluxio-user-properties).

## Configure an Alluxio Cluster

### `alluxio-site.properties` Files (Recommended)

Alluxio admins can create and edit the properties file `conf/alluxio-site.properties` to
configure Alluxio masters or workers.
If this file does not exist, it can be copied from the template file under `${ALLUXIO_HOME}/conf`:

```shell
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
```

Make sure that this file is distributed to `${ALLUXIO_HOME}/conf` on every Alluxio master
and worker before starting the cluster.
Restarting Alluxio processes is the safest way to ensure any configuration updates are applied.

### Environment Variables

Alluxio supports defining a few frequently used configuration settings through environment
variables, including:

<table class="table table-striped">
<tr><th>Environment Variable</th><th>Description</th></tr>
{% for env_var in site.data.table.en.env_vars %}
<tr>
<td markdown="span">`{{env_var.name}}`</td>
<td markdown="span">{{env_var.description}}</td>
</tr>
{% endfor %}
</table>

For example, the following example will set up:
- an Alluxio master at `localhost`
- the root mount point as an HDFS cluster with a namenode also running at `localhost`
- defines the maximum heap space of the VM to be 30g
- enable Java remote debugging at port 7001

```shell
$ export ALLUXIO_MASTER_HOSTNAME="localhost"
$ export ALLUXIO_MASTER_MOUNT_TABLE_ROOT_UFS="hdfs://localhost:9000"
$ export ALLUXIO_MASTER_JAVA_OPTS="-Xmx30g"
$ export ALLUXIO_MASTER_ATTACH_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7001"
```

Users can either set these variables through the shell or in `conf/alluxio-env.sh`.
If this file does not exist yet, it can be copied from the template file under `${ALLUXIO_HOME}/conf`:

```shell
$ cp conf/alluxio-env.sh.template conf/alluxio-env.sh
```

### Cluster Defaults

When different client applications (Alluxio Shell CLI, Spark jobs, MapReduce jobs)
Expand Down Expand Up @@ -177,8 +69,8 @@ Alluxio properties can be configured from multiple sources.
A property's final value is determined by the following priority list, from highest priority to lowest:
1. [JVM system properties (i.e., `-Dproperty=key`)](http://docs.oracle.com/javase/jndi/tutorial/beyond/env/source.html#SYS)
2. [Environment variables](#environment-variables)
3. [Property files](#alluxio-siteproperties-files-recommended):
2. [Environment variables]({{ '/en/reference/Environment-List.html' | relativize_url}})
3. [Property files]({{ '/en/reference/Properties-List.html' | relativize_url }}):
When an Alluxio cluster starts, each server process including master and worker searches for
`alluxio-site.properties` within the following directories in the given order, stopping when a match is found:
`${CLASSPATH}`, `${HOME}/.alluxio/`, `/etc/alluxio/`, and `${ALLUXIO_HOME}/conf`
Expand Down
14 changes: 14 additions & 0 deletions docs/en/operation/User-CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,20 @@ Usage: alluxio [COMMAND]
...
```
Alluxio shell users can put JVM system properties `-Dproperty=value` after the `fs` command and
before the subcommand to specify Alluxio user properties from the command line.
For example, the following Alluxio shell command sets the write type to `CACHE_THROUGH` when copying
files to Alluxio:
```shell
$ ./bin/alluxio fs -Dalluxio.user.file.writetype.default=CACHE_THROUGH \
copyFromLocal README.md /README.md
```
Note that, as a part of Alluxio deployment, the Alluxio shell will also take the configuration in
`${ALLUXIO_HOME}/conf/alluxio-site.properties` when it is run from Alluxio installation at
`${ALLUXIO_HOME}`.
## General operations
This section lists usages and examples of general Alluxio operations
Expand Down
37 changes: 37 additions & 0 deletions docs/en/reference/Environment-List.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---
layout: global
title: List of Environment Variables
---

Alluxio supports defining a few frequently used configuration settings through environment
variables.

<table class="table table-striped">
<tr><th>Environment Variable</th><th>Description</th></tr>
{% for env_var in site.data.table.en.env_vars %}
<tr>
<td markdown="span">`{{env_var.name}}`</td>
<td markdown="span">{{env_var.description}}</td>
</tr>
{% endfor %}
</table>

For example, the following example will set up:
- an Alluxio master at `localhost`
- the root mount point as an HDFS cluster with a namenode also running at `localhost`
- defines the maximum heap space of the VM to be 30g
- enable Java remote debugging at port 7001

```shell
$ export ALLUXIO_MASTER_HOSTNAME="localhost"
$ export ALLUXIO_MASTER_MOUNT_TABLE_ROOT_UFS="hdfs://localhost:9000"
$ export ALLUXIO_MASTER_JAVA_OPTS="-Xmx30g"
$ export ALLUXIO_MASTER_ATTACH_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7001"
```

Users can either set these variables through the shell or in `conf/alluxio-env.sh`.
If this file does not exist yet, it can be copied from the template file under `${ALLUXIO_HOME}/conf`:

```shell
$ cp conf/alluxio-env.sh.template conf/alluxio-env.sh
```
24 changes: 19 additions & 5 deletions docs/en/reference/Properties-List.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,26 @@ layout: global
title: List of Configuration Properties
---

An Alluxio cluster can be configured by setting the values of Alluxio
configuration properties within
`${ALLUXIO_HOME}/conf/alluxio-site.properties`. If this file does not exist, it can be copied from the template file under `${ALLUXIO_HOME}/conf`:

```shell
$ cp conf/alluxio-site.properties.template conf/alluxio-site.properties
```

Make sure that this file is distributed to `${ALLUXIO_HOME}/conf` on every Alluxio master
and worker before starting the cluster.
Restarting Alluxio processes is the safest way to ensure any configuration updates are applied.

All Alluxio configuration settings fall into one of the five categories:
[Common](#common-configuration) (shared by Master and Worker),
[Master specific](#master-configuration), [Worker specific](#worker-configuration),
[User specific](#user-configuration), and
[Security specific](#security-configuration) (shared by Master, Worker, and User).
- properties prefixed with `alluxio.master` affect the Alluxio master processes
- properties prefixed with `alluxio.worker` affect the Alluxio worker processes
- properties prefixed with `alluxio.user` affect Alluxio client operations (e.g. compute applications)

## Common Configuration

Expand All @@ -18,7 +32,7 @@ The common configuration contains constants shared by different components.
<tr><th>Property Name</th><th>Default</th><th>Description</th></tr>
{% for item in site.data.generated.common-configuration %}
<tr>
<td><a class="anchor" name="{{ item.propertyName }}"></a> {{ item.propertyName }}</td>
<td markdown="span"><a class="anchor" name="{{ item.propertyName }}"></a> `{{ item.propertyName }}`</td>
<td>{{ item.defaultValue }}</td>
<td>{{ site.data.generated.en.common-configuration[item.propertyName] }}</td>
</tr>
Expand All @@ -34,7 +48,7 @@ the port number.
<tr><th>Property Name</th><th>Default</th><th>Description</th></tr>
{% for item in site.data.generated.master-configuration %}
<tr>
<td><a class="anchor" name="{{ item.propertyName }}"></a> {{ item.propertyName }}</td>
<td markdown="span"><a class="anchor" name="{{ item.propertyName }}"></a> `{{ item.propertyName }}`</td>
<td>{{ item.defaultValue }}</td>
<td>{{ site.data.generated.en.master-configuration[item.propertyName] }}</td>
</tr>
Expand All @@ -50,7 +64,7 @@ the port number.
<tr><th>Property Name</th><th>Default</th><th>Description</th></tr>
{% for item in site.data.generated.worker-configuration %}
<tr>
<td><a class="anchor" name="{{ item.propertyName }}"></a> {{ item.propertyName }}</td>
<td markdown="span"><a class="anchor" name="{{ item.propertyName }}"></a> `{{ item.propertyName }}`</td>
<td>{{ item.defaultValue }}</td>
<td>{{ site.data.generated.en.worker-configuration[item.propertyName] }}</td>
</tr>
Expand All @@ -65,7 +79,7 @@ The user configuration specifies values regarding file system access.
<tr><th>Property Name</th><th>Default</th><th>Description</th></tr>
{% for item in site.data.generated.user-configuration %}
<tr>
<td><a class="anchor" name="{{ item.propertyName }}"></a> {{ item.propertyName }}</td>
<td markdown="span"><a class="anchor" name="{{ item.propertyName }}"></a> `{{ item.propertyName }}`</td>
<td>{{ item.defaultValue }}</td>
<td>{{ site.data.generated.en.user-configuration[item.propertyName] }}</td>
</tr>
Expand All @@ -81,7 +95,7 @@ user. Settings for file permission only take effect for master.
<tr><th>Property Name</th><th>Default</th><th>Description</th></tr>
{% for item in site.data.generated.security-configuration %}
<tr>
<td><a class="anchor" name="{{ item.propertyName }}"></a> {{ item.propertyName }}</td>
<td markdown="span"><a class="anchor" name="{{ item.propertyName }}"></a> `{{ item.propertyName }}`</td>
<td>{{ item.defaultValue }}</td>
<td>{{ site.data.generated.en.security-configuration[item.propertyName] }}</td>
</tr>
Expand Down

0 comments on commit 915f19a

Please sign in to comment.