Steps to configure Elasticsearch snapshots

[derekenos]

I'm writing down the steps that I take to configure ES snapshots for later inclusion in the disk image and documentation.

1. Install the S3 repository plugin (es docs)

./usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3

That's ^ supposed to work but it just hangs after:

-> Installing repository-s3
-> Downloading repository-s3 from elastic
[=================================================] 100%   

Manually installing repository-s3-7.7.0.zip like this worked:

./usr/share/elasticsearch/bin/elasticsearch-plugin install file:///tmp/repository-s3-7.7.0.zip

2. Configure S3 plugin endpoint
   Add this line to /etc/elasticsearch/elasticsearch.yml:

s3.client.default.endpoint: sfo2.digitaloceanspaces.com

^ replacing sfo2.digitaloceanspaces.com with whatever's appropriate for your DO Space.

3. Create DO Space credentials for the ES snapshots

4. Add the DO Space credentials to the ES keystore:
   Set the access_key:

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
{ enter the access key + hit ENTER}

Set the secret_key:

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key
{ enter the secret key + hit ENTER }

Note that the default in these key names corresponds to the name of the Elasticsearch S3 repository that we have yet to create.

5. Restart ES:

systemctl restart elasticsearch

[derekenos]

Bundled up the server-side stuff in a bash script:

#!/bin/bash

CONFIG_PATH=/etc/elasticsearch/elasticsearch.yml

# Define a helper function that will prompt for confirmation to continue.
function prompt_for_confirmation () {
  msg=$1
  echo -n "$msg (Y/n)? "
  read response
  # Use wierd ${<var-name>,,} syntax to downcase the response.
  response="${response,,}"
  if [[ "$response" != "y" && "$response" != "" ]]
    then
      echo "Aborting"
      exit 1
  fi
}

# Check whether elasticsearch.yml already defines an s3.client endpoint.
res=`grep ^s3\.client $CONFIG_PATH`

# Abort if a value is already defined.
if [ "$?" -eq 0 ]
  then
    echo "S3 client already configured in $CONFIG_PATH: \"$res\""
    exit 1
fi

# Install the repository-s3 plugin if necessary.
echo -n "Checking whether S3 snapshot repository plugin is installed - "
res=`/usr/share/elasticsearch/bin/elasticsearch-plugin list | grep repository-s3`
if [ "$?" -eq 0 ]
then
  echo "installed"
else
  echo "not installed"
  echo "Installing S3 snapshot repository plugin..."
  ES_VERSION=`/usr/share/elasticsearch/bin/elasticsearch --version | grep -o -P '^Version:\s[^,]+' | cut -d' ' -f2`
  PLUGIN_FILENAME="repository-s3-$ES_VERSION.zip"
  curl --silent https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/$PLUGIN_FILENAME -O
  /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch --silent file://`pwd`/$PLUGIN_FILENAME
  rm $PLUGIN_FILENAME
fi


SETTING_KEY="s3.client.default.endpoint"

# Prompt user for endpoint.
echo -n "Enter your S3-compatible endpoint value: "
read endpoint
prompt_for_confirmation "Set \"$SETTING_KEY\" to \"$endpoint\""

# Append the setting to elasticsearch.yml
echo "$SETTING_KEY: $endpoint" >> $CONFIG_PATH

# Add the access key to the keystore.
echo "Preparing to configure S3 credentials..."
/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key

# Restart Elasticsearch
prompt_for_confirmation "Restart Elasticsearch now"
systemctl restart elasticsearch