Skip to content

Elasticsearch setup guide

Jump to bottom Edit New page
Blake Niemyjski edited this page Jan 22, 2017 · 18 revisions

This guide will walk you through setting up an Elasticsearch production instance on Linux (please note that we assume that you have some Linux experience). We use this guide to setup and configure our own nodes on Azure.

Feedback

We'd love your feedback on how we can improve our setup and configuration. It would be greatly appreciated if a docker guru could create some docker images based on the following tutorial :).

Operating System

Let's start by creating a new virtual machine and select the latest 64bit Ubuntu Operating System. After your up and running lets ensure it's running the latest software:

sudo apt-get update
sudo apt-get upgrade

Installing Elasticsearch

Please note that this guide will install Elasticsearch 1.7.x and not the recent 2.x release.

Install the latest version of Java:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -version

Install Elasticsearch:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

sudo apt-get install elasticsearch

For more information on running Elasticsearch as a service (using SystemD) please read this.

Installing Elasticsearch as a docker container

Elasticsearch can run in a docker container. The official elasticsearch repository is located at docker hub. Note that max version supported is 1.7.x (per Exceptionless requirement), so make sure you don't go with the :latest. Follow the given instructions on docker hub (elasticsearch.yml has to be copied to /usr/share/elasticsearch/config directory in container) or use this docker-compose.yml sample:

version: '2'

services:
    elastic:
        image: elasticsearch:1.7.5
        restart: always
        volumes:
            - [DIRECTORY]/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
            - [DIRECTORY]/data:/usr/share/elasticsearch/data
        ports:
            - 9200:9200
            - 9300:9300

where [DIRECTORY] is directory on the host and should contain the elasticsearch.yml configuration file. Start the container using

sudo docker-compose up -d

When running Elasticsearch in a docker container the steps below have to be modified appropriately.

Add secondary storage

You will want to attach a secondary hard disk/storage to your virtual machine before continuing. We use this disk to store the elastic search indexes. We create the largest one possible in azure as we only pay for space that is actually used. A plus side of doing this is you only have to pay for what is actually allocated on disk.

Get a list of the attached SCSI devices.

dmesg | grep SCSI

Make sure it’s sdc and that we are formatting the correct one.

sudo fdisk /dev/sdc

Command n then p and all defaults then w to write it

sudo mkfs -t ext4 /dev/sdc1

Mount the new drive to /mnt/data

sudo mkdir /mnt/data
sudo mount /dev/sdc1 /mnt/data

Auto mount the drive on reboot.

sudo -i blkid

Grab the GUID for /dev/sdc1 and open fstab.

sudo nano /etc/fstab

Paste in under the existing UUID:

UUID=YOUR_GUID       /mnt/data        ext4   defaults        0 0

Create the storage folders by creating a db, log and work directory in /mnt/data

cd /mnt/data
mkdir db
mkdir log
mkdir work

Make elasticsearch user the owner of the folders

sudo chown -R elasticsearch:elasticsearch /mnt/data/

sudo chown -R elasticsearch:elasticsearch /mnt/data/log
sudo chown -R elasticsearch:elasticsearch /mnt/data/work
sudo chown -R elasticsearch:elasticsearch /mnt/data/db

Installing plugins

Lets install the Cloud Azure, HQ and Marvel plugins

cd /usr/share/elasticsearch
sudo bin/plugin -i elasticsearch/elasticsearch-cloud-azure/2.8.2
sudo bin/plugin -i royrusso/elasticsearch-HQ
sudo bin/plugin -i elasticsearch/marvel/latest

Configuration

It's important that you decide early on roughly how many nodes and how much ram the nodes will have so you can configure it properly. It's recommend that you at least three nodes with two master nodes. Having lots of ram and faster storage will help greatly.

Update the Elasticsearch configuration. We have our configuration file located here:

sudo nano /etc/elasticsearch/elasticsearch.yml

Edit the environment config and set ES_HEAP_SIZE to half of the ram size:

sudo nano /etc/default/elasticseach

Set MAX_LOCKED_MEMORY=unlimited

sudo nano /etc/init.d/elasticsearch

Update system limits

sudo nano /etc/security/limits.conf

With these values

elasticsearch - nofile 65535
elasticsearch - memlock unlimited

Update SystemD configuration settings

sudo nano /usr/lib/systemd/system/elasticsearch.service

With these values

LimitMEMLOCK=infinity

Restart the service to ensure the configuration is picked up

sudo /bin/systemctl restart elasticsearch

Finally, lets verify that mlockall is true and maxfiles is 65535.

curl http://localhost:9200/_nodes/process?pretty

Configure Elasticsearch to run as a service

Ensure Elasticsearch starts after reboot via SystemD:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

Backups

This section assumes that you've configured the Cloud-Azure plugin in the previous configuration step with your Azure blob storage access keys. The cleanup scripts require you to install curator.

Create new snapshot repositories

We'll create a new snapshot repository. You'll need to follow this step as well if you wish to restore production data to a secondary cluster.

PUT _snapshot/ex_stacks
{
  "type": "azure",
    "settings": {
       "base_path": "stacks"
    }
}

PUT _snapshot/ex_organizations
{
  "type": "azure",
    "settings": {
       "base_path": "organizations"
    }
}

PUT _snapshot/ex_events
{
  "type": "azure",
    "settings": {
       "base_path": "events"
    }
}

Create a manual backup

To create a backup and view the status of a snapshot:

GET _snapshot/ex_stacks/_status
PUT /_snapshot/ex_stacks/2020-01-01-12-00
{
 "indices": "stacks*",
 "ignore_unavailable": "true"
}

GET _snapshot/ex_events/_status
PUT /_snapshot/ex_events/2020-01-01-12-00
{
 "indices": "events*",
 "ignore_unavailable": "true"
}

GET _snapshot/ex_organizations/_status
PUT /_snapshot/ex_organizations/2020-01-01-12-00
{
 "indices": "organizations*",
 "ignore_unavailable": "true"
}

Create the backup and cleanup scripts

We recommend creating these files on one of your elastic nodes:

Let's navigate to the data directory:

cd /mnt/data

Create the events snapshot script

touch events_snapshot
chmod +x events_snapshot
nano stacks_snapshot

With the content:

#!/bin/bash

DATE=`date +%Y-%m-%d-%H-%M`
curl -XPUT "localhost:9200/_snapshot/ex_events/$DATE?wait_for_completion=true" -d '{
    "indices": "events*",
    "ignore_unavailable": "true"
}'

Create the stacks snapshot script

touch stacks_snapshot
chmod +x stacks_snapshot
nano stacks_snapshot

With the content:

#!/bin/bash

DATE=`date +%Y-%m-%d-%H-%M`
curl -XPUT "localhost:9200/_snapshot/ex_stacks/$DATE?wait_for_completion=true" -d '{
    "indices": "stacks*",
    "ignore_unavailable": "true"
}'

Create the organizations snapshot script

touch organizations_snapshot
chmod +x organizations_snapshot
nano organizations_snapshot

With the content:

#!/bin/bash

DATE=`date +%Y-%m-%d-%H-%M`
curl -XPUT "localhost:9200/_snapshot/ex_organizations/$DATE?wait_for_completion=true" -d '{
    "indices": "organizations*",
    "ignore_unavailable": "true"
}'

Create the snapshot cleanup script

touch cleanup_snapshots
chmod +x cleanup_snapshots
nano cleanup_snapshots

With the content:

#!/bin/bash

/usr/local/bin/curator --timeout 600 delete snapshots --older-than 7 --time-unit days --timestring %Y-%m-%d --repository ex_events
/usr/local/bin/curator --timeout 600 delete snapshots --older-than 7 --time-unit days --timestring %Y-%m-%d --repository ex_stacks
/usr/local/bin/curator --timeout 600 delete snapshots --older-than 7 --time-unit days --timestring %Y-%m-%d --repository ex_organizations

Create the index cleanup script

touch cleanup_indexes
chmod +x cleanup_indexes
nano cleanup_indexes

With the content:

#!/bin/bash

curator --host localhost delete indices --older-than 3 --time-unit months --timestring %Y%m

Run the scripts on a Cron job

Edit the Cron job

crontab -e # choose option 2

Add the following Cron jobs

0 */12 * * * /mnt/data/events_snapshot >/dev/null 2>&1
40 * * * *  /mnt/data/stacks_snapshot >/dev/null 2>&1
30 * * * * /mnt/data/organizations_snapshot >/dev/null 2>&1
5 */1 * * * /mnt/data/cleanup_snapshots >/dev/null 2>&1
10 */2  * * * /mnt/data/cleanup_indexes >/dev/null 2>&1

You can verify that your cronjob has ran by running: tail -n 20 /var/log/syslog

Restoring from a backup

You'll first want to setup the snapshot repositories as well as install and configure the Cloud-Azure plugin before restoring to a new cluster.

List of all snapshots:

GET _snapshot/ex_stacks/_all
GET _snapshot/ex_events/_all
GET _snapshot/ex_organizations/_all

To do a restore of all indices run the following command (please take a look at the Elasticsearch documentation on how to restore a single index):

POST _snapshot/ex_organizations/2015-12-01-12-30/_restore
{
  "include_global_state": false
}

Removing a node from a cluster:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.exclude._ip": "<IP ADDRESS OF A NODE>"
  }
}

Updating a node:

Add the latest repository and update packages:

add-apt-repository "deb http://packages.elasticsearch.org/elasticsearch/1.7/debian stable main"

apt-get update
apt-get upgrade

Updating plugins.

cd /usr/share/elasticsearch

bin/plugin -r cloud-azure && bin/plugin -i elasticsearch/elasticsearch-cloud-azure/2.8.2
bin/plugin -r HQ && bin/plugin -i royrusso/elasticsearch-HQ
bin/plugin -r marvel && bin/plugin -i elasticsearch/marvel/latest

Restart the service

sudo /bin/systemctl restart elasticsearch

Logging config (Optional)

Elasticsearch uses Log4j for logging. By default it adds a timestamp to old files, but doesn't delete them. If instead you want Elasticsearch to clean them up, you can change the logging.yml inside the config folder of you Elasticsearch installation. Below is example that will allow up to 10 logging files with 10MB max file size each. maxFileSize determines the max file of each file, maxBackupIndex determines how many logging files we can have.

# you can override this using by setting a system property, for example -Des.logger.level=DEBUG
es.logger.level: INFO
rootLogger: ${es.logger.level}, console, file
logger:
  # log action execution errors for easier debugging
  action: DEBUG
  # reduce the logging for aws, too much is logged under the default INFO
  com.amazonaws: WARN
  org.apache.http: INFO

  # gateway
  #gateway: DEBUG
  #index.gateway: DEBUG

  # peer shard recovery
  #indices.recovery: DEBUG

  # discovery
  #discovery: TRACE

  index.search.slowlog: TRACE, index_search_slow_log_file
  index.indexing.slowlog: TRACE, index_indexing_slow_log_file

additivity:
  index.search.slowlog: false
  index.indexing.slowlog: false

appender:
  console:
    type: console
    layout:
      type: consolePattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  file:
    type: rollingFile
    file: ${path.logs}/${cluster.name}.log
    maxFileSize: 10MB
    maxBackupIndex: 10
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  # Use the following log4j-extras RollingFileAppender to enable gzip compression of log files. 
  # For more information see https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html
  #file:
    #type: extrasRollingFile
    #file: ${path.logs}/${cluster.name}.log
    #rollingPolicy: timeBased
    #rollingPolicy.FileNamePattern: ${path.logs}/${cluster.name}.log.%d{yyyy-MM-dd}.gz
    #layout:
      #type: pattern
      #conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  index_search_slow_log_file:
    type: rollingFile
    file: ${path.logs}/${cluster.name}_index_search_slowlog.log
    maxFileSize: 10MB
    maxBackupIndex: 10
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

  index_indexing_slow_log_file:
    type: rollingFile
    file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log
    maxFileSize: 10MB
    maxBackupIndex: 10
    layout:
      type: pattern
      conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n"

Tips

Some good tips for making sure you setup Azure correctly:

Documentation has been moved to https://exceptionless.com/docs/

Exceptionless Documentation

Individual Client Docs

Clone this wiki locally