Skip to content

Latest commit

 

History

History
198 lines (155 loc) · 5.81 KB

README.md

File metadata and controls

198 lines (155 loc) · 5.81 KB
Raw

InCal Elastic + Akka HTTP Seed Project

This project is meant to demonstrate functionality of InCal Elastic library by integrating it with a simple Akka HTTP server and providing a skeleton / seed app to fork and further customize.

Installation

  • Java 1.8

(e.g. Oracle JDK)

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt install oracle-java8-set-default

  • Elastic Search

    • Install ES (5.6.10)
    sudo apt-get update
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.10.deb
sudo dpkg -i elasticsearch-5.6.10.deb
sudo systemctl enable elasticsearch.service
    • Modify the configuration in /etc/elasticsearch/elasticsearch.yml
      cluster.name: seed_app-cluster (if not changed "elasticsearch" is used by default)
  bootstrap.memory_lock: true
  network.host: x.x.x.x           (set to a non-localhost ip address if the db should be accessible within a network)
  thread_pool:
    index:
      queue_size: 8000
    search:
      queue_size: 8000 
    bulk:
      queue_size: 500
  indices.query.bool.max_clause_count: 4096
    • If you want to store data in a special directory set also
    path.data: /your_custom_path

    (Note that you need to make your_custom_path writeable for the elasticsearch user)

    • Create a limits file /etc/security/limits.d/elasticsearch.conf
    elasticsearch    soft    nofile          1625538
elasticsearch    hard    nofile          1625538
elasticsearch    soft    memlock         unlimited
elasticsearch    hard    memlock         unlimited

    • To force your new limits to be loaded log out of all your current sessions and log back in.

    • Configure open-file and locked memory constraints in /etc/init.d/elasticsearch (and/or /etc/default/elasticsearch)

    MAX_OPEN_FILES=1625538
MAX_LOCKED_MEMORY=unlimited
    • Depending on the Linux installation configure also /usr/lib/systemd/system/elasticsearch.service
    LimitNOFILE=1625538
LimitMEMLOCK=infinity
    • Set a reasonable heap size; recommended to 50% of available RAM, but no more than 31g in /etc/elasticsearch/jvm.options, e.g.
    -Xms4g
-Xmx4g
    • and finally apply the settings for systemd
    sudo systemctl daemon-reload
    • Start ES
    sudo service elasticsearch start

    • To check if everything works as expected see the log file(s) at /var/log/elasticsearch/ and/or curl the server info by curl -XGET localhost:9200.

    • Recommendation: For convenient DB exploration and query execution you might want to install the cerebro app:

     sudo wget https://github.com/lmenezes/cerebro/releases/download/v0.8.3/cerebro-0.8.3.zip
 sudo unzip cerebro-0.8.3.zip
    • Open the configuration file conf/application.conf and add the host and name of your Elastic server to the hosts section, e.g.,
      {
    host = "http://127.0.0.1:9200"
    name = "seed_app-cluster"
  }
    • Run Cerebro, for example at the port 9209
     sudo ./bin/cerebro -Dhttp.port=9209

    (Cerebro web client is then accessible at http://localhost:9209)

    • Alternatively you can install Kibana, which also allows execution of Elastic queries and many more, as follows (more info here):
    echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo apt-get update
sudo apt-get install kibana
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service
sudo service start kibana

Configuration

To configure the example app server pls. set the host and port properties in application.conf. Example:

############
# Seed App #
############

seed_app.host = "localhost"

seed_app.port = 8080

##################
# Elastic search #
##################

elastic {
  host = "127.0.0.1"
  port = 9200
  cluster.name = "seed_app-cluster"

  type = "transport"
  client.transport.sniff = true
  client.transport.ping_timeout = "80s"
  thread_pool.index.size = 4
  thread_pool.index.queue_size = 10000
  thread_pool.bulk.size = 4
  thread_pool.bulk.queue_size = 1000
}

Deployment

To create an executable jar with all dependencies run

sbt assembly

This will produce a file such as incal-access-elastic-akka-http-seed-assembly-0.0.2.jar

Usage / Examples

By default the seed app runs at http://localhost:8080, which is the url used in all the examples bellow.

 

  1. Add a person to the app and returns its id

    Endpoint: add
    Example: curl -X POST --header "Content-Type: application/json" --data '{"name": "Robert Mugabe", "age": 95, "gender": "Male", "died": true, "timeCreated": 312415324324532}' http://localhost:8080/add
    Response: 2bc51223-613c-42d4-8254-de23654b9f34

  2. Count the stored persons

    Endpoint: count
    Example: curl http://localhost:8080/count
    Response: 3

  3. Get a person by id

    Endpoint: get
    Example: curl http://localhost:8080/get/2bc51223-613c-42d4-8254-de23654b9f34
    Response: {"age":95,"died":true,"gender":"Male","id":"2bc51223-613c-42d4-8254-de23654b9f34","name":"Robert Mugabe","timeCreated":312415324324532}

  4. Search persons by a given name

    Endpoint: search
    Example: curl http://localhost:8080/search/John
    Response: ["2025e587-2656-4ea2-9f26-6b6f75840461","b2332005-ab0e-4210-8c42-55b4fa9230e1","c04a3785-bbc9-40a4-a1f6-4dd96f028e62"]

  5. Clear/remove the stored persons

    Endpoint: clear
    Example: curl -X POST http://localhost:8080/clear
    Response: All persons deleted.

Note: You can upload items from a resource csv persons_upload.csv or a csv of your choice by executing UploadPersonsFromFile.