Skip to content

Install Manager

Jump to bottom
Joongi Kim edited this page Dec 29, 2017 · 33 revisions

Install system packages

We assume that your system is configured with a sudoable admin user named devops.

Ubuntu

Ensure your system is up-to-date:

$ sudo apt-get update
$ sudo apt-get upgrade

Install dependencies for Backend.AI daemonization:

$ sudo apt-get install -y \
  ca-certificates nginx \
  redis-server redis-tools \
  supervisor

Install dependencies for Python builds:

$ sudo apt-get install -y \
  build-essential git-core \
  libreadline-dev libsqlite3-dev libssl-dev libbz2-dev libzmq3-dev tk-dev

Here are some optional but useful packages:

$ sudo apt-get install vim tmux htop

CentOS / RHEL

(TODO)

Install latest Python

Use pyenv or your favorite method to install Python 3.6 or later. Check out Install Python via pyenv for instructions on using pyenv.

Then, create a virtualenv named "venv-manager".

Install Backend.AI Manager from Source

$ pyenv shell venv-manager
$ git clone --branch=master "https://github.com/lablup/backend.ai-manager" "backend.ai-manager"
$ cd backend.ai-manager
$ pip install -U pip setuptools
$ pip install -U -r requirements.txt

Configure NGINX

$ sudo vi /etc/nginx/sites-available/gateway
map http_connection connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    server_name ENDPOINT;
    charset utf-8;
    client_max_body_size 32M;
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "ENDPOINT";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    location ~ ^/v\d+/stream/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "ENDPOINT";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 60s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

(optional) Configure NGINX to support ssl

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;

map http_connection connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 443 ssl;
    server_name ENDPOINT;
    charset utf-8;
    client_max_body_size 32M;
    ssl_certificate SSLCERT;
    ssl_certificate_key SSLPRIVKEY;
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "ENDPOINT";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    location ~ ^/v\d+/stream/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "ENDPOINT";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 60s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Install Datadog agent (optional)

$ DD_API_KEY=DDAPIKEY bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/dd-agent/master/packaging/datadog-agent/source/install_agent.sh)"

Load initial etcd data

Edit image-metadata.sample.yml and image-aliases.sample.yml according to your setup. By default you can pull the images listed in the sample via docker pull lablup/kernel-xxxx:tag as they are hosted on the public Docker registry.

Load image registry metadata

$ python -m ai.backend.manager.cli etcd update-kernels \
  --namespace=NS --etcd-addr=ETCDADDR \
  -f image-metadata.sample.yml

Load image aliases

$ python -m ai.backend.manager.cli etcd update-aliases \
  --namespace=NS --etcd-addr=ETCDADDR \
  -f image-aliases.sample.yml

Set the default storage mount for virtual folders

$ ETCDCTL_API=3 etcdctl put /sorna/NS/volumes/_vfroot mystorage

Configure supervisord

$ sudo vi /etc/supervisor/conf.d/apps.conf
[program:backendai-manager]
user = devops
stopsignal = TERM
stopasgroup = true
command = /home/devops/run-manager.sh
$ sudo vi /home/devops/run-manager.sh
#!/bin/bash
source /home/devops/init-venv.sh
umask 0002
export AWS_ACCESS_KEY_ID="S3AKEY"
export AWS_SECRET_ACCESS_KEY="S3SKEY"
export DATADOG_API_KEY=DDAPIKEY
export DATADOG_APP_KEY=DDAPPKEY
export RAVEN_URI="SENTRYURL"
python -m ai.backend.gateway.server \
       --db-addr=DBHOST:DBPORT \
       --db-user=DBUSER \
       --db-password='DBPASS' \
       --db-name=backend \
       --etcd-addr ETCDHOST:ETCDPORT\
       --namespace NS \
       --redis-addr REDISHOST:REDISPORT \
       --events-port 5002 \
       --service-ip 127.0.0.1 \
       --service-port 8081

Initialize database and load intial fixtures

Create a new database

$ psql -h DBHOST -U DBPASS
postgres=# CREATE DATABASE backend;
postgres=# \q

Set up database schema

Backend.AI uses alembic to manage database schema and its migration during version upgrades. First, localize the sample config:

$ cp alembic.ini.sample alembic.ini

Modify the line where sqlalchemy.url is set. You may use the following shell command: (ensure that special characters in your password are properly escaped)

$ sed -i'' -e 's!^sqlalchemy.url = .*$!sqlalchemy.url = postgresql://DBUSER:DBPASS@DBHOST/backend!' alembic.ini
$ python -m ai.backend.manager.cli schema oneshot head

NOTE: All sub-commands under "schema" uses alembic.ini to establish database connections.

Load initial fixtures

Edit ai/backend/manager/models/fixtures.py so that you have a randomized admin keypair.

**(TODO: automate here!)**

Then pour it to the database:

$ python -m ai.backend.manager.cli \
  --db-addr=DBHOST:DBPORT --db-user=DBUSER --db-password=DBPASS --db-name=backend \
  fixture populate example_keypair

restart nginx

$ sudo cp /etc/nginx/sites-enabled/default ../nginx.site-enabled.default
$ sudo rm /etc/nginx/sites-enabled/default
$ ln -s ../sites-available/gateway /etc/nginx/sites-enabled/gateway
$ sudo systemctl restart nginx

Finally, Run!

$ sudo supervisorctl start backendai-manager
$ sudo service nginx restart
Clone this wiki locally