Install Manager

We assume that your system is configured with a sudoable admin user named devops. Also, you should already have a running etcd cluster, PostgreSQL and Redis servers as guided in previous step guides.

Guide variables

⚠️ Prepare the values of the following variables before working with this page and replace their occurrences with the values when you follow the guide.

{NS}	The etcd namespace (just create a unique string like domain names)
{ETCDADDR}	The etcd cluster address ({ETCDHOST}:{ETCDPORT}, localhost:2379 for development setup)
{REDISADDR}	The Redis server address ({REDISHOST}:{REDISPORT}, localhost:6389 for development setup)
{DBADDR}	The PostgreSQL server address ({DBHOST}:{DBPORT}, localhost:5442 for development setup)
{DBUSER}	The database username (e.g., postgres for development setup)
{DBPASS}	The database password (e.g., develove for development setup)
{ENDPOINT}	The DNS hostname of the API server (depending on your environment, this may be either a publicly registered domain or a local private domain)

Optional variables

{SSLCERT}	The path to your SSL certificate (bundled with CA chain certificates)
{SSLPKEY}	The path to your SSL private key
{S3AKEY}	The access key for AWS S3 or compatible services[1]
{S3SKEY}	The secret key for AWS S3 or compatible services
{DDAPIKEY}	The Datadog API key
{DDAPPKEY}	The Datadog application key
{SENTRYURL}	The private Sentry report URL

^[1] AWS S3 is used to store the output files generated by the user code in kernels' /home/work/.output directory. If not specified, Backend.AI will just skip uploading generated files.

Install dependencies for daemonization

Ubuntu

$ sudo apt-get -y update
$ sudo apt-get -y dist-upgrade
$ sudo apt-get install -y ca-certificates git-core nginx supervisor

Here are some optional but useful packages:

$ sudo apt-get install -y vim tmux htop

CentOS / RHEL

(TODO)

Prepare Python 3.6+

Check out Install Python via pyenv for instructions.
Create a virtualenv named "venv-manager".

Install Backend.AI Manager from Source

$ pyenv shell venv-manager
$ git clone --branch=master "https://github.com/lablup/backend.ai-manager" "backend.ai-manager"
$ cd backend.ai-manager
$ pip install -U pip setuptools
$ pip install -U -r requirements.txt

Configure NGINX

$ sudo vi /etc/nginx/sites-available/gateway

(For testing) Without SSL

map http_connection connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    server_name {ENDPOINT};
    charset utf-8;
    client_max_body_size 32M;
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    location ~ ^/v\d+/stream/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 60s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

(For production) With SSL

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;

map http_connection connection_upgrade {
    default upgrade;
    ''      close;
}
server {
    listen 443 ssl;
    server_name {ENDPOINT};
    charset utf-8;
    client_max_body_size 32M;
    ssl_certificate {SSLCERT};
    ssl_certificate_key {SSLPKEY};
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    location / {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    location ~ ^/v\d+/stream/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_pass_request_headers on;
        proxy_set_header Host "{ENDPOINT}";
        proxy_redirect off;
        proxy_buffering off;
        proxy_read_timeout 60s;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

Monitoring and Logging (optional)

Check out the Install Monitoring and Logging Tools guide.

Configure supervisord

supervisord eases the management of daemonization lifecycles of user-defined programs.

supervisord application config

$ sudo vi /etc/supervisor/conf.d/apps.conf

[program:backendai-manager]
user = devops
stopsignal = TERM
stopasgroup = true
command = /home/devops/run-manager.sh

pyenv + venv initialization script for non-login shells

$ vi /home/devops/init-venv.sh

#!/bin/bash
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
pyenv shell venv-manager

The main program managed by supervisord

$ vi /home/devops/run-manager.sh

#!/bin/bash
source /home/devops/init-venv.sh
umask 0002
export AWS_ACCESS_KEY_ID="{S3AKEY}"
export AWS_SECRET_ACCESS_KEY="{S3SKEY}"
export DATADOG_API_KEY={DDAPIKEY}
export DATADOG_APP_KEY={DDAPPKEY}
export RAVEN_URI="{SENTRYURL}"
exec python -m ai.backend.gateway.server \
            --db-addr={DBHOST}:{DBPORT} \
            --db-user={DBUSER} \
            --db-password='{DBPASS}' \
            --db-name=backend \
            --etcd-addr {ETCDHOST}:{ETCDPORT} \
            --namespace NS \
            --redis-addr {REDISHOST}:{REDISPORT} \
            --events-port 5002 \
            --service-ip 127.0.0.1 \
            --service-port 8081

Initialize database and load intial fixtures

See the Prepare Databases for Manager guide.

Restart nginx

$ sudo cp /etc/nginx/sites-enabled/default ../nginx.site-enabled.default
$ sudo rm /etc/nginx/sites-enabled/default
$ ln -s ../sites-available/gateway /etc/nginx/sites-enabled/gateway
$ sudo systemctl restart nginx

Finally, Run!

$ sudo supervisorctl reread
$ sudo supervisorctl start backendai-manager
$ sudo service nginx restart