Deployment

Local VMs

Please see the Cloud Images Wiki page for the list of images available. If you intend to deploy a local instance (without GitHub), you can use one of the images above but add following directives to rcloud.conf:

gist.backend: gitgist
gist.git.root: ${ROOT}/data/gists
Exec.auth: as-local

and make sure permissions leading to /data/rcloud/data/gists are sufficient:

sudo chgrp -R rcloud /data/rcloud/data
sudo chmod -R g+rwx /data/rcloud/data

Multiple local users

If you want to support multiple users in the instance, you will have to run the QAP service as root and enable PAM authentication. To do that, make sure SessionKeyServer has been compiled with PAM support and then in rcloud.conf use:

Exec.auth: pam
Exec.match.user: login

In order to run the QAP service as root edit /data/rcloud/services/rcloud-qap and replace

exec su - rcloud /data/rcloud/services/rcloud-start

with

exec /data/rcloud/services/rcloud-start

Again, adjust the permissions of the gist directory:

sudo chmod a+rx /data/rcloud/data
sudo chmod 0777 /data/rcloud/data/gists

Finally, run

sudo usermod -a -G shadow ubuntu

(see below why) and restart.

PAM and SKS

On some systems PAM using pam_unix.so will fail to authenticate users due to a restriction in unix_chkpwd. The error message will be cryptic:

unix_chkpwd[1444]: check pass; user unknown 

despite the fact that the user is actually correct and known. The full explanation is in this post. The most practical fix is to add the user running SKS into the shadow group (or whatever group owns /etc/shadow), e.g. for our Ubuntu cloud images that would be

sudo usermod -a -G shadow ubuntu

System redis

redis (valkey or redict) can be run directly in the services directory, or it is possible to use the system-managed service. However, modern Linux distributions run the redis-server service in a sandbox, so it is important to adjust the service settings when changing the data directory. For example, if using /data/redis then the following steps are required:

systemctl edit redis-server

add the following section at the top:

[Service]
ReadWriteDirectories=/data/redis

Then reload and restart the service:

systemctl daemon-reload
systemctl restart redis-server

Note that incorrect settings lead to silent failures: the redis service will work for some time, but will start issuing errors when it finds out that the dump cannot be saved. This may manifest itself by wrong star counts or that new notebooks are not visible after reload (they can still be loaded by their ID).

Use nginx reverse proxy

For highly scalable deployments it is advisable to use nginx as reverse-proxy for load-balancing in front of a pool of RCloud compute instances. This is an example configuration:

# enable websocket upgrades
map $http_upgrade $connection_upgrade {
    	default	upgrade;
	''	close;
}

upstream rcloud_servers {
    # least_conn is load balancing based on selecting a server from the pool which has the
    # least active sessions from nginx. Default is round robin.
    least_conn;

    # List all of the rcloud server pool machines/VMs in the format
    # server <FQDN>:<PORT>;
    # For automatic scaling use externally managed file:
    include /etc/rcloud-servers;
}

server {
        ## put your basic server details here ... (protocols, ports, etc.)

	location / {
		try_files $uri $uri/ @proxy;
	}

	location @proxy {
		proxy_pass		http://rcloud_servers;
		proxy_set_header	X-Real-IP  $remote_addr;

		# allow connection upgrade to WebSockets
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection $connection_upgrade;

		# we may serve long-running queries,
		# so we set this really high
		proxy_read_timeout 30m;
	}
}

The above assumes you have a file /etc/rcloud-servers which lists all currently available compute instances, e.g.:

server 10.2.0.2:8080;
server 10.2.0.3:8080;

This list can be automatically managed, e.g., the forward proxy has support for srvmgr (see forward -S <host>[:<port>] option) which adds instances when forward is started and removes them when they shut down. Thus when a compute VM is started it automatically registers with the RCloud instance and later removes itself form the list when shut down.

If only simple setup is required, you can just the servers directly in the nginx config file.

Optionally, you can have the root point to the rcloud/htdocs directory in which can nginx will serve static content directly instead of asking the RCloud forward proxy.