Installing a Clojupyter notebook on a Digital Ocean droplet

Backstory

My daughter has a maths assignment for her IB school that needs to be ready in a few months. There is quite some text involved and also some maths, obviously. They normally use a combination of office tools: word, excel etc. A lot of screen shots and copypasting.

Although I like to do my stuff in emacs, even the R stuff, I showed her how Jupyter notebooks with Python. She thought it was interesting, but as we have been doing some Clojure puzzles together for the past year, she asked if one could do the notebook thing using Clojure. That of course warmed this old Lisp heart so I replied that yes we can make that happen.

To be honest it is not a very simple process to currently get access to notebooks that support Clojure. First, I did some searching for cloud notebooks, and could not find any free or cheap that would do. So I started looking into hosting one myself. The other option would be to install it on her laptop locally, but as I am guessing this will need some maintenance effort, a cloud option is more handy for this use case.

Eventually, I settled for a Digital Ocean droplet. They are easy to set up and maintain and have a good command line tool to automate a big part of the process.

I’ll try to capture and document the entire setup process here, to help myself in the future and perhaps lower the barrier for others to try. Provided here is also a small script that does most of the heavy lifting in terms of installing the conda stuff as it is not entirely trivial.

Next to Clojupyter we will also install R and some packages there to be able to play with both the new python and R interop possibilities.

Preparations

Sign Up to Digital Ocean

Follow the instructions here.

Generate and upload your ssh key

Install the xclip utility with the package manager of your distribution.

Ubuntu

$ apt install -y xclip

ArchLinux

$ pacman -S xclip

and so forth…

If you havent already done so, generate an ssh-key:

$ ssh-keygen -t rsa -b 4096

Copy the public key to your clip board

$ xclip -o -sel clip < ~/.ssh/id_rsa.pub

Add the key to your account:

1. Log in to your Digital Ocean account with your browser.
2. Click on the ‘Security’ button on the left sidebar underneath ‘Billing’
3. Click on ‘Add SSH Key’ button
4. Paste the key from your clipboard to the field
5. Add a name to the key as well. For instance ‘Home Machine’

Install and set up doctl the command line utility for Digital Ocean

Use your package manager to install doctl:

Ubuntu

$ apt install -y doctl

Retrieve your DigitalOcean access token and save it locally, run:

$ doctl auth login

Check your account to see that everything works:

$ doctl account get

Provisioning the Clojupyter droplet and installing the system

Start by cloning this repository and cd into the folder

$ git clone https://github.com/skallinen/do-clojupyter-provisioning.git
$ cd do-clojupyter-provisioning

Run the following commands, by copypasting them to your terminal, to create a new droplet and copy the installation script into it:

$ read  -p 'Enter the name of the droplet, something like 'do-clojupyter': ' NAME; doctl compute droplet create $NAME --size s-1vcpu-2gb --image ubuntu-18-04-x64 --region fra1 --ssh-keys  $(doctl compute ssh-key list --format FingerPrint --no-header) --enable-backups; echo "One moment, waiting for the droplet to spin up...";sleep 60; scp -o StrictHostKeyChecking=no notebook-install.sh root@$(doctl compute droplet list $NAME --format "PublicIPv4" --no-header):/root/; doctl compute ssh $NAME

It will ask for a name for the droplet. Give it a name that makes sense. I use do-clojupyter, the do referring to digital ocean. The droplet specified in the commands is a minimum size droplet needed to run the notebook but also some useful R libraries such as tidyverse if you are playing with r-interop. In January 2020 this cost $10 per month. You can increase and decrease the ram size and number of vCPUs according to use cases. Note that you will be billed for the droplet regarless if it is on or not. You need to destroy the droplet in order to stop the billing. So remember to do that after you have finished with this if you just want to try it out. It is easy to reprovision a droplet when you need it again.

Now you should be inside the droplet as root, run the installation script:

$ ./notebook-install.sh

It will start by asking the system username. Give it a name. I used “grace”.

Next it will ask for you to type the UNIX password, type it into the prompt and then repeat it when it asks you to. Then press enter until the user configuration is finished.

The system will now be installing. This will take quite some time. About 45 minutes last time I tried. Most of the time is taken up by installing R Tidyverse. Take the opportunity and have some lunch. :-)

Finally when it is finished log in as the user you defined above:

$ su - <youruser>

Change directory to home

$ cd

Now set the password for the notebooks.

$ conda activate notebook
$ jupyter notebook password

It will ask you to type the your password twice.

You can now start the notebook:

$ screen -d -m -S clojupyter bash -c 'jupyter notebook --no-browser --port=8889'

Next, exit the remote terminal

# first exit from user
$ exit
# then from the remote machine to get back to local terminal
$ exit

Evaluate the following command on your local machine to port forward the traffic from your localhost to the remote localhost:

ssh -N -f -L localhost:9999:localhost:8889 <youruser>@$(doctl compute droplet list $NAME --format "PublicIPv4" --no-header)

Now you should be able to point your browser to localhost:9999 and see the Jupyter interface.

Save the last command as you will have to re-establish the port forwarding every time you loose connection, restart your droplet or your local machine.

I hope to do another write-up explaining how you do data science with Clojure by doing R and Python interop.

Destroying the droplet

In order to stop Digital Ocean billing for the droplet, if you are not using it, you need to destroy it. This can be done simply by issuing this command:

$ doctl compute droplet delete <the-name-of-your-droplet>

Provisioning a new one is simple, just jump back and repeat the steps in the previous section. And you can have lunch again!