Kubernetes HA cluster installer

This repository contains a kubeadm-based installer for a highly available Kubernetes cluster with a configurable number of master and worker nodes. HA is on the roadmap for kubeadm and this script should be considered a workaround until kubeadm is HA-capable. This installer is largely based on Creating HA clusters with kubeadm.

The installer should run on any Linux system which meets the software prerequisites outlined below.

Just like kubeadm, this installer should work with both bare-metal and cloud servers. To make use of cloudprovider-specific features, like volumes or load-balancers, some cloudprovider config may be needed. Hooks are provided to pass small configuration fragments for such purposes.

The installer script assumes that Ubuntu (16.04) servers on which to install Kubernetes have a already been provisioned. Infrastructure providers are used to create cloud infrastructure (such as servers and apiserver load-balancers). Some examples are available under infra-providers. At the moment, two infrastructure providers are implemented:

• an AWS infra-provider, which creates a cloud infrastructure via Terraform.
• a Vagrant infra-provider, which uses Vagrant to start cluster VMs locally on a single host.

The general workflow is to

0. Use an infra-provider (or any other means) to provision cluster machines for the Kubernetes masters and workers. Note that at least 3 masters are required for a HA setup to work. A smaller number prevents etcd from forming a quorum. A load-balancer also needs to be provisioned to distribute HTTPS traffic over the masters on port 6443 (the kube-apiserver port).
1. Declare a cluster definition (in JSON) for the cluster installer. The format is described below.
2. Use the installer to render cluster assets. The cluster assets consists of secrets (cluster token, certificates, ssh keys) and boot scripts for each master and worker.
3. After inspecting the boot scripts, use the installer to install the cluster by running the boot scripts (over SSH) against the master and workers declared in the cluster definition.

Quick start

The simplest way to get started is to try out a HA cluster on your local machine.

0. Follow the instructions in infra-providers/vagrant/README.md to create a cluster of local VMs fronted by a HAProxy load-balancer.

1. Prepare the installer: create a virtual environment, install the installer and its dependencies and enter a sub-shell for the virtual environment:

     # optional: create virtualenv in $PWD instead of ~/.local/share/
     export PIPENV_VENV_IN_PROJECT=true
   
     pipenv install .
     pipenv shell
   

2. Run the installer script using the vagrant-cluster.json.

     # render cluster assets (certs, keys, bootscripts)
     hakube-installer render samples/vagrant-cluster.json
     # install nodes over ssh
     hakube-installer install samples/vagrant-cluster.json
   

3. Follow the post-installation step (see below) to run kubectl commands.

Once you have familiarized yourself with the scripts, try it out on AWS using the AWS infra-provider. A cluster definition template is available under samples/aws-cluster.json.

Cluster definition

A cluster definition describes the nodes that constitute the cluster and may optionally contain hooks to configure the cluster for a particular cloudprovider.

An example cluster definition for an AWS cluster may look as follows:

NOTE: the bracket-prefixed comments are provided for clarity, they are NOT permitted by the json parser.

{

    # The name to be assigned to the cluster. This is the name that will be
    # set in the kubeconfig file generated by kubeadm.
    "clusterName": "awskube",
    # (Optional) A list of DNS names intended to be used to access the
    # API server. This should, for example, in the AWS case include
    # the FQDN of a public master loadbalancer, and any additional
    # domain names one would like to access it through.
    # All FQDNs in the list are added as subject alternate names to the
    # generated master certificates.
    "masterFQDNs":  ["kube-402233249.us-east-1.elb.amazonaws.com"],
    # A stable IP address or DNS name for the control plane endpoint.
    # This will be used internally within the cluster, for example by
    # workers to connect to the apiservers. You would typically set it
    # to the private address of a master load-balancer. In case of a
    # single-master cluster, set this to the private IP/DNS name of the
    # master node.
    "masterLoadBalancerAddress": "internal-kube-lb-199111610.us-east-1.elb.amazonaws.com",
    # (Optional) The username to use when logging in over SSH. Typically
    # 'ubuntu'.
    "sshLoginUser": "ubuntu",
    # (Optional) A private SSH login key to use when connecting to nodes.
    # Can be overridden on a per-node basis (see below).
    "sshLoginKey": "~/.ssh/id_rsa",

    # (Optional) The docker version to use.
    "dockerVersion": "17.03.2~ce-0~ubuntu-xenial",
    # (Optional) The Kubernetes version to use.
    "kubernetesVersion": "1.11.2",
    # (Optional) The etcd version to use. Preferred versions are:
    # k8s 1.11 => 3.2.18+, k8s 1.10 => 3.2.14+, k8s 1.9 => v3.1.10+,
    # k8s 1.8 => v3.0.17.
    "etcdVersion": "3.2.18",

    # (Optional) Directory under assets dir where etcd certificates will
    # be written.
    "etcdPKIDir": "pki/etcd",
    # (Optional) The Common Name of the etcd CA cert.
    "etcdCACommonName": "etcd-ca",
    # (Optional) The Common Name of the etcd client cett.
    "etcdClientCommonName": "etcd-client",
    # (Optional) The CA certificate expiry time. Default: 20 years.
    "etcdCACertExpiry": "175200h",

    # (Optional) Directory under assets dir where ssh keys for the nodes will
    # be written.
    "sshKeysDir": "pki/ssh",
    # (Optional) Directory under assets dir where node boot scripts will
    # be written.
    "scriptsDir": "scripts",

    # (Optional) The address range (in CIDR notation) to use for the pod
    # network. Note that this setting may require adjustment depending on the
    # chosen network provider (.hooks.networkProvider)
    "podNetworkCIDR": "10.32.0.0/12",


    # The list of master nodes in the cluster.
    "masters": [
        {
            # The node's name.
            "nodeName": "ip-10-1-0-10.ec2.internal",
            # The node's private IP. In some cases, this may be the same IP as
            # the publicIP. Used for cluster-internal communication.
            "privateIP": "10.1.0.10",
            # The node's public IP. Used to run boot scripts.
            "publicIP": "35.168.152.188"
        },
        {
            "nodeName": "ip-10-1-1-10.ec2.internal",
            "privateIP": "10.1.1.10",
            "publicIP": "34.236.94.133"
        },
        {
            "nodeName": "ip-10-1-2-10.ec2.internal",
            "privateIP": "10.1.2.10",
            "publicIP": "34.225.224.64"
        }
    ],

    # The list of worker nodes in the cluster.
    "workers": [
        {
            "nodeName": "ip-10-1-0-40.ec2.internal",
            "privateIP": "10.1.0.40",
            "publicIP": "34.239.205.162",
            "sshLoginKey": "~/.ssh/worker_rsa"
        },
        {
            "nodeName": "ip-10-1-1-40.ec2.internal",
            "privateIP": "10.1.1.40",
            "publicIP": "52.72.31.142",
            "sshLoginKey": "~/.ssh/worker_rsa"
        }
    ],


    "hooks": {
        # The name of a script to include as the first step in the generated
        # boot scripts, and will therefore run prior to anything else.
        # Assumed to be a fragment located under templates/hooks/preinstall/
        "preinstall": "aws.sh.j2",
        # Can be used to configure Kubernetes for a particular cloudprovider
        # (will be passed to apiserver, kubelet, and controller-manager).
        # For example, 'aws', 'azure', or 'openstack'.
        "cloudProvider": "aws",
        # A cloud-config to be used to configure the specified cloudprovider.
        # Some cloudproviders (aws) do not require a configuration file.
        # Assumed to be a fragment located under templates/hooks/cloudconfig/
        "cloudProviderConfig": null,
        # The name of a bash script used to set up a pod network provider once
        # kubeadm has installed the control plane. The script will run on a
        # master node with kubectl set up to talk to the apiserver. A script
        # fragment must be located under templates/hooks/network/<name>.sh.j2
        # Depending on the specifics of the network provider, you may need to
        # adjust the value of .podNetworkCIDR.
        "networkProvider": "weave"
    }
}

For more details on the configuration format, refer to the source code.

A couple of sample cluster definitions are available under samples.

Prepare the installer

0. Make sure you have the following software installed:

   • bash
   • Python 3.5+
   • pip and pipenv
   • cfssl and cfssljson
   • ssh
   • ssh-keygen

1. Install software prerequisites for the installer in a virtual environment:

    # optional: create virtualenv in $PWD instead of ~/.local/share/
    export PIPENV_VENV_IN_PROJECT=true
   
    pipenv install --three
    pipenv shell
   

Render cluster assets

Render cluster assets. The cluster assets consists of secrets (cluster token, certificates, ssh keys) and boot scripts for each master and worker. The assets are written to assets/ by default (run --help for flags).

    hakube-installer render cluster.json

You can skim through the generated assets to convince yourself that they are acceptable.

Install cluster software

Use the installer to install the cluster by running the boot scripts (over SSH) against the master and workers declared in the cluster definition:

    hakube-installer install cluster.json

By default all cluster machines are installed at once. One can also run the installer against a subset of the machines in the cluster definition. For example:

    hakube-installer install cluster.json ip-10-1-0-10.ec2.internal

This can be useful if something went wrong with a node or it needs to be updated or re-installed. Depending on the state of the node, it may be necessary to first log in to the node and run sudo kubeadm reset to clear a prior installation (depending on the network provider used, additional steps may be required to reset the node's iptables.

Post-installation

Copy the ~/.kube/config file from one of the masters, edit it to make sure that the server field refers to the load-balancer's IP/hostname. You may then interact as usual with the cluster using kubectl:

kubectl --kubeconfig kubeconfig get nodes
... etc