production-kops

Setup KOPS on AWS with RBAC enabled and initial role bindings in private topology with calico networking.

Pre-Flight:

• NOTE - SpotInst kops is now officially a part of the official kops tree.
• Download and install latest spotinst kops binary compiled just for SpotInst:
• This repo was last updated for use with kops v1.11.0

kops downloads

• Next link them to your bin dir ln -s /usr/local/bin/kops kops

IAM:

• Setup an IAM user for KOPS (goes inside 00-env.sh script) as outlined here.

Create Cluster && VPC:

• Scripts provided by spotinst kops.

• Edit the values in 00-env.sh

• cd bin && ./01-create.sh

• Wait about 5-10 minutes.

Create in Existing VPC:

• Edit the values in 00-env.sh

• cd bin && ./02-create-wo-apply.sh

• . 00-env.sh && kops get clusters -o yaml > config.yaml

• . 00-env.sh && kops edit cluster

  • Change your subnet CIDRs to non-conflicting CIDRs; IE those that aren't currently used in your VPC.
  • You could also manuall edit vim config.yaml.

• kops replace -f config.yaml

• cd bin && ./07-update.sh

Update ig:

• . 00-env.sh && kops edit ig <instance group>
• . 00-env.sh && kops update cluster --name $KOPS_CLUSTER_NAME
• . 00-env.sh && kops rolling-update cluster --name $KOPS_CLUSTER_NAME --master-interval 3m --node-interval 3m --yes --fail-on-validate-error="false"

When satisfied with your cluster save the overall manifest for later use / record-keeping:

kops get $KOPS_CLUSTER_NAME -o yaml > <some-output.txt>

Export your admin level config for use elsewhere:

• Set your KUBECONFIG environment variable so you don't overwrite or merge to a current config:

  • export KUBECONFIG=$HOME/tools.prominentedge.com

• Export:

  • `kops export kubecfg --name tools.prominentedge.com

Deploy Spaceship:

Creating pull secret for dockerhub registry (only run once):

• Create DockerHub Image Pull Secrets (Jenkins username / password are available in Jenkins or 1Password):

./bin/create_secret.sh <DockerHub user> <DockerHub pass> <DockerHub email>

• Now patch the default service account to automagically add pull secret to all pods in the default namespace:

kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "registrykey"}]}'

• Note that this also requires you add the appropriate collaborators under each DockerHub repo.

• If you have issues, see here-image-pull-secrets.

HELM - Initializing helm after cluster boot:

• After deploying a new kubernetes cluster via kops (and configuring your env locally to interact with it properly), be sure to give helm permissions to act on namespaces with:

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

• helm init - you should get the message HAPPY TILLING, and you are now ready to rock 'n roll.

Install Nginx Ingress:

• Follow the Generic Deployment Instructions, and also use the Install with RBAC roles.
  • The above are installed under the ingress-nginx namespace.
  • For a good example of ingress with a service please look at build-files/statengine.

Install External DNS:

• Update IAM roles by creating and attaching the IAM policy specified here.

cd ./external-dns
kubectl create -f external*

• Enjoy spaceship.

---

Where next?

• Install Heapster, Grafana, and Influxdb.

---

Repos:

• SpotInst Kops Fork.