Merge pull request #164 from ruecarlo/eks-1.21

Upgrade to 1.21 and fixes on a managed group sections

.vscode/settings.json

@@ -0,0 +1,9 @@
+{
+    "cSpell.words": [
+        "Autoscaler",
+        "Kubernetes",
+        "Rebalance",
+        "Rebalancing",
+        "Tolerations"
+    ]
+}

content/running_spark_apps_with_emr_on_spot_instances/cloud9-awscli.md

@@ -32,4 +32,4 @@ aws --version
 ```bash
 aws ec2 create-key-pair --key-name emr-workshop-key-pair --query "KeyMaterial" --output text > emr-workshop-key-pair.pem
 chmod 400 emr-workshop-key-pair.pem
-```
+```

content/using_ec2_spot_instances_with_eks/prerequisites/_index.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/_index.md

@@ -1,13 +1,13 @@
 ---
-title: "Start the workshop..."
+title: "Start the workshop"
 chapter: true
 weight: 10
 ---
 
 # Getting Started
 To start the workshop, follow one of the following depending on whether you are...
 
-* ...[running the workshop on your own (in your own account)]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/self_paced.md" >}}), or
-* ...[attending an AWS hosted event (using AWS provided hashes)]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/aws_event.md" >}})
+* ...[running the workshop on your own (in your own account)]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/self_paced.md" >}}), or
+* ...[attending an AWS hosted event (using AWS provided hashes)]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/aws_event.md" >}})
 
-Once you have completed with either setup, continue with **[Create a Workspace]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/workspace.md" >}})**
+Once you have completed with either setup, continue with **[Create a Workspace]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/workspace.md" >}})**

content/using_ec2_spot_instances_with_eks/prerequisites/at_an_aws_updateiam.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/at_an_aws_updateiam.md

@@ -7,7 +7,8 @@ hidden: true
 
 ## Attach the IAM role to your Workspace
 
-1. Follow this [deep link to find your Cloud9 EC2 instance](https://console.aws.amazon.com/ec2/v2/home?#Instances:tag:Name=aws-cloud9-eksworkshop;sort=desc:launchTime).
+1. Click the grey circle button (in top right corner) and select **Manage EC2 Instance**.
+![cloud9Role](/images/using_ec2_spot_instances_with_eks/prerequisites/cloud9-role.png)
 1. Select the instance, then choose **Actions / Security / Modify IAM role**
 ![c9instancerole](/images/using_ec2_spot_instances_with_eks/prerequisites/c9instancerole.png)
 1. Choose **TeamRoleInstance** from the **IAM role** drop down, and select **Save**

content/using_ec2_spot_instances_with_eks/prerequisites/aws_event.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/aws_event.md

@@ -9,7 +9,7 @@ weight: 20
 {{% notice warning %}}
 Only complete this section if you are at an AWS hosted event (such as re:Invent,
 Kubecon, Immersion Day, or any other event hosted by an AWS employee). If you 
-are running the workshop on your own, go to: [Start the workshop on your own]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/self_paced.md" >}}).
+are running the workshop on your own, go to: [Start the workshop on your own]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/self_paced.md" >}}).
 {{% /notice %}}
 
 ### Login to the AWS Workshop Portal
@@ -42,8 +42,6 @@ We have deployed the below resources required to get started with the workshop u
 
 + An EKS cluster with the name `eksworkshop-eksctl` and a [EKS managed node group](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)  with 2 on-demand instances.
 
-{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/eksctl/create_eks_cluster_eksctl_command.md" %}}
-
 
 #### Use your resources 
 
@@ -65,9 +63,9 @@ you will notice additional Cloudformation stacks were also deployed which is the
 
 - Click on the url against `Cloud9IDE` from the outputs
 
-{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/workspace_at_launch.md" %}}
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/workspace_at_launch.md" %}}
 
-{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/update_workspace_settings.md" %}}
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/update_workspace_settings.md" %}}
 
 ### Validate the IAM role {#validate_iam}
 
@@ -78,10 +76,10 @@ aws sts get-caller-identity
 
 ```
 
-{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/at_an_aws_validaterole.md" %}}
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/at_an_aws_validaterole.md" %}}
 
 {{% notice note %}}
-Since we have already setup the prerequisites, **you can head straight to [Test the Cluster]({{<  relref "/using_ec2_spot_instances_with_eks/eksctl/test.md"  >}})**
+Since we have already setup the prerequisites, **you can head straight to [Test the Cluster]({{<  relref "/using_ec2_spot_instances_with_eks/020_eksctl/test.md"  >}})**
 {{% /notice %}}
 
 {{% /expand%}}

content/using_ec2_spot_instances_with_eks/prerequisites/k8stools.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/k8stools.md

@@ -16,7 +16,7 @@ for the download links.](https://docs.aws.amazon.com/eks/latest/userguide/gettin
 #### Install kubectl
 
 ```
-export KUBECTL_VERSION=v1.20.0
+export KUBECTL_VERSION=v1.21.2
 sudo curl --silent --location -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl
 sudo chmod +x /usr/local/bin/kubectl
 ```

content/using_ec2_spot_instances_with_eks/prerequisites/on_your_own_updateiam.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/on_your_own_updateiam.md

@@ -16,7 +16,8 @@ hidden: true
 
 ## Attach the IAM role to your Workspace
 
-1. Follow this [deep link to find your Cloud9 EC2 instance](https://console.aws.amazon.com/ec2/v2/home?#Instances:tag:Name=aws-cloud9-eksworkshop;sort=desc:launchTime).
+1. Click the grey circle button (in top right corner) and select **Manage EC2 Instance**.
+![cloud9Role](/images/using_ec2_spot_instances_with_eks/prerequisites/cloud9-role.png)
 1. Select the instance, then choose **Actions / Security / Modify IAM role**
 ![c9instancerole](/images/using_ec2_spot_instances_with_eks/prerequisites/c9instancerole.png)
 1. Choose **eksworkshop-admin** from the **IAM role** drop down, and select **Save**

content/using_ec2_spot_instances_with_eks/prerequisites/prerequisites.files/eks-spot-workshop-quickstart-cnf.yml → content/using_ec2_spot_instances_with_eks/010_prerequisites/prerequisites.files/eks-spot-workshop-quickstart-cnf.yml

@@ -28,22 +28,22 @@ Parameters:
   C9KubectlVersion:
     Description: Cloud9 instance kubectl version
     Type: String
-    Default: v1.18.12
+    Default: v1.21.2
     ConstraintDescription: Must be a valid kubectl version
   C9KubectlVersionTEST:
     Description: Cloud9 instance kubectl version
     Type: String
-    Default: v1.18.12
+    Default: v1.21.2
     ConstraintDescription: Must be a valid kubectl version
   C9EKSctlVersion:
     Description: Cloud9 instance eksctl version
     Type: String
-    Default: 0.31.0
+    Default: v0.68.0
     ConstraintDescription: Must be a valid eksctl version
   EKSClusterVersion:
     Description: EKS Cluster Vesion
     Type: String
-    Default: 1.18
+    Default: 1.21
     ConstraintDescription: Must be a valid eks version
   EKSClusterName:
     Description: EKS Cluster Name
@@ -216,8 +216,7 @@ Resources:
                       cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, 'CustomResourcePhysicalID')
 
                   except Exception as e:
-                      # logger.error(e, exc_info=True)
-                      responseData = {'Error': traceback.format_exc(e)}
+                      responseData = {'Error': str(e)}
                       cfnresponse.send(event, context, cfnresponse.FAILED, responseData, 'CustomResourcePhysicalID')
 ################## SSM BOOTSRAP HANDLER ###############
   C9OutputBucket:
@@ -277,7 +276,10 @@ Resources:
             - sudo mv -v /tmp/eksctl /usr/local/bin
             - sudo -H -u ec2-user /usr/local/bin/eksctl version
             - echo '=== Create EKS Cluster ==='
-            - !Sub 'sudo -H -u ec2-user /usr/local/bin/eksctl create cluster --version=${EKSClusterVersion} --name=${EKSClusterName} --node-private-networking  --managed --nodes=2 --alb-ingress-access --region=${AWS::Region} --node-labels="lifecycle=OnDemand,intent=control-apps" --asg-access'
+            - sudo -H -u ec2-user curl --silent --location -o /home/ec2-user/environment/eksworkshop.yaml "https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/using_ec2_spot_instances_with_eks/010_prerequisites/prerequisites.files/eksworkshop.yaml"
+            - !Sub sed -i.bak -e 's/--AWS_REGION--/${AWS::Region}/' /home/ec2-user/environment/eksworkshop.yaml
+            - !Sub sed sed -i.bak -e 's/--EKS_VERSION--/${EKSClusterVersion}/' /home/ec2-user/environment/eksworkshop.yaml
+            - sudo -H -u ec2-user /usr/local/bin/eksctl create cluster -f /home/ec2-user/environment/eksworkshop.yaml'
             - sudo -H -u ec2-user /usr/local/bin/kubectl get nodes
 
   C9BootstrapAssociation:

content/using_ec2_spot_instances_with_eks/010_prerequisites/prerequisites.files/eksworkshop.yaml

@@ -0,0 +1,37 @@
+---
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+metadata:
+  name: eksworkshop-eksctl
+  region: --AWS_REGION--
+  version: --EKS_VERSION--
+
+managedNodeGroups:
+- amiFamily: AmazonLinux2
+  instanceType: m5.large
+  name: mng-od-m5large
+  desiredCapacity: 2
+  maxSize: 3
+  minSize: 0
+  labels:
+    alpha.eksctl.io/cluster-name: eksworkshop-eksctl
+    alpha.eksctl.io/nodegroup-name: mng-od-m5large
+    intent: control-apps
+  tags:
+    alpha.eksctl.io/nodegroup-name: mng-od-m5large
+    alpha.eksctl.io/nodegroup-type: managed
+    k8s.io/cluster-autoscaler/node-template/label/intent: control-apps
+  ssh:
+    enableSsm: true
+  iam:
+    withAddonPolicies:
+      autoScaler: true
+      cloudWatch: true
+      albIngress: true
+  privateNetworking: true
+
+# To enable all of the control plane logs, uncomment below:
+# cloudWatch:
+#  clusterLogging:
+#    enableTypes: ["*"]

content/using_ec2_spot_instances_with_eks/prerequisites/self_paced.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/self_paced.md

@@ -5,7 +5,7 @@ weight: 10
 ---
 
 {{% notice warning %}}
-Only complete this section if you are running the workshop on your own. If you are at an AWS hosted event (such as re:Invent, Kubecon, Immersion Day, etc), go to [Start the workshop at an AWS event]({{< ref "/using_ec2_spot_instances_with_eks/prerequisites/aws_event.md" >}}).
+Only complete this section if you are running the workshop on your own. If you are at an AWS hosted event (such as re:Invent, Kubecon, Immersion Day, etc), go to [Start the workshop at an AWS event]({{< ref "/using_ec2_spot_instances_with_eks/010_prerequisites/aws_event.md" >}}).
 {{% /notice %}}
 
 ### Running the workshop on your own
@@ -34,4 +34,4 @@ as an IAM user with administrator access to the AWS account:
 ![Login URL](/images/using_ec2_spot_instances_with_eks/prerequisites/iam-4-save-url.png)
 
 
-Once you have completed the step above, **you can head straight to [Create a Workspace]({{<  ref "/using_ec2_spot_instances_with_eks/prerequisites/workspace.md"  >}})**
+Once you have completed the step above, **you can head straight to [Create a Workspace]({{<  ref "/using_ec2_spot_instances_with_eks/010_prerequisites/workspace.md"  >}})**

content/using_ec2_spot_instances_with_eks/prerequisites/sshkey.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/sshkey.md

@@ -10,7 +10,7 @@ Starting from here, when you see command to be entered such as below, you will e
 
 Please run this command to generate SSH Key in Cloud9. This key will be used on the worker node instances to allow ssh access if necessary.
 
-```bash
+```
 ssh-keygen
 ```
 
@@ -20,6 +20,6 @@ Press `enter` 3 times to take the default choices
 
 Upload the public key to your EC2 region:
 
-```bash
+```
 aws ec2 import-key-pair --key-name "eksworkshop" --public-key-material fileb://~/.ssh/id_rsa.pub
 ```

content/using_ec2_spot_instances_with_eks/prerequisites/update_workspaceiam.md → content/using_ec2_spot_instances_with_eks/010_prerequisites/update_workspaceiam.md

@@ -4,7 +4,7 @@ chapter: false
 weight: 60
 ---
 
-{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/update_workspace_settings.md" %}}
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/update_workspace_settings.md" %}}
 
 We should configure our aws cli with our current region as default:
 ```
@@ -17,5 +17,5 @@ aws configure set default.region ${AWS_REGION}
 aws configure get default.region
 ```
 
-{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/prerequisites/validate_workspace_role.md" %}}
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/validate_workspace_role.md" %}}
 

content/using_ec2_spot_instances_with_eks/020_eksctl/create_eks_cluster_eksctl_command.md

@@ -0,0 +1,62 @@
+---
+title: "Create EKS cluster Command"
+chapter: false
+disableToc: true
+hidden: true
+---
+<!--
+This markdown file is used as part of another file using 'insert-md-from-file' shortcode
+-->
+
+Create an eksctl deployment file (eksworkshop.yaml) to create an EKS cluster:
+
+```
+cat << EOF > eksworkshop.yaml
+---
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+metadata:
+  name: eksworkshop-eksctl
+  region: ${AWS_REGION}
+  version: "1.21"
+
+managedNodeGroups:
+- amiFamily: AmazonLinux2
+  instanceType: m5.large
+  name: mng-od-m5large
+  desiredCapacity: 2
+  maxSize: 3
+  minSize: 0
+  labels:
+    alpha.eksctl.io/cluster-name: eksworkshop-eksctl
+    alpha.eksctl.io/nodegroup-name: mng-od-m5large
+    intent: control-apps
+  tags:
+    alpha.eksctl.io/nodegroup-name: mng-od-m5large
+    alpha.eksctl.io/nodegroup-type: managed
+    k8s.io/cluster-autoscaler/node-template/label/intent: control-apps
+  iam:
+    withAddonPolicies:
+      autoScaler: true
+      cloudWatch: true
+      albIngress: true
+  privateNetworking: true
+
+# To enable all of the control plane logs, uncomment below:
+# cloudWatch:
+#  clusterLogging:
+#    enableTypes: ["*"]
+
+EOF
+```
+
+Next, use the file you created as the input for the eksctl cluster creation.
+
+```
+eksctl create cluster -f eksworkshop.yaml
+```
+
+{{% notice info %}}
+Launching EKS and all the dependencies will take approximately 15 minutes
+{{% /notice %}}

content/using_ec2_spot_instances_with_eks/020_eksctl/launcheks.md

@@ -0,0 +1,39 @@
+---
+title: "Launch EKS"
+date: 2018-08-07T13:34:24-07:00
+weight: 20
+---
+
+
+{{% notice warning %}}
+**DO NOT PROCEED** with this step unless you have [validated the IAM role]({{< relref "../010_prerequisites/update_workspaceiam.md#validate_iam" >}}) in use by the Cloud9 IDE. You will not be able to run the necessary kubectl commands in the later modules unless the EKS cluster is built using the IAM role.
+{{% /notice %}}
+
+#### Challenge:
+**How do I check the IAM role on the workspace?**
+
+{{%expand "Expand here to see the solution" %}}
+
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/010_prerequisites/validate_workspace_role.md" %}}
+
+If you do not see the correct role, please go back and **[validate the IAM role]({{< relref "../010_prerequisites/update_workspaceiam.md" >}})** for troubleshooting.
+
+If you do see the correct role, proceed to next step to create an EKS cluster.
+{{% /expand %}}
+
+
+### Create an EKS cluster
+
+{{% insert-md-from-file file="using_ec2_spot_instances_with_eks/020_eksctl/create_eks_cluster_eksctl_command.md" %}}
+
+`eksctl create cluster` command allows you to create the cluster and managed nodegroups in sequence. There are a few things to note in the configuration that we just used to create the cluster and a managed nodegroup.
+
+ * Nodegroup configurations are set under the **managedNodeGroups** section, this indicates that the node group is managed by EKS.
+ * Nodegroup instance type is **m5.large** with **minSize** to 0, **maxSize** to 3 and **desiredCapacity** to 2. This nodegroup has capacity type set to On-Demand Instances by default.
+ * Notice that the we add 3 node labels:
+
+  * **alpha.eksctl.io/cluster-name**, to indicate the nodes belong to **eksworkshop-eksctl** cluster.
+  * **alpha.eksctl.io/nodegroup-name**, to indicate the nodes belong to **mng-od-m5large** nodegroup.
+  * **intent**, to allow you to deploy control applications on nodes that have been labeled with value **control-apps**
+
+ * Amazon EKS adds an additional Kubernetes label **eks.amazonaws.com/capacityType: ON_DEMAND**, to all On-Demand Instances in your managed node group. You can use this label to schedule stateful applications on On-Demand nodes.

content/using_ec2_spot_instances_with_eks/eksctl/prerequisites.md → content/using_ec2_spot_instances_with_eks/020_eksctl/prerequisites.md

@@ -6,7 +6,7 @@ weight: 10
 
 For this module, we need to download the [eksctl](https://eksctl.io/) binary:
 ```
-export EKSCTL_VERSION=0.51.0
+export EKSCTL_VERSION=v0.68.0
 curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/${EKSCTL_VERSION}/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
 
 sudo mv -v /tmp/eksctl /usr/local/bin