Adding FIS to the EMR Workshop awslabs#187

content/running_spark_apps_with_emr_on_spot_instances/analyzing_costs.md

@@ -1,6 +1,6 @@
 ---
 title: "Analyzing costs"
-weight: 145
+weight: 110
 ---
 
 In this section we will use AWS Cost explorer to look at the costs of our EMR cluster, including the underlying EC2 Spot Instances.

content/running_spark_apps_with_emr_on_spot_instances/automations_monitoring.md

@@ -1,6 +1,6 @@
 ---
 title: "Automations and monitoring"
-weight: 110
+weight: 145
 ---
 
 When adopting EMR into your analytics flows and data processing pipelines, you will want to launch EMR clusters and run jobs in a programmatic manner. There are many ways to do so with AWS SDKs that can run in different environments like Lambda Functions, invoked by AWS Data Pipeline or AWS Step Functions, with third party tools like Apache Airflow, and more.

content/running_spark_apps_with_emr_on_spot_instances/conclusions_and_cleanup.md → content/running_spark_apps_with_emr_on_spot_instances/calltoaction.md

@@ -1,26 +1,20 @@
----
-title: "Conclusions and cleanup"
-weight: 150
----
-
-**Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, you verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances.
-
-
-#### Cleanup
-
-Select the correct tab, depending on where you are running the workshop:
-{{< tabs name="EventorOwnAccount" >}}
-    {{< tab name="In your own account" include="cleanup_ownaccount" />}}
-    {{< tab name="In an AWS event" include="cleanup_event.md" />}}
-{{< /tabs >}}
-
-
-#### Thank you
-
-We hope you found this workshop educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR, in order to optimize your costs.  
-If you have any feedback or questions, click the "**Feedback / Questions?**" link in the left pane to reach out to the authors of the workshop.
-
-#### Other Resources:
-Visit the [**Amazon EMR on EC2 Spot Instances**](https://aws.amazon.com/ec2/spot/use-case/emr/) page for more information, customer case studies and videos.  
-Read the blog post: [**Best practices for running Apache Spark applications using Amazon EC2 Spot Instances with Amazon EMR**](https://aws.amazon.com/blogs/big-data/best-practices-for-running-apache-spark-applications-using-amazon-ec2-spot-instances-with-amazon-emr/)  
-Watch the AWS Online Tech-Talk: [**Best Practices for Running Spark Applications Using Spot Instances on EMR - AWS Online Tech Talks**](https://www.youtube.com/watch?v=u5dFozl1fW8)
+---
+title: "Call to Action"
+weight: 170
+---
+
+**Congratulations!** you have reached the end of the workshop. In this workshop, you learned about the need to be flexible with EC2 instance types when using Spot Instances, and how to size your Spark executors to allow for this flexibility. You ran a Spark application solely on Spot Instances using EMR Instance Fleets, you verified the results of the application, and saw the cost savings that you achieved by running the application on Spot Instances.
+
+#### Thank you
+
+We hope you found this workshop educational, and that it will help you adopt Spot Instances into your Spark applications running on Amazon EMR, in order to optimize your costs.  
+If you have any feedback or questions, click the "**Feedback / Questions?**" link in the left pane to reach out to the authors of the workshop.
+
+#### Other Resources:
+Visit the [**Amazon EMR on EC2 Spot Instances**](https://aws.amazon.com/ec2/spot/use-case/emr/) page for more information, customer case studies and videos.  
+
+Read the blog post: [**Best practices for running Apache Spark applications using Amazon EC2 Spot Instances with Amazon EMR**](https://aws.amazon.com/blogs/big-data/best-practices-for-running-apache-spark-applications-using-amazon-ec2-spot-instances-with-amazon-emr/)  
+
+Watch the AWS Online Tech-Talk: 
+
+{{< youtube u5dFozl1fW8 >}}

content/running_spark_apps_with_emr_on_spot_instances/cleanup.md

@@ -0,0 +1,13 @@
+---
+title: "Cleanup"
+weight: 150
+---
+
+#### Cleanup
+
+Select the correct tab, depending on where you are running the workshop:
+{{< tabs name="EventorOwnAccount" >}}
+    {{< tab name="In your own account" include="cleanup_ownaccount" />}}
+    {{< tab name="In an AWS event" include="cleanup_event.md" />}}
+{{< /tabs >}}
+

content/running_spark_apps_with_emr_on_spot_instances/cleanup_ownaccount.md

@@ -5,8 +5,16 @@ disableToc: true
 hidden: true
 ---
 
-1. In the EMR Management Console, check that the cluster is in the **Terminated** state. If it isn't, then you can terminate it from the console.
+1. Go the EMR Management Console and termiate the cluster.
 2. Go to the [Cloud9 Dashboard](https://console.aws.amazon.com/cloud9/home) and delete your environment.
 3. Delete the VPC you deployed via CloudFormation, by going to the CloudFormation service in the AWS Management Console, selecting the VPC stack (default name is Quick-Start-VPC) and click the Delete option. Make sure that the deletion has completed successfully (this should take around 1 minute), the status of the stack will be DELETE_COMPLETE (the stack will move to the Deleted list of stacks).
 4. Delete your S3 bucket from the AWS Management Console - choose the bucket from the list of buckets and hit the Delete button. This approach will also empty the bucket and delete all existing objects in the bucket.
-5. Delete the Athena table by going to the Athena service in the AWS Management Console, find the **emrworkshopresults** Athena table, click the three dots icon next to the table and select **Delete table**.
+5. Delete the Athena table by going to the Athena service in the AWS Management Console, find the **emrworkshopresults** Athena table, click the three dots icon next to the table and select **Delete table**.
+6. Delete the CloudFormation stack for the FIS experiment templates. Run the following command:
+```
+aws cloudformation delete-stack --stack-name fis-spot-interruption
+```
+7. Delete the CloudFormation stack for tracking the Spot interruptions. Run the following command:
+```
+aws cloudformation delete-stack --stack-name track-spot-interruption
+```

content/running_spark_apps_with_emr_on_spot_instances/examining_cluster.md

@@ -90,9 +90,6 @@ Some notable metrics:
 * **ContainerAllocated** - this represents the number of containers that are running on your cluster, on the Core and Task Instance Fleets. These would the be Spark executors and the Spark Driver.  
 * **MemoryAllocatedMB** & **MemoryAvailableMB** - you can graph them both to see how much memory the cluster is actually consuming for the wordcount Spark application out of the memory that the instances have.  
 
-#### Terminate the cluster
-When you are done examining the cluster and using the different UIs, terminate the EMR cluster from the EMR management console. This is not the end of the workshop though - we still have some interesting steps to go.
-
 #### Number of executors in the cluster
 With 32 Spot Units in the Task Instance Fleet, EMR launched either 8 * xlarge (running one executor) or 4 * 2xlarge instances (running 2 executors) or 2 * 4xlarge instances (running 4 executors), so the Task Instance Fleet provides 8 executors / containers to the cluster.  
 The Core Instance Fleet launched one xlarge instance, able to run one executor.

content/running_spark_apps_with_emr_on_spot_instances/spot_interruption_experiment.md

@@ -0,0 +1,111 @@
+---
+title: "Creating the Spot Interruption Experiment"
+weight: 97
+---
+
+In this section, you're going to start creating the experiment to [trigger the interruption of Amazon EC2 Spot Instances using AWS Fault Injection Simulator (FIS)](https://aws.amazon.com/blogs/compute/implementing-interruption-tolerance-in-amazon-ec2-spot-with-aws-fault-injection-simulator/). When using Spot Instances, you need to be prepared to be interrupted. With FIS, you can test the resiliency of your workload and validate that your application is reacting to the interruption notices that EC2 sends before terminating your instances. You can target individual Spot Instances or a subset of instances in clusters managed by services that tag your instances such as ASG, Fleet and EMR.
+
+You're going to use the CLI, so launch your terminal to run the commands included in this section.
+
+#### What do you need to get started?
+
+Before you start launching Spot interruptions with FIS, you need to create an experiment template. Here is where you define which resources you want to interrupt (targets), and when you want to interrupt the instance. 
+
+You're going to use the following CloudFormation template which creates the IAM role (`FISSpotRole`) with the minimum permissions FIS needs to interrupt an instance, and the experiment template (`FISExperimentTemplate`) you're going to use to trigger a Spot interruption:
+
+```
+AWSTemplateFormatVersion: 2010-09-09
+Description: FIS for Spot Instances
+Parameters:
+  InstancesToInterrupt:
+    Description: Number of instances to interrupt
+    Default: 3
+    Type: Number
+
+  DurationBeforeInterruption:
+    Description: Number of minutes before the interruption
+    Default: 2
+    Type: Number
+
+Resources:
+
+  FISSpotRole:
+    Type: AWS::IAM::Role
+    Properties:
+      AssumeRolePolicyDocument:
+        Statement:
+        - Effect: Allow
+          Principal:
+            Service: [fis.amazonaws.com]
+          Action: ["sts:AssumeRole"]
+      Path: /
+      Policies:
+        - PolicyName: root
+          PolicyDocument:
+            Version: "2012-10-17"
+            Statement:
+              - Effect: Allow
+                Action: 'ec2:DescribeInstances'
+                Resource: '*'
+              - Effect: Allow
+                Action: 'ec2:SendSpotInstanceInterruptions'
+                Resource: 'arn:aws:ec2:*:*:instance/*'
+
+  FISExperimentTemplate:
+    Type: AWS::FIS::ExperimentTemplate
+    Properties:       
+      Description: "Interrupt multiple random instances"
+      Targets: 
+        SpotIntances:
+          ResourceTags: 
+            ResourceTagKey: ResourceTagValue
+          Filters:
+            - Path: State.Name
+              Values: 
+              - running
+          ResourceType: aws:ec2:spot-instance
+          SelectionMode: !Join ["", ["COUNT(", !Ref InstancesToInterrupt, ")"]]
+      Actions: 
+        interrupt:
+          ActionId: "aws:ec2:send-spot-instance-interruptions"
+          Description: "Interrupt multiple Spot instances"
+          Parameters: 
+            durationBeforeInterruption: !Join ["", ["PT", !Ref DurationBeforeInterruption, "M"]]
+          Targets: 
+            SpotInstances: SpotIntances
+      StopConditions:
+        - Source: none
+      RoleArn: !GetAtt FISSpotRole.Arn
+      Tags: 
+        Name: "FIS_EXP_NAME"
+
+Outputs:
+  FISExperimentID:
+    Value: !GetAtt FISExperimentTemplate.Id
+```
+
+Here are some important notes about the template:
+
+* You can configure how many instances you want to interrupt with the `InstancesToInterrupt` parameter. In the template it's defined that it's going to interrupt **three** instances.
+* You can also configure how much time you want the expriment to run with the `DurationBeforeInterruption` parameter. By default, it's going to take two minutes. This means that as soon as you launch the experiment, the instance is going to receive the two-minute notification Spot interruption warning.
+* The most important section is the `Targets` from the experiment template. The template has two placeholders `ResourceTagKey` and `ResourceTagValue` which are basically the key/value for the tags to use when choosing the instances to interrupt. We're going to run a `sed` command to replace them with the proper values for this workshop.
+* Notice that instances are **chosen randomly**, and only those who are in the `running` state.
+
+#### Create the EC2 Spot Interruption Experiment with FIS
+
+Let's continue by creating the Spot interruption experiment template using Cloudformation. You can view the CloudFormation template (**fisspotinterruption.yaml**) at GitHub [here](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/fis/fisspotinterruption.yaml). To download it, you can run the following command:
+
+```
+wget https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/fis/fisspotinterruption.yaml
+```
+
+Now, simply run the following commands to create the FIS experiment:
+
+```
+export FIS_EXP_NAME=fis-spot-interruption
+sed -i -e "s#ResourceTagKey#aws:elasticmapreduce:instance-group-role#g" fisspotinterruption.yaml
+sed -i -e "s#ResourceTagValue#TASK#g" fisspotinterruption.yaml
+sed -i -e "s#FIS_EXP_NAME#$FIS_EXP_NAME#g" fisspotinterruption.yaml
+aws cloudformation create-stack --stack-name $FIS_EXP_NAME --template-body file://fisspotinterruption.yaml --capabilities CAPABILITY_NAMED_IAM
+aws cloudformation wait stack-create-complete --stack-name $FIS_EXP_NAME
+```