Merge pull request #1 from hpg-cepetro/heuristic

Heuristic

README.md

@@ -1 +1,83 @@
-# CloudPITS
+# CloudPITS / Instance Selection
+
+This is the initial idea behind Cloud-PITS, which involves selecting a group of instances that provides a price per 
+cost ratio that will execute a given SPITS code in less than an user-defined input amount.
+Since this is a prototype it still has a lot of limitations and is not ready for general usage. However the general
+idea behind the algorithm is here.
+
+ #### Disclaimer
+
+ This is still a proof of concept. The SPITS program used for validation computes the Zero Offset Non-hyperbolic 
+ Common Reflection Surface (DOI:10.1111/j.1365-2478.2012.01055.x) parameters, therefore there are some parameters that
+  are currenlty tailored for that specific software. This is not yet generic for any SPITS program!
+
+ #### How it works
+
+ The instance selection algorithm works with a stored database containing previous executions performance 
+ measurements. By using previous executions information for the same experiment it is possible to infer how the 
+ current experiment will perform, therefore allows us to create a good initial instance poll. Additionally, by 
+ verifying periodically how the instances are performing and how expensive they are at a given moment, it is possible
+  to select the types that offer better performance for their money.
+
+ The three main files are the following (with their input/output described in their own headers):
+
+    launch/create_ondemand_jm.py
+    populate/simulation.py
+    populate/to_execute.py
+
+ The first one (launch/create_ondemand_zocrsomp_jm.py) is used to launch an On-Demand instance that should work as 
+ the master instance (Job Manager and instance selector). Therefore it is necessary that in some part of the 
+ user_data variable inserting the code for calling the "to_execute.py" code to call this instance selection algorithm
+ and start the Job Manager process.
+
+ The second one simulates an execution, given previous the information stored in the database and the price log 
+ stored in a file (populate/log_prices.csv). This allows the user to simulate how much would be spent for each input 
+ price.
+
+ The last one is the actual Python script to be executed, in which the script will start a new poll of instances and 
+ then verify from time to time for instances that are performing below the desired cost vs performance threshold. 
+ Replacing bad performing types with better ones. 
+
+ #### Database
+
+ The algorithm extracts information from a database as configured in the "rds_config.py" file. The database should be
+  called "experimentos" and have the tables as displayed in the picture below. There is a SQL script to generate it 
+  in the folder "databases/".
+ ![](database/experimentos_db.png)
+
+ #### Performance measurement report
+
+ The worker instances must report their performance to the Job Manager via CloudWatch. To do so, the user needs to 
+ create a new metric with the following characteristics:
+
+    Namespace='Performance',
+    MetricName='perf_sec',
+    Dimensions=[{'Name': 'Instance Id', 'Value': instance_id},
+                {'Name': 'Type', 'Value': instance_type}]
+
+    Namespace='Performance',
+    MetricName='perf_sec_stdev',
+    Dimensions=[{'Name': 'Instance Id', 'Value': instance_id},
+                {'Name': 'Type', 'Value': instance_type}]
+
+    Namespace='Performance',
+    MetricName='tasks_completed',
+    Dimensions=[{'Name': 'Instance Id', 'Value': instance_id},
+                {'Name': 'Type', 'Value': instance_type}]
+
+ This report also goes to the database aforementioned so that future executions use them. Also, they are mandatory so
+  that the instance selection Python script can select the initial poll of instances for the SPITS program being 
+  optimized.
+
+ #### Dependencies
+
+ The program have a few dependencies. They can be installed using the python pip command:
+
+    python3 -m pip install --user -r requirements.txt
+
+ Most Python packages already come with the some dependencies installed, however, two of them are not included, namely:
+
+    boto3: To access the AWS instances
+    pymysql: To enable Python to access a MySQL database and perform queries
+
+  Further information about all files are in their own documentations.

database/experimentos_db.sql

@@ -0,0 +1,147 @@
+-- The MIT License (MIT)
+--
+-- Copyright (c) 2019 Nicholas Torres Okita <[email protected]>
+--
+-- Permission is hereby granted, free of charge, to any person obtaining a copy
+-- of this software and associated documentation files (the "Software"), to
+-- deal in the Software without restriction, including without limitation the
+-- rights to use, copy, modify, merge, publish, distribute, sublicense,
+-- and/or sell copies of the Software, and to permit persons to whom the
+-- Software is furnished to do so, subject to the following conditions:
+--
+-- The above copyright notice and this permission notice shall be included in
+-- all copies or substantial portions of the Software.
+--
+-- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+-- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+-- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+-- THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+-- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+-- FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+-- IN THE SOFTWARE.
+--
+-- MySQL script to create the database for performance reports
+
+SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0;
+SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0;
+SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION';
+
+-- -----------------------------------------------------
+-- Schema experimentos
+-- -----------------------------------------------------
+
+-- -----------------------------------------------------
+-- Schema experimentos
+-- -----------------------------------------------------
+CREATE SCHEMA IF NOT EXISTS `experimentos` DEFAULT CHARACTER SET utf8 ;
+USE `experimentos` ;
+
+-- -----------------------------------------------------
+-- Table `experimentos`.`data`
+-- -----------------------------------------------------
+CREATE TABLE IF NOT EXISTS `experimentos`.`data` (
+  `iddata` INT(11) NOT NULL AUTO_INCREMENT,
+  `name` VARCHAR(45) NULL DEFAULT NULL,
+  `hash` VARCHAR(45) NULL DEFAULT NULL,
+  PRIMARY KEY (`iddata`))
+ENGINE = InnoDB
+AUTO_INCREMENT = 13
+DEFAULT CHARACTER SET = utf8;
+
+
+-- -----------------------------------------------------
+-- Table `experimentos`.`instance`
+-- -----------------------------------------------------
+CREATE TABLE IF NOT EXISTS `experimentos`.`instance` (
+  `name` VARCHAR(45) NOT NULL,
+  PRIMARY KEY (`name`))
+ENGINE = InnoDB
+DEFAULT CHARACTER SET = utf8;
+
+
+-- -----------------------------------------------------
+-- Table `experimentos`.`parameters`
+-- -----------------------------------------------------
+CREATE TABLE IF NOT EXISTS `experimentos`.`parameters` (
+  `idparameters` INT(11) NOT NULL AUTO_INCREMENT,
+  `aph` FLOAT NULL DEFAULT NULL,
+  `apm` FLOAT NULL DEFAULT NULL,
+  `window` DOUBLE NULL DEFAULT NULL,
+  `np` FLOAT NULL DEFAULT NULL,
+  `gens` FLOAT NULL DEFAULT NULL,
+  PRIMARY KEY (`idparameters`))
+ENGINE = InnoDB
+AUTO_INCREMENT = 49
+DEFAULT CHARACTER SET = utf8;
+
+
+-- -----------------------------------------------------
+-- Table `experimentos`.`experiment`
+-- -----------------------------------------------------
+CREATE TABLE IF NOT EXISTS `experimentos`.`experiment` (
+  `idperformance` INT(11) NOT NULL AUTO_INCREMENT,
+  `data_iddata` INT(11) NOT NULL,
+  `parameters_idparameters` INT(11) NOT NULL,
+  `instance_name` VARCHAR(45) NOT NULL,
+  PRIMARY KEY (`idperformance`),
+  INDEX `fk_performance_data1_idx` (`data_iddata` ASC) VISIBLE,
+  INDEX `fk_performance_parameters1_idx` (`parameters_idparameters` ASC) VISIBLE,
+  INDEX `fk_performance_instance1_idx` (`instance_name` ASC) VISIBLE,
+  CONSTRAINT `fk_performance_data1`
+    FOREIGN KEY (`data_iddata`)
+    REFERENCES `experimentos`.`data` (`iddata`),
+  CONSTRAINT `fk_performance_instance1`
+    FOREIGN KEY (`instance_name`)
+    REFERENCES `experimentos`.`instance` (`name`),
+  CONSTRAINT `fk_performance_parameters1`
+    FOREIGN KEY (`parameters_idparameters`)
+    REFERENCES `experimentos`.`parameters` (`idparameters`))
+ENGINE = InnoDB
+AUTO_INCREMENT = 5731
+DEFAULT CHARACTER SET = utf8;
+
+
+-- -----------------------------------------------------
+-- Table `experimentos`.`interpols`
+-- -----------------------------------------------------
+CREATE TABLE IF NOT EXISTS `experimentos`.`interpols` (
+  `idinterpols` INT(11) NOT NULL AUTO_INCREMENT,
+  `parameters_idparameters` INT(11) NOT NULL,
+  `data_iddata` INT(11) NOT NULL,
+  `interpols` FLOAT NULL DEFAULT NULL,
+  `datetime` DATETIME NULL DEFAULT CURRENT_TIMESTAMP,
+  PRIMARY KEY (`idinterpols`),
+  INDEX `fk_interpols_parameters1_idx` (`parameters_idparameters` ASC) VISIBLE,
+  INDEX `fk_interpols_data1_idx` (`data_iddata` ASC) VISIBLE,
+  CONSTRAINT `fk_interpols_data1`
+    FOREIGN KEY (`data_iddata`)
+    REFERENCES `experimentos`.`data` (`iddata`),
+  CONSTRAINT `fk_interpols_parameters1`
+    FOREIGN KEY (`parameters_idparameters`)
+    REFERENCES `experimentos`.`parameters` (`idparameters`))
+ENGINE = InnoDB
+AUTO_INCREMENT = 71
+DEFAULT CHARACTER SET = utf8;
+
+
+-- -----------------------------------------------------
+-- Table `experimentos`.`interpsec`
+-- -----------------------------------------------------
+CREATE TABLE IF NOT EXISTS `experimentos`.`interpsec` (
+  `idinterpsec` INT(11) NOT NULL AUTO_INCREMENT,
+  `interpsec` FLOAT NULL DEFAULT NULL,
+  `experiment_idperformance` INT(11) NOT NULL,
+  `creationDate` DATETIME NULL DEFAULT CURRENT_TIMESTAMP,
+  PRIMARY KEY (`idinterpsec`),
+  INDEX `fk_interpsec_experiment1_idx` (`experiment_idperformance` ASC) VISIBLE,
+  CONSTRAINT `fk_interpsec_experiment1`
+    FOREIGN KEY (`experiment_idperformance`)
+    REFERENCES `experimentos`.`experiment` (`idperformance`))
+ENGINE = InnoDB
+AUTO_INCREMENT = 45971
+DEFAULT CHARACTER SET = utf8;
+
+
+SET SQL_MODE=@OLD_SQL_MODE;
+SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS;
+SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS;

launch/create_ondemand_jm.py

@@ -0,0 +1,126 @@
+#!/usr/bin/python
+
+# The MIT License (MIT)
+#
+# Copyright (c) 2018-2019 Nicholas Torres Okita <[email protected]>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+# This function creates a Job Manager instance
+
+import boto3
+import logging
+import sys
+from datetime import datetime
+
+# function getLogger
+#
+# \param name: Name of the logger
+# \return Logger object
+#
+# This function simply creates a new logger object to output information
+def getLogger(name):
+    now = datetime.now()
+    #Logging configuration
+    logger = logging.getLogger(name)
+    logger.setLevel(logging.INFO)
+    #Log formatter
+    formatter = logging.Formatter("[%(asctime)s] %(levelname)-8s %(message)s")
+    #Log File handler
+    handler = logging.FileHandler("create_spot.log")
+    handler.setLevel(logging.INFO)
+    handler.setFormatter(formatter)
+    logger.addHandler(handler)
+    #Screen handler
+    screenHandler = logging.StreamHandler(stream=sys.stdout)
+    screenHandler.setLevel(logging.INFO)
+    screenHandler.setFormatter(formatter)
+    logger.addHandler(screenHandler)
+    return logger
+
+# function main
+# \param (command line input) instance type: name of instance type to be created
+#
+# This function is the one responsible for creating the instance. The user needs
+# to complete information regarding the user_data (script to be run when the
+# instance starts), and other parameters for the instance creation, such as
+# ImageId and SecurityGroupId (the others are document through out the code).
+def main():
+    # Variable containing the user script to initialize and execute the program
+    user_data = """#!/bin/bash"""
+
+    if len(sys.argv) <= 1:
+        logger.error('Please insert the instance type')
+        return
+
+    logger.info('Starting the instance deployment from the template "'+sys.argv[1]+'"')
+    instance_type_in = sys.argv[1]
+
+    ec2 = boto3.resource('ec2',region_name='us-east-1')
+    instances = ec2.create_instances(
+                InstanceType=instance_type_in,
+                ImageId='', # Image AMI id
+                InstanceInitiatedShutdownBehavior='terminate',
+                SecurityGroupIds=[''], # Security group ID to create instances
+                SubnetId='', # Subnet Id
+                UserData=user_data,
+                MaxCount=1,
+                MinCount=1,
+                KeyName='', # Instance key name
+                Monitoring={
+                    'Enabled': True
+                },
+                BlockDeviceMappings=[ # This is the root disk, can be
+                                      # reconfigured to what is more
+                                      # convenient
+                {
+                    'DeviceName': '/dev/sda1',
+                    'VirtualName': 'eth0',
+                    'Ebs': {
+                        'DeleteOnTermination': True,
+                        'VolumeSize': 20,
+                        'VolumeType': 'io1',
+                        'Iops': 1000
+                    },
+                    'NoDevice':''
+                },
+                ]
+            )
+
+    # Get the instance id from the created instance
+    instance_id = instances[0].id
+    logger.info('Instance deployed! Instance id: '+instance_id)
+
+    # Set tags, user can edit to add more tags or rename them
+    result = ec2.create_tags(Resources=[instance_id],
+            Tags=[
+                {
+                    'Key': 'Type',
+                    'Value': 'jobmanager-ondemand'
+                },
+                {
+                    'Key': 'Name',
+                    'Value': 'JobManager'
+                }
+                ]
+            )
+
+if __name__ == '__main__':
+    logger = getLogger(__name__)
+    main()