Spark Mllib Performance Tests for OpenShift

This is based from the databricks spark-perf test harness

It runs on Openshift and uses pbench to collect telemetry data for.

It assumes you have already build your spark cluster in OpenShift.

peformance analyses of the entire cluster while each benchmark is running.

Pbench archives the performance data in elastic search

IMPORTANT: Pbench agents must be installed on all nodes in the spark cluster before running these tests

This test harness must be run by 'root' because pbench requires root privileges

The following machine learning algorithms are run with the config settings in config/config.py

• Machine Learning
  • glm-regression: Generalized Linear Regression Model
  • glm-classification: Generalized Linear Classification Model
  • naive-bayes: Naive Bayes
  • als: Alternating Least Squares
  • kmeans: K-Means clustering
  • pearson: Pearson's Correlation
  • spearman: Spearman's Correlation