Launching Cloudera Manager with Whirr
=====================================

Follow these instructions to start a cluster on EC2 running Cloudera Manager.
Cloudera Manager Express allows you to install, run, and manage a Hadoop cluster.

This method uses Whirr to start a cluster with
 * one node running the Cloudera Manager Admin Console, and
 * a user-selectable number of nodes for the Hadoop cluster itself

Once Whirr has started the cluster, you use Cloudera Manager in the usual way.

Note that you can omit the CDH client node if you want to run programs entirely
from Hue.

It is not currently possible to launch Hadoop programs from your local machine
that use the cluster, due to the way Cloudera Manager manages host addresses.
To work around this limitation you can add a Gateway role which installs CDH
client components on a node in the cloud to run programs from.

1. Install Whirr
================

Run the following commands from you local machine.

Set your AWS credentials as environment variables:

% export AWS_ACCESS_KEY_ID=...
% export AWS_SECRET_ACCESS_KEY=...

Download and install Whirr:

% curl -O http://www.apache.org/dist/incubator/whirr/whirr-0.7.1-incubating/whirr-0.7.1-incubating.tar.gz
% tar zxf whirr-0.7.1-incubating.tar.gz
% export PATH=$PATH:$(pwd)/whirr-0.7.1-incubating/bin

Create a password-less SSH keypair for Whirr to use:

% ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_cm

2. Install the Whirr Cloudera Manager Service Plugin
====================================================

Download the plugin from

https://github.com/tomwhite/whirr-cm/archives/master

and copy it into the lib directory of your Whirr installation.

3. Launch a Cloudera Manager Cluster
====================================

The following command will start a cluster with 5 Hadoop nodes. To change this
number edit the cm-ec2.properties file. Edit the same file if you don't want to
launch a CDH client node.

% whirr launch-cluster --config cm-ec2.properties

Whirr will report progress to the console as it runs. The command will exit when
the cluster is ready to be used.

4. Configure the Hadoop cluster
===============================

The next step is to run the Cloudera Manager Admin Console -- at the URL printed
by the Whirr command -- to install and configure Hadoop, using the instructions
at

https://ccp.cloudera.com/display/FREE400BETA/Automated+Installation+of+Cloudera+Manager+and+CDH

The output of the Whirr command includes settings for the cluster hosts
and the authentication method to be used while running the Cloudera Manager
Admin Console.

5. Use the cluster
==================

Once the Hadoop cluster is up and running you can use it via Hue (the URL
is printed by the launch cluster command), or from a CDH gateway machine. In
the latter case, follow these instructions to add a gateway role

https://ccp.cloudera.com/display/FREE400BETA/Adding+Role+Instances

Then SSH to the gateway machine. Now you can interact with the cluster,
e.g. to list files in HDFS:

% hadoop fs -ls /tmp

6. Shutdown the cluster
=======================

Finally, when you want to shutdown the cluster, run the following command. Note
that all data and state stored on the cluster will be lost.

% whirr destroy-cluster --config cm-ec2.properties