Simple Hadoop Yarn (With Use Case: Send Email using Yarn)
Tested on -
- Mac OS X 10.9 using java version "1.7.0_40"
and
- Hortonworks Sandbox Version 2.0
(http://hortonworks.com/products/hortonworks-sandbox/)
using java version "1.6.0_24"
PREREQ:
You need to have installed Hadoop 2 on your Mac,
or be using Hortonworks Sandbox Version 2.0
NOTE:
Prior knowledge of Hadoop source code,
especially within this area helps -
hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
RUNNING:
Just run the BASH script (run-simple-yarn-app.sh)
It will do a maven build, and then run the yarn client.
USAGE:
$ ./run-simple-yarn-app.sh
=> Usage: ./run-simple-yarn-app.sh -c <unix_command> -n <number_of_yarn_containers>
SAMPLE RUN:
$ ./run-simple-yarn-app.sh -c /usr/bin/who -n 9
EXTRAS_1 (Use Case: Send Email Demo):
Want to send email from all the Yarn Containers?
Place the sendemail ruby script in accessible location
(/bin, /usr/bin, or some other standard location of PATH,
or mention the full path in run script).
$ ./run-simple-yarn-app.sh -c 'sendemail <your_email_address> <your_email_password>' -n 9
EXTRAS_2 (Use Case: Let Yarn Container run MapReduce Job):
Want to run WordCount MapReduce job under Yarn by dedicating a container for it?
$ ./run-simple-yarn-app.sh -c "/bin/cp $LOCAL_DIRS/*/appMaster.jar /tmp; hadoop --config /tmp/hadoop_mr_cfg jar /tmp/appMaster.jar com.hw.yarn.WordCount in out" -n 1