diff --git a/website/content/docs/contributors/codebase.md b/website/content/docs/contributors/codebase.md index c54736963a3..653c5702e0c 100644 --- a/website/content/docs/contributors/codebase.md +++ b/website/content/docs/contributors/codebase.md @@ -1,5 +1,5 @@ --- -title: The Heron Codebase +title: Heron Code Organization --- This document contains information about the Heron codebase intended primarily @@ -14,16 +14,15 @@ cluster, see [Building Topologies](../developers/topologies.html) instead. The primary programming languages for Heron are C++, Java, and Python. * **C++ 11** is used for most of Heron's core components, including the -[Topology Master](../../concepts/architecture#topology-master), [Stream -Manager](../../concepts/architecture#stream-manager), and [Metrics -Manager](../../concepts/architecture#metrics-manager). +[Topology Master](../../concepts/architecture#topology-master), and +[Stream Manager](../../concepts/architecture#stream-manager). * **Java 8** is used primarily for Heron's [topology -API](../../concepts/topologies) and is currently the only language in which -topologies can be written. Instructions can be found in [Building -Topologies](../developers/topologies.html), while API documentation for the Java -API can be found [here](../api/topology/index.html). Please note that Heron -topologies do not require Java 8 and can be written in Java 7 or later. +API](../../concepts/topologies), and [Heron Instance](../../concepts/architecture#heron-instance). +It is currently the only language in which topologies can be written. Instructions can be found +in [Building Topologies](../developers/topologies.html), while API documentation for the Java +API can be found [here](../api/topology/index.html). Please note that Heron topologies do not +require Java 8 and can be written in Java 7 or later. * **Python 2** (specifically 2.7) is used primarily for Heron's [CLI interface](../..//operators/heron-cli) and UI components such as [Heron @@ -42,7 +41,7 @@ communication between components. Most `.proto` definition files can be found in [`heron/proto`]({{% githubMaster %}}/heron/proto). * **Cluster coordination** — Heron relies heavily on ZooKeeper for cluster -coordination, be it for [Mesos/Aurora](../../operators/deployment/aurora), +coordination for distributed deployment, be it for [Mesos/Aurora](../../operators/deployment/aurora), [Mesos alone](../../operators/deployment/mesos), or for a [custom scheduler](../custom-scheduler) that you build. More information on ZooKeeper components in the codebase can be found in the [State @@ -61,7 +60,7 @@ Heron supports three cluster schedulers out of the box: [Aurora](../../operators/deployment/aurora), and a [local scheduler](../../operators/deployment/local). The Java code for each of those schedulers, as well as for the underlying scheduler API, can be found in -[`heron/scheduler`]({{% githubMaster %}}/heron/scheduler). +[`heron/schedulers`]({{% githubMaster %}}/heron/schedulers). Info on custom schedulers can be found in [Implementing a Custom Scheduler](../custom-scheduler); info on the currently available schedulers @@ -142,28 +141,13 @@ Java. Those examples can be found in Heron has a tool called `heron` that is used to both provide a CLI interface for [managing topologies](../../operators/heron-cli) and to perform much of the heavy lifting behind assembling physical topologies in your cluster. - The Python code for `heron` can be found in -[`heron/cli2`]({{% githubMaster %}}/heron/cli2). The entire logic of `heron` is -contained in [`cli.py`]({{% githubMaster %}}/heron/cli2/src/python/cli.py). - -The default configuration for Heron schedulers is found in -[`scheduler.conf`]({{% githubMaster %}}/heron/cli2/src/python/scheduler.conf), -while configuration for Heron's [local -scheduler](../../operators/deployment/local) can be found in -[`local_scheduler.conf`]({{% githubMaster %}}/heron/cli2/src/python/local_scheduler.conf). - -### Heron UI +[`heron/cli`]({{% githubMaster %}}/heron/cli). -The Python code for the [Heron UI](../../operators/heron-ui) can be found in -[`heron/web`]({{% githubMaster %}}/heron/web). +Sample configurations for different Heron schedulers -Like Heron Tracker, Heron UI is a web server written in Python that relies on -the [Tornado](http://www.tornadoweb.org/en/stable/) framework. You can add new -HTTP routes to Heron UI in -[`main.py`]({{% githubMaster %}}/heron/web/source/python/main.py) and corresponding -handlers in the [`handlers`]({{% githubMaster %}}/heron/web/source/python/handlers) -directory. +* [Local scheduler](../../operators/deployment/local) config can be found in [`heron/config/src/yaml/conf/local`]({{% githubMaster %}}/heron/config/src/yaml/conf/local), +* [Aurora scheduler](../../operators/deployment/aurora) config can be found [`heron/config/src/yaml/conf/aurora`]({{% githubMaster %}}/heron/config/src/yaml/conf/aurora). ### Heron Tracker @@ -177,6 +161,18 @@ routes to the Tracker in corresponding handlers in the [`handlers`]({{% githubMaster %}}/heron/tracker/src/python/handlers) directory. +### Heron UI + +The Python code for the [Heron UI](../../operators/heron-ui) can be found in +[`heron/ui`]({{% githubMaster %}}/heron/ui). + +Like Heron Tracker, Heron UI is a web server written in Python that relies on +the [Tornado](http://www.tornadoweb.org/en/stable/) framework. You can add new +HTTP routes to Heron UI in +[`main.py`]({{% githubMaster %}}/heron/web/source/python/main.py) and corresponding +handlers in the [`handlers`]({{% githubMaster %}}/heron/web/source/python/handlers) +directory. + ### Heron Shell The Python code for the [Heron Shell](../../operators/heron-shell) can be diff --git a/website/content/docs/contributors/testing.md b/website/content/docs/contributors/testing.md index 2fe6085a02d..6cc183bf485 100644 --- a/website/content/docs/contributors/testing.md +++ b/website/content/docs/contributors/testing.md @@ -6,7 +6,7 @@ Heron uses [Bazel](../../developers/compiling#installing-bazel) for building and running unit tests. Before running tests, first set up your build environment as described in [Compiling Heron](../../developers/compiling/compiling). -## Running Tests +### Running Unit Tests The following command will run all tests: @@ -21,7 +21,7 @@ target](http://bazel.io/docs/test-encyclopedia.html), pass the test target name. $ bazel test --config=darwin heron/statemgrs/tests/java:localfs-statemgr_unittest ``` -## Discovering Test Targets +### Discovering Unit Test Targets To see a full listing of all Bazel test targets: @@ -46,3 +46,21 @@ For **Python** targets: ```bash $ bazel query 'kind("pex_test rule", ...)' ``` + +### Running Integration Tests + +Integration tests are divided into two categories + +* Functional integration tests +* Failure integration tests + +To run the failure integration tests on your Mac OS X, do the following: + +``` +bazel run --config=darwin -- scripts/packages:heron-client-install.sh --user + +bazel build --config=dawrin integration-test/src/... + +python integration-test/src/python/local_test_runner/main.py +``` + diff --git a/website/content/docs/developers/compiling/docker.md b/website/content/docs/developers/compiling/docker.md index f86a57c5082..1d9bb06732f 100644 --- a/website/content/docs/developers/compiling/docker.md +++ b/website/content/docs/developers/compiling/docker.md @@ -1,20 +1,20 @@ --- -title: Docker +title: Compiling With Docker --- -To use Heron, you will need to compile it for the environment that you +For developing Heron, you will need to compile it for the environment that you want to use it in. If you'd like to use Docker to create that build environment, -Heron provides a convenient script to make that process easier. +Heron provides a convenient script to make that process easier. -Currently, only Ubuntu 14.04 and CentOS 7 are supported, but if you need another -platform there are instructions for adding new ones +Currently, only Ubuntu 14.04, Ubuntu 15.10, and CentOS 7 are supported, but if you +need another platform there are instructions for adding new ones [below](#contributing-new-environments). -## Requirements +### Requirements * [Docker](https://docs.docker.com) -## Recommendations for Running Docker in a Virtual Machine +### Running Docker in a Virtual Machine If you are running Docker in a virtual machine (VM), it is recommended that you adjust your settings to help speed up the build. To do this, open @@ -40,14 +40,14 @@ $ docker/build-artifacts.sh Running the script by itself will display usage information: ``` -Usage: docker/build-artifacts.sh [source-tarball] - -Platforms: ubuntu14.04, centos7 - +Usage: docker/build-artifacts.sh [source-tarball] + +Platforms Supported: darwin, ubuntu14.04, ubuntu15.10, centos7 + Example: - docker/build-artifacts.sh ubuntu14.04 0.1.0-SNAPSHOT ~/heron-release - -NOTE: If running on OSX, the output directory will need to + ./build-artifacts.sh ubuntu14.04 0.12.0 . + +NOTE: If running on OSX, the output directory will need to be under /Users so virtualbox has access to. ``` @@ -64,7 +64,7 @@ The following arguments are required: Here's an example usage: ```bash -$ docker/build-artifacts.sh ubuntu14.04 0.1.0-SNAPSHOT ~/heron-release +$ docker/build-artifacts.sh ubuntu14.04 0.12.0 ~/heron-release ``` This will build a Docker container specific to Ubuntu 14.04, create a source @@ -87,13 +87,13 @@ of the generated artifacts: ```bash $ ls ~/heron-release -heron-0.1.0-SNAPSHOT.tar.gz -heron-api-0.1.0-SNAPSHOT.tar.gz -heron-bin-0.1.0-SNAPSHOT.tar.gz -heron-cli-0.1.0-SNAPSHOT.tar.gz -heron-conf-0.1.0-SNAPSHOT.tar.gz -heron-core-0.1.0-SNAPSHOT.tar.gz -# etc +heron-api-0.12.0-ubuntu14.04.tar.gz +heron-client-0.12.0-ubuntu14.04.tar.gz +heron-tools-0.12.0-ubuntu14.04.tar.gz +heron-client-install-0.12.0-ubuntu.sh +heron-tools-install-0.12.0-ubuntu.sh +heron-api-install-0.12.0-ubuntu.sh +heron-core-0.12.0-ubuntu.tar.gz ``` ## Contributing New Environments @@ -112,26 +112,23 @@ documentation](https://docs.docker.com/engine/articles/dockerfile_best-practices You should make sure that your `Dockerfile` specifies *at least* all of the following: -1. The OS being used in a - [`FROM`](https://docs.docker.com/engine/reference/builder/#from) statement. +#### Step 1 - The OS being used in a [`FROM`](https://docs.docker.com/engine/reference/builder/#from) statement. Here's an example: - ```dockerfile - FROM centos:centos7 - ``` -2. A `TARGET_PLATFORM` environment variable using the - [`ENV`](https://docs.docker.com/engine/reference/builder/#env) instruction. +```dockerfile +FROM centos:centos7 + ``` +#### Step 2 - A `TARGET_PLATFORM` environment variable using the [`ENV`](https://docs.docker.com/engine/reference/builder/#env) instruction. Here's an example: - ```dockerfile - ENV TARGET_PLATFORM centos - ``` -3. A general dependency installation script using a - [`RUN`](https://docs.docker.com/engine/reference/builder/#run) instruction. +```dockerfile +ENV TARGET_PLATFORM centos +``` +#### Step 3 - A general dependency installation script using a [`RUN`](https://docs.docker.com/engine/reference/builder/#run) instruction. Here's an example: - ```dockerfile - RUN apt-get update && apt-get -y install \ +```dockerfile +RUN apt-get update && apt-get -y install \ automake \ build-essential \ cmake \ @@ -148,35 +145,36 @@ following: python-setuptools \ unzip \ wget - ``` +``` -4. An installation script for Java 8 and a `JAVA_HOME` environment variable. +#### Step 4 - An installation script for Java 8 and a `JAVA_HOME` environment variable. Here's an example: - ```dockerfile - RUN \ +```dockerfile +RUN \ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \ add-apt-repository -y ppa:webupd8team/java && \ apt-get update && \ apt-get install -y oracle-java8-installer && \ rm -rf /var/lib/apt/lists/* && \ rm -rf /var/cache/oracle-jdk8-installer - ENV JAVA_HOME /usr/lib/jvm/java-8-oracle - ``` -5. An installation script for [Bazel](http://bazel.io/) version 0.1.2 or - above. Here's an example: +ENV JAVA_HOME /usr/lib/jvm/java-8-oracle +``` + +#### Step 5 - An installation script for [Bazel](http://bazel.io/) version 0.1.2 or above. +Here's an example: - ```dockerfile - RUN wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-linux-x86_64.sh \ +```dockerfile +RUN wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-linux-x86_64.sh \ && chmod +x /tmp/bazel.sh \ && /tmp/bazel.sh - ``` +``` -6. Add the `bazelrc` configuration file for Bazel and the `compile.sh` +#### Step 6 - Add the `bazelrc` configuration file for Bazel and the `compile.sh` script (from the `docker` folder) that compiles Heron: - ```dockerfile - ADD bazelrc /root/.bazelrc - ADD compile.sh /compile.sh - ``` +```dockerfile +ADD bazelrc /root/.bazelrc +ADD compile.sh /compile.sh +``` diff --git a/website/content/docs/developers/compiling/linux.md b/website/content/docs/developers/compiling/linux.md index 25febac6aca..439df7f7c2f 100644 --- a/website/content/docs/developers/compiling/linux.md +++ b/website/content/docs/developers/compiling/linux.md @@ -6,28 +6,28 @@ title: Linux Platforms This is a step by step guide for building Heron on a fresh Ubuntu 14.04 installation. -First update Ubuntu. +#### Step 1 - First update Ubuntu. ```bash sudo apt-get update -y sudo apt-get upgrade -y ``` -Install required libraries such as build tools, automake, cmake, libtool, zip and libuwind +#### Step 2 - Install required libraries ```bash sudo apt-get install git build-essential automake cmake libtool zip \ libunwind-setjmp0-dev zlib1g-dev unzip pkg-config -y ``` -Set the following environment variables +#### Step 3 - Set the following environment variables ```bash export CC=/usr/bin/gcc-4.8 export CCX=/usr/bin/g++-4.8 ``` -Install JDK +#### Step 4 - Install JDK 8 ```bash sudo add-apt-repository ppa:webupd8team/java @@ -35,37 +35,39 @@ sudo apt-get update -y sudo apt-get install oracle-java8-installer -y ``` -Install Bazel +#### Step 5 - Install Bazel ```bash wget https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-linux-x86_64.sh chmod +x bazel-0.1.2-installer-linux-x86_64.sh ./bazel-0.1.2-installer-linux-x86_64.sh --user ``` -Make sure Bazel bin is in the PATH + +#### Step 6 - Make sure Bazel bin is in the PATH ```bash export PATH="$PATH:$HOME/bin" ``` -Get the latest version of heron + +#### Step 7 - Get the latest version of heron ```bash git clone https://github.com/twitter/heron.git && cd heron ``` -Configure Heron for build +#### Step 8 - Configure Heron for build ```bash ./bazel_configure.py ``` -Build the project +#### Step 9 - Build the project ```bash bazel build --config=ubuntu heron/... ``` -Build the packages +#### Step 10 - Build the packages ```bash bazel build --config=ubuntu scripts/packages:binpkgs @@ -74,20 +76,22 @@ bazel build --config=ubuntu scripts/packages:tarpkgs This will build the packages below the `bazel-bin/scripts/packages/` directory. +### Manually Installing Libraries + If you encounter errors with libunwind, libtool, or gperftools install them manually -Libtool https://www.gnu.org/software/libtool/ +Compiling and installing [libtool] (https://www.gnu.org/software/libtool) ```bash wget http://ftpmirror.gnu.org/libtool/libtool-2.4.6.tar.gz -tar -xvf libtool-2.4.6.tar.gz && cd libtool-2.4.6 +tar -xvf libtool-2.4.6.tar.gz +cd libtool-2.4.6 ./configure make sudo make install ``` -Libunwind http://www.nongnu.org/libunwind/ +Compiling and installing [libunwind] (http://www.nongnu.org/libunwind) ```bash -cd /home/ubuntu wget http://download.savannah.gnu.org/releases/libunwind/libunwind-1.1.tar.gz tar -xvf libunwind-1.1.tar.gz cd libunwind-1.1 @@ -95,16 +99,14 @@ cd libunwind-1.1 make sudo make install ``` -Install gperftools -https://github.com/gperftools/gperftools/releases +Compiling and installing [gperftools] (https://github.com/gperftools/gperftools/releases) ```bash -cd /home/ubuntu wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.5/gperftools-2.5.tar.gz tar -xvf gperftools-2.5.tar.gz cd gperftools-2.5 ./configure make sudo make install -``` \ No newline at end of file +``` diff --git a/website/content/docs/developers/compiling/mac.md b/website/content/docs/developers/compiling/mac.md index c2059d5c29e..7e781ba5938 100644 --- a/website/content/docs/developers/compiling/mac.md +++ b/website/content/docs/developers/compiling/mac.md @@ -1,21 +1,23 @@ --- -title: Max OS X (10.10) +title: Building on Mac OS X --- -Install brew +This is a step by step guide for building Heron on Mac OS (10.10 and 10.11). + +#### Step 1 - Install brew, if already not installed ```bash /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" ``` -Install the required libraries +#### Step 2 - Install the required libraries ```bash brew install automake brew install cmake ``` -Set the following environment variables +#### Step 3 - Set the following environment variables ```bash $ export CC=/usr/bin/clang @@ -23,7 +25,7 @@ $ export CXX=/usr/bin/clang++ $ echo $CC $CXX ``` -Install Bazel +#### Step 4 - Install Bazel ```bash curl -O -L https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-darwin-x86_64.sh @@ -31,34 +33,33 @@ chmod +x bazel-0.1.2-installer-linux-x86_64.sh ./bazel-0.1.2-installer-linux-x86_64.sh --user ``` -Make sure Bazel bin is in the PATH +#### Step 5 - Make sure Bazel bin is in the PATH ```bash export PATH="$PATH:$HOME/bin" ``` -Get the latest version of heron +#### Step 6 - Get the latest version of heron ```bash git clone https://github.com/twitter/heron.git && cd heron ``` -Configure Heron for build +#### Step 7 - Configure Heron for build ```bash ./bazel_configure.py ``` -If the configure scripts fails with missing dependencies, -brew can be used to install the dependencies. +If the configure scripts fails with missing dependencies, brew can be used to install the dependencies. -Build the project +#### Step 8 - Build the project ```bash bazel build --config=darwin heron/... ``` -Build the packages +#### Step 9 - Build the packages ```bash bazel build --config=darwin scripts/packages:binpkgs diff --git a/website/content/docs/getting-started.md b/website/content/docs/getting-started.md index ec8a8df56b1..c2a99ce516d 100644 --- a/website/content/docs/getting-started.md +++ b/website/content/docs/getting-started.md @@ -5,7 +5,7 @@ title: Getting Started - Local (Single Node) Run topologies locally using pre-compiled Heron binaries (Mac OSX, Ubuntu >= 14.04, Centos7) -### Step 1 - Download pre-compiled Heron binaries with install scripts +### Step 1 - Download Heron binaries with install scripts Navigate to [Twitter Heron Releases](https://github.com/twitter/heron/releases) and download the following self extracting binary install scripts for your platform. @@ -68,7 +68,7 @@ $ heron-ui ``` In local browser, Heron UI is available at http://localhost:8889 -### Step 5 - Explore activate, deactivate, and kill topology commands +### Step 5 - Explore topology management commands ```bash $ heron activate local ExclamationTopology diff --git a/website/content/docs/upgrade-storm-to-heron.md b/website/content/docs/upgrade-storm-to-heron.md index acb9605d57a..1bfc2a0c0f1 100644 --- a/website/content/docs/upgrade-storm-to-heron.md +++ b/website/content/docs/upgrade-storm-to-heron.md @@ -5,7 +5,7 @@ title: Upgrade Storm Topologies to Heron Heron is designed to be fully backward compatible with existing [Apache Storm](http://storm.apache.org/index.html) projects, allowing simple [Maven POM.xml](https://maven.apache.org/pom.html) changes to migrate existing Storm [topologies](../concepts/topologies). -### Step 1 - Download pre-compiled Heron API binaries with install script +### Step 1 - Download Heron API binaries with install script Navigate to [Twitter Heron Releases](https://github.com/twitter/heron/releases) and download the following self extracting binary API install script for your platform. @@ -119,4 +119,4 @@ $ heron submit local ~/.heron/examples/heron-examples.jar com.twitter.heron.exam [Deploying Existing topologies](../operators/deployment) in clustered, scheduler-driven environments (Aurora, Mesos, Local) -[Developing Topologies](../concepts/architecture) with the Architecture of Heron \ No newline at end of file +[Developing Topologies](../concepts/architecture) with the Architecture of Heron diff --git a/website/data/toc.yaml b/website/data/toc.yaml index df60b1408d1..59387fadf44 100644 --- a/website/data/toc.yaml +++ b/website/data/toc.yaml @@ -57,7 +57,7 @@ sections: url: /docs/operators/heron-tracker - name: Heron UI url: /docs/operators/heron-ui - - name: Topology Developers + - name: Topology Writers sublinks: - name: Heron Topologies url: /docs/concepts/topologies @@ -73,10 +73,6 @@ sections: url: /docs/javadoc - name: Heron Developers sublinks: - - name: Heron Architecture - url: /docs/concepts/architecture - - name: Codebase - url: /docs/contributors/codebase - name: Compiling url: /docs/developers/compiling/compiling - name: Compiling on Linux @@ -85,11 +81,19 @@ sections: url: /docs/developers/compiling/mac - name: Compiling With Docker url: /docs/developers/compiling/docker - - name: Testing + - name: Running Tests url: /docs/contributors/testing - - name: Implementing a Custom Scheduler + - name: Code Organization + url: /docs/contributors/codebase + - name: Heron Concepts + sublinks: + - name: Heron Architecture + url: /docs/concepts/architecture + - name: Extending Heron + sublinks: + - name: Custom Scheduler url: /docs/contributors/custom-scheduler - - name: Implementing a Custom Metrics Sink + - name: Custom Metrics Sink url: /docs/contributors/custom-metrics-sink - name: Heron Resources sublinks: