Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Commit

Permalink
fixed several docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@kramasamy
kramasamy committed May 8, 2016
1 parent ba97081 commit 3a0854f
Show file tree
Hide file tree
Showing 8 changed files with 145 additions and 126 deletions.
56 changes: 26 additions & 30 deletions website/content/docs/contributors/codebase.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: The Heron Codebase
title: Heron Code Organization
---

This document contains information about the Heron codebase intended primarily
Expand All @@ -14,16 +14,15 @@ cluster, see [Building Topologies](../developers/topologies.html) instead.
The primary programming languages for Heron are C++, Java, and Python.

* **C++ 11** is used for most of Heron's core components, including the
[Topology Master](../../concepts/architecture#topology-master), [Stream
Manager](../../concepts/architecture#stream-manager), and [Metrics
Manager](../../concepts/architecture#metrics-manager).
[Topology Master](../../concepts/architecture#topology-master), and
[Stream Manager](../../concepts/architecture#stream-manager).

* **Java 8** is used primarily for Heron's [topology
API](../../concepts/topologies) and is currently the only language in which
topologies can be written. Instructions can be found in [Building
Topologies](../developers/topologies.html), while API documentation for the Java
API can be found [here](../api/topology/index.html). Please note that Heron
topologies do not require Java 8 and can be written in Java 7 or later.
API](../../concepts/topologies), and [Heron Instance](../../concepts/architecture#heron-instance).
It is currently the only language in which topologies can be written. Instructions can be found
in [Building Topologies](../developers/topologies.html), while API documentation for the Java
API can be found [here](../api/topology/index.html). Please note that Heron topologies do not
require Java 8 and can be written in Java 7 or later.

* **Python 2** (specifically 2.7) is used primarily for Heron's [CLI
interface](../..//operators/heron-cli) and UI components such as [Heron
Expand All @@ -42,7 +41,7 @@ communication between components. Most `.proto` definition files can be found in
[`heron/proto`]({{% githubMaster %}}/heron/proto).

* **Cluster coordination** — Heron relies heavily on ZooKeeper for cluster
coordination, be it for [Mesos/Aurora](../../operators/deployment/aurora),
coordination for distributed deployment, be it for [Mesos/Aurora](../../operators/deployment/aurora),
[Mesos alone](../../operators/deployment/mesos), or for a [custom
scheduler](../custom-scheduler) that you build. More information on ZooKeeper
components in the codebase can be found in the [State
Expand All @@ -61,7 +60,7 @@ Heron supports three cluster schedulers out of the box:
[Aurora](../../operators/deployment/aurora), and a [local
scheduler](../../operators/deployment/local). The Java code for each of those
schedulers, as well as for the underlying scheduler API, can be found in
[`heron/scheduler`]({{% githubMaster %}}/heron/scheduler).
[`heron/schedulers`]({{% githubMaster %}}/heron/schedulers).

Info on custom schedulers can be found in [Implementing a Custom
Scheduler](../custom-scheduler); info on the currently available schedulers
Expand Down Expand Up @@ -142,28 +141,13 @@ Java. Those examples can be found in
Heron has a tool called `heron` that is used to both provide a CLI interface
for [managing topologies](../../operators/heron-cli) and to perform much of
the heavy lifting behind assembling physical topologies in your cluster.

The Python code for `heron` can be found in
[`heron/cli2`]({{% githubMaster %}}/heron/cli2). The entire logic of `heron` is
contained in [`cli.py`]({{% githubMaster %}}/heron/cli2/src/python/cli.py).

The default configuration for Heron schedulers is found in
[`scheduler.conf`]({{% githubMaster %}}/heron/cli2/src/python/scheduler.conf),
while configuration for Heron's [local
scheduler](../../operators/deployment/local) can be found in
[`local_scheduler.conf`]({{% githubMaster %}}/heron/cli2/src/python/local_scheduler.conf).

### Heron UI
[`heron/cli`]({{% githubMaster %}}/heron/cli).

The Python code for the [Heron UI](../../operators/heron-ui) can be found in
[`heron/web`]({{% githubMaster %}}/heron/web).
Sample configurations for different Heron schedulers

Like Heron Tracker, Heron UI is a web server written in Python that relies on
the [Tornado](http://www.tornadoweb.org/en/stable/) framework. You can add new
HTTP routes to Heron UI in
[`main.py`]({{% githubMaster %}}/heron/web/source/python/main.py) and corresponding
handlers in the [`handlers`]({{% githubMaster %}}/heron/web/source/python/handlers)
directory.
* [Local scheduler](../../operators/deployment/local) config can be found in [`heron/config/src/yaml/conf/local`]({{% githubMaster %}}/heron/config/src/yaml/conf/local),
* [Aurora scheduler](../../operators/deployment/aurora) config can be found [`heron/config/src/yaml/conf/aurora`]({{% githubMaster %}}/heron/config/src/yaml/conf/aurora).

### Heron Tracker

Expand All @@ -177,6 +161,18 @@ routes to the Tracker in
corresponding handlers in the
[`handlers`]({{% githubMaster %}}/heron/tracker/src/python/handlers) directory.

### Heron UI

The Python code for the [Heron UI](../../operators/heron-ui) can be found in
[`heron/ui`]({{% githubMaster %}}/heron/ui).

Like Heron Tracker, Heron UI is a web server written in Python that relies on
the [Tornado](http://www.tornadoweb.org/en/stable/) framework. You can add new
HTTP routes to Heron UI in
[`main.py`]({{% githubMaster %}}/heron/web/source/python/main.py) and corresponding
handlers in the [`handlers`]({{% githubMaster %}}/heron/web/source/python/handlers)
directory.

### Heron Shell

The Python code for the [Heron Shell](../../operators/heron-shell) can be
Expand Down
22 changes: 20 additions & 2 deletions website/content/docs/contributors/testing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Heron uses [Bazel](../../developers/compiling#installing-bazel) for building
and running unit tests. Before running tests, first set up your build environment
as described in [Compiling Heron](../../developers/compiling/compiling).

## Running Tests
### Running Unit Tests

The following command will run all tests:

Expand All @@ -21,7 +21,7 @@ target](http://bazel.io/docs/test-encyclopedia.html), pass the test target name.
$ bazel test --config=darwin heron/statemgrs/tests/java:localfs-statemgr_unittest
```

## Discovering Test Targets
### Discovering Unit Test Targets

To see a full listing of all Bazel test targets:

Expand All @@ -46,3 +46,21 @@ For **Python** targets:
```bash
$ bazel query 'kind("pex_test rule", ...)'
```

### Running Integration Tests

Integration tests are divided into two categories

* Functional integration tests
* Failure integration tests

To run the failure integration tests on your Mac OS X, do the following:

```
bazel run --config=darwin -- scripts/packages:heron-client-install.sh --user
bazel build --config=dawrin integration-test/src/...
python integration-test/src/python/local_test_runner/main.py
```

102 changes: 50 additions & 52 deletions website/content/docs/developers/compiling/docker.md
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
---
title: Docker
title: Compiling With Docker
---

To use Heron, you will need to compile it for the environment that you
For developing Heron, you will need to compile it for the environment that you
want to use it in. If you'd like to use Docker to create that build environment,
Heron provides a convenient script to make that process easier.
Heron provides a convenient script to make that process easier.

Currently, only Ubuntu 14.04 and CentOS 7 are supported, but if you need another
platform there are instructions for adding new ones
Currently, only Ubuntu 14.04, Ubuntu 15.10, and CentOS 7 are supported, but if you
need another platform there are instructions for adding new ones
[below](#contributing-new-environments).

## Requirements
### Requirements

* [Docker](https://docs.docker.com)

## Recommendations for Running Docker in a Virtual Machine
### Running Docker in a Virtual Machine

If you are running Docker in a virtual machine (VM), it is recommended that you
adjust your settings to help speed up the build. To do this, open
Expand All @@ -40,14 +40,14 @@ $ docker/build-artifacts.sh
Running the script by itself will display usage information:

```
Usage: docker/build-artifacts.sh <platform> <version-string> [source-tarball] <output-directory>
Platforms: ubuntu14.04, centos7
Usage: docker/build-artifacts.sh <platform> <version_string> [source-tarball] <output-directory>
Platforms Supported: darwin, ubuntu14.04, ubuntu15.10, centos7
Example:
docker/build-artifacts.sh ubuntu14.04 0.1.0-SNAPSHOT ~/heron-release
NOTE: If running on OSX, the output directory will need to
./build-artifacts.sh ubuntu14.04 0.12.0 .
NOTE: If running on OSX, the output directory will need to
be under /Users so virtualbox has access to.
```

Expand All @@ -64,7 +64,7 @@ The following arguments are required:
Here's an example usage:

```bash
$ docker/build-artifacts.sh ubuntu14.04 0.1.0-SNAPSHOT ~/heron-release
$ docker/build-artifacts.sh ubuntu14.04 0.12.0 ~/heron-release
```

This will build a Docker container specific to Ubuntu 14.04, create a source
Expand All @@ -87,13 +87,13 @@ of the generated artifacts:

```bash
$ ls ~/heron-release
heron-0.1.0-SNAPSHOT.tar.gz
heron-api-0.1.0-SNAPSHOT.tar.gz
heron-bin-0.1.0-SNAPSHOT.tar.gz
heron-cli-0.1.0-SNAPSHOT.tar.gz
heron-conf-0.1.0-SNAPSHOT.tar.gz
heron-core-0.1.0-SNAPSHOT.tar.gz
# etc
heron-api-0.12.0-ubuntu14.04.tar.gz
heron-client-0.12.0-ubuntu14.04.tar.gz
heron-tools-0.12.0-ubuntu14.04.tar.gz
heron-client-install-0.12.0-ubuntu.sh
heron-tools-install-0.12.0-ubuntu.sh
heron-api-install-0.12.0-ubuntu.sh
heron-core-0.12.0-ubuntu.tar.gz
```

## Contributing New Environments
Expand All @@ -112,26 +112,23 @@ documentation](https://docs.docker.com/engine/articles/dockerfile_best-practices
You should make sure that your `Dockerfile` specifies *at least* all of the
following:

1. The OS being used in a
[`FROM`](https://docs.docker.com/engine/reference/builder/#from) statement.
#### Step 1 - The OS being used in a [`FROM`](https://docs.docker.com/engine/reference/builder/#from) statement.
Here's an example:

```dockerfile
FROM centos:centos7
```
2. A `TARGET_PLATFORM` environment variable using the
[`ENV`](https://docs.docker.com/engine/reference/builder/#env) instruction.
```dockerfile
FROM centos:centos7
```
#### Step 2 - A `TARGET_PLATFORM` environment variable using the [`ENV`](https://docs.docker.com/engine/reference/builder/#env) instruction.
Here's an example:
```dockerfile
ENV TARGET_PLATFORM centos
```
3. A general dependency installation script using a
[`RUN`](https://docs.docker.com/engine/reference/builder/#run) instruction.
```dockerfile
ENV TARGET_PLATFORM centos
```
#### Step 3 - A general dependency installation script using a [`RUN`](https://docs.docker.com/engine/reference/builder/#run) instruction.
Here's an example:

```dockerfile
RUN apt-get update && apt-get -y install \
```dockerfile
RUN apt-get update && apt-get -y install \
automake \
build-essential \
cmake \
Expand All @@ -148,35 +145,36 @@ following:
python-setuptools \
unzip \
wget
```
```

4. An installation script for Java 8 and a `JAVA_HOME` environment variable.
#### Step 4 - An installation script for Java 8 and a `JAVA_HOME` environment variable.
Here's an example:

```dockerfile
RUN \
```dockerfile
RUN \
echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
add-apt-repository -y ppa:webupd8team/java && \
apt-get update && \
apt-get install -y oracle-java8-installer && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
```

5. An installation script for [Bazel](http://bazel.io/) version 0.1.2 or
above. Here's an example:
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
```

#### Step 5 - An installation script for [Bazel](http://bazel.io/) version 0.1.2 or above.
Here's an example:

```dockerfile
RUN wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-linux-x86_64.sh \
```dockerfile
RUN wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-linux-x86_64.sh \
&& chmod +x /tmp/bazel.sh \
&& /tmp/bazel.sh
```
```

6. Add the `bazelrc` configuration file for Bazel and the `compile.sh`
#### Step 6 - Add the `bazelrc` configuration file for Bazel and the `compile.sh`
script (from the `docker` folder) that compiles Heron:

```dockerfile
ADD bazelrc /root/.bazelrc
ADD compile.sh /compile.sh
```
```dockerfile
ADD bazelrc /root/.bazelrc
ADD compile.sh /compile.sh
```
Loading

0 comments on commit 3a0854f

Please sign in to comment.