Skip to content

Latest commit

 

History

History
 
 

grok

README.rst

Grok

Grok is an application for monitoring IT infrastructure and notifying on anomalous behavior.

Setup

Dependencies:

cd ~/nta/grok

NOTE: Do not use sudo with pip.

Python dependencies are documented in requirements.txt and installed automatically with grok.

python setup.py develop --install-dir="$HOME/Library/Python/2.7/lib/python/site-packages"

The install directory can be any directory on your PYTHONPATH.

If the above does not work for you, try the older steps (pip 1.4.1 required):

pip install --install-option="--prefix=$NUPIC_ENV" -e .

Config

Bootstrap your Grok configuration files and initialize the database:

NOTE: Remember to set APPLICATION_CONFIG_PATH environment variable to the directory where generated Grok configuration files should be stored by python setup.py init. If not set, it presently defaults to the location of Grok configuraiton template files (e.g., grok/conf/)

You'll need to install/start MySQL server prior to running setup.

python setup.py init

Review your grok configuration file and make changes as necessary.

Delete old checkpoints:

If you have an old copy of grok then you need to clean up the old model checkpoints.

rm -fr ~/grok_model_checkpoints/

Common Grok Environment Variables

APPLICATION_CONFIG_PATH: directory path where active Grok application configuration files are located

AWS_ACCESS_KEY_ID: The AWS access key

AWS_SECRET_ACCESS_KEY: The AWS secret key

GROK_API_KEY: If defined, grok's setup.py init uses its value to initialize the [security] apikey setting in grok's application.conf; if not defined, grok's setup.py init will generate the API key automatically.

GROK_HOME: the directory path of the parent directory of the grok package directory; the pipeline code relies on this environment variable being set. The application code falls back to a computed path if GROK_HOME is not in the environment.

GROK_LOG_DIR: Directory path where application logs should be stored; application code falls back to the parent directory of grok's package directory.

Grok Environment Variables Specific to the Build System

JOB_NAME: Name of the build job; used by build pipeline and behavioral_tests_driver;

BUILD_NUMBER: This is set automatically by Jenkins to manage the workspace. If this is not set, the current SHA will be used by the pipeline.

WORKSPACE: This is an internal use parameter that is automatically set by Jenkins and _**should not be populated manually**_. It defines where on the machine Jenkins will checkout the repository and run the pipeline.

BUILD_WORKSPACE: This is an optional parameter. If pipeline does not find BUILD_WORKSPACE it will create one for you inside WORKSPACE as follows: ${WORKSPACE}/${BUILD_NUMBER}. Setting this env var ensures you build in a specific location if needed.

JENKINS_HOME: This is automatically set by Jenkins and should not be populated manually.

PRODUCTS: directory path of the products tree; used by the build pipeline

Run

Start MySQL

Individual configurations may vary. Be sure to start MySQL however best works with the installation path you followed.

Start RabbitMQ

rabbitmq-server -detached

Reset RabbitMQ

If you have an old copy of grok then you need to clean up RabbitMQ queues.

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

Start nGinx

sudo nginx -p . -c conf/grok-api.conf

Start Grok Services

supervisord -c conf/supervisord.conf

Use Grok

Test

Setup AWS Credentials for Integration Tests

AWS Credentials needs to be setup for Grok application before running integration tests. You can either use the Grok Web UI to set those up or use Grok CLI to do the same.

Specify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY CLI options.

grok credentials GROK_SERVER_URL --AWS_ACCESS_KEY_ID=... --AWS_SECRET_ACCESS_KEY=...

Set AWS credentials from a specific file using the -d, or --data CLI options.

grok credentials GROK_SERVER_URL -d PATH_TO_FILE
grok credentials GROK_SERVER_URL --data=PATH_TO_FILE

For more details refer Grok CLI readme.

Python

Run Python unit tests:

./run_tests.sh -l py

Run Python integration tests:

# Prepare for integration tests by restarting supervisor
# and initializing.
supervisorctl shutdown
python setup.py init
./bin/set_edition.py standard
./bin/update_quota.py
supervisord -c conf/supervisord.conf
# Run with --num=X option to multithread. (TODO: MER-2177)
./run_tests.sh -l py -i

Javascript

Install NodeJS and NPM.

If using homebrew:

brew install node

Install node dev/test npm module dependencies:

npm install

Run Javascript tests locally on dev laptop (Mac OS X):

open tests/js/unit/*.html

Run Javascript tests on SauceLabs Browser testing cloud (logs stored in /tmp):

export SAUCE_USERNAME=<username>
export SAUCE_ACCESS_KEY=<accesskey>
./run_tests.sh -l js

Documentation

Complete documentation can be built using Sphinx:

python setup.py build_sphinx

Once built, docs will be in build/sphinx/html. Periodically, you should run sphinx-apidoc and commit new .rst files that it creates:

sphinx-apidoc -f -o docs/ grok
git add docs/*.rst
git commit