API V3

This documentation is on a best efforts basis. Please do not use the management tools without prior instruction from the developer team.

Requirements for running local server:

• mongodb instance running on localhost without auth.
• redis running on localhost
• python-dev

Installation: Before running the server or management commands locally you must do the following. The env.sh file is pinned to the api channel in Slack.

git clone ssh://github.com/YoApp/api/api.git # (first time)
virtualenv --no-site-packages venv # (first time)
. venv/bin/activate # (In between terminal/computer restarts)
pip install -r requirements.txt # (first time and when its updated)
. /path/to/env.sh

Running the server:

# Against local databases
python server.py --config yoapi.config.Default

# Against production databases
python server.py --config yoapi.config.Production

Running the workers:

# Against local databases
python worker.py --config yoapi.config.Default --pool-size 10 --queue [queue]

# Against production databases and local redis
python worker.py --config yoapi.config.LocalRedis --pool-size 10 --queue [queue]

# Against production databases and production redis
# NOTE: Doing this will consume production worker jobs. That is, your local machine
# will process yos/devices that are hitting the Production server. This should be done
# very rarely and only if you know EXACTLY what you are doing.
python worker.py --config yoapi.config.Production --pool-size 10 --queue [queue]

Update staging server at api-dev.herokuapp.com

heroku git:remote -a api-dev (first time only)
git push [-f] heroku [branch]:master

Update live server at newapi.justyo.co:

# Update master branch.
git push origin master

# CircleCI performs nosetests and pushes update to Heroku.
open https://circleci.com/gh/YoApp/api  

# NOTE: It is highly discouraged to push directly to production.
# There are a lot of tests and you never know what you may break.

Runtime arguments for gunicorn on heroku are specified in:

• bin/web
• bin/workerlow
• bin/workermedium
• bin/workerhigh
• bin/scheduler

Redis Cache cluster The redis cache cluster is implemented with (Twitter Twemproxy)[https://github.com/twitter/twemproxy]. In order to enable this on heroku a (custom buildpack was created)[https://github.com/YoApp/heroku-buildpack-python-twemproxy/commits/master] that bootstraps twemproxy on heroku dynos. The redis servers in the cluster are controlled via environment variables and are compiled into the twemproxy config file on app start. The config file is generated by (this script)[https://github.com/YoApp/api/blob/master/nutcracker_run.sh]. In order for any new app/process to use the production cache cluster it MUST use this custom buildpack and a ProcFile configuration similar to those in the bin folder of this repo.

Process descriptions:

• web - The main web process receiving requests.

run: python server.py --config yoapi.config.Default

• workerlow - The backend worker process connected to low Redis Queue. This process is mainly responsible for sending yos.

run: python worker.py --config yoapi.config.LocalRedis --pool-size 10 --queue low

• workermedium - The backend worker process connected to medium Redis Queue. This process is mainly responsible for queuing broadcast yos to send.

run: python worker.py --config yoapi.config.LocalRedis --pool-size 10 --queue medium

• workerhigh - The backend worker process connected to high Redis Queue. This process is mainly responsible for (un)registering devices.

run: python worker.py --config yoapi.config.LocalRedis --pool-size 10 --queue high

• scheduler - The backend process responsible for sending yos in the future.

run: python scheduler.py --config yoapi.config.LocalRedis

Config descriptions:

• Default - Run everything against your local computer (redis, mongodb, ratelimiter, cache, redis-queues).
• Production - Run everything against production (redis, mongodb, ratelimiter, cache, redis-queues).

NOTE: The Production redis cache you will be connecting to is NOT the same as the cache cluster used for production.

• ProductionWithPrettyJson - Same as Production with pretty json output.
• LocalRedis - Run production with pretty json with the your local computer's redis (redis, ratelimiter, redis-queues)

Useful commands To trail the logs in pretty print mode, ensure brew install node, make sure you have the heroku heroku toolbelt installed and do the following:

heroku login
heroku git:clone -a api-dev
npm install -g underscore-cli
cd api-dev
heroku logs -s app -n 100 --tail |  cut -d" " -f3- | while read x; do echo $x | underscore print; done

There is a Flask-Script manager in the API Flask application. It is used to run arbitrary jobs in the context of a running server, constructing both application contexts and making virtual requests.

NOTE: Most if not all of the scripts are either outdated, deprecated, and/or made for one time use. However, the shells are very handy and can be used freely. Some examples are below.

# Install dependencies using pip
pip install -r requirements.txt

# Enter a shell and create indexes on a collection.
>>> from yoapi import models
>>> models.User.ensure_indexes()
>>> quit()

# Enter a shell and query the db.
>>> from yoapi import models
>>> models.User.objects(username='BUZZFEED').get()
<User: 5489bea8f4a5052f69616801>
>>> quit()

# Enter a shell as a specific user.
python manage.py --config yoapi.config.Production --impersonate BUZZFEED loggedinyoshell
Now impersonating BUZZFEED: ObjectId('5489bea8f4a5052f69616801')

>>> me.username
u'BUZZFEED'
>>> quit()

Notes on models The models (i.e MongoDB collections) map object attributes (and their types) to MongoDB fields.

The current mongoengine library settings/version allows MongoDB objects from containing fields that are not currently mapped in a model.

The type of the field on various MongoDB objects should always match up to those in the model and if they need to be changed the database should be updated first.

We are pinned to an out of date version of mongoengine. All of the patches in the version have been merged to upstream master. When attempting to upgrade our version to the pip current stable, it disallwed MongoDB objects from containing fields that are not currently mapped in a model. This needs to be researched more so that we can upgrade to the current stable on pip.

If a new model is created and it needs indexes be sure to ensure_indexes on them before going live. An example of how this can be done via manage.py is above.

NOTE: ONLY modify the indexes of large collections (Yo/User) in background mode. This can be done in the compose.io web gui.

Notes about the Payload class The payload class is arguably the most complex and important class in the code. It is in charge of taking a Yo and converting it to a push notification payload that can be understood by Windows Phone, Android, and iOS. It is also charged with ensuring older devices get older versions of the payload while newer devices get newer versions. Currently this is done via NotificationEndpoint payload support dicts that map functionality with device versions.

Whenever modifying the payload please be sure to keep device versioning in mind.

Other development notes and tips: Scanning through lists of objects with ReferenceFields is VERY slow. If this is necessary, the scanning portion should be cached.

If creating new data structures, prevent these scenarios as much as possible.

For example:

# Bad
contacts = get_contact_objects(user)
contacts_trimmed = [c for c in contacts if c.target.in_store]

# Better (but still not ideal)
@cache.memoize()
def get_trimmed_contacts(user):
   contacts = [custom contacts query]
   contacts_trimmed = [c for c in contacts if c.target.in_store]
   return contacts_trimmed

When caching objects by a string key ensure it is always access as a string OR unicode string. Caching treats them differently. For example:

content = u'test'
old_items = get_cached_items(content)
...
add_cache_item_and_clear_cache('test', item)
new_items = get_cached_items(content)
# This should have been changed but it isn't.
new_items != old_items

# to fix this simply do this:
content = str(u'test')