Skip to content

Developer Setup

Jump to bottom
lhcramer edited this page Feb 1, 2018 · 1 revision

Warning - This page needs updating

Developer setup is described in the code repository README files - you should always check there for up to date instructions. This wiki will be updated to provide additional clues, examples and procedural advice.

Todo: how does mapstory and geonode development environments differ - and in particular how do you integrate and test additional modules for mapstory that are developed against geonode core?

Deprecated content

The testing and update sections may still be relevant however.

Requirements

This guide assumes you are running a Linux/OSX terminal and have the following commands available:

  • git
  • python
  • pip
  • virtualenv
  • vagrant
  • virtualbox
  • fabric
  • ansible

Create a local working directory

Note: We are installing everything inside the user's home directory (~); if you want to install somewhere else just replace ~ with your custom path.

Run the following commands in your terminal.

  • cd ~
  • mkdir mapstory
  • cd mapstory

Clone repositories to working directory

Inside our root mapstory/ directory run:

  • git clone https://github.com/MapStory/mapstory.git

This will result in the following directory structure:

~/mapstory/
     mapstory/

Install and provision a vagrant virtual machine

Install Vagrant

Important Note: You only need to provision the machine once. In the future, you can start the machine with vagrant reload.

  • cd ~/mapstory/mapstory/scripts/provision/
  • vagrant up --provision

This will download and install a virtual machine and might take a while.

If you encounter any issues during this process, please create an issue following the Creating Issues guide.

Install & configure venv

From the project root directory (mapstory/), run pip install virtualenv. Next, run virtualenv venv to create the virtual environment for the project.

Now your project should be configured so that you can run source venv/bin/activate from the project root directory for local development.

Fix Elastic Search

Sometimes when running vagrant provision, you may run across an error like this:

TASK [web : create or update django super user] ********************************
fatal: [192.168.56.151]: FAILED! => {"changed": true, "cmd": "PYTHONPATH=. DJANGO_SETTINGS_MODULE=mapstory.settings /home/mapstory/.virtualenvs/mapstory/bin/python -c \"import django; from geonode.people.models import Profile; django.setup(); p,_ = Profile.objects.get_or_create(username='admin'); p.is_staff=p.is_superuser=True; p.set_password('admin'); p.save()\"", "delta": "0:00:03.734981", "end": "2017-04-24 16:05:55.996408", "failed": true, "rc": 1, "start": "2017-04-24 16:05:52.261427", "stderr": "Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/django/db/models/base.py\", line 734, in save\n    force_update=force_update, update_fields=update_fields)\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/django/db/models/base.py\", line 771, in save_base\n    update_fields=update_fields, raw=raw, using=using)\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py\", line 189, in send\n    response = receiver(signal=self, sender=sender, **named)\n  File \"mapstory/search/signals.py\", line 53, in handle_save\n    index.update_object(instance, using=using)\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/haystack/indexes.py\", line 282, in update_object\n    backend.update(self, [instance])\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py\", line 191, in update\n    bulk_index(self.conn, prepped_docs, index=self.index_name, doc_type='modelresult')\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py\", line 182, in bulk\n    for ok, item in streaming_bulk(client, actions, **kwargs):\n  File \"/home/mapstory/.virtualenvs/mapstory/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py\", line 124, in streaming_bulk\n    raise e\nelasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4012b1ed90>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f4012b1ed90>: Failed to establish a new connection: [Errno 111] Connection refused)", "stdout": "", "stdout_lines": [], "warnings": []}

The Failed to establish a new connection: [Errno 111] Connection refused) error is an indicator that Elastic Search is no longer running and needs to be restarted, which you can do manually like so from mapstory/mapstory/scripts/provision:

vagrant ssh
sudo supervisorctl restart elasticsearch

Or you can follow the steps to set up an Elastic Search autofix below (which you'll need to do each time you re-provision).

Setup Elastic Search autofix

There is a known problem with elasticsearch and it often needs to be restarted. This trick will automatically fix and restart elastic search anytime the virtual vagrant shell is started.

This step is optional but recommended so you don't have to manually restart elasticsearch.

Step 1: Start a vagrant shell

Inside ~/mapstory/mapstory/scripts/provision Run

  • vagrant ssh

Note: You must do all this as a regular user, not mapstory user

This will start a shell on the virtual server

Step 2: Edit bash profile

Inside the virtual shell run:

  • nano ~/.bashrc

At the bottom of the file insert this text:

./autofixElastic.sh
  • Save with ctrl + o
  • Exit with ctrl + x

Step 3: Create the fix script

Inside the virtual server shell run:

  • touch ~/autofixElastic.sh
  • chmod +x ~/autofixElastic.sh
  • nano ~/autofixElastic.sh

Add this text to the file:

clear
echo ""
echo "------------------------"
echo "Elastic Search Auto-Fix"
echo "------------------------"
echo ""
echo "[*] Creating Directory ..."
sudo mkdir /var/run/elasticsearch
echo ""
echo "[*] Chowning Directory ..."
sudo chown elasticsearch:elasticsearch /var/run/elasticsearch
echo ""
echo "[*] Restarting elastic search"
sudo supervisorctl restart elasticsearch
echo ""
echo "-----------------"
echo "Success fixing!!"
echo "-----------------"
echo ""
  • Save with ctrl + o
  • Exit with ctrl + x

Step 4: Reload bash profile

Run this on the virtual shell

  • source .bashrc

The elastic search auto-fix should now run every time you start a virtual shell.

Running test cases

./manage.py test

To run with coverage report:

coverage run ./manage.py test && coverage report && coverage html -d cover

This will generate documents. Open cover/index.html to view coverage report in browser

Running browser tests

Requires Protractor

To run automated browser tests:

Start protractor service

Start the web driver and leave it running

cd mapstory/mapstory/mapstory/tests/
./startWebDriver.sh

Run tests

On a separate shell start the testing suite

cd mapstory/mapstory/mapstory/tests/
./runE2ETests.sh

How to

Start the mapstory virtual server

Your vagrant machine should now be installed and can be started like this:

  • cd ~/mapstory/mapstory/scripts/provision
  • vagrant reload

Run the server for local development

Assuming you've already provisioned vagrant, run vagrant up from /mapstory/scripts/provision/.

Then, from the root project folder run source venv/bin/activate. This should create your virtual environment.

With (venv) active, from /mapstory run fab dev runserver. If you haven't already, you may also need to run pip install ansible and pip install fabric after source venv/bin/activate.

You'll be prompted to enter a password:

[192.168.56.151] Login password for 'vagrant':

The password is vagrant.

Now your server should be running at http://192.168.56.151/.

From mapstory/static/ you can run grunt watch to update assets as you save changes.

If you have trouble and aren't seeing your changes reflected, try running fab dev collect restart from /mapstory before rerunning fab dev runserver. Note that image assets and test data are not provided in the local build and you'll have to add those yourself via the site (or from Django Admin).

Restart elastic search

From mapstory/scripts/provision, run the following commands:

vagrant ssh
sudo supervisorctl restart elasticsearch

If that doesn't fix the problem, it may because there is no directory that exists for elastic search, in which case, try sudo supervisorctl tail elasticsearch. If the output includes this error (or something similar):

- FileNotFoundException[/var/run/elasticsearch/elasticsearch.pid (No such file or directory)]
java.io.FileNotFoundException: /var/run/elasticsearch/elasticsearch.pid (No such file or directory)

...then run the following commands:

sudo mkdir /var/run/elasticsearch
sudo chown elasticsearch:elasticsearch /var/run/elasticsearch
sudo supervisorctl restart elasticsearch

If you re-provisioned and lost your local theme settings

If you reprovisioned, you may have lost additions you made locally to mapstory/settings/local_settings.py. For example, you may need to add back the line THEME = 'orange' if you want your local MapStory to show the orange theme that's used on production.

If port 8000 is already in use

Sometimes the server doesn't shut down properly. In that case, you may see an error when you try to fab dev runserver that looks like this:

[192.168.56.151] out: Error: That port is already in use.
[192.168.56.151] out:

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: python manage.py runserver
Executed: sudo -S -p 'sudo password:'  -u "www-data"  /bin/bash -l -c "cd /srv/git/mapstory/mapstory >/dev/null && source /home/mapstory/.virtualenvs/mapstory/bin/activate && python manage.py runserver"

Aborting.
Disconnecting from 192.168.56.151... done.

To fix this, run vagrant ssh from /mapstory/scripts/provision/.

Then, search for the server task: ps aux | grep runserver.

Locate the pid (in www-data 9392 0.1 4.2 337708 86892 pts/0 S+ 13:54 0:01 python manage.py runserver, the pid is 9392), and enter sudo kill -9 [your pid here]. So if your pid is 9392, you'd run sudo kill -9 9392. You can check to see that you killed that task by running ps aux | grep runserver. When you run this, you shouldn't see any tasks matching runserver except for your grep.

Once you've killed the process within vagrant, you can exit to return to your local terminal, where you should be able to run the server normally.

If importer hangs

If importing layers locally hangs and will not complete, run vagrant provision from mapstory/mapstory/scripts/provision/ to restart celery without doing a total rebuild.

Start a virtual server shell

Step 1: Start mapstory server:

If your mapstory virtual server is not up yet:

  • cd ~/mapstory/mapstory/scripts/provision
  • vagrant reload

Step 2: Start a vagrant shell:

  • vagrant shell

Run django management commands

Step 1: Start a vagrant shell

  • cd ~/mapstory/mapstory/scripts/provision/
  • Optional: (If your vagrant machine isn't up run: vagrant reload)
  • vagrant ssh

Step 2: Work on mapstory as mapstory

This step is crucial! do not forget to workon mapstory as mapstory user only!

  • cd /srv/git/mapstory/mapstory/
  • sudo su mapstory
  • workon mapstory

Step 3: Run a management command

Now you should be able to run Django's management commands with ./manage.py

Start an ipython session with Django modules preloaded like this:

  • ./manage.py shell_plus

Run the automated tests:

  • ./manage.py test mapstory

Run MapLoom locally within MapStory to work on Composer locally

Step 1: Clone the MapLoom project Currently, MapStory uses a fork of MapLoom for the UI and underlying functionality of Composer. You can find the repo for that fork here. The branch is feature/composer-wip.

To set up the fork for local development within the MapStory project, clone the repo into your MapStory parent folder:

$ git clone [email protected]:MapStory/MapLoom.git

Assuming you've set up a parent folder that houses all related MapStory repos, an ls within that parent folder may look something like this after you've cloned the MapLoom fork:

~/dev/mapstory $ ls
developer-tools	mapstory	node_modules
MapLoom		geonode		mapstory.wiki	venv

Step 2: Configure MapLoom Assuming you already have Node.js installed, run the following commands to configure MapLoom:

$ cd MapLoom
$ sudo npm -g install grunt-cli karma bower
$ npm install
$ bower install

Step 3: Configure MapStory to recognize the local instance of MapLoom

  • Create the MapLoom directory in vagrant. First, vagrant ssh, then:
cd /srv/git/mapstory/mapstory/mapstory/static/
mkdir maploom
exit
  • Run ./clean-build from mapstory/MapLoom
  • Run fab dev map_loom_django_dev from mapstory/mapstory Note that your virtual machine should be running, or this command will time out.
  • Run fab dev collect restart from mapstory/mapstory
  • From /MapLoom dir, grunt watch should keep html changes updated in browser if you add new html files, another fab dev collect restart from mapstory might be needed

Fixing StoryTools console errors for when Layer/Viewer/Composer hangs on "Loading..."

Sometimes if changes are made to StoryTools, the virtual machine doesn't pick up those changes and you need to run bower install manually in Vagrant.

These errors in the browser console are an indicator of this problem:

vendor-assets-min.js:4 Uncaught TypeError: a.indexOf is not a function
    at r.fn.init.r.fn.load (vendor-assets-min.js:4)
    at geonode:american_civil_war?showMetadata=true:1365
story-tools-core-all.css Failed to load resource: the server responded with a status of 404 (Not Found)
2story-tools-vendor-all.js Failed to load resource: the server responded with a status of 404 (Not Found)
util.js:1171 Uncaught ReferenceError: ol is not defined
    at Object.7 (util.js:1171)
    at s (util.js:7)
    at util.js:7
    at Object.1../time/boxes (util.js:8)
    at s (util.js:7)
    at e (util.js:7)
    at util.js:7
    at util.js:7
    at util.js:7
jquery.min.js:2 Uncaught Error: [$injector:modulerr] http://errors.angularjs.org/1.3.15/$injector/modulerr?p0=viewer&p1=Error%3A…F%2F192.168.56.151%2Fstatic%2Fvendor%2Fangular%2Fangular.min.js%3A18%3A179)
    at angular.js:38
    at angular.js:4138
    at r (angular.js:323)
    at g (angular.js:4099)
    at ab (angular.js:4025)
    at d (angular.js:1452)
    at uc (angular.js:1473)
    at Jd (angular.js:1367)
    at HTMLDocument.<anonymous> (angular.js:26304)
    at j (jquery.min.js:2)
story-tools-core-all.css Failed to load resource: the server responded with a status of 404 (Not Found)

In the UI, this may show as a layer that displays "Loading..." indefinitely.

A) To fix from bower:

from mapstory/mapstory/mapstory/static/, run bower update story-tools then run fab dev collect from your mapstory project directory

B) To fix from your VM:

vagrant ssh
cd /srv/git/mapstory/mapstory/mapstory/static/vendor
rm -rf story-tools

Then, exit vagrant ssh and run bower install from mapstory/mapstory/mapstory/static/. You'll then need to run fab dev collect restart. On hard refresh, this should fix the layer viewer and eliminate console errors.

It's possible that a bower update story-tools and fab dev collect may also be a solution without requiring a vagrant ssh.

Installing and running StoryTools within MapStory locally

TODO: CONFIRM STEPS BELOW Set up StoryTools in the root mapstory directory and include this fab command in your fabfile:

def viewer_dev():
    sudo('rm -rf /srv/git/mapstory/mapstory/mapstory/static_root/vendor/story-tools/dist/*')
    sudo('cp /srv/git/mapstory/story-tools/dist/* /srv/git/mapstory/mapstory/mapstory/static_root/vendor/story-tools/dist/')

It copies your story-tools dev dir into vendor files. You could also symlink.

Project Vision

User Personas

Beta Baseline and Testing

Creating Issues

How to Contribute

Project Architecture

MapStory Road Map

Clone this wiki locally