Go Botany is a web site that encourages informal, self-directed education in botany for science students and beginning and amateur botanists.
First, check out the repository and run dev/setup to install the
application and its dependencies in a Python virtual environment (that
lives inside of dev/venv in case you ever need to access it):
git clone [email protected]:newfs/gobotany-app.git
cd gobotany-app
dev/setup
Next, make sure that you can access your local PostgreSQL server, which
you can confirm with a quick psql -l, and then start up a Solr full
text index server with:
dev/start-solr
At this point the application should at least run, even though most pages will give errors if your database is not set up yet. To start the application, simply run:
dev/django runserver
You should then be able to visit the application at:
http://localhost:8000/
Before importing data into your database, ensure you have your AWS credentials set in your environment variables, which is often accomplished by sourcing a shell script kept outside the repository. This is needed to ensure that image importing will work.
If you are starting fresh and have no database set up yet, or want to
start over because some tables have changed or Sid had released a new
CSV file, then you can rebuild the database with these commands (the
first one will give an error if you do not have a gobotany database
already sitting in your way; in that case, ignore the error):
dropdb gobotany
createdb -E UTF8 gobotany
dev/load
At this point you are done installing and should be able to test and develop the application!
If you ever need to activate the virtual environment so that Python prompts or scripts run from your shell have access to the Go Botany application and its dependencies, then enter:
source dev/activate
If you want to rebuild our minified JavaScript in preparation for a deploy to production, run:
dev/jsbuild
Our various tests can be run with three commands:
dev/test-browser
dev/test-js
dev/test-python
Start by checking out this "gobotany-app" repository on your machine:
git clone [email protected]:newfs/gobotany-app.git
cd gobotany-app
Follow steps 1–3 at http://devcenter.heroku.com/articles/quickstart_
so that you can run the heroku command, then use the following
command to create and provision a new app on Heroku:
heroku create
heroku addons:add heroku-postgresql:basic
heroku addons:add memcache:5mb
heroku addons:add websolr:cobalt
heroku pg:wait
git push heroku master
Once the Postgres database is up and running, note its color (like "RED" or "SILVER"), and promote it to being the "main database" for the app:
heroku pg:promote <color>
Add three configuration variables to your Heroku app, so that Go Botany will be able to scan its S3 image repository:
heroku config:add AWS_ACCESS_KEY_ID=...
heroku config:add AWS_SECRET_ACCESS_KEY=...
heroku config:add AWS_STORAGE_BUCKET_NAME=newfs
The application will now be up and running. You can find its URL with
the heroku apps:info command. When you visit, you will see an
exception, because the database tables that it needs have not yet been
created. To set up the database, run these commands:
heroku config:add DJANGO_SETTINGS_MODULE=gobotany.settings
heroku run django-admin.py syncdb --noinput
heroku run python -m gobotany.core.importer zipimport
heroku run bin/import-images.sh
heroku run bin/import-dkey.sh
Prepare Solr by first generating your Solr schema:
heroku run django-admin.py build_solr_schema > schema.xml
Once this file exists, you can visit the Heroku web site, navigate to
your app's configuration, select the addon "Websolr", choose the section
"Advanced Configuration", and paste in the contents of schema.xml
that you just created. Once the schema is installed (give it a few
minutes to make sure the change has the chance to propagate to WebSolr's
servers), you can build the Solr index and thereby activate the Go
Botany site's search field:
heroku run django-admin.py rebuild_index --noinput
If you would like to server static/admin assets from S3 or Cloudfront directly instead of from Heroku's dynamic webserver you need to set a couple environment variables. First you need to set your local AWS variables and export the files to S3:
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_STORAGE_BUCKET_NAME=newfs
export S3_STATIC=True
dev/django collectstatic
To let Heroku we want to serve these files from S3 and to avoid having it run time-consuming collectstatic on every deploy, we need the following settings:
heroku config:add S3_STATIC=True
heroku config:add DISABLE_COLLECTSTATIC=1
To run our Python tests you can either:
dev/test-python # to run all tests
dev/test-python api site # to hand-pick Django apps to test
Note: if you get failures with the Python tests, try running the tests again for just the app where failures occured. There is a known problem where some tests can fail when running tests for multiple apps at once.
To run our JavaScript tests, run:
dev/test-js # to run all tests
dev/test-js test/Filter.js # to select which modules to test
Our selenium-powered browser tests are intended to cover things that cannot be tested without a browser and JavaScript. To run them:
dev/test-browser # to run all tests
dev/test-browser FilterFunctionalTests # select which tests
Note: if you get failures with the browser tests, try running the test that failed individually. Sometimes a failure or two occurs when running the entire suite, but the tests pass when run individually.
Detailed notes about testing under selenium can be found in:
gobotany-app/gobotany/simplekey/testdir/README-SELENIUM.txt
To check for test coverage using coverage.py, run:
$ export DJANGO_SETTINGS_MODULE=gobotany.settings
$ coverage run --source=gobotany dev/venv/bin/django-admin test
$ coverage html
$ open htmlcov/index.html
If you don't specify the --source, you'll have stats for the entire site-packages directory, the dev directory, etc.
The first pass generates a .coverage text file with coverage statistics.
The second command creates nicely formatted, highlighted, and linked HTML reports.
To see the same data on the command line, use:
$ coverage report
The Go Botany search feature uses Haystack and Solr.
Our unit and functional tests aim to ensure various aspects of the search feature including desired ranking.
Ranking relies mostly on Haystack document boost, as seen in several
places in our search_indexes.py. For more fine-grained control where
boost is not enough, some hidden repeated keywords are added to search
indexes such as in the search_text_species.txt template.
To adjust ranking: cycle through running the functional tests, adjusting
the boosts in search_indexes.py, and, if necessary, adjusting the
hidden-keyword sections at the end of search_*.txt templates. The Solr
Admin full interface, which allows examining details including ranking
scores, may also be helpful:
http://localhost:8983/solr/admin/form.jsp
When developing locally, the default is to use the individual non-minified JS files, as they require no separate build step before testing.
After working on these JS files, you build the combined and minified production JS file to be used on the site.
It is a good idea to test with this file to make sure the changes work as expected. Sometimes things load a bit differently when using this file, and it can happen that things worked with the individual JS files fail with the production JS file.
To temporarily use the production JS file, edit the file: gobotany/app/gobotany/site/templates/gobotany/_js.html
Change the if statement there to evaluate to False, such as with 'if 1 == 0'. Ensure Django reloads (if not, restart Django), then reload the page.