Skip to content

Latest commit

 

History

History
310 lines (212 loc) · 9.61 KB

developer_guide.md

File metadata and controls

310 lines (212 loc) · 9.61 KB

Developer Guide

Website is deployed in Kubernetes cluster. A deployment contains the following containers:

  • website: A Flask app with static files complied by Webpack.
  • mixer: The Data Commons API server.
  • esp: Google Extensive Service Proxy used for gRPC to Json transcoding.

Mixer is a submodule of this Git repo. The exact commit of the submodule is deployed together with the website so it may not be the same API version as api.datacommons.org. Make sure to update and track the mixer change for a new deployment:

git submodule foreach git pull origin master
git submodule update --init --recursive

Prerequisites

Basic Development

For development that only involves minor bug fixes or feature additions, the following steps are required:

  • Install nodejs

    • Install nvm
    • Install node 18.4.0: nvm install 18.4.0

  • Install protoc

Place Search

Development that involves place search needs the following additional requirements:

  • Contact Data Commons team to get dev maps api key.

  • Install gcloud

  • Get GCP authentication

    gcloud auth application-default login

Develop with Kubernetes

This is only needed for development with a local Kubernetes cluster (e.g. for changes dependent on mixer changes).

Run Tests

Install web browser and webdriver

❗IMPORTANT: Make sure that your ChromeDriver version is compatible with your local Google Chrome version.

Before running the tests, install the browser and webdriver. Here we recommend you use Google Chrome browser and ChromeDriver.

  • Chrome browser can be downloaded here.

  • ChromeDriver can be downloaded here, or you can download it using package manager directly:

    npm install chromedriver

You can view the latest ChromeDriver version here. Also make sure PATH is updated with ChromeDriver location.

If using Linux system, you can run the following commands to download Chrome browser and ChromeDriver, this will also include the path setup:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb; sudo apt-get -fy install
CHROMEDRIVERV=$(curl https://chromedriver.storage.googleapis.com/LATEST_RELEASE)
wget https://chromedriver.storage.googleapis.com/${CHROMEDRIVERV}/chromedriver_linux64.zip
unset CHROMEDRIVERV
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
sudo chown root:root /usr/bin/chromedriver
sudo chmod +x /usr/bin/chromedriver

Run all tests

./run_test.sh -a

Update React test snapshots

cd static
npm test testfilename -- -u

Develop with Flask (simple/lite)

This way the website talks to the autopush Mixer.

Note: the autopushed mixer can deviate from the mixer submodule and may not be fully compatible with website.

Package javascript and static assets

./run_npm.sh

This will watch static files change and re-build on code edit.

Start the Flask Server

Start the flask webserver locally at localhost:8080

./run_server.sh

If you don't have DataCommons GCP permissions, run

./run_server.sh -e lite

Other options for -e are private for private data commons, or iitm for iitm data commons.

This will bring up local website without place search functionality.

There are now multiple environments for the server. To start multiple instances, bind each server instance to a different port. The following example will start localhost on port 8081. The default is 8080.

./run_server.sh -p 8081

Please note the strict syntax requirements for the script, and leave a space after the flag. So ./run_server.sh -p 8081 but not ./run_server.sh -p=8081.

Develop with local Kubernetes

This is an alternative way to bring up website stack locally and this is close to how the production server is deployed in GKE.

Local Kubernetes cluster has similar configurations as the production deployment, for example, it brings up a local mixer instead of talking to the autopush/staging mixer.

This is useful for local development that involves data version, mixer and website changes.

Start website in Minikube

This takes a few minutes to complete, as it involves building several docker images. If only website code is changed, can use the alternative approach.

NOTE Make sure the local Docker engine has more than 12G of memory.

minikube start --memory=11G
minikube addons enable gcp-auth
eval $(minikube docker-env)
kubectl config use-context minikube
skaffold dev --port-forward -n website

This exposes the local website at localhost:8080.

Hot reload

All the code change is synced to the containers through "File Sync" of Skaffold.

Run the following command to get the javascript code recompiled when changed:

./run_npm.sh

Python code change will trigger a restart of the Flask server automatically.

Monitoring the containers

Run minikube dashboard in a separate terminal to start the dashboard, which is useful for monitoring and controlling the containers.

Deploy local instance to GCP

Commit all changes locally. Then run

gcloud auth login
gcloud auth configure-docker
./scripts/push_image.sh
./scripts/deploy_gke.sh dev us-central1

The script will build docker image locally and tag it with the local git commit hash at HEAD, then deploy to GKE.

View the deployoment at link

Other Developing Tips

GKE config

The GKE configuration is stored here.

Private Instance

Create a pub/sub topic for mixer to listen to data change.

gsutil notification create -t tmcf-csv-reload -f json gs://<BUCKET_NAME>

Redis memcache

Redis memcache is used for production deployment. Each cluster has a Redis instance located in the same region.

Adding new charts

  1. Update server/config/chart_config/<category>.json with the new chart.

       {
     "category": "", // The top level category this chart belongs to. Order of charts in the spec matters.
     "topic": "",  // Strongly encouraged - A page-level grouping for this chart.
     "titleId": "", // Strictly for translation purposes.
     "title": "", // Default (EN) display string
     "description": "", // Strictly for translation purposes.
     "statsVars": [""], // List of stat vars to include in the chart
     "isOverview": true, // Optional - default false. If the chart should be added to the overview page.
     "isChoropleth": true, // Optional - default false. If a map should be used to display the data
     "unit": "",
     "scaling": 100,
     "relatedChart": {  // Defined if there should be comparison charts added
       // All chart fields from above can be specified. If unspecified, it will be inherited.
     }
   }

  2. Update related files.

    Note: Please add very detailed descriptions to guide our translators. See localization.md for more details.

  3. Run these commands:

    ./scripts/extract_messages.sh
./scripts/compile_messages.sh

  4. IMPORTANT: Manually restart the flask or minikube instance to reload the config and translations. Most likely, this means re-running run_server.py

  5. Test the data on a place page!

Debugging Webdriver tests

  • Disable headless mode in webdriver to follow the test in Chrome. Chrome features like the dev inspector are available in this mode which is useful combined with sleep() to give you time to inspect the page. To enter this mode, comment out this line in base_test.py:

    chrome_options.add_argument('--headless')

  • Another option is to save a screenshot at various points of the test:

    self.driver.save_screenshot(filename)