NEEM Hub support (#56)

* removed init_webapp

* removed experiments admin page

* removed mongo admin interface

* towards neem-hub

* fixed user_data route

* add menu item for user info

* quickly bootstrapped the user_info page

* user password can be changed

* Reformatting global variable name

* added .dockerignore to avoid docker COPY command including .git dirs in local node_modules

* require @openease/charts and @openease/canvas-three

* depend on @openease/charts and @openease/canvas-three

* load layout.css

* remove imports that are part of node modules @openease/charts and @openease/canvas-three

* cop local node_modules directory into the image

* added empty node_modules directory

* moved some openEASE specific JS code back into this source tree

* require @openease/ros-clients and @openease/rosprolog-console

* depend on @openease/ros-clients and @openease/rosprolog-console

* JS modules have moved to statis/js

* updated README

* removed unused env

* moved highlightElement into this template

* updated JS imports

* logging interface app.logger is used

* show_user_data is renamed as user_data in app.py

* css files are moved into openease_flask container and thus openease_css is removed

* variable name changed due to typo

* added QueryCard widget

* css for split.js, todo: seems not working on small devices from the scratch

* removed table definitions

* added ease logo with light colors

* depend on split.s

* +MAX_HISTORY_LINES

* only use one volume for neems (neems-ws is not used anymore)

* removed some definitions used in QA page

* updated imports

* refactor

* serve meshes from neem directory

* set fixed frame to map

* reorganized navbar a bit

* work on QA page

* work on QA page

* quick fix for api endpoints, for now disabled starting user container

* work on query console

* need to escape single quotes in query string

* wait on KnowRob before loading a NEEM

* scalable pages

* minor changes: smaller-screen-size class added and unused code removed.

* qa-container should have w-100 class

* use only one navbar-brand since it does not look right for small devices

* minor changes: smaller-screen-size class removed from QA html and split sizes changed based on window width.

* Fixed bug by using Ubuntu 18.04

* NEEM instances are created from mongo docs

* env variable name is changed

* neem meta information updated: name and url paramters are added to neem

* neem id is set from mongo meta data

* Explore link now works and redirects to QA Page.

* make explore button work again

* neem_id type is checked before using it for retrieving neem from mongoDB.

* Updated some js packages

* updated versions of node packages

* renamed env variable

* Environment variables used for mongoclient set-up.

* Int data is required as mongo port.

* robot image as background-image of card-body

* show neem id

* removed unused code

* removed unused code

* connect to host knowrob

* neem discovery

* Added usage of server for meshes

* Update settings.py

* discovery page updated

* crated by is displayed in more readable format

* modified meta data is red from mongo client

* final changes for neem discovery page with different kinds of neems

* neem_id is now set by the choice on the discovery page

* repo path is shown on top of neem card with clickable link

* neem card header text color is changed

* added more examples

* implemented some example queries

* example queries

* Initial commit with tutorial page. TODO : Fix the column in the right

* - Right page
- query console is not editable

* Set the code from the tutorial to the console

* Modified text and queries

* switch to npm v12

* re-enabled route for video page

* implemented filtering via simple mongo query

* video page

* canvas snapshot

* cleanup

* canvas snapshot works now

* handle neem id in QA.js

* fixed some errors

* removed unused code

* some minor things

* new logo is added to navbar

* Links for Admin dropdown list are fixed. However pages needs to be properly defined.

* footer position is fixed. Also background color of some pages are changed.

* navbar on top and footer on bottom are fixed. Also user details page updated.

* links are replaced by buttons in user details page

* margins for containers classes are added for top and bottom side

* minor changes

* redesinged QA page

* minor

* added empty template for query examples page

* work on blackboard

* blackboard

* border color

* light blue background for card header

* Update ROSCanvas.js

* Update ROSCanvas.js

* Update ROSCanvas.js

* new logo is added to other remaining pages.

* typos removed and padding class is added

* event playback

* NEEM Hub Configuration (#52)

* first commit with NEEM Hub configuration

* Postgresql is configured for storing NEEMHubSettings data

* NEEM configuration is checked on application start

* Go to configuration helper page added

* Final working changes

* neemHubSettings are passed at container creation time. Also code cleaned.

* updates with working version

* code cleaned: unused boolean variable removed

* code cleaning: unused variables are removed.

* code cleaned: unused method removed.

* code cleaned. suggestions implemented.

* changes related to neem_manager singleton

* minor update

* suggestions are applied

* TODOs are removed

* Final comments are added where exception handling is done on method caller side

* Custom neem meta exception class is added to handle exceptions during connecting to mongodb neem meta collection

* custom SQL Alchemy exception class is added

* all exceptions are handled by flask handler so do not need to handle them individually.

* added button bar to canvas

* changed reverted for navbar and footer position fixed. Now footer gets fixed position for some pages but not all. Also openease logo is reverted to old logo. (#53)

* canvas links

* few more changes (#54)

* changed reverted for navbar and footer position fixed. Now footer gets fixed position for some pages but not all. Also openease logo is reverted to old logo.

* few more changes: QA logo fixed and footer script is arranged properly

* implemented pause button in playback widget (#55)

* cleanup

* cleanup

* removed some unused files

* QA

* playback widget

* fix fir woff files by Abhijit

* code cleanup

* refactoring

* more general invalid settings page

* removed unused mesh_data volume

* cleanup

* refactoring

* refactor

* removed unused file

* fix for db admin pages

* refactor

* removed unused template

* fixed package structure

* removed unused file

* added README

* refactor

* removed unused code

* refactor

* removed unused code

* refactor

* remove screen.css and user.css

* refactor

* refactor

* refactor

* removed unused dependency

* CSS cleanup

* no need to use openease/postgres

* removed unused files

* refactor

* examples page

* query examples

* example queries

* clickable example queries

* clickable example queries

* css cleanup

* renamed docker image openease/flask to openease/app

Co-authored-by: Abhijit Vyas <[email protected]>
Co-authored-by: Sascha Jongebloed <[email protected]>
Co-authored-by: Daniel Beßler <[email protected]>
Co-authored-by: kaviyachandran <[email protected]>

Author: 4 people

.codecov.yml

@@ -0,0 +1,3 @@
+certs/*
+node_modules/*/.*
+node_modules/*/*/.*

.dockerignore

@@ -22,7 +22,9 @@ setup.cfg
 build
 
 # node and javascript stuff
-node_modules/
+/node_modules/*
+!/node_modules/README.md
+
 npm-debug.log
 
 # ignore sass-cache

.gitignore

@@ -1,19 +1,21 @@
-FROM ubuntu:16.04
+FROM ubuntu:18.04
 MAINTAINER Daniel Beßler, [email protected]
 
 # install python and flask
 RUN apt-get -qq update
 RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -q curl python-all python-pip python-dev wget gcc imagemagick mongodb libffi-dev libpq-dev
 RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -q subversion git
 
-RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -q nodejs nodejs-legacy npm
+RUN curl -sL https://deb.nodesource.com/setup_12.x | bash
+RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -q nodejs
 RUN DEBIAN_FRONTEND=noninteractive apt-get -qq install -y -q postgresql
 
-WORKDIR /opt/webapp
+RUN apt-get update && \
+  DEBIAN_FRONTEND=noninteractive apt-get install -y ruby-full build-essential rubygems && \
+  gem install sass --no-user-install -v 3.5.5 && \
+  apt-get clean
 
-# flag used in nginx configuration
-# TODO: is this still needed?
-ENV OPEN_EASE_WEBAPP true
+WORKDIR /opt/webapp
 
 # install python-dependencies including flask
 COPY requirements.txt .
@@ -24,27 +26,34 @@ RUN useradd -m -d /home/ros -p ros ros && chsh -s /bin/bash ros
 ENV HOME /home/ros
 
 ## install npm dendencies
-RUN mkdir /tmp/npm
-ADD ./webrob/static/index.js ./webrob/static/package.json /tmp/npm/
-RUN cd /tmp/npm && npm install && npm run build && chown -R ros:ros /tmp/npm
+RUN mkdir -p /tmp/npm/node_modules
+WORKDIR /tmp/npm
+COPY static/package.json /tmp/npm/
+RUN npm install
+# copy local node modules into the image
+# FIXME: needs to be done after npm install
+COPY ./node_modules /tmp/npm/node_modules
+
+COPY static/index.js /tmp/npm/
+RUN npm run build
+RUN chown -R ros:ros /tmp/npm
+WORKDIR /opt/webapp
 
 ## copy this folder to the container
-ADD . /opt/webapp/
+RUN mkdir /opt/webapp/webrob
+COPY . /opt/webapp/webrob
 RUN chown -R ros:ros /opt/webapp/
 
-RUN mkdir /home/ros/mesh_data
-RUN chown -R ros:ros /home/ros/mesh_data
-
 USER ros
 
-# install JS libraries using npm
-# TODO why need to copy?
-# RUN cd /opt/webapp/webrob/static && npm install
+# install JS libraries to static dir of webserver
 RUN mv /tmp/npm/openease*.js /opt/webapp/webrob/static/
 
 RUN cd /home/ros
-# Expose volumes for maintenance
-VOLUME /opt/webapp/
+
+# configure scss to css file conversion here with sass
+WORKDIR /opt/webapp/webrob/static/css/SCSS
+RUN sass --update .:.
 
 EXPOSE 5000
 

Dockerfile

@@ -1,12 +1,4 @@
 openease/flask
 ================
 
-### Master: [![Build Status](https://travis-ci.org/navidJadid/openease_flask.svg?branch=master)](https://travis-ci.org/navidJadid/openease_flask) [![codecov](https://codecov.io/gh/navidJadid/openease_flask/branch/master/graph/badge.svg)](https://codecov.io/gh/navidJadid/openease_flask)
-
-### Development: [![Build Status](https://travis-ci.org/navidJadid/openease_flask.svg?branch=development)](https://travis-ci.org/navidJadid/openease_flask) [![codecov](https://codecov.io/gh/navidJadid/openease_flask/branch/development/graph/badge.svg)](https://codecov.io/gh/navidJadid/openease_flask)
-
-Repository which contains the front & back-end code for the openEASE project.
-
-Cannot be run by itself as of right now. Either build the [whole project](https://github.com/ease-crc/openease) with docker-compose (will pull the most recent docker-build from [`docker-hub`](https://hub.docker.com/r/openease/flask)) or check out the [development project](https://github.com/navidJadid/openease_webserver_development).
-
-Apart from that, tests will of course still run in CI-environment.
+### Master: [![Build Status](https://travis-ci.org/ease-crc/openease_flask.svg?branch=master)](https://travis-ci.org/ease-crc/openease_flask) [![codecov](https://codecov.io/gh/ease-crc/openease_flask/branch/master/graph/badge.svg)](https://codecov.io/gh/ease-crc/openease_flask)

README.md

@@ -6,6 +6,7 @@
 #
 # Authors: Ling Thio <[email protected]>
 
+import os
 from flask import Flask
 from flask_sqlalchemy import SQLAlchemy
 import logging
@@ -15,5 +16,10 @@
 app.logger.addHandler(logging.StreamHandler())
 app.logger.setLevel(logging.INFO)
 
+# set up flask SQLAlchemy db uri
+app.config['SQLALCHEMY_DATABASE_URI'] =  'postgresql://docker@' + \
+os.environ['POSTGRES_PORT_5432_TCP_ADDR'] + ':' + \
+os.environ['POSTGRES_PORT_5432_TCP_PORT'] + '/docker'
+
 # This is the SQLAlchemy ORM object
 db = SQLAlchemy(app)

webrob/__init__.py __init__.py

@@ -0,0 +1,4 @@
+# Ignore certificate files and keys, those must not be commited/published!
+*.crt
+*.csr
+*.key

webrob/app_and_db.py app_and_db.py

@@ -0,0 +1,8 @@
+-----BEGIN DH PARAMETERS-----
+MIIBCAKCAQEAm+GxaTqFwuw/rcg++CQ4YBNE+QVIpAicQKFSanWjIYSrIAMJ+MOL
+DYaRxQw8pzUhF0RoGPK41ap+61SRPP4YQXUhCmbH+mRc+SD4HOJm1Yf6PNY0Ze7X
+SW+ZhDImAiIpMwfqK/hClVEWJ3jBkaVnmtUnKeIuBohNlhoW+nskrH3nRgcJWuVh
+gDXATKQf2jAD9l2Yr7HUogBM9JnnYJ5HMK1jpPDmhVeO/WuQB1qHTb5EZrGZV2s6
+53SNZrMpMkNaRqyCLuafONBtucJRRP2hZjXK345r0+9n8Z4vs8e/hHLlZlBHwOg/
+C7Ny0wdwKAM5v2V43DCpl7VhAjbivgolUwIBAg==
+-----END DH PARAMETERS-----

certs/.gitignore

@@ -0,0 +1,6 @@
+#!/bin/sh
+NAME=default
+openssl genrsa -out $NAME.key 4096
+openssl req -new -key $NAME.key -out $NAME.csr -subj "/OU=IAI/CN=localhost/O=Universitaet Bremen/C=DE"
+openssl x509 -req -days 365 -in $NAME.csr -signkey $NAME.key -out $NAME.crt
+rm $NAME.csr

certs/dhparams.pem

@@ -0,0 +1,98 @@
+# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
+# scheme used to connect to this server
+map $http_x_forwarded_proto $proxy_x_forwarded_proto {
+  default $http_x_forwarded_proto;
+  ''      $scheme;
+}
+
+# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
+# Connection header that may have been passed to this server
+map $http_upgrade $proxy_connection {
+  default upgrade;
+  ''      '';
+}
+
+gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
+
+# Maximum upload size (affects size of text a user can type in the editor). Default is 1 MB
+client_max_body_size 20M;
+
+log_format vhost '$host $remote_addr - $remote_user [$time_local] '
+                 '"$request" $status $body_bytes_sent '
+                 '"$http_referer" "$http_user_agent"';
+
+access_log /proc/self/fd/1 vhost;
+error_log /proc/self/fd/2;
+
+# HTTP 1.1 support
+proxy_http_version 1.1;
+proxy_buffering off;
+proxy_set_header Host $http_host;
+proxy_set_header Upgrade $http_upgrade;
+proxy_set_header Connection $proxy_connection;
+proxy_set_header X-Real-IP $remote_addr;
+proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
+
+proxy_read_timeout 600s;
+{{ $cert := "default" }}
+
+server {
+  listen 80;
+  server_name localhost data.open-ease.org;
+  # entrypoint that connects to ros service on the host
+  # TODO: figure out howto make the IP a variable so that openEASE can connect
+  #   to remote ROS master.
+  #location ~ ^/ws/remote/(.*)$ {
+    #rewrite /ws/remote/.*/(.*) /$1 break;
+    #proxy_pass http://$1:9090;
+  #}
+  # Allow connections to ROS master on docker host via /ws/remote/172.17.42.1
+  location /ws/host {
+    rewrite /ws/host/(.*) /$1 break;
+    proxy_pass http://172.17.0.1:9090;
+  }
+
+  # iterates over all containers and creates 'location' entries
+  # websocket endpoints: /ws/<container-name>
+  {{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
+    {{ range $container := $containers }}
+      {{ if $container.Env.VIRTUAL_PORT }}
+        {{ range $i, $address := $container.Addresses }}
+        {{ $network := index $container.Networks 0 }}
+          {{ if eq $address.Port $container.Env.VIRTUAL_PORT }}
+  # Container {{ $host }}
+            {{ if eq $host "openease" }}
+  location / {
+            {{ else }}
+  location /ws/{{ $host }}/ {
+    rewrite /ws/{{ $host }}/(.*) /$1 break;
+            {{ end }}
+    proxy_pass http://{{ $network.IP }}:{{ $address.Port }};
+          {{ end }}
+        {{ end }}
+      {{ end }}
+    {{ end }}
+  }
+  {{ end }}
+}
+
+{{ if (and (ne $cert "") (exists (printf "/etc/nginx/certs/%s.crt" $cert)) (exists (printf "/etc/nginx/certs/%s.key" $cert))) }}
+server {
+  server_name localhost data.open-ease.org;
+  listen 443 ssl;
+  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
+  ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
+  ssl_prefer_server_ciphers on;
+  ssl_session_timeout 10m;
+  ssl_session_cache shared:SSL:50m;
+  ssl_certificate /etc/nginx/certs/{{ (printf "%s.crt" $cert) }};
+  ssl_certificate_key /etc/nginx/certs/{{ (printf "%s.key" $cert) }};
+  ssl_dhparam /etc/nginx/certs/dhparams.pem;
+  add_header Strict-Transport-Security "max-age=31536000";
+  proxy_redirect http:// https://;
+  location / {
+    proxy_pass http://localhost;
+  }
+}
+{{ end }}

certs/gencert.sh

@@ -18,6 +18,8 @@
 MAIL_PASSWORD = os.environ.get('OPENEASE_MAIL_PASSWORD')
 MAIL_DEFAULT_SENDER = '"Sender" <[email protected]>'
 
+USE_HOST_KNOWROB = bool(os.environ.get('OPENEASE_USE_HOST_KNOWROB', 'False'))
+
 FACEBOOK_APP_TOKENS = (os.environ.get('FACEBOOK_APP_ID'), os.environ.get('FACEBOOK_APP_SECRET'))
 TWITTER_APP_TOKENS  = (os.environ.get('TWITTER_APP_ID'), os.environ.get('TWITTER_APP_SECRET'))
 GITHUB_APP_TOKENS   = (os.environ.get('GITHUB_APP_ID'), os.environ.get('GITHUB_APP_SECRET'))
@@ -29,7 +31,6 @@
 
 MAX_HISTORY_LINES = 100
 
-MESH_REPOSITORIES = map(lambda x: tuple(x.split(' ')),
-                        os.getenv('OPENEASE_MESHES', 'git https://github.com/PR2/pr2_common').split(','))
-
-ROS_DISTRIBUTION = os.getenv('OPENEASE_ROS_DISTRIBUTION', 'indigo')
+MESH_URDF_SERVER = os.environ.get('MESH_URDF_SERVER')
+if bool(MESH_URDF_SERVER)==False:
+    MESH_URDF_SERVER='http://neem-1.informatik.uni-bremen.de/data/'