Merge pull request #155 from hatchbed/feature/cors-and-range-support-…

…rebase

Support integrating with external applications

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Bag Database changelog
 
+Forthcoming
+- Add CORS and Range support to /bags/download
+- Add context menu item to open bags in external applications
+
 3.3.0
 - Refactor bag storage to allow multiple backends with different sources
 - Add S3 storage backend implementation

docs/configuration/configuration.md

@@ -49,6 +49,13 @@ ldapServer:
 ldapUserPattern: uid={0},ou=People,dc=example,dc=com
 metadataTopics: 
 - /metadata
+openWithUrls:
+  'Webviz':
+      - 'https://webviz.io/app/?'
+      - 'remote-bag-url'
+  'Foxglove Studio':
+      - 'https://studio.foxglove.dev/?'
+      - 'remote-bag-url'
 vehicleNameTopics: 
 - /vms/vehicle_name
 - /vehicle_name

docs/index.md

@@ -36,6 +36,8 @@ for bags, downloading them, and doing post-processing on them.
   - [Map View](web-interface/list-view#map-bag): Display all of a bag's (or a series
   of bags') GPS coordinates on an interactive map so you can see where your vehicle
   went.
+  - [Open With External Apps](web-interface/open-with): Bag files can be streamed to external
+  web applications such as [Webviz](https://webviz.io/) for more complex visualizations.
 - **[Authentication](configuration/ldap)**: If you don't want everybody on your
   network to have open access to your database, the Bag Database can be connected
   to an LDAP server for authenticating users.

docs/installation/docker/behind-a-reverse-proxy.md

@@ -3,7 +3,7 @@ layout: default
 title: Behind a Reverse Proxy
 parent: Docker
 grand_parent: Installation
-nav_order: 4
+nav_order: 5
 description: "Running the Bag Database behind a Reverse HTTP Proxy"
 permalink: /installation/docker/behind-a-reverse-proxy
 ---

docs/installation/docker/docker.md

@@ -54,6 +54,7 @@ see [Storage](../../configuration/storage.md) for an example.
 | `LDAP_SERVER` | The LDAP server for authentication.  If set to an empty string, LDAP authentication will not be enabled, and anonymous users may connect. | |
 | `LDAP_USER_PATTERN` | The pattern for finding user DNs in the LDAP server.  `{0}` will be replaced with the username from the login form. | |
 | `METADATA_TOPICS` | A comma-separated list of `std_msgs/String` topics in bag files that will be searched for metadata.  The messages on the topic should be newline-separated tags that are made of colon-separated key/value pairs; see [Metadata Example](#metadata-example) for an example. | |
+| `OPEN_WITH_URLS` | A block of YAML describing external applications that can open bag files. See [Open With Format](#open-with-format). | `"{'Webviz':['https://webviz.io/app/?', 'remote-bag-url']}"` |
 | `SCRIPT_TMP_PATH` | Path to write temporary script files and bag files downloaded from remote storage backends.  This can be empty if you do not intend to run scripts or use remote storage. It must be writable by the bag database, and the Docker service that runs the scripts **must have it mounted as a volume at the same location as the Bag Database**. | /scripts | 
 | `TILE_HEIGHT_PX` | The height of the tiles returned from the tile map in pixels. | 256 |
 | `TILE_MAP_URL` | If `USE_TILE_MAP` is `true`, this URL will be used as a template for retrieving map tiles from a WMTS tile server.  See the documentation for the `url` property of OpenLayers' [ol.source.XYZ](http://openlayers.org/en/latest/apidoc/ol.source.XYZ.html) class.  The default value will use the terrain map provided by [Stamen](http://maps.stamen.com/). | http://{a-d}.tile.stamen.com/terrain/{z}/{x}/{y}.jpg |
@@ -76,3 +77,31 @@ another with the name `email` and value `[email protected]`.
 
 Tag names are unique.  If a topic has multiple messages and there are any tags with duplicate names, later ones
 will overwrite earlier ones.
+
+#### Open With Format
+
+By default, the Bag Database will show an "Open With Webviz" item in context menus that can be used to
+open bag files with https://webviz.io/app/ .  This can be customized to disable it or supply a
+custom list of applications.
+
+The configuration should be a YAML map in a single line so that it can be passed through as
+an environment variable.  For clarity, here's another example that is formatted more cleanly:
+
+```yaml
+'Webviz': 
+- 'https://webviz.io/app/?'
+- 'remote-bag-url'
+'Foxglove Studio':
+- 'https://studio.foxglove.dev/?'
+- 'remote-bag-url'
+```
+
+Each object in the map has a key that is used as the label in the UI, and each object has a list with
+two strings; the first is the base URL of the application, and the second is a parameter that should
+be set to the URL of the bag file.  In the above example, if a user is accessing a bag database at
+`http://localhost:6080/bagdb/` and selects a bag file with an ID of "10", it will open this URL
+in a new window: `https://webviz.io/app/?remote-bag-url=http%3A%2F%2Flocalhost%3A6080%2Fbagdb%2Fbags%2Fdownload%3FbagId%3D10`
+
+If a user selects multiple bags at once, the Bag Database will supply additional parameters based
+on what Webviz expects; for example, `remote-bag-url-2`.  A URL for two bags might look like:
+`https://webviz.io/app/?remote-bag-url=http%3A%2F%2Flocalhost%3A6080%2Fbagdb%2Fbags%2Fdownload%3FbagId%3D10&remote-bag-url-2=http%3A%2F%2Flocalhost%3A6080%2Fbagdb%2Fbags%2Fdownload%3FbagId%3D11`

docs/installation/docker/with-a-private-registry.md

@@ -3,7 +3,7 @@ layout: default
 title: With a Private Registry
 parent: Docker
 grand_parent: Installation
-nav_order: 2
+nav_order: 3
 description: "Using docker-compose to start up a Bag Database with a private Docker registry"
 permalink: /installation/docker/with-a-private-registry
 ---

docs/installation/docker/with-ldap-authentication.md

@@ -3,7 +3,7 @@ layout: default
 title: With LDAP Authentication
 parent: Docker
 grand_parent: Installation
-nav_order: 2
+nav_order: 4
 description: "Using docker-compose to start up a Bag Database with an LDAP server"
 permalink: /installation/docker/with-ldap-authentication
 ---

docs/installation/docker/with-s3-storage.md

@@ -3,7 +3,7 @@ layout: default
 title: With S3 Storage
 parent: Docker
 grand_parent: Installation
-nav_order: 5
+nav_order: 6
 description: "Using docker-compose to start up a Bag Database that stores files in an S3 bucket"
 permalink: /installation/docker/with-s3-storage
 ---

docs/installation/docker/with-webviz.md

@@ -0,0 +1,258 @@
+---
+layout: default
+title: With Webviz
+parent: Docker
+grand_parent: Installation
+nav_order: 7
+description: "Running the Bag Database with a Self-Hosted Webviz Instance behind a Reverse Proxy"
+permalink: /installation/docker/with-webviz
+---
+
+# With a Self-Hosted Webviz Instance behind a Reverse Proxy
+
+This is a fairly complex example that resembles something more like what you might use
+in an actual production environment.  It combines quite a bit from the other, simpler examples.
+
+This Bag Database will:
+- Use LDAP for authentication
+- Run the Bag Database and [Webviz](https://webviz.io/) behind a [Traefik](https://traefik.io/) reverse proxy
+- Use Traefik to enforce SSL on all connections
+- Automatically generate a signed SSL certificate for `bagdb.example.com` through [Let's Encrypt](https://letsencrypt.org/)
+- Use the self-hosted Webviz instance for opening bag files
+
+To make the docker-compose.yml file a bit cleaner, environment variables for many of the containers
+have been pulled out into separate files.  All of these files should be saved in the same directory,
+and if you're going to run a server based on them, make sure you edit them to add your own passwords,
+domain name, and user information.
+
+## Files
+
+### `bagdb.env`
+
+Environment variable for the Bag Database.
+
+```
+ADMIN_PASSWORD=random_bagdb_password
+DB_DRIVER=org.postgresql.Driver
+DB_PASS=random_postgres_password
+DB_URL=jdbc:postgresql://postgres/bag_database
+DB_USER=bag_database
+DOCKER_HOST=http://docker:2375
+GPS_TOPICS=/localization/gps, gps, /vehicle/gps/fix, /localization/sensors/gps/novatel/raw, /localization/sensors/gps/novatel/fix, /imu_3dm_node/gps/fix, /local_xy_origin
+LDAP_BINDDN=cn=admin,dc=example,dc=com
+LDAP_BIND_PASSWORD=random_ldap_password
+LDAP_SEARCH_BASE=ou=People,dc=example,dc=com
+LDAP_SERVER=ldap://openldap
+LDAP_USER_PATTERN=uid={0},ou=People,dc=example,dc=com
+METADATA_TOPICS=/metadata
+OPEN_WITH_URLS={'Webviz':['https://bagdb.example.com/webviz/?', 'remote-bag-url']}
+VEHICLE_NAME_TOPICS=/vms/vehicle_name, /vehicle_name
+```
+
+### `openldap.env`
+
+Environment variables for the LDAP server.
+
+```
+LDAP_ORGANIZATION=My Company
+LDAP_DOMAIN=example.com
+LDAP_ADMIN_PASSWORD=random_ldap_password
+```
+
+### `postgres.env`
+
+Environment variables for the PostGIS server.
+
+```
+POSTGRES_PASSWORD=random_postgres_password
+POSTGRES_USER=bag_database
+POSTGRES_DB=bag_database
+```
+
+### `webviz-default.conf`
+
+A custom configuration file for Webviz's nginx server.  We need to change the `location` here
+because it will be running under an alias at `/webviz` in our reverse proxy.
+
+```
+server {
+    listen 8080;
+    server_name  localhost;
+
+    #access_log  /var/log/nginx/host.access.log  main;
+
+    location /webviz {
+        alias  /usr/share/nginx/html;
+        index  index.html index.htm;
+    }
+
+    #error_page  404              /404.html;
+
+    # redirect server error pages to the static page /50x.html
+    #
+    error_page   500 502 503 504  /50x.html;
+    location = /50x.html {
+        root   /usr/share/nginx/html;
+    }
+
+    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
+    #
+    #location ~ \.php$ {
+    #    proxy_pass   http://127.0.0.1;
+    #}
+
+    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
+    #
+    #location ~ \.php$ {
+    #    root           html;
+    #    fastcgi_pass   127.0.0.1:9000;
+    #    fastcgi_index  index.php;
+    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
+    #    include        fastcgi_params;
+    #}
+
+    # deny access to .htaccess files, if Apache's document root
+    # concurs with nginx's one
+    #
+    #location ~ /\.ht {
+    #    deny  all;
+    #}
+}
+```
+
+### `docker-compose.yml`
+
+The main docker-compose.yml file.  If everything is in place, you can start the server with
+`docker-compose up -d`.
+
+After everything is running, you will be able to access the server at `https://bagdb.example.com`.
+
+```yaml
+version: '3.6'
+services:
+    traefik:
+        image: traefik:v2.4
+        command:
+            - --api.insecure=true
+            - --providers.docker
+            - --entrypoints.web.address=:80
+            - --entrypoints.websecure.address=:443
+            - --entrypoints.web.http.redirections.entrypoint.to=websecure
+            - --entrypoints.web.http.redirections.entrypoint.scheme=https
+            - --certificatesresolvers.bagdbresolver.acme.email=youremail@example.com
+            - --certificatesresolvers.bagdbresolver.acme.storage=/etc/traefik/acme/acme.json
+            - --certificatesresolvers.bagdbresolver.acme.httpchallenge.entrypoint=web
+        networks:
+            - bagdb
+        ports:
+            - "80:80"
+            - "443:443"
+            - "8080:8080"
+        volumes:
+            - /var/run/docker.sock:/var/run/docker.sock
+            - acme:/etc/traefik/acme
+    docker:
+        image: docker:dind
+        privileged: yes
+        networks:
+            - bagdb
+        volumes:
+            - bags:/bags:ro # Needs to match the path in the bagdb container
+            - scripts:/scripts
+            - docker_cache:/var/lib/docker
+        command: ["dockerd", "--host=tcp://0.0.0.0:2375"]
+    bagdb:
+        image: ghcr.io/swri-robotics/bag-database:latest
+        networks:
+            - bagdb
+        depends_on:
+            - postgres
+            - openldap
+        labels:
+            - "traefik.http.routers.bagdb.rule=Host(`bagdb.exaple.com`)"
+            - "traefik.http.routers.bagdb.tls=true"
+            - "traefik.http.routers.bagdb.tls.certresolver=bagdbresolver"
+        volumes:
+            - bags:/bags
+            - indexes:/root/.ros-bag-database/indexes
+            - scripts:/scripts
+        env_file:
+            - bagdb.env
+    postgres:
+        image: postgis/postgis:11-2.5
+        networks:
+            - bagdb
+        volumes:
+            - postgres:/var/lib/postgresql/data
+        env_file:
+            - postgres.env
+    openldap:
+        image: osixia/openldap:1.3.0
+        networks:
+            - bagdb
+        volumes:
+            - ldap:/var/lib/ldap
+            - slapd:/etc/ldap/slapd.d
+        ports:
+            - "389:389"
+            - "636:636"
+            # These ports are exposed so you can easily connect to the container to add
+            # data by, for example, using the LDIF files below.  You can hide these ports
+            # if you don't need to connect to the container, or perhaps even add another
+            # web server behind the reverse proxy to act as an LDAP administration interface.
+        env_file:
+            - openldap.env
+    webviz:
+        image: cruise/webviz
+        networks:
+            - bagdb
+        volumes:
+            - ./webviz-default.conf:/etc/nginx/conf.d/default.conf:ro
+        labels:
+            - "traefik.http.routers.webviz.rule=Host(`bagdb.example.com`) && PathPrefix(`/webviz`)"
+            - "traefik.http.routers.webviz.tls=true"
+            - "traefik.http.routers.webviz.tls.certresolver=bagdbresolver"
+            - "traefik.http.services.webviz.loadbalancer.server.port=8080"
+networks:
+    bagdb: {}
+volumes:
+    acme:
+    bags:
+    docker_cache:
+    postgres:
+    ldap:
+    slapd:
+    indexes:
+    scripts:
+        driver_opts:
+            type: 'tmpfs'
+            device: 'tmpfs'
+```
+
+### `people.ldif`
+
+An example LDIF file for creating a "People" group in your LDAP server.  After starting the server,
+run `ldapadd -x -D cn=admin,dc=example,dc=com -W -f people.ldif` to add this group.
+
+```
+dn: ou=People,dc=example,dc=com
+objectClass: organizationalUnit
+ou: People
+```
+
+### `user.ldif`
+
+An example LDIF file that defines a single user.  Customize this for each user, and after you've
+added the People group, run `ldapadd -x -D cn=admin,dc=example,dc=com -W -f user.ldif` to add
+this person to the server.
+
+```
+dn: uid=jdoe,ou=People,dc=example,dc=com
+objectClass: inetOrgPerson
+uid: jdoe
+sn: Doe
+givenName: John
+cn: jdoe
+displayName: jdoe
+userPassword: letmein
+```

docs/installation/docker/without-compose.md

@@ -3,7 +3,7 @@ layout: default
 title: Without Compose
 parent: Docker
 grand_parent: Installation
-nav_order: 3
+nav_order: 2
 description: "How to run it without docker-compose"
 permalink: /installation/docker/without-compose
 ---

docs/web-interface/administration.md

@@ -2,7 +2,7 @@
 layout: default
 title: Administration
 parent: Web Interface
-nav_order: 6
+nav_order: 1
 description: "Administration"
 permalink: /web-interface/administration
 ---

docs/web-interface/bag-details.md

@@ -2,7 +2,7 @@
 layout: default
 title: Bag Details
 parent: Web Interface
-nav_order: 4
+nav_order: 2
 description: "Bag Details"
 permalink: /web-interface/bag-details
 ---