Applications
Here are the applications in the SAGE3 system.
Experiment on triggering AI models from the UI.
Simple viewer application from CSV files. No operations on the data is available. CSV files can be uploaded to the asset manager and are automatically opened using this application. CSV files are parse on the client side using the csv-parse library. It could potentially parse large files using its stream API, but it is not used yet. The table is built using React Virtuoso which is a high-performance virtualized list component for React.
Internal application to generate charts in chartjs for the HCDP (Hawaii Mesonet) data visualization application.
Internal application (Arti) to generate charts from datasets using a natural language processing (NLP) module.
Clock is a simple application that displays time using an SVG clock face (current time of the client). D3.js is used to render the clock face and the hands.
An input box allows the user to type a location to show the time at that location. Google maps is used to geocode the location and timezonedb is used to get the time zone information.
The location/timezone process is buggy right now.
An experiement in cobrowsing. The user starting the session is the "driver" and streams the pixels of the browser window to the other users. The other users are "passengers" and see the driver's browser window. Pixels are captured using Electron API of the webview and sent using websockets. A webRTC peer-to-peer approach could be used instead of websockets.
Application not functional yet.
A simple counter application that can be used to test the SAGE3 system and a tutorial for creating new applications, showing how to maintain state and communicate with the server.
Experiment on displaying tabular data and applying operations on the data (filter, sort, ...) using Python backend.
A viewer for DeepZoom images. DeepZoom images are a format for tiled images that can be used to display large images. The DeepZoom format is supported by many other frameworks.
Large images are processed using VIPS and converted to DeepZoom format. It resuls in a directory with a dzi file and a files directory with the tiles, that needs to be hosted on a web server.
We use the OpenSeadragon library to display the images.
Interaction:
- Click and drag to pan
- Scroll to zoom in and out
- Click to zoom in, Shift+click to zoom out
- Keys - and = to zoom in and out
- Keyboard arrows to pan
- Minimap on the top-right shows the current view
A simple 3D model viewer using the Three.js library. We also use the React render for Three.js react-three/fiber. The model is loaded using the GLTFLoader. GLTF files can be uploaded to the asset manager and are automatically opened using this application. The 3D file can be downloaded from the application.
This is a work in progress, to be integrated with the HDM VR work in Unity.
Internal data visualization application for the Hawaii Mesonet project.
Application is displaying images. Valid formats are handled when the file is uploaded to the asset manager (JPEG, PNG, ...). HEIC are currently not supported because they require a native library to be installed on the server. After upload, all images are scaled to multiple resolutions and converted to the efficent web-compatible WebP format. The original image is also kept for download, but converted to JEPG for compatibility. The viewer matches dynamically the window size, dpi of the display, and scale of the board to an optimal resolution.
The application supports displaying bounding boxes generated by AI models. The application has also access to all the metadata collected on the initial file (EXIF, IPTC, XMP, ...).
(ref: NASA)
Webview into the Jupyter instance running in the backend. Might be deprecated in the future (security and collaboration issues). Migtht be replaced by a standard notebook render (PDF, image, ...) or a nbviewer instance.
Application to create and monitor Jupyter kernels in the backend.
Deprecated, to be converted to a panel.
Simple map application using the Leaflet library and the react-leaflet module react-leaflet. The user can move the map using the mouse or the keyboard. The navigation is synchronized across all users. Street or satellite view can be selected. An input box allow a user to type a location to show the map at that location. ESRI geocoder is used to geocode the location (using a SAGE3 API key).
GeoJSON data can be uploaded to the asset manager and displayed on the map. The data is displayed using a Leaflet.GeoJSON data layer.
Notepad is a collaborative text editor. It uses the Quill library to display the text. Styling can be applied to the text. Produced text can be downloaded as a HTML file. Collaboration is implemented peer-to-peer using Yjs where text deltas are exchanged between peers. The text is stored in the SAGE3 application state as needed.
Internal application to display the results of a PDF analysis. Multiple Python libraries are used to extract information from the PDF file in the backend. The results are displayed in a table: metadata, links, references, ...
PDF viewing application. PDF files are uploaded to the asset manager and are automatically opened using this application. The PDF file can be downloaded from the application. PDF files are processed in the backend using pdf-js and converted to images (at multiple resolutions). The images are then displayed in the application. Navigation across pages is synchronized across all users. Multiple pages can be displayed at the same time to leverage the available screen space. The file can be downloaded as a PDF.
Shortcut keys:
- Next page: Arrow right, Arrow down
- Previous page: Arrow left, Arrow up
- First page: 1 key
- Last page: 0 key
- Add a page: + key
- Remove a page: - key
IFrame application to allow the use of external applications. It can be coupled with the @sage3/sageplugin npm package to synchronize the state of the application across all users.
All kind of HTML/CSS/JS applications can be used. The application plugin is uploaded and hosted on the SAGE3 server (see plugin documentation).
The computational cell application. Python code can be entered in the cell and executed in a Jupyter kernel in the backend. Other languages possible (R, Julia, etc). The results are displayed in the cell: we support most of the cell output formats such as text, markdown, images, PDF, HTML and plotly. The cell can be resized and moved around the board. The cell is synchronized with all users. The code can be downloaded as a '.py' Python file and uploaded to the asset manager. Opening a Python file from the asset manager will open a new cell with the code. A code cell can be duplicated for quick prototyping. The editor used the powerful Monaco editor.
A user can instatiate many kernels in the backend (using the Kernel panel) and switch each cell to any of the available kernels. Execution can be stopped at any time. The cell can be cleared to remove all the output.
Implemented in the TwilioScreenshare application. Screensharing is backed by a Twilio backend service to manage users and rooms in a peer-to-peer fashion. Your server has to be configured with a Twilio account and a valid API key. The application is using the twilio-video npm package to manage the screensharing session.
A user can select a monitor or a window to share. The quality of the video is adjusted automatically to the available bandwidth. The video is synchronized across all users. Because of the associated costs, a session is limit to 75 minutes by default, but another session can be started right away. When the user sharing the screen leaves the board, the session is automatically closed. The user can also close the session manually. Multiple users can share their screen at the same time.
Experiment in AI, NLP and Programming.
Internal data visualization application for the Hawaii Mesonet project.
Post-it note application. A user can create a post-it note and move it around the board. The note is synchronized across all users. The note can be resized and the font size can be adjusted. The note can be downloaded as a Text or Markdown file. Opening a '.md' file from the asset manager will open a new note with the text.
User can select the color of the stickie, the font size, and to lock the stickie to prevent other users from editing it. While editing, 'Shift+Tab' will create a new stickie next to the current one. The name of the user is shown in the title bar of the stickie.
Editor for the Vega-Lite visualization language. The editor used the powerful Monaco editor. It loads a JSON specification and displays the visualization in the VegaLiteViewer application. JSON files from the asset manager can be opened in the editor.
Internal data visualization application for the VegaLite application. Allows the visualization to be saved as PNG image file. The viewer is using the vega-embed to render a Vega-Lite visualization.
Video viewing application. Web-compatible video files are uploaded to the asset manager and are automatically opened using this application. MP4 format is the best option. MOV files might work if MP4 codec is used internally. The video file can be downloaded from the application. Navigation is synchronized across all users in a best effort fashion. The file can be downloaded as a video. Video are muted by default to prevent audio feedback.
Webview is browser window that can be opened in the board. It can be used to display most web pages. The navigation is synchronized across all users, but NOT the scroll position. Non-deterministic web pages (such as pages with ads) or pages with wegbl/canvas (such as games) might not synchronize properly. Login pages might not work properly. It is best to use public content inside webviews (like public page of a Google doc).
Basic navigation and scaling is provided in the toolbar. Webviews are muted by default to prevent audio feedback. Using 'Ctrl-click / Cmd-click' when clicking on a link in a webview will open the link in a new webview next to the current one.
The webview is using the webview API from Electron.
