RealWear Collaborate and AI

This repository contains the code for RealWear Collaborate and the RealWear AI Demos. Internally at RealWear we use the same framework for both applications as the codebase can be used to create custom solutions utilising both AI and Microsoft Teams. While these are separate apps, you can create bespoke experiences that utilise functions from either to create compelling hands-free solutions.

Note: Windows Users - All of our build scripts are written for bash and not PowerShell. Please use WSL2 (Windows Subsystem for Linux) or the VSCode Dev Container to build and run any of the server components

RealWear Collaborate

To build and run RealWear Collaborate you need both an Android APK and a Server. We have created build scripts for both.

Components:

• Android APK - A device app which hosts the RealWear Collaborate hands-free application
• Loader - An Angular "loader" which is compiled into the APK. This checks for the existence of the server, and provides basic retry and connectivity logic if required
• RWT Device - An Angular front-end web app which displays the UI for signing-in and joining a meeting
• RWT Teams Addon - An Angular front-end web app which hosts the Microsoft Teams addon for joining a call with a 5 digit code
• RWT Backend - A NodeJS Express Server used for generating ACS (Azure Communication Services) Tokens and facilitating the Teams Addon connect codes

The RealWear Collaborate server builds as a self-contained Docker image with a corresponding Helm chart. You can choose to host the Docker image manually, or within a Kubernetes cluser using Helm.

Prerequisites:

• Bash - terminal environment for scripts (ie: NOT Windows)
• Android Studio - If you wish to modify the Android APK
• Docker - To build the APK and Server components
  • If you are not modifying the APK, you only need Docker and not Android Studio
• VS Code - To host the Dev Container - A fully enclosed development environment with all required dependencies to build the server components
  • Redis - If you opt to develop outside of VSCode Dev Containers, a running Redis server is required
• Azure Tenant - RealWear Collaborate uses a Microsoft EntraID Application to authenticate the user
• Azure Communication Services - Used to connect to Microsoft Teams for meetings and calls - See Setting up Azure Communication Services
• Minikube - To host locally the server

Building the Android APK

We have provided a Docker build script for easily building the APK. This will the following parameters:

SITE_URL: the location where your server UI will be hosted. DATADOG_CLIENT_TOKEN: (optional) client token for Datadog log reporting.

• From your Terminal:

  SITE_URL="http://localhost:4200" \
    DATADOG_CLIENT_TOKEN="pub1234567890" \
    .devops/rwt/rwtbuildapk.sh
  

• The APK will be dropped in dist/android/build folder

Note: The unit tests are run by default, and must succeed for the Docker build to complete

Building in Android Studio

Before you open in Android Studio, you must build the loader project. This will automatically copy the loader app into the assets folder of the Android file structure. Run yarn rwt:deploy_loader from your VSCode Dev Container to start this process.

Open Android Studio at android/apps/rwt and build/deploy as usual. The SITE_URL defaults to http://localhost:4200 which is fine for debugging, however, you can modify the build.gradle.kts file if you wish to use a different SITE_URL.

Preparing your Server Environment Variables

Before you start, you need to make sure that you have identifed and created the following environment variables:

Environment Variable	Description
AZURE_CLIENT_ID	An Azure EntraID Client ID for your application.
AZURE_CLIENT_SECRET	A corresponding secret for the EntraID application. Used for the Teams Addon when joining a meeting.
AZURE_TENANT_ID	(Optional) Used if you wish to restrict the tenant that the users can sign in to.
ACS_CONNECTION_STRING	A fully provisioned Azure Communication Services connection string with the ability to create anonymous tokens.

Create a copy of the .localConfigs.template file and rename to .localConfigs and fill in your values. Many of the scripts assume that a valid .localConfigs file exists.

Double check these values before continuing to Buildings or Debugging

Debugging the Server

The easiest thing to do is to use the Built-In DevContainer. Open VSCode and launch the project in the DevContainer (your must have Docker installed). Then, from your terminal, simply launch:

• yarn rwt:local

This will build and run all 3 required projects (backend server, device frontend app and teams addon).

To test that this works:

• http://localhost:4200 - The Teams addon
• http://localhost:4300 - The Device frontend app

Building the Microsoft Teams addon package

See Setting up the Microsoft Teams Addon.

Exposing Localhost to Android

If your Android device (RealWear Navigator-520) is plugged in via USB to your host computer, you can expose local ports through ADB (Android Debug Bridge). Simply run: adb reverse tcp:4300 tcp:4300 and when your device hits localhost, it will be reverse proxied through to your development machine.

Building the Server Docker image

From the root directory run: docker build -f .devops/rwt/Dockerfile.web -t rwt . (replace rwt with the desired name).

To host the Docker image, please make sure you have a well-defined .localConfigs file and then start the docker image with:

TBD - Coming in future revision

Hosting in Minikube

For a near-production environment, you can host RealWear Collaborate in a Minikube environment. Minikube is a development (not for production) Kubernetes environment that you can run locally on your machine.

Pre-requisites:

• A .localConfigs file in your project root directory with the following environment variables (see above)
• A running Minikube cluster with KUBE_CONTEXT set to minikube

Run the script .devops/rwt/rwtfullminikube.sh from the root project directory. This script will:

1. Build the RWT Docker image
2. Build the Helm Chart
3. Provision the Minikube cluster (and install Redis if necessary)
4. Deploy and Test the Helm Chart
5. Expose port 4200 on localhost for accessing the service

If successful, you should be able to browse to localhost:4200 from your browser, or, build the Android APK and point to your dev machine

Note: be sure to adb reverse tcp:4200 tcp:4200 to access localhost from your Android device

RealWear AI Demos

Similar to RealWear Collaborate, the RealWear AI Demos are hosted as an Angualar front-end app powered by a NodeJS backend.

The AI Demos do not currently run within a Kubernetes environment (they are much simpler), however we do provide a Docker image for simple hosting.

Components:

• Android APK - A device app which hosts the RealWear AI Demos
• Loader - An Angular "loader" which is compiled into the APK. This checks for the existence of the server, and provides basic retry and connectivity logic if required (this is shared between Collaborate and AI Demos)
• AI - An Angular front-end web app which displays the UI for signing-in and the basic AI Demos
• Backend - A NodeJS Express Server used for connecting to the relevant services

The RealWear Collaborate server builds as a self-contained Docker image with a corresponding Helm chart. You can choose to host the Docker image manually, or within a Kubernetes cluser using Helm.

Prerequisites:

• Bash - terminal environment for scripts (ie: NOT Windows)
• Android Studio - If you wish to modify the Android APK
• Docker - To build the APK and Server components
  • If you are not modifying the APK, you only need Docker and not Android Studio
• VS Code - To host the Dev Container - A fully enclosed development environment with all required dependencies to build the server components
• Azure Tenant - RealWear Collaborate uses a Microsoft EntraID Application to authenticate the user for the AI Demos - See Setting up EntraID
• Azure Speech Service - RealWear AI Demos uses Azure AI Speech for Dictation - See Setting up Azure Speech
• Azure OpenAI Service - RealWear AI Demos uses Azure OpenAI for the LLM interactions - See Setting up Azure OpenAI

Environment Variables

To debug or execute the application, the following environment variables must be present in a .localConfigs file. Copy the existing .localConfigs.template and use as a reference:

Environment Variable	Description
AZURE_CLIENT_ID	An Azure EntraID Client ID for your application.
AZURE_CLIENT_SECRET	A corresponding secret for the EntraID application. Used for the Teams Addon when joining a meeting.
AZURE_TENANT_ID	(Optional) Used if you wish to restrict the tenant that the users can sign in to.
AZURE_OPENAI_ENDPOINT	An Azure OpenAI Service Endpoint
AZURE_OPENAI_KEY	(Optional) A secret key used to authenticate with the Azure OpenAI Service
AZURE_SPEECH_KEY	A secret key for authenticating to Azure Speech Services
AZURE_SPEECH_REGION	The region that hosts the Azure Speech Services (ie: eastus2)

Using Managed Identity

RealWear AI supports authenticating to the Azure services (AI / Speech) through Environment or Managed Identity. If you are deploying this service to an Azure PaaS environment which includes Managed Identity, then you can authenticate automatically to the Azure OpenAI Service without requiring an authentication key (AZURE_OPENAI_KEY).

Alternatively, if you are developing locally and have logged in via the Azure CLI (provided within the Dev Container) then RealWear AI will attempt to authenticate to the Azure OpenAI Service via your developer credentials.

See DefaultAzureCredential class on the Azure SDK for JavaScript documentation.

Debugging the Server

To launch the server, use yarn ai:local. This will run both the front and back end applications and serve through http://localhost:4200.

Note: Please ensure that you have a valid .localConfigs file setup with the appropriate keys

To expose your localhost server to a running Android APK, call adb reverse tcp:4200 tcp:4200

Running in Production

RealWear AI is deployed using Docker. You can build the Docker file using the rwaiserver.sh. This will output a docker file with a tag rwai which you can run directly or push to a container registry and run on your cloud provider.

Note: Be sure to load the appropriate environment variables at runtime

License and Usage Guidelines

RealWear Collaborate is licensed under the Apache License, Version 2.0. This permissive license allows use, reproduction, and distribution of the software with or without modifications, provided that the terms of the license are met.

Key Points:

• You may use, modify, and distribute this software in source or object form.
• Redistributions must include a copy of the license and retain the copyright notice.
• Any significant contributions you make to this project are subject to the same license.

Commercial Licensing

If your use case cannot comply with the terms of the Apache 2.0 license or you require additional support, commercial licenses are available. A commercial license includes:

• Access to proprietary extensions and tools.
• Dedicated support and consulting.