Real-time Object Detection & Translation Web App for TravelAR

XR Project for COMP4461 Human Computer Interaction Spring 24 Authors: Tiffany Fu, Nicole Lucas, Sherie Lam, Edith Leung

This is a web app that could detect & identify object and provide translation in real-time. It is built using Next.js, ONNXRuntime, and YOLOv7 model.

deployed

https://main.d1oms94am8cez.amplifyapp.com

video paper

https://www.youtube.com/watch?v=otRh5jk1fiw&t=10s

1. Getting Started

How to run this project on your local machine for development and testing purposes.

Prerequisites

In order to run this project, you will need to have the following software installed on your machine:

• Node.js
• A web browser

Installation

1. Install the necessary dependencies by running:

npm install
# or 
yarn install

2. Set up .env.local on root:

3. Start the development server by running:

npm run dev
# or
yarn dev

4. Open your web browser and navigate to http://localhost:3000 to view the application.

Installation as PWA

This app can also be installed on your device (desktop or mobile) as a progressive web app (PWA). Here's how:

1. Visit the app's URL in a web browser that supports PWAs (such as Google Chrome or Firefox).
2. Look for the "Install" or "Add to Homescreen" button in the browser's interface.
3. Click the button and follow the prompts to install the app.
4. The app will now be installed on your device and can be launched from the homescreen like any other app.

Deployment

This project is deployed using AWS Amplify for public access.

• How to use AWS Amplify
• How to deploy a Next.js application

2. Built With

• ONNXRuntime - An open-source project for running inferences using pre-trained models in a variety of formats.
• YOLOv7 - A Object detection model which is used in this project.
• Next.js - A JavaScript framework for building server-rendered React applications.
• PWA - A progressive web app that can be installed on a user's device and run offline, providing a native-like experience.

Acknowledgement

This project is built based on the following two repos:

@juanjaho - Object Detection Web App by Juan Sebastian @colbyfayock - AI Translator by Colby Fayock

Citation for YOLOv7

@article{wang2022yolov7,
  title={{YOLOv7}: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors},
  author={Wang, Chien-Yao and Bochkovskiy, Alexey and Liao, Hong-Yuan Mark},
  journal={arXiv preprint arXiv:2207.02696},
  year={2022}
}