Skip to content

Educational augmented reality AI lab assistant and simulation with multi-user in the same environment support. Hands On offers a 3D virtual lab sandbox, allowing them to conduct scientific experiments using realistic virtual tools. One of its standout features is AI assistance, which provides real-time guidance, feedback, and explanations .

License

Notifications You must be signed in to change notification settings

project-slot/HandsOn

Repository files navigation

Notice of possible occurrence:

After publishing this Github repo, I saw that my Azure credit was significantly dropped, resulting in Speech API deny. This might be due to bad people taking advantage of my Azure keys and endpoints, which are hard coded (although a bad practice but due to time constrain). I'm trying to fix it so that the speech-to-text will work again.

Docs

0 2 Development architecture. Server architecture is underdevelopment.

Hands On AI Argumented Lab Demo

Laboratory equipment are expensive. Learning on 2D blackboard is unengaging. Aiming to make laboratories accessible to students around the world; Helping teachers to make lesson more engaging. Hands On is AR simulation equipped with AI for both teachers & students to interact, experiment, and make mistakes freely.

Screenshot 2023-10-30 094619

Video

Group 3 Note: Suggested UI (under development).

Interacting with the environment using natural language

a. Speak to Hands On:

  • “I need a weight[object] made of iron[material] to conduct a buoyancy force experiment”.
  • “Change the simulation gravity[field] to the moon’s gravity[value]”.
  • “Set the height[field] of liquid[target] to 57cm[value]”.
  • “Spawn something to measure length[description]”.

b. Best Practices: i. It’s good to keep it concise about what your intentions are. ii. You can ask Hands On to assist you with details. Example: “Show me the forces analysis of objects.” iii. Beside speech interaction, you can also interact with the simulation using real-world physical movement. Example: “grab object, move object around.” iv. Users are also able to take notes anywhere in the environment, so the world is your infinite, limitless canvas.

What's next for Hands On

  • Real-time AI feedback
  • Multi User Collaboration on the same AR environment
  • Tool recognition using computer vision
  • IOS ARKit development
  • Classroom / lab safety practice

Inspiration

I remember the first time I went to my physics lab during my high school years to experiment with electronic circuits. The equipment looks nothing familiar to me even though I have been studying it for a year. Turn out I was working with 2D symbol drawings the whole time, without touching or experiencing myself.

What it does

Laboratory equipment are expensive. Learning on the 2D blackboard is unengaging. Making Physic Experiment accessible anywhere, Hands On is an AR simulation for both teachers and students to interact, experiment, and make mistakes freely. It makes classroom experience more engaging for everyone. Better yet, Hands On is equipped with AI to instruct, explain to students in real-time: like "You should turn off the power source before plugging in", or perform actions like "Give me the set of equipment for Archimedes experiment", which can enhance creativity, safety, and familiarity.

Screenshot 2023-10-30 094740

How we built it

Input utterances, label data, train, test, improve, and deploy AI model using Azure CLU. Connected Cognitive Search and SQL database. Code 3D physic simulations from scratch using Unity C# Develop for Android AR Core

Many challenges still to come

Screenshot 2023-10-30 140037 It's just a super early state. The model is still improving and hasn't generalized things well enough. Currently, the project is just a prototype with some simple requests.

Future goals

  1. AI Real-time feedback (Anticipated example use cases):
  • “Why is my spectrometer not giving accurate readings?”

  • Real-Time Feedback: Hands On can suggest checking the alignment, the quality of the light source, or the calibration of the spectrometer.\

  • "Is it safe to heat this substance with the Bunsen burner?"

  • Real-Time Feedback: Hands On can provide safety guidelines, including the substance's properties and suitable temperature ranges for heating.

  1. AR Multi-user in the same environment / Tools recognition using computer vision
  • Enhance collaboration that allows students to work together and share experiences.
  • Allow teachers to demonstrate how the experiment should be conducted.
  • Import tools into the simulation.

About

Educational augmented reality AI lab assistant and simulation with multi-user in the same environment support. Hands On offers a 3D virtual lab sandbox, allowing them to conduct scientific experiments using realistic virtual tools. One of its standout features is AI assistance, which provides real-time guidance, feedback, and explanations .

Resources

License

Stars

Watchers

Forks