Light sensing AR moth controlling a lightbulb through a relay

In this example we will create a mixed reality application that uses a physical button to turn on an AC-powered lightbulb and a digital button to turn it off using a relay. Once the light is on, a digital moth emerges from a flower and turns the light off.

Concept

In this mini-project we adapt Bill Verplank's Do-Feel-Know-loop that underpins interaction design for AR.

Bill Verplank's Do-Feel-Know-loop

The metaphor is that moths are attracted by light. What could that mean for an AR moth? Can it sense light in the conventional reality? - The answer for this project was "No". Though we exploit the well understood light switch concept to make it interact with the AR moth: A human's triggering of a real light switch triggers a change in the virtual world that the AR moth senses. Whenever the real light goes on, a virtual light sitting on top of the light switch is activated. The AR moth then acts as virtual light switch. Whenever it touches the virtual light, the real light goes off.

Hardware Needed

2x Raspberry Pi
1x 2 channel 5V relay module
1x Sense HAT
AC-powered lightbulb
Breadboard
Jumper wires
Adafruit Pi Cobbler
GPIO Ribbon Cable

Software and Frameworks Needed

For the Raspis: Current distro with Python 2.7 and Spacebrew, WiFi/network is assumed to be working
For the AR scene: Unity 2017.3.0f3 with Vuforia and the YUXI examples
One Spacebrew server (we had a local instance running at 192.168.1.191)

Unity assets

For the "moth": Butterfly (animated)
For the virtual light: Vive controller
For the flower: Mystical flower

System Overview

Creating the scene

STEP 1: Wiring "Devil's switch"

There are two parts:

a) From the 2 Relay module connect to the breadboard

Connect GND to GND
Connect IN1 to GPIO pin 21
Connect VCC to +5V

b) From the lamp to the relay

Split the wiring of the lamp so that one wire runs through the relay as shown below: Relay connected to a lightbulb ((c) by howtomechatronics.com)

ATTENTION: 220 voltage! Before handling the AC-Powered cable please disconnect from the mains. Do not touch exposed wires. Use only connectors and cables that are rated for 220 volts. Make sure that all the parts are in a shielded casing that is properly attached to a solid structure.

STEP 2: Software on "Devil's switch"

The following python script is run. It is based on the script YUXI-SenseHat.py. Minor changes were made for sending the button presses through Spacebrew (search for "switch" in the declaration-part and in the While-True-loop). Also, to synchronize the light-switch-state, the script reads the state messages through switchState:

import sys
import time
from pySpacebrew.spacebrew import Spacebrew
from sense_hat import SenseHat

switchOn = False

brew = Spacebrew("Holy Ghost", description="Oye",  server="192.168.1.191", port=9000)
brew.addSubscriber("letters", "string")
brew.addSubscriber("switchState", "boolean")
brew.addPublisher("joystick", "string")
brew.addPublisher("switch", "boolean")

sense = SenseHat()
sense.set_rotation(180)

w = [255, 255, 255]
e = [0, 0, 0]
current_rate = 1.0
sense.clear()

def handleString(value):
    letter = value[0]
    print(letter)
    sense.show_letter(letter)
    time.sleep(0.5)
    sense.clear()

def handleBoolean(value):
    global switchOn
    print("Received: "+str(value))
    if (value == 'true' or str(value) == 'True'):
        switchOn = True
    else:
        switchOn = False

brew.subscribe("letters", handleString)
brew.subscribe("switchState", handleBoolean)

try:
    
    print("Press Ctrl-C to quit.")
    brew.start()

    while True:
        for event in sense.stick.get_events():
            if event.action == "pressed":

                if event.direction == "up":
                    sense.show_letter("U")      # Up arrow
                elif event.direction == "down":
                    sense.show_letter("D")      # Down arrow
                elif event.direction == "left": 
                    sense.show_letter("L")      # Left arrow
                elif event.direction == "right":
                    sense.show_letter("R")      # Right arrow
                elif event.direction == "middle":
                    sense.show_letter("M")      # Enter key
                
                switchOn = not switchOn

                brew.publish('joystick', event.direction)

                brew.publish('switch', "true")

                if switchOn:
                    brew.publish('switch', "true")
                else:
                    brew.publish('switch', "false")

                time.sleep(0.5)
                sense.clear()

finally:
    brew.stop()

STEP 3: Software on "Holy ghost"

The following python script is run to control the relay. It is based on YUXI_LightsButtons.py. Important is that the relay is controlled through GPIO pin 21 - this has to match STEP 1 above:

import sys
import time
from pySpacebrew.spacebrew import Spacebrew

import RPi.GPIO as GPIO

brew = Spacebrew("Devil's Switch", description="Python Light control", server="192.168.1.191", port=9000)
brew.addSubscriber("flipLight", "boolean")

CHECK_FREQ = 0.1 # How often to check the hardware


GPIO.setwarnings(False)
GPIO.setmode(GPIO.BCM) # GPIO Numbers instead of board numbers
 
RELAIS_1_GPIO = 21
GPIO.setup(RELAIS_1_GPIO, GPIO.OUT)
GPIO.output(RELAIS_1_GPIO, GPIO.HIGH)

lightOn = False 
alreadySent = False # to 'debounce' the button


def handleBoolean(value):
    global lightOn
    print("Received: "+str(value))
    if (value == 'true' or str(value) == 'True'):
        lightOn = True
    else:
        lightOn = False

brew.subscribe("flipLight", handleBoolean)

try:
    brew.start()
    print("Press Ctrl-C to quit.")
    while True:
        GPIO.output(RELAIS_1_GPIO, not lightOn)
        time.sleep(CHECK_FREQ)

finally:
    GPIO.cleanup()
    brew.stop()

STEP 4: Unity scene

The unity scene is based on YUXI's BaseBoARd_LightsButtons scene. To reproduce our project:

Move all assets out of sight except for the FiducialObjects
Setup Spacebrew client's name (server address and client name need to be changed accordingly)
Add the assets listed above under path BaseBoARd/YourObjectsGoHere
The butterfly animation is set not to play automatically.
The red glow around the Vive object is deactivated, path: BaseBoARd/YourObjectsGoHere/ViveRed/HTCVIVE20
The FiducialObjects is given a Rigidbody component for the OnTrigger events but with Use gravity turned off
Add a sphere collider to Butterfly/butterfly_gruppe/koerper. Center is X: 0, Y: -2, Z: -4 with a radius = 10. The collider is set up with Is Trigger checked (we use OnTrigger events)

In SpacebrewEvents.cs: Change the function OnSpacebrewEvent where flipLight messages are handled to:

// Look for messages to turn the virtual lamp light on
if (_msg.name == "flipLight") {
	if (_msg.value == "true") {
		lightState = true;
		StartCoroutine(DelayedLightStart());
	}
}

We use a Coroutine - a kind of program part that runs in a non-blocking way in parallel - to add delays to make the micro-interactions feel more plausible.

Also, we add a couple of functions for the light control:

public void switchLightOff() {
	lightState = false;
	sbClient.sendMessage ("buttonPress", "boolean", "false");
}

IEnumerator DelayedLightStart() {
	yield return new WaitForSeconds(0.4F);
	GameObject go = GameObject.Find("BaseBoARd/YourObjectsGoHere/ViveRed/HTCVIVE20");
	go.gameObject.SetActive(lightState);
	yield return new WaitForSeconds(1.0F);
	go = GameObject.Find("BaseBoARd/YourObjectsGoHere/Butterfly");
	Animation anim = go.GetComponent<Animation> ();
	anim.Play();
}

public IEnumerator switchLightOffWithDelay() {
	switchLightOff ();
	yield return new WaitForSeconds(0.3F);
	GameObject go = GameObject.Find("BaseBoARd/YourObjectsGoHere/ViveRed/HTCVIVE20");
	go.gameObject.SetActive(lightState);
}

For the collision trigger we add a script called FiducialTriggers (FiducialTriggers.cs) with the following code:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class FiducialTriggers : MonoBehaviour {

	public GameObject spacebrew;

	// Use this for initialization
	void Start () {
		Debug.Log ("Hello. I'm alive. I switch the light off");
		spacebrew.GetComponent<SpacebrewEvents> ().switchLightOff ();
		GameObject go = GameObject.Find("BaseBoARd/YourObjectsGoHere/Butterfly");
		go.GetComponent<Animation> ().Stop ();
	}

	// Update is called once per frame
	void Update () {
	}

	void OnTriggerExit(Collider collider){
		Debug.Log("Trigger executed");
		StartCoroutine(spacebrew.GetComponent<SpacebrewEvents> ().switchLightOffWithDelay ());
	}			
}

Once you've added this script to FiducialObjects in your scene graph, make sure that the public variable spacebrew refers to SpacebrewObject (path: BaseBoARd/Spacebrew/SpacebrewObject).

STEP 5: Spacebrew configuration

Prerequisites:

Make sure that the two Raspis are up and the respective script is running.
Make sure that the unity scene is running

Then:

Connect Devil's switch flipLight to Holy Ghost's switch
Connect Devil's switch flipLight to Flying Spaghetti Monster's buttonPress
Connect Flying Spaghetti Monster's buttonPress to HolyGhost's switchState

(cf. the system overview above)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly