Skywriter SpaceBrew Unity Tutorial: (The allegory of the Virtual Cave with real rubber duck)
The concept is to create the experience of acknowledging the virtual world in the real world without looking into screens. There is a virtual maze and a real robot Duck. So, you need to figure out how to get out of the virtual maze that you can't see but you can feel it as a real Duck will stop when it hits the walls of the maze. For the demo, we built only one wall of the maze to demonstrate the idea and show how to work with software and hardware.
- 1x Skywriter HAT - Pimoroni Detect position data and gesture information (flicks, taps) and the motion of your hand with X, Y and Z positions. Skywriter HAT Python library https://github.com/pimoroni/skywriter-hat
- 2x Raspberry Pi 3 (Model B)
- 1x Adafruit 16-Channel PWM/Servo HAT & Bonnet for Raspberry Pi
- 2x Continuous Rotation Servos
The Skywriter HAT can sense position and a few gestures using electrical near-field 3D. The outputs you can expect to receive out of the box are (a) position of a object in (x,y,z), flicks (up, down, left, right), ‘air wheels’, taps, touches and double taps. To access these easily we downloaded the pre-made examples to our RPI using “git clone https://github.com/pimoroni/skywriter-hat”, the read me details all the python libraries and files we needed and how to load them on RPI terminal.
- NOTE: We encountered known issues with “sudo pip install autopy” that we have yet to resolve. However the basic test.py functions without autopy and can be repurposed for your project *
Off the bat we noticed that the resolution of the (x,y,z) position coordinates that the skywriter returned were very good but the range in 3D space above the hat is very narrow. For us air wheels, taps, touches and double taps were fairly unreliable but flicks gave us robust signals reliably. Knowing we wanted a responsive control input for the duck and that SpaceBrew does not like too many inputs at once, we opted to use the flicks as the controller.
We pulled the test.py from the Skywriter examples and refitted it with a publish script from the native SpaceBrew examples. The python script below is what we used to publish our flicks to Spacebrew.
#!/usr/bin/env python
import signal
import skywriter
import sys
import time
import subprocess
from pySpacebrew.spacebrew import Spacebrew
brew = Spacebrew("Bird & Worm", description="publish flicks", server="sandbox.spacebrew.cc", port=9000)
brew.addPublisher("move", "string")
connected = False
CHECK_FREQ = 2 # check mail every 60 seconds
CURR_INDEX = 0 # Create a queue of items
print("sweet mercy, someone find me out here in cyberspace")
try:
brew.start()
while True:
if (connected == True):
@skywriter.flick()
def flick(start,finish):
print('Got a flick!', start, finish)
if ((start == "north") and (finish == "south")):
brew.publish("move", "down")
if ((start == "south") and (finish == "north")):
brew.publish("move", "up")
if ((start == "west") and (finish == "east")):
brew.publish("move", "right")
if ((start == "east") and (finish == "west")):
brew.publish("move", "left")
connected = True
time.sleep(CHECK_FREQ)
signal.pause()
finally:
brew.stop()- NOTE: If you load the file into another folder, insure that there is a pySpacebrew folder in the level.
- NOTE: At this point you should have your skywriter/RPI talking to Spacebrew at the server you specified. You can test the values using “http://spacebrew.github.io/spacebrew.js/spacebrew_string/index.html?server=sandbox.spacebrew.cc&name=cloudString”
Once we built our unity file with the proper “maze” (or in our case just a wall) with all the proper physics and collision events it was time to subscribe to the flicks of the skywriter to move the virtual object. The following c# script contains the code to listen to Spacebrew – this will involve a new script (that we call littleguy.cs) and an edit to SpacebrewEvents.cs. It is important to create a SpacebrewClient component in your space brew objects with all the proper naming and structure – the following photo shows how we filled that out.
**littleguy.cs **
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class littleguy : MonoBehaviour {
public float moveSpeed;
public string virtualstep = "none";
GameObject remoteBrew;
SpacebrewEvents brewbrewSpace;
bool shouldMove = true;
// Use this for initialization
void Start () {
print("yo I'm your guy");
moveSpeed = 100f;
// GameObject remoteBrew = GameObject.Find ("SpacebrewObject"); // the name of your client object
// SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
}
// Update is called once per frame
void Update () {
if (virtualstep == "left") {
GameObject remoteBrew = GameObject.Find ("SpacebrewObject");
SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
brewbrewSpace.moveL();
virtualstep = "none";
}
if (virtualstep == "right") {
GameObject remoteBrew = GameObject.Find ("SpacebrewObject");
SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
brewbrewSpace.moveR();
virtualstep = "none";
}
if (virtualstep == "forward") {
if (shouldMove) {
GameObject remoteBrew = GameObject.Find ("SpacebrewObject");
SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
brewbrewSpace.moveFore();
virtualstep = "none";
}
}
if (virtualstep == "back") {
GameObject remoteBrew = GameObject.Find ("SpacebrewObject");
SpacebrewEvents brewbrewSpace = remoteBrew.GetComponent <SpacebrewEvents> ();
brewbrewSpace.moveBackward();
virtualstep = "none";
}
}
public void moveLeft() {
print("moveLEFT");
transform.Translate(Vector3.left * moveSpeed * Time.deltaTime);
virtualstep = "left";
}
public void moveRight()
{
print("moveRIGHT");
transform.Translate(Vector3.right * moveSpeed * Time.deltaTime);
virtualstep = "right";
}
public void moveForward()
{
print("moveFORWARD");
transform.Translate(Vector3.forward * moveSpeed * Time.deltaTime);
virtualstep = "forward";
}
public void moveBack()
{
print("moveBACK");
transform.Translate(Vector3.back * moveSpeed * Time.deltaTime);
virtualstep = "back";
}
void OnCollisionEnter(Collision collision) {
if (collision.gameObject.name == "theWall") {
Debug.Log ("Hit a wall");
shouldMove = false;
//virtualstep = "back";
}
}
}**littleguy.cs **
using UnityEngine;
using System.Collections;
public class SpacebrewEvents : MonoBehaviour {
SpacebrewClient sbClient;
bool lightState = false;
// Use this for initialization
void Start () {
GameObject go = GameObject.Find ("SpacebrewObject"); // the name of your client object
sbClient = go.GetComponent <SpacebrewClient> ();
sbClient.addEventListener (this.gameObject, "move");
}
// Update is called once per frame
void Update () {
if (Input.GetKeyDown ("space")) {
print ("Sending Spacebrew Message");
// name, type, value
// COMMON GOTCHA: THIS MUST MATCH THE NAME VALUE YOU TYPED IN THE EDITOR!!
sbClient.sendMessage("buttonPress", "boolean", "true");
}
foreach (char c in Input.inputString) {
print("Just pressed: "+c.ToString());
if (c >= 'a' && c <= 'z') {
sbClient.sendMessage("letters", "string", c.ToString());
GameObject go = GameObject.Find ("MatrixContainer"); // the name of your client object
MatrixMaker grid = go.GetComponent <MatrixMaker> ();
grid.ParseIncomingLetter(c.ToString());
grid.delayLayer();
//grid.CreateLayer(true);
}
}
if (Input.touchCount > 0 && Input.touches[0].phase == TouchPhase.Began) {
sbClient.sendMessage("buttonPress", "boolean", "true");
}
}
public void moveFore() {
sbClient.sendMessage("move", "string", "forward");
}
public void moveBackward() {
sbClient.sendMessage("move", "string", "back");
}
public void moveL() {
sbClient.sendMessage("move", "string", "left");
}
public void moveR() {
sbClient.sendMessage("move", "string", "right");
}
public void OnSpacebrewEvent(SpacebrewClient.SpacebrewMessage _msg) {
// print ("Received Spacebrew Message");
// print("message name" + _msg.name);
// print("value" + _msg.value);
// skywriter flicks test
if (_msg.name == "move") {
if (_msg.value == "up") {
//print("up detected");
GameObject go = GameObject.Find("littleguy");
littleguy s = go.GetComponent <littleguy> ();
s.moveForward();
}
if (_msg.value == "down")
{
print("down detected");
GameObject go = GameObject.Find("littleguy");
littleguy s = go.GetComponent <littleguy> ();
s.moveBack();
}
if (_msg.value == "left")
{
print("left detected");
GameObject go = GameObject.Find("littleguy");
littleguy s = go.GetComponent <littleguy> ();
s.moveLeft();
}
if (_msg.value == "right")
{
print("right detected");
GameObject go = GameObject.Find("littleguy");
littleguy s = go.GetComponent <littleguy> ();
s.moveRight();
}
}
}
}- NOTE: Remember your noodles (in Spacebrew) *
In Unity, you need to create the maze (in our case only one wall) and a virtual avatar of robot Duck (Virtual Object), which can be just a sphere or any object. It is required to add Rigidbody to both objects. The mass of the wall should be much bigger than the mass of the avatar (100:1). Add gravity to both the avatar and the wall. Than add plane as a ground to place objects with gravity on it. Later you can disable plane (make it invisible) in Inspector.
To move the avatar, you will use data from Skywriter HAT, but just for testing, you can use KeyArrows. Code for it:
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class MovingObjectCube : MonoBehaviour {
public float moveSpeed=100f;
public float moveTime = 0.1f;
bool shouldMove = true;
void Start () {
}
void Update () {
if (Input.GetKey (KeyCode.DownArrow)&&shouldMove) {
transform.Translate (Vector3.back * moveSpeed * Time.deltaTime)
print ("move back");
}
if (Input.GetKey (KeyCode.UpArrow)&&shouldMove) {
transform.Translate (Vector3.forward * moveSpeed * Time.deltaTime)
print ("move forward");
}
if (Input.GetKey (KeyCode.LeftArrow)&&shouldMove) {
transform.Translate (Vector3.left * moveSpeed * Time.deltaTime);
print ("move left");
}
if (Input.GetKey (KeyCode.RightArrow)&&shouldMove) {
transform.Translate (Vector3.right * moveSpeed * Time.deltaTime);
print ("move right")
}
}To stop the object (the avatar) when it hits another object (the wall), you need to use OnCollisionEnter script:
void OnCollisionEnter(Collision collision) {
if (collision.gameObject.name == "Wall") {
Debug.Log ("Hit Wall");
shouldMove = false;
}
}
This is where things all come together, or fall apart. For much better references on how the Adafruit PMW Servo hat work see the servo examples in the wiki (i.e. hotdog not hotdog). However on the Adafruit PMW Servo hat GitHub you can download the examples by “git clone https://github.com/adafruit/Adafruit_Python_PCA9685.” Suffice to say we tested a few values that got our duck to move close to forward, backward, left and right without understanding the PMW logic. We wrote a script for the RPI to receive values from Spacebrew and translate them into servo moves.
- NOTE: Beware the cold solder *
from __future__ import division
import sys
import time
from pySpacebrew.spacebrew import Spacebrew
import Adafruit_PCA9685
pwm = Adafruit_PCA9685.PCA9685()
# Configure min and max servo pulse lengths
servo_min = 0 # Min pulse length out of 4096
servo_max = 3000 # Max pulse length out of 4096
letter = "emptystring"
CHECK_FREQ = 0.2
brew = Spacebrew("duckfeed", description="duckweed", server="sandbox.spacebrew.cc", port=9000)
brew.addSubscriber("thebeak", "string")
brew.addPublisher("eyeoftheduck", "string")
def handleString(value):
global letter
letter = value
print(letter)
print("this is a duck prayer")
brew.subscribe("thebeak", handleString)
def set_servo_pulse(channel, pulse):
pulse_length = 1000000 # 1,000,000 us per second
pulse_length //= 60 # 60 Hz
print('{0}us per period'.format(pulse_length))
pulse_length //= 4096 # 12 bits of resolution
print('{0}us per bit'.format(pulse_length))
pulse *= 1000
pulse //= pulse_length
pwm.set_pwm(channel, 0, pulse)
pwm.set_pwm_freq(60)
try:
print("Press Ctrl-C to quit.")
brew.start()
while True:
if (letter == "forward"):
pwm.set_pwm(3, 0, 200)
pwm.set_pwm(0, 0, 3000)
time.sleep(1)
pwm.set_pwm(3, 0, servo_min)
pwm.set_pwm(0, 0, servo_min)
letter = "null"
if (letter == "back"):
pwm.set_pwm(3, 0, 3000)
pwm.set_pwm(0, 0, 3000)
time.sleep(2)
pwm.set_pwm(3, 0, 200)
pwm.set_pwm(0, 0, 3000)
time.sleep(1)
pwm.set_pwm(3, 0, servo_min)
pwm.set_pwm(0, 0, servo_min)
letter = "null"
if (letter == "left"):
pwm.set_pwm(3, 0, 300)
pwm.set_pwm(0, 0, 300)
time.sleep(1)
pwm.set_pwm(3, 0, 200)
pwm.set_pwm(0, 0, 3000)
time.sleep(1)
pwm.set_pwm(3, 0, servo_min)
pwm.set_pwm(0, 0, servo_min)
letter = "null"
if (letter == "right"):
pwm.set_pwm(3, 0, 1000)
pwm.set_pwm(0, 0, 1000)
time.sleep(1)
pwm.set_pwm(3, 0, 200)
pwm.set_pwm(0, 0, 3000)
time.sleep(1)
pwm.set_pwm(3, 0, servo_min)
pwm.set_pwm(0, 0, servo_min)
letter = "null"
if (letter == "null"):
pwm.set_pwm(3, 0, servo_min)
pwm.set_pwm(0, 0, servo_min)
time.sleep(CHECK_FREQ)
finally:
brew.stop()**First and very foremost – unless you're comfortable with internet protocol, OSC/UDP/Websockets and processing – We wouldn't recommend trying to connect a MKR1000 to Spacebrew via WiFi. **