weathAR: Weather AR simulation using Raspberry pi, Unity and Spacebrew

The design of the AR experience was built around the idea of having access to cities’ weather around a physical object. The team used objects from Google Poly kit for weather icons and designed the layouts around the pre setup fiducial objects.

To build the weather roulette you will need the following equipment

HARDWARE

• Raspberry pi 3
• DC Motor with a hat -https://www.adafruit.com/product/2348
• Breadboard
• Jumper Cables
• A platform (cardboard, MDF or lego blocks) to secure the DC motor and hold the fiduciary
• External 5V DC power supply for the DC Motor.
• Ipad/Iphone (optional)

SOFTWARE

• Unity Software on a suitable computer. (Download the personal version from https://unity3d.com/)
• Spacebrew library
• Python (for the Raspberry pi)
• Associated libraries

Additionally we will be try to connect the Raspberry pi and Unity to the Spacebrew server to allow for online input control.

General Schematics

This is a general schematic of the prototype. Please note the iPad is optional. The same can be done with your computer (with its internal camera or a USB connected web camera).

SCHEMATIC

Overall, we will try to spin the DC Motor platform (square platform for this project) for a random amount of time so the edge facing the camera is variable. Each platform edge represents a particular city. The DC Motor can also be controlled through another user through the space brew platform. Once the spinning stops, you will see the weather for that particular edge popping out as you see it through unity.

The AR weather app required a mix of physical components and software. In this tutorial, we’ll walk you through it all. First, we’ll begin with the graphical design, followed by the construction of the device, and finally the code.

1.BUILDING THE CIRCUIT AND THE CODE:

CONNECT DC MOTOR TO RASP PI: Connected the DC Motor with the Raspberry pi and upload data to the server.

Instructions on assembling the circuit and example code can be found at https://www.adafruit.com/product/2348. Please note the needed libraries and PIN numbers for connecting the Motor. Here is a picture of the connected circuit. You will need an external 5V power supply for the DC Motor.

Circuit Diagram

• Uploaded is the code used to connect the raspberry pi with the DC motor. This code also included the associated space brew libraries and code to subscribe and publish data from the raspberry pi module.
• The current code takes in; input ‘a’ + return to start motor, and ; input ‘s’ + return to stop motor
• You can use spacebrew to send input over the server to start the motor as well.
• Note the publisher and subscriber names and types.
• Make sure pySpacebrew.spacebrew is in the same folder as the python script.

DCTEST3.py

#!/usr/bin/python
# -*- coding: utf-8 -*-
from Adafruit_MotorHAT import Adafruit_MotorHAT, Adafruit_DCMotor
from pySpacebrew.spacebrew import Spacebrew
import time
import atexit
import random
import sys,tty
# publish start status - bool
# listen for input to motor - bool
# double check server address
brew = Spacebrew("Weather_Roulette", description="dc motor controller",  server="192.168.1.165", port=9000)
brew.addSubscriber("motorOn", "boolean")
brew.addPublisher("startOn", "boolean")
  
tty.setcbreak(sys.stdin)  
key = ord(sys.stdin.read(1))  
# key captures the key-code
# based on the input we do something - in this case print something
#not needed for this code
# create a default object, no changes to I2C address or frequency
mh = Adafruit_MotorHAT(addr=0x60)
lightOn = False
keyPress = "s"
# recommended for auto-disabling motors on shutdown!
def turnOffMotors():`
    mh.getMotor(1).run(Adafruit_MotorHAT.RELEASE)
    mh.getMotor(2).run(Adafruit_MotorHAT.RELEASE)
    mh.getMotor(3).run(Adafruit_MotorHAT.RELEASE)
    mh.getMotor(4).run(Adafruit_MotorHAT.RELEASE)
 atexit.register(turnOffMotors)
 ################################# DC motor test!
 myMotor = mh.getMotor(3)
 # set the speed to start, from 0 (off) to 255 (max speed)
 myMotor.setSpeed(150)
 myMotor.run(Adafruit_MotorHAT.FORWARD);
 # turn on motor
 myMotor.run(Adafruit_MotorHAT.RELEASE);
 #print(randomTime);

 def handleBoolean(value):
     global lightOn
     global keyPress
     print("Received: "+str(value))
     if (value == 'true' or str(value) == 'True'):
        lightOn = not lightOn
        keyPress == "a"
        myMotor.run(Adafruit_MotorHAT.FORWARD)
        print("Start Motor")
        myMotor.setSpeed(100)
     elif (value == 'false' or str(value) == 'false'):
        keyPress == "s"
        print("Motor stopped")
        myMotor.setSpeed(1)
        myMotor.run(Adafruit_MotorHAT.RELEASE)

 brew.subscribe("motorOn", handleBoolean)

 try:
     brew.start()
     #print("Should be looping")
     print("Press Ctrl-C to quit.")

     while True:
         print("hi")
         keyPress = raw_input("enter command")
         keyPress = keyPress`
         if keyPress=="a":
        
            `print("Forward! ")`
            `myMotor.run(Adafruit_MotorHAT.FORWARD)`
            `randomTime = int(random.uniform(50,255))`
            `print(randomTime)`
            `print("\tSpeed up...")`
            `#for i in range(randomTime):`
            `print("in loop")`
            `myMotor.setSpeed(randomTime)`
            `brew.publish('startOn',True)`
            `time.sleep(0.1)`
        
         elif keyPress=="s": 
             print("you pressed s")
             #break
             print("outofloop")
             myMotor.setSpeed(1)
             #print("Release")
             myMotor.run(Adafruit_MotorHAT.RELEASE)
             brew.publish('startOn',False)
             time.sleep(1.0)
 finally:
     brew.stop()

2. PLATFORM CONSTRUCTION

To allow the viewer to scan between the weather of different cities, the team constructed a sturdy platform that was spun by a DC motor. Firstly, a lego structure was created so that the DC could be firmly housed.

https://github.com/shalinster/weathAR/blob/master/construction%20pic.jpg

The team then laser cut 5 MDF rings to fit tightly on the head of the motor. The rings were 150 mm in diameter, and have a small hole in the middle (3 mm by 5 mm) so that they can fit snug onto the motor. The rings were glued together. Following this, the fiducial marker was glued onto the top ring, so that when the motor turned, so did the fiducial. From a graphical perspective, this allowed the different cities’ weather to smoothly change when the motor turned.

3. DESIGN AND CREATE ASSETS IN UNITY

Design:

Create assets and scenes in Unity. Unity is a game design engine very commonly used by game designers. It also provides a great platform for mixed reality projects like these. Remember you will need to have a Fiducial marker on the platform for Unity to project your assets on the platform. Here you can learn more about Unity if you have never used it before. https://unity3d.com/learn/courses

Animation:

The animations were scripted for each object for a richer experience. The rain object was created using 4 casule game objects to individually animate them.

Here are the scripts

1. Cloud animation:

 using UnityEngine;
 using System.Collections;
 public class cloudExperiment2 : MonoBehaviour
 {
    public AnimationCurve myCurve;

public float speed = 2f;
float height = 2.4f;
float newy = 0f;
  
void Update() {
//get the objects current position and put it in a variable so we can access it later with less code
Vector3 pos = transform.position;
//calculate what the new Y position will be
             
               if(pos.y > -40f){
              newy = pos.y - speed;
               }
              else{
                 newy = 30f;
              }
              transform.position = new Vector3(pos.x, newy, pos.z);

              float newY = Mathf.Sin(Time.deltaTime * speed); 
               //set the object's Y to the new calculated Y 
      } 
 }

Tornado animation:

   using System.Collections;
  using System.Collections.Generic;
   using UnityEngine;

   public class capsuleExperiments : MonoBehaviour {

       public float speed = 100.0f;
           public Renderer rend;


       // Use this for initialization
       void Start () {
           rend = GetComponent<Renderer>();

       }
      
      // Update is called once per frame
       void Update () {
          

 // rotation example
          transform.Rotate(Vector3.up, speed * Time.deltaTime);
          
       }
   }

4. ADD INTERACTION SCRIPTS TO THE UNITY SOFTWARE:

To be able to just see the weather for the edge closest to the camera, a script is written in C# and attached to associated game objects. 4 Gameobjects (set as invisible)are created to match each edge and hence each city. They are placed at the centre of each respective edge. Then a script is written to evaluate the distance of each object to the camera. The gameobject closest to the camera is part of the platform edge facing the camera. It is the only edge that should display the weather. The weather assets from all other edges are set to inactive.

The associated scripts are attached:

FindClosest.cs (attached to main camera and calculates closest distance)

using System.Collections;
  using System.Collections.Generic;
  using UnityEngine;

public class FindClosest : MonoBehaviour {

   void Start () {

       }
 // Update is called once per frame
    void Update () {
     FindClosestEnemy ();
 	 } 

    void FindClosestEnemy()
    {
   	
// define and locate your gameobjects
  GameObject triangleEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/triangle/");
 		
  GameObject squareEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/square");
 		
  GameObject circleEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/circle/");
		
  GameObject polygonEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/polygon/");

  // gameobjects for weather assets	
 GameObject rainEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/rain/");
 		
 GameObject sunEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/sun");
 		
  GameObject lightEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/Lightning2/");
 		
 GameObject tornadoesEnemy = GameObject.Find("BaseBoARd/YourObjectsGoHere/TestObjectsForBaseboARd/Container/Capsule/Tornadoes");

  // set all objects to Active initially
 		 sunEnemy.gameObject.SetActive(true);
 		 rainEnemy.gameObject.SetActive(true);
 		 lightEnemy.gameObject.SetActive(true); 
 		 tornadoesEnemy.gameObject.SetActive(true);


 		 Enemy something1;
 		 Enemy something2;
		 Enemy something3;
		 Enemy something4;



  // calculate closest distance		
  float distanceToClosestEnemy = Mathf.Infinity;
 		
  Enemy closestEnemy = null;
 		
  Enemy[] allEnemies = GameObject.FindObjectsOfType<Enemy>();

 		 foreach (Enemy currentEnemy in allEnemies) {
 			 float distanceToEnemy = (currentEnemy.transform.position - this.transform.position).sqrMagnitude;
			 if (distanceToEnemy < distanceToClosestEnemy) {
 				 distanceToClosestEnemy = distanceToEnemy;
                                closestEnemy = currentEnemy;
}
}

		
// print on screen to test 
print ("nearesrt enemy is "+closestEnemy);
print ("triangle is "+triangleEnemy);

something1 = triangleEnemy.GetComponent<Enemy>();
something2 = squareEnemy.GetComponent<Enemy>();
something3 = circleEnemy.GetComponent<Enemy>();
something4 = polygonEnemy.GetComponent<Enemy>();

if (closestEnemy == something1){
print("do a thing with the triange!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
//set all other objects to inactive

sunEnemy.gameObject.SetActive(false);
rainEnemy.gameObject.SetActive(false);
lightEnemy.gameObject.SetActive(false);
			
}

				
if (closestEnemy == something2){
print("do a thing with the Square/////////////////./../.s/./s.ds/.");
// set other objects to inactive

sunEnemy.gameObject.SetActive(false);
lightEnemy.gameObject.SetActive(false);
tornadoesEnemy.gameObject.SetActive(false);

}

 				
if (closestEnemy == something3){
print("do a thing with the circle!0000000000000000000");
// set other objects to inactive
sunEnemy.gameObject.SetActive(false);
tornadoesEnemy.gameObject.SetActive(false);
rainEnemy.gameObject.SetActive(false);


}


if (closestEnemy == something4){
print("do a thing with the polygon____________________________");
// set other objects to inactive

tornadoesEnemy.gameObject.SetActive(false);
rainEnemy.gameObject.SetActive(false);
lightEnemy.gameObject.SetActive(false);
}


//Debug.DrawLine (this.transform.position, closestEnemy.transform.position);
}

}

Enemy.cs (placed on the invisible gameobjects. Does not do anything)

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

   public class Enemy : MonoBehaviour {

 	 // Use this for initialization
  	 void Start () {
  		
  	 }
  	
 	 // Update is called once per frame
 	 void Update () {
  	 }
 }

5. SPACEBREW FILES for interfacing with the spacebrew server

The associated SpacebrewEvents.cs and SpacebrewClient.cs scripts are also included to help unity connect to the spacebrew or a local server. Make sure you have the right server address and portnumbers when connecting to spacebrew or a local server.

SpacebrewClient.cs

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using WebSocketSharp;
using SimpleJSON;
using System;
using System.Reflection;

public class SpacebrewClient : MonoBehaviour {

	[Serializable]
	public class Publisher
		{
			public string name;
			public enum type{
				BOOLEAN = 0,
				STRING = 1,
				RANGE = 2
			};
			public type pubType;
			//public string defaultValue;
		}

	[Serializable]
	public class Subscriber
	{
		public string name;
		public enum type{
			BOOLEAN = 0,
			STRING = 1,
			RANGE = 2
		};
		public type subType;
	}

	public class SpacebrewMessage
	{
		public string name;
		public string type;
		public string value;
		public string clientName;
	}

	public class SpacebrewEvent
	{
		public GameObject sbGo;
		public string sbEvent;
	}

	public WebSocket conn;
	public string serverAddress; // you can include the port number so ws://192.168.7.2:9000
	public Publisher[] publishers;
	public Subscriber[] subscribers;
	public string clientName;
	public string descriptionText;
	public ArrayList SpacebrewEvents;
	List<SpacebrewEvent> spacebrewEvents = new List<SpacebrewEvent>();
	List<SpacebrewMessage> spacebrewMsgs = new List<SpacebrewMessage>();

	void Awake() {
		conn = new WebSocket (serverAddress); // removed WebSocket on begin
		conn.OnOpen += (sender, e) => {
						print ("Attempting to open socket");
				};

		conn.OnMessage += (sender, e) => {
			print (e.Data);

			// parse the incoming json message from spacebrew
			var N = JSON.Parse(e.Data);
			var cMsg = new SpacebrewMessage();
			cMsg.name = N["message"]["name"];
			cMsg.type = N["message"]["type"];
			cMsg.value = N["message"]["value"];
			cMsg.clientName = N["message"]["clientName"];
		
			print (cMsg);
			spacebrewMsgs.Add(cMsg);
			//ProcessSpacebrewMessage(cMsg);

//			if (e.Type == Opcode.Text) {
//				// Do something with e.Data
//				print (e);
//				print (e.Data);
//				return;
//			}
//			
//			if (e.Type == Opcode.Binary) {
//				// Do something with e.RawData
//				return;
//			}

		};

		conn.OnError += (sender, e) => {
			print ("THERE WAS AN ERROR CONNECTING");
			print (e.Message);
		};

		conn.OnClose += (sender, e) => {
			print ("Connection closed");
		};

		print ("Attemping to connect to " + serverAddress);
		conn.Connect ();

		//addPublisher ("power", "boolean", "0");
		//addSubscriber ("hits", "boolean");

		// Connect and send the configuration for the app to Spacebrew
		conn.Send (makeConfig().ToString());
	}

	// You can use these to programatically add publisher and subsribers
	// otherwise you should do it through the editor interface.
	void addPublisher(string _name, string _type, string _default) {
		var P = new JSONClass();
		P ["name"] = _name;
		P ["type"] = _type;
//		if (_default != "") {
//			P ["default"] = _default;
//		}
		//publishers.Add(P);
	}
	
	void addSubscriber(string _name, string _type) {
		var S = new JSONClass();
		S ["name"] = _name;
		S ["type"] = _type;
		//subscribers.Add(S);
	}
	
	private JSONClass makeConfig() {
		// Begin the JSON config
		var I = new JSONClass();
		I["name"] = clientName;
		I["description"] = descriptionText;
		
		// Add all the publishers
		print ("there are " + publishers.Length);
		for (int i = 0; i < publishers.Length; i++) // Loop through List with for
		{
			var O = new JSONClass();
			O["name"] = publishers[i].name;
			string tType = "empty";
			switch ((int)publishers[i].pubType) {
			case 0:
				tType = "boolean";
				break;
			case 1:
				tType = "string";
				break;
			case 2:
				tType = "range";
				break;
			}
			O["type"] = tType;
			O["default"] = "";
			I["publish"] ["messages"][-1] = O;
		}
		
		// Add all the subscribers
		for (int i = 0; i < subscribers.Length; i++) // Loop through List with for
		{
			var Q = new JSONClass();
			Q["name"] = subscribers[i].name;
			string tType = "empty";
			switch ((int)subscribers[i].subType) {
			case 0:
				tType = "boolean";
				break;
			case 1:
				tType = "string";
				break;
			case 2:
				tType = "range";
				break;
			}
			Q["type"] = tType;
			I["subscribe"] ["messages"][-1] = Q;
		}

		
		// Add everything to config
		var C = new JSONClass();
		C ["config"] = I;
		
		print("Connection:");
		print(C.ToString());
		print("");
		
		return C;
	}
	
	void onOpen() {
		
	}
	
	void onClose() {
		
	}

	public void addEventListener(GameObject _sbGo, string _event) {
		print ("Adding a listener for " + _event);
		SpacebrewEvent evt = new SpacebrewEvent();
		evt.sbGo = _sbGo;
		evt.sbEvent = _event;
		spacebrewEvents.Add(evt);
	}

	public void sendMessage(string _name, string _type, string _value) {
		var M = new JSONClass();
		M["clientName"] = clientName;
		M["name"] = _name;
		M["type"] = _type;
		M["value"] = _value;

		var MS = new JSONClass ();
		MS ["message"] = M;
		conn.Send(MS.ToString());
		//        conn.Send (makeConfig().ToString());
		
		//       {
		//         "message":{
		//           "clientName":"CLIENT NAME (Must match the name in the config statement)",
		//           "name":"PUBLISHER NAME (outgoing messages), SUBSCRIBER NAME (incoming messages)",
		//           "type":"DATA TYPE",
		//           "value":"VALUE",
		//       }
		//   }
		
	}

	void ProcessSpacebrewMessage(SpacebrewMessage _cMsg) {
//		foreach (SpacebrewEvent element in spacebrewEvents)
//		{
//			//This will now work because you've constrained the generic type V
//			print(element.sbEvent);
//			if (_cMsg.name == element.sbEvent) {
//
//				// if this element subscribes to this event then call it's callback
//				//element.eventCallback
//				//element.sbGo.OnSpacebrewEvent(_cMsg);
//				element.sbGo.SendMessage("OnSpacebrewEvent", _cMsg);
//				//this.GetComponent<SpacebrewEvents>().OnSpacebrewEvent(_cMsg);
//				//this.GetComponent<MyScript>().MyFunction();
//				//print(element.sbGo);
//				//element.sbGo.gameObject.SpacebrewEvent(_cMsg);
//				//element.sbGo.
////				MethodInfo mi = element.sbGo.GetType().GetMethod(element.eventCallback);
//				//mi.Invoke(element.sbGo, null);
//			}
//		}
		if (_cMsg.name == "hits") {
			if (_cMsg.value == "true"){
				print ("do something");
					//pillVisible = !pillVisible;
				}
			}		
	}

	// Use this for initialization
	void Start () {
		
	}
	
	// Update is called once per frame
	void Update () {

		// go through new messages
		foreach (SpacebrewMessage element in spacebrewMsgs)
		{
			//This will now work because you've constrained the generic type V
			//print(element.sbEvent);
			//if (_cMsg.name == element.sbEvent) {
				
				// if this element subscribes to this event then call it's callback
				//element.eventCallback
				//element.sbGo.OnSpacebrewEvent(_cMsg);
				//element.sbGo.SendMessage("OnSpacebrewEvent", _cMsg);
				this.GetComponent<SpacebrewEvents>().OnSpacebrewEvent(element);
				//this.GetComponent<MyScript>().MyFunction();
				//print(element.sbGo);
				//element.sbGo.gameObject.SpacebrewEvent(_cMsg);
				//element.sbGo.
				//				MethodInfo mi = element.sbGo.GetType().GetMethod(element.eventCallback);
				//mi.Invoke(element.sbGo, null);
			//}
		}
		spacebrewMsgs.Clear();

		if (Input.GetKeyDown ("space")) {
			print ("Sending Spacebrew Message");
			//sendMessage();
		}
		//GameObject.Find("pill").renderer.enabled = pillVisible;
	}
}

SpacebrewEvents.cs

using UnityEngine;
using System.Collections;

public class SpacebrewEvents : MonoBehaviour {

	`SpacebrewClient sbClient;
	`bool lightState = false;

/*

Ok, let's do this all in one script?

BaseExample: None?
HelloWorld
	P: None
	S: launchSatellite, string
LightsButtons
	P: buttonPress, boolean
	S: flipLight, boolean
SenseHat
	P: letter, string
	S: direction, string
		up,down,left,right,middle (default)
	S: layer, string

p:2
s:4

 */


	// Use this for initialization
	void Start () {
		GameObject go = GameObject.Find ("SpacebrewObject"); // the name of your client object`
		sbClient = go.GetComponent <SpacebrewClient> ();

		// register an event with the client and a callback function here.
		// COMMON GOTCHA: THIS MUST MATCH THE NAME VALUE YOU TYPED IN THE EDITOR!!
		sbClient.addEventListener (this.gameObject, "launchSatellite");
		sbClient.addEventListener (this.gameObject, "lightOn");
		sbClient.addEventListener (this.gameObject, "letters");
	}

	// Update is called once per frame
	void Update () {
		if (Input.GetKeyDown ("space")) {
			print ("Sending Spacebrew Message");
		// name, type, value
			// COMMON GOTCHA: THIS MUST MATCH THE NAME VALUE YOU TYPED IN THE EDITOR!!
			sbClient.sendMessage("buttonPress", "boolean", "true");
		}`
		foreach (char c in Input.inputString) {
			print("Just pressed: "+c.ToString());
			if (c >= 'a' && c <= 'z') {
				sbClient.sendMessage("letters", "string", c.ToString());
				GameObject go = GameObject.Find ("MatrixContainer"); // the name of your client object
				MatrixMaker grid = go.GetComponent <MatrixMaker> ();
				grid.ParseIncomingLetter(c.ToString());
				grid.delayLayer();
				//grid.CreateLayer(true);
				
			}
		}

		if (Input.touchCount > 0 && Input.touches[0].phase == TouchPhase.Began) {
			sbClient.sendMessage("buttonPress", "boolean", "true");
		}
	}

	public void OnSpacebrewEvent(SpacebrewClient.SpacebrewMessage _msg) {


		print ("Received Spacebrew Message");
		print (_msg);

		// Look for incoming Satellite messages
		if (_msg.name == "launchSatellite") {
			if (_msg.value == "true") {
				GameObject go = GameObject.Find("BaseBoARd/YourObjectsGoHere/CenterOfUniverse");
				OrbitManager om = go.GetComponent <OrbitManager> ();
				om.makeSatellite();
				print("Tried to launch Satellite");
			}
		}

		// Look for messages to turn the virtual lamp light on
		if (_msg.name == "letters") {
			GameObject go = GameObject.Find ("MatrixContainer"); // the name of your client object
			MatrixMaker grid = go.GetComponent <MatrixMaker> ();
			grid.ParseIncomingLetter(_msg.value[0].ToString());
			grid.delayLayer();
		}

		// Look for messages to turn the virtual lamp light on
		if (_msg.name == "lightOn") {
			//print(go);
			if (_msg.value == "true") {
				GameObject go =
GameObject.Find("BaseBoARd/YourObjectsGoHere/Lamp/SpacebrewSpotlight");
				lightState = !lightState;
				go.gameObject.SetActive(lightState);
			}
		}



		`//if (_msg.name == "letters") {`
		`//print(go);`
		`//if (_msg.value == "true") {`
		`// GameObject go = GameObject.Find ("MatrixContainer"); // the name of your client object`
		`// MatrixMaker grid = go.GetComponent <MatrixMaker> ();`
		`// grid.CreateLayer(true);`
		`// grid.ParseIncomingString(_msg.value);`
	}

}

Here’s a glimpse at the final product!