SIM2VR is an extension of the User-in-the-Box (UitB) framework for integrating biomechanical simulations directly into Unity applications. This is achieved by running the user simulation and the Unity application in separate processes (in parallel), while ensuring their temporal and spatial synchronicity.
By integrating biomechanical simulations into Unity applications, SIM2VR enables real and simulated users to perceive and control exactly the same virtual environment. This increases the ecological validity of simulation-based predictions for VR interaction. SIM2VR can be used to predict and analyse differences in performance and ergonomics between specific design choices, and to anticipate potential user strategies for a given VR application.
Paper link: SIM2VR: Towards Automated Biomechanical Testing in VR (UIST 2024)
Please cite the following paper when using SIM2VR:
@inproceedings{FischerIkkala24,
author = {Fischer, Florian and Ikkala, Aleksi and Klar, Markus and Fleig, Arthur and Bachinski, Miroslav and Murray-Smith, Roderick and H\"{a}m\"{a}l\"{a}inen, Perttu and Oulasvirta, Antti and M\"{u}ller, J\"{o}rg},
title = {SIM2VR: Towards Automated Biomechanical Testing in VR},
year = {2024},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3654777.3676452},
doi = {10.1145/3654777.3676452},
booktitle = {Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology},
numpages = {15},
location = {Pittsburgh, PA, USA},
series = {UIST '24}}
SIM2VR requires that the Unity application has an OpenXR plugin (min. version 1.5.3) to handle the VR device interaction. For the user simulation, any simulated user instance created within the UitB framework can be used (min. version 2.0). Note that this requires the biomechanical model to be implemented in the MuJoCo physics engine.
The current focus of SIM2VR is on movement-based VR interaction using VR controllers and an HMD. Since the UitB framework only includes visual and proprioceptive sensor modalities, SIM2VR is currently limited to the transmission of visual output signals from the VR application to the simulated user. However, we plan to support other feedback modalities such as auditory and haptic output in the future.
SIM2VR provides two new components to the UitB framework -- UnityEnv and UnityHeadset -- and a Unity asset SIM2VR Asset.
-
UnityEnv is a UitB task module that enables Unity applications to be used as the simulation environment. Specifically, UnityEnv communicates with the SIM2VR asset and exchanges information between the simulations: UnityEnv sends the HMD and controllers' pose to the Unity application, and receives the image rendered in the virtual HMD, a scalar reward, and other stateful information. Additionally, the UnityEnv module equips the biomechanical model with the virtual HMD and controllers.
-
UnityHeadset is a UitB perception module that receives the visual observation from the Unity application through the UnityEnv module. UnityHeadset is a simple perception module that can e.g. filter color channels and stack multiple observations to allow the control policy to infer movement.
-
SIM2VR Asset handles the communication with the UnityEnv task module, and provides an RLEnv class. This class must be inherited and implemented separately for each Unity application. All necessary functionality required for the RL training (e.g. proper initialisation and resetting, as well as the reward function) need to be defined within the inherited class.
In the following, we demonstrate how SIM2VR can be used to generate user simulations for a given Unity application. As an example, we consider the Beat Saber-style game implemented in the VR Beats Kit, which is freely available on the Unity Asset Store.
-
Set up your Unity project for VR development with OpenXR. Follow e.g. this video. Most importantly, you need to install the OpenXR Plugin (min version 1.5.3) and XR Interaction Toolkit through the Package Manager (Window -> Package Manager).
-
Clone or download this repository, and import the "sim2vr.unitypackage" into your Unity project (Assets -> Import Package -> Custom Package ...).
-
Add the XR Origin rig through GameObject -> XR -> XR Origin (Action-based)
-
Add the sim2vr prefab as a game object into the desired scene.
-
Connect the SimulatedUser fields to the LeftHand Controller, RightHand Controller and Main Camera of the XR Origin rig, and to the RLEnv of the sim2vr game object.
To train a simulated user interact with the Unity application, appropriate reward and reset methods need to be defined. For this purpose, an application-specific class must be inherited from the RLEnv class provided by the SIM2VR asset. Note that all game objects and variables relevant for the reward calculation must be accessible from this class. For example, if the distance of a game object to the controller is used as a reward, the game object's and controller's positions should be fields of this class.
The task-specific reward needs to be computed by the method CalculateReward and stored in the variable _reward. If a game score is provided by the VR application, this score can be directly used as reward (note that game scores typically accumulate points throughout the round, so the reward signal should be set to the increase in that score since the last frame). If necessary, the typically sparse game reward can be augmented by additional, more sophisticated terms, as described in the accompanying paper. In the VR Beats game, a player receives rewards for hitting approaching boxes. Each hit is worth 50-200 points, depending on the hit speed and direction. As the rewards accumulate throughout the game play, we use the difference between successive frames as the RL training reward.
The method Reset needs to ensure that the entire scene is reset to a (reproducible) initial state at the end of each round (i.e., RL training episode). This usually includes the destruction of game objects created during runtime and resetting variables required to compute the game reward. All code related to resetting the game reward should be defined in the method InitialiseReward. Preparations for the next round, such as choosing a game level or defining variables required for the reward calculations, can also be defined here. Actions and settings that should be taken only once when starting the game can be defined in the method InitialiseGame. For the VR Beats game, it is sufficient to simply invoke the existing onRestart game event, which triggers the restart of the level in the VR_BeatManager and the ScoreManager, and set the current game score for reward calculation to 0 in the method Reset
Finally, the simulated user needs to be informed about whether the current episode has ended, i.e., the variable _isFinished needs to be updated accordingly within the method UpdateIsFinished. This is the approapriate method for logging other variables of interest as well, by saving them into the _logDict dictionary, which will be then uploaded into Weights&Biases. In the Beats VR game, we make use of the method getIsGameRunning of the VR_BeatManager instance to track whether the episode has ended.
Since including an application- and task-dependent time feature (i.e., information regarding elapsed time or time remaining in the episode) as "stateful" information in the observation may help in training the simulated user, the RLEnv class provides a method GetTimeFeature to define this time feature. Note that implementing this method for deterministic environments may lead to the simulated user exploiting this information instead of e.g. relying on visual perceptions. As the Beats VR game is such a deterministic environment, we did not define the time feature.
Often, a Unity application commences with an initial scene, such as a menu, rather than directly starting the game. As SIM2VR does not provide a mechanism to switch scenes, this needs to be manually implemented. If possible, this should be implemented in the Reset function, otherwise, the application source code may need to be modified.
As the biomechanical model, we use a bimanual version of the MoBL_ARMS model. Furthermore, to speed up the training, we replaced the muscle actuators with joint-wise torque actuators. Additionally, the Beats VR game includes walls that approach the player and must be dodged. As MoBL_ARMS model is a static upper body model, we chose to play only the first 10 seconds of the game, which do not contain these walls. The first 10 seconds include four boxes that approach the player and must be hit; if a box passes by the player without a hit, the game terminates.
The Logger is optional and can be used to log individual trials, for example, when collecting data from a user study. It needs to be defined separately for each application, as the logged variables will be different.
From the resulting Unity project augmented by the SIM2VR scripts and game objects, a standalone Unity Application can be built. This application is then used as interaction environment for the UitB simulated user during training.
After preparing the VR Interaction environment for running user simulations, a simulator needs to be created in UitB.
All relevant information can be defined in the YAML config file (see here). The config file used for the Beats VR game can be found here. Defining the YAML mainly involves:
- selecting a biomechanical user model (bm_model), including the effort model (effort_model) and effort cost weight (weight)
- selecting perception modules, including the vision.UnityHeadset module provided by SIM2VR (perception_modules)
- providing the path of the standalone Unity Application to interact with (unity_executable)
Other optional parameters include:
- optional arguments to be passed to the VR application (e.g., to set a specific game level or difficulty) (app_args)
- the VR hardware setup (gear)
- the position and orientation of the VR controllers (left_controller_relpose, right_controller_relpose) relative to a body part included in the biomechanical user model (left_controller_body, right_controller_body)
- the position and orientation of the HMD (headset_relpose) relative to a body part included in the biomechanical user model (headset_body)
- RL hyperparameters (e.g., network size, time steps, batch size, etc.)
For the Beats VR game, we use the MoblArmsBimanualMotor biomechanical model with neural effort costs, which penalize the sum of the squared muscle control signals at each time step. As perception modules, we use the default proprioception module and the UnityHeadset vision module provided by SIM2VR. The former allows the simulated user to infer joint angles, velocities and accelerations, as well as muscle activations and index finger position. The latter is configured to include red color and depth channels of the RGB-D image, and stacked with a delayed (0.2 seconds prior) visual observation to allow the control policy to distinguish between left and right arm targets, and to infer object movement.
The training can then be started by running the UitB Python script uitb/train/trainer.py and passing the configuration file as an argument.
Similarly, a trained simulator can be evaluated using the standard UitB script uitb/test/evaluator.py. The script runs the simulator/policy and optionally saves log files and videos of the evaluated episodes.
For better logging and evaluation, we recommend to connect a Weights and Biases account to the trainer.
For the Beats VR game, we trained three simulated users with neural effort cost weights 0.001, 0.01 and 0.05. This was done to demonstrate how the behaviour of the simulated user can range from an "explorative" or "enthusiastic" player to a more "lazy" player. The simulated users learned to hit the incoming targets using different strategies depending on the effort cost weight. For instance, the simulated user trained with the highest effort cost weight (0.05) learned to do only the bare minimum to move its arms and hit the incoming targets. In particular, it hit the targets by relying mostly on wrist movements, suggesting that the game design could benefit from modifying the target trajectories to force the players to move their arms more. The simulated users trained with lower effort costs learned to move their arms more, with the lowest effort leading to a policy where the arm movements are exaggerated.
The trained simulators can be found in the UitB repo, named beats_vr_* where the suffix indicates the neural effort cost magnitude. Below you can find example videos from the trained simulators. The right-hand side videos show the game play from an external camera, and do not represent what the simulated user sees.
To train and evaluate simulated users for the Whac-A-Mole Unity game from the SIM2VR paper, please use the mobl_arms_whacamole_* config files to be found here. The trained simulators can be found in the UitB repo; see below for example videos from the trained simulators. Right-hand side of the videos show the camera feed from the UnityHeadset camera to illustrate how the simulated user perceives the environment. Note that the head of the simulated user is not tilted for the different target area placements, but the camera angle is (incorporating neck movements into this model is left for future work).
The source code of the game as well as the user data collected from the accompanying VR study can be found in the Whac-A-Mole repository.
Both the Reach Envelope and the Reward Scaling tool described in the SIM2VR paper are publicly available from the uitb-tools repo. After installing the uitb-tools Python package, the respective Jupyter notebook files can be executed.
Florian Fischer*
Aleksi Ikkala*
Markus Klar
Arthur Fleig
Miroslav Bachinski
Roderick Murray-Smith
Perttu Hämäläinen
Antti Oulasvirta
Jörg Müller
(*equal contribution)