Virtual Reality Jenga

A Port of the Block Game

In an effort to understand the limitations and features of new input methods, we used Unity to create a minimalist port of the popular block game Jenga with a Leap Motion and Oculus DK2.

UX Designer
August 2015 to October 2015
While At
Carnegie Mellon Human-Computer Interaction Institute, Research Issues in Game Development
Daniel Hua, Swetha Kannan, Wole Idowu, Madeline Duque, Zhen Geng


  • Create a version of Jenga in Unity in about seven weeks on a team including a developer, artist, designer, and project manager
  • Begin with mouse and keyboard controls, eventually move to Leap
  • Motion and then add Oculus DK2
  • Make updates to game based on weekly playtesting and class critiques
  • Game must be usable and fun

Near Final Experience & Code

Samples of our playtesting.

Code on

My Role

  • Led user experience design and usability
  • Created prototypes of the visual design to be critiqued by the team
  • Conducted design experiments for a standing version of Jenga


We were put in teams with a developer, artist, designer, and project manager. Roles were fluid as there were only three UX designers in the class (the rest of the students were either developers or artists) and the majority of time was spent on programming. Each week the teams would present their work in front of the class for critique. We showed off our current state of development, shared problems and insights, talked about our weekly playtesting, and gave demos. In addition to our classmates, our professor (Tom Corbett of Electronic Arts) and occasionally outside game developers/designers would critique our work. The whole project was about seven weeks: two weeks for mouse and keyboard, two weeks for Leap Motion integration, and three weeks to integrate the Oculus DK2.

In the days leading up to working with a new technology like the Leap Motion or Oculus DK2, I spent a weekend downloading and playing games designed for those devices. As a UX designer, it is important that I understand how others have solved the problems I will encounter — while keeping an open mind for new and better solutions.

Understanding Roles

This was my first experience building a project in VR and my second experience with Unity (my first being a “Let’s Learn Unity” demo project). The class was largely video game developers and artists with three students from HCII (including myself). Having no previous experience working in the video game world, I struggled to find my place. It was also the first time students entrenched in video game design and development worked with user experience designers.

We worked around our field jargon to create a workflow: I created initial designs, we all critiqued them, and then the programmers started implementing what we all agreed on. Throughout development, I explored games built for the Leap Motion and Oculus DK2 and made relevant UX suggestions (e.g. how to best position your hand for the Leap Motion to read it).

Prototyping Visuals & UX

Early on I suggested to the team that we use a minimalist aesthetic. Since none of us were performance graphic specialists nor were our laptops powerful (which were to later power a DK2), it made sense to find a new aesthetic that would be fun to build and play and would still run at a high frame rate. Additionally, the minimalist aesthetic would differentiate us from the other teams who were building (1) a cell-shaded, kid-esque Jenga and (2) a Godzilla, city sized Jenga.

For week one, I designed an interactive wireframe for the menu and some of the interactions. Since we were aiming for minimalism, the designs were not too far from the final desired aesthetic.

An early wireframe prototype of our Jenga game

In the first week, we had our first version of Jenga running. It wasn’t beautiful and it wasn’t the easiest to use. An example: when you clicked a block just a bit, it would respond by turning red. This confused users because of red’s usual correlation with doing something wrong. I refined the color palette and we had more positive feedback.

Left: First week. Right: later weeks after a few color palette refinements.
Near final color palette. In the last week, we decided that this palette was too low contrast for some of the public TVs we were demoing the game on. We scrapped some of this palette for a higher contrast palette, making it easier to differentiate the tower from the background.

On the interface side, I had initially designed a start screen that included both a side menu and a semi-playable version of Jenga on the right. This also confused users. “Is this Jenga the game I’m playing?” “Do I need to press start if it already began?”

For the second version, we added what began as an end game error. Before the developer finished writing the game’s end-state, there was an error where blocks kept falling from the sky like a waterfall. Finding the waterfall playful, we added it to the start screen in lieu of the playable tower. This worked well until we started working with the Oculus. In virtual reality, waterfalling blocks are too disorienting.

Interface elements unique to a digital version of Jenga, useful for mouse and keyboard or Leap Motion.

I also explored the use of small elements that would give more information about what was happening in the game. In the top right of the above image, you can see a tilt icon: how much and which way the tower is swaying. In the bottom left, there’s an element that shows the user’s score. The bottom right has a clickable icon for keyboard shortcuts, looks like the standard walking keys in video games, and happens to use the first letter of each name of the original team.

Interacting with the tower proved more difficult than we expected. With mouse and keyboard controls, there does not exist a great way to move in three spatial dimensions. Leap Motion helped, but once we were working with the Oculus, the Leap’s position on the headset reduced hand-tracking confidence. I suggested we work around this by reaching into the tower with an open faced hand then grasping a block with a whole fist. These exaggerated motions improved hand-tracking, but distanced our game from normal Jenga.

Poor hand tracking also caused players to accidentally drop a block when they pulled it from the tower. Then, when trying to grab the block on the ground and put it back on the top of the tower, there was a good chance hand-tracking would fail and the user would have to try again. To get around this, we put blocks back on the top of the tower for the users.

Experimenting with Standing Jenga

When people first try VR, they usually don’t realize they can move their head in and out, toward and away objects. To prompt this motion, I wanted to create a version of Jenga where a floating tower would appear in front of the user. To reach the pieces and look around, they’d be inclined to move their body, trigging a deeper level of immersion. To find our design boundaries, I ran a design experiment.

The Oculus VR demo scene allows you to examine a small space around a table. Source:

Standing in the Oculus demo scene, I tested the limits of head tracking for position and rotation away from the positional tracker. I found that:

  • The positional tracker has to be level.
  • The head-mounted display needs to be about 4–5ft away from the positional tracker to give the user enough room to walk around and still be sensed inside of the sensing cone.
  • To be sensed well, the head-mounted display (at 4–5ft) needs to be at no more than an angle close to 45˚ from the centerline of the positional tracker. To accommodate this, I suggested we use lighting and sound to suggest the walkable space to the user.
  • If the user tries to get something off of the ground, they’ll likely bend their head too much away from the positional tracker to be read well. Therefore, we should create a “table” the Jenga tower can rest on. This allows the user to only bend down a small amount and the positional tracker can continue to sense with a high level of tracking.

My two design documents, created to help my team understand our limitations. Left: a profile view of the positional tracker and our virtual tower. Right: a top-down view of the same setup.

We never realized this version of our Jenga project due to time and human resource constraints.

Sound Design

In addition to my UX work, I also created custom sound effects for the game. Because of our minimal aesthetic, bare-bones electronic music seemed the most fitting. I created a short tune with three levels of intensity to correlate with more intense gameplay. Due to development constraints, I created another, longer version to loop in the background during game play.

Increasing intensity sounds for the background music.
Previous Project


Back to Top