Samsung Research America

Defining & prototyping the future of AR glasses across OS, apps, and interactions

Role

Team Lead Process, hiring, exec demos Senior Designer OS, apps, interactions Prototyper Unity / C# / Blender Researcher User studies, de-risking

Date

February 2017 - May 2021

While at

Samsung Research America, Mobile Platform Solutions Lab

With

My coworkers in XR Design Group (XRDG)

A precursor to Galaxy XR. Explored the opportunities and constraints of spatial glasses across OS, apps, and interactions which informed efforts across Samsung and with external partners. Led an AR/VR design and development group. Regularly met with executives, partners, and visiting dignitaries. Created and managed our design process and hiring process. Designed and built prototypes and user studies, de-risking explorations. Submitted 10+ patents with 9 granted.

The work is obfuscated enough to comply with my NDA. Some personal work is shown that is characteristic of work done at Samsung.

What should AR glasses be?

What, How, Why

The challenge

Make spatial computing work for real humans in an environment with rapidly changing constraints.

At the time, Samsung was figuring out what kinds of products could exist. Many teams were focused on semi-isolated technical problems.

My group focused on finding real value and validating that with prototypes, across interactions, apps, and the operating system.

Our work shaped the trajectory of Samsung’s spatial products, a precursor to Galaxy XR.

Core focus

A new “responsive design” where spatial content is responsive to a user’s distance and the user’s context.

How to cohesively unite interaction and representation systems like:

  • Mixed user representations: avatar, volumetric video, flat video
  • Mixed content: meshs, flat windows, flat stickers, interactive
  • Mixed and extensible device interaction systems: AR glasses (with varying capabilities), mobile phones

Extensible/flexible designs that adapt to hardware and OS capabilities. “The device” was actually a slew of potential devices, internally and with external partners.

Spatial design systems that are cohesive, opinionated, and extensible. For instance, we may define a default behavior at the OS that can be modified by an app or user settings. This becomes more complex when the computing environment is also the user’s real world, expected to have multiple apps running at once.

And generally, finding the details that must be solved for true everyday use, not just the surface level work seen in marketing. What can we only do with spatial computers?

Sample questions I tackled

What is an AR app?

Can messaging apps be spatial?

How should phone and headset users collaborate in a shared space?

How should the OS be context-responsive?

Can a phone act as a controller?

How does device input capability change spatial interface design?

What are the responsibilities of the OS, app developers, and users?

And more…

Patents

The work was mysterious & important

I can’t show my designs, demos, systems…
but, I can show the patents that came out of our work

10+ submitted. 9 granted.

2022 US20220385617A1 Granted
Contextually-aware notification management system for AR glasses.

Notification system that takes into account the user's current context. Context may be determined from visual or audio data, directly or via inference. Notifications may be filtered or delayed to prioritize user focus, reduce distractions, and increase safety.

Contextually-aware notification management system for AR glasses. Contextually-aware notification management system for AR glasses.
2021 US11995776B2 Granted
Synchronous AR interactions across heterogeneous devices (phone + glasses)

Interactions and rendering techniques for smoother synchronous AR experiences.

Synchronous AR interactions across heterogeneous devices (phone + glasses) Synchronous AR interactions across heterogeneous devices (phone + glasses) Synchronous AR interactions across heterogeneous devices (phone + glasses) Synchronous AR interactions across heterogeneous devices (phone + glasses)
2021 US20220229535A1 Abandoned
Systems and Methods for Manipulating Views and Shared Objects in XR Space

Interactions around embodied "video calls" in AR, from a system that can support headsets and phones.

Systems and Methods for Manipulating Views and Shared Objects in XR Space Systems and Methods for Manipulating Views and Shared Objects in XR Space Systems and Methods for Manipulating Views and Shared Objects in XR Space Systems and Methods for Manipulating Views and Shared Objects in XR Space Systems and Methods for Manipulating Views and Shared Objects in XR Space
2020 US20210407215A1 Granted
Automatic representation toggling based on depth camera field of view

Dynamically blend between depth video and an avatar in a AR-HMD and depth capture enabled phone system, based on if the user is within the capture volume.

Automatic representation toggling based on depth camera field of view Automatic representation toggling based on depth camera field of view Automatic representation toggling based on depth camera field of view Automatic representation toggling based on depth camera field of view
2020 US20220005215A1 Granted
Efficient depth data encoding across devices

In an AR-HMD and depth-capture-enabled phone system, decrease the depth data that needs to be encoded by using dynamic min-max culling.

Efficient depth data encoding across devices Efficient depth data encoding across devices
2020 US20210319617A1 Granted
Avatar-based AR messaging app interactions

Interactions for an avatar-based AR chat app.

Avatar-based AR messaging app interactions Avatar-based AR messaging app interactions Avatar-based AR messaging app interactions Avatar-based AR messaging app interactions
2019 US20200202629A1 Granted
Head/body movement as UI input for AR glasses

Utilizing head or body movement as a discrete or continuous input. Useful in limited contexts for non-primary input.

Head/body movement as UI input for AR glasses Head/body movement as UI input for AR glasses
2019 US20200320906A1 Granted
Mobile device with a foldable display and method of providing user interfaces on the foldable display

Novel interactions to manage multiple open apps/windows, especially for foldable devices.

Mobile device with a foldable display and method of providing user interfaces on the foldable display Mobile device with a foldable display and method of providing user interfaces on the foldable display
2018 US20190279407A1 Granted
System and method for augmented reality interaction

Rendering and organization of virtual objects based on priority levels in AR.

System and method for augmented reality interaction System and method for augmented reality interaction
2018 US20190172262A1 Granted
Distance-informed UI for AR glasses

Increased legibility and contextual controls based on user distance from UI. Additional system to prevent rapid toggling when user stands on boundary line. Akin to model LODs, but for interfaces.

Distance-informed UI for AR glasses Distance-informed UI for AR glasses Distance-informed UI for AR glasses Distance-informed UI for AR glasses

My role

I was a Team Lead and Senior Designer in the XR Design Group (XRDG), a cross-functional team of designers and engineers figuring out what would make spatial AR glasses compelling.

Team Lead 🙋🏻‍♂️

  • Demonstrated vision to executives, partners, and visiting dignitaries through conversations, presentations, and demos
  • Created and managed the process that took us from ideas, to rapid prototypes, to high fidelity prototypes, to video documentation
  • Set team priorities with input from team, managers, and execs
  • Led and supported hiring process
  • Mentored teammates through spatial design problems and navigating Samsung

Designer 👨🏻‍🎨

  • Led investigations across the operating system, apps, and interactions
  • Designed novel interactions and product strategies for the new spatial medium
  • Bridged design and engineering as device capabilities kept changing
  • Ran user studies to understand sentiment
  • Rendered pre-vis imagery to quickly show concepts

Prototyper & Engineer 👨🏻‍💻

  • Built on-device prototypes (Unity/C#) that de-risked explorations
  • Documented the team’s work — internal catalogs and a custom site showcasing our decisions and history

To support the work, I used all sorts of tools from Unity, Blender, volumetric capture (Depthkit, Meshroom, Polycam), AR/VR design tools (Maquette, Tvori, Tilt Brush, Blocks, Quill), and 2D design tools (Sketch, Adobe Creative Suite, Procreate).

Exploration process

Or, how to find diamonds

One of my largest contributions was creating a new rapid iteration process that let us experiment both wide and deep, as part of our partnership with a group in HQ and external partners.

It had three goals

  1. Give the team time to explore wide. Narrow focus too early will trap us in a local optimum.
  2. Give the team time to explore deep. Shallow work risks giant unknowns.
  3. Jointly answer UX, engineering, and product questions. Isolated tracks will never find the true opportunities and constraints.

Process components

Depending on the timeline and problem we were addressing, we could shift time between components.

User, design, tech, and market research to understand opportunities and constraints in a space.

1. Early research

User, design, tech, and market research to understand opportunities and constraints in a space.

Renders and high level documentation to sell the idea internally and to guide the team.

2. Product concept

Renders and high level documentation to sell the idea internally and to guide the team.

Define our MVP while aligning the team. What can be made real? What should be faked? What needs to be tested?

3. Scoping

Define our MVP while aligning the team. What can be made real? What should be faked? What needs to be tested?

Exploration, definition, and then refinement – each stage with mockups and prototypes, in/validated with user studies.

4. Design & prototyping

Exploration, definition, and then refinement – each stage with mockups and prototypes, in/validated with user studies.

When sharing our work across the company, we create polished decks and demo videos of our prototypes.

5. Presentation

When sharing our work across the company, we create polished decks and demo videos of our prototypes.

We learn something new. Requirements or goals change. We pivot to the next thing.

6. Iteration

We learn something new. Requirements or goals change. We pivot to the next thing.

Characteristic work

A mix of Samsung & personal work

With spatial displays and the proper imaging pipeline, sonograms could look like x-rays. (Personal work that is representitive of early concepting I would do at Samsung.)

Patent imagry is adapted from working apps, prototypes.

Flexible user representation in a volumetric call based on device capabilities: flat 2D screen, cutout 2D screen, volumetric projection.

Flexible user representation in a volumetric call based on device capabilities: flat 2D screen, cutout 2D screen, volumetric projection.

Part of the flow for 2D users sharing content in a mixed-device spatial call. We made it feel simple with contextual and scoped interactions.

Part of the flow for 2D users sharing content in a mixed-device spatial call. We made it feel simple with contextual and scoped interactions.

Async messaging app exploring volumetric content and interactions, like grabbing a friend’s tiny avatar to start a new chat.

Async messaging app exploring volumetric content and interactions, like grabbing a friend's tiny avatar to start a new chat.

Headset users can move around 3D content easily. To support the same for 2D users, we created a “model inspector” view.

Headset users can move around 3D content easily. To support the same for 2D users, we created a "model inspector" view.

In a volumetric call, 2D users can change their 3D viewing perspective.

In a volumetric call, 2D users can change their 3D viewing perspective.

Unified sharing space for mixed-device calls. Headset user is pointing to shared “wall” while mobile user can see the wall in a volumetric view or as a 2D region on their screen.

Unified sharing space for mixed-device calls. Headset user is pointing to shared "wall" while mobile user can see the wall in a volumetric view or as a 2D region on their screen.

High five with tiny avatars in an async messaging system.

High five with tiny avatars in an async messaging system.

Volumetric volume slider exploration.

Volumetric volume slider exploration.

Exploration for a memory palace. (Personal)

Exploration for a memory palace. (Personal)

Look development for various materials. (Personal)

Look development for various materials. (Personal)

<em>Siteless</em> is a book full of abstract architectural forms. I use it for modeling inspiration. (Personal)

Siteless is a book full of abstract architectural forms. I use it for modeling inspiration. (Personal)

Hardware VR project for independently controlled eyes, like in <em>Pan’s Labyrinth</em>. (Personal)

Hardware VR project for independently controlled eyes, like in Pan's Labyrinth. (Personal)

When a project needed a tent model and I happened to be learning photogrammetry, I captured and processed my tent for use.

Other work

XD Immersive presentation: From 2D to 3D product design. What changes and what stays the same in a spatial context? (~25min)

For other examples of my spatial computing work, you can look at Humane Virtuality and Moral Decisions & Haptics in VR as well as my sporadic YouTube uploads.

© Andrew R McHugh turns R&D ambiguity into meaningful products.