Samsung+ 🤓
Spatial computing & magic glasses
A mixture of details about Samsung and my personal 3D work, respecting my NDA.
Senior Designer & Team Lead
February 2017 - Now
Samsung Research America, Mobile Platform Solutions Lab
My coworkers in XR Design Group (XRDG)
Contents
The general problem I solve
Spatial computing is an entirely new medium. I figure out how to make it more human.
I create novel interactions and product strategies. I think beyond our current black rectangles. I figure out how to meet or overcome our constraints. I design for moving targets and design for various device capabilities simultaneously.
It’s a new medium and I have a moral duty to help make it a good one. There are few opportunities – if any – in your career to help shape the direction of a new medium. Each new medium presents new opportunities and challenges to humanity. I do what I can to create more opportunities in spatial computing than challenges.
Everything is challenging and ambiguous. While the high-level design principles are the same in spatial computing, there aren’t clear answers for any particular problem. Each decision requires investigation. And, I’ve had to learn new tools to ideate and build spatial products.
XD Immersive talk
A pretty good overview of what I work on and how I think about spatial designFrom 2D to 3D product design: a kinda personal journey. (~25min)
First, this talk showed how high level design principles don’t change when working on spatial computing problems (e.g. Neilson’s usability heuristics, Case’s principles of calm technology, Rams’ principles for good design, Maeda’s laws of simplicity).
Second, I walk through the new constraints and a sample critique of an AR mockup.
Lastly, I close with a few pointers on how to transition from 2D design into a 3D design career.
My team
My team is the XR Design Group (XRDG). We work under MPS Lab under Samsung Research America (SRA). Together, we find meaningful ways to build future facing technologies.
Our core team consists of around eight designers and engineers (though we collaborate with teams across Samsung). We all participate in the entire design and development process, but focus on our specialties. Our tight collaboration leads us to better solutions.
I am the most senior product designer, though not the most senior designer (that honor would go to one of the two artist-engineers on our team from the games industry).
My role
And characteristic workWith spatial displays and the proper imaging pipeline, sonograms could look like x-rays.
Sometimes AR examples are limited to things we can already do well on flat devices. My work tries to go beyond these simple examples – into experiences we can only have with spatial computers.
Here’s the process my team usually goes through for a project:
1. Early research
User, design, tech, and market research to understand opportunities and constraints in a space.
2. Product concept
Renders and high level documentation to sell the idea internally and to guide the team.
3. Scoping
Define our MVP while aligning the team. What can be made real? What should be faked? What needs to be tested?
4. Interaction design
Exploration, definition, and then refinement – each stage with mockups and prototypes, in/validated with user studies.
5. Presentation
When sharing our work across the company, we create polished decks and demo videos of our prototypes.
6. Iteration
We learn something new. Requirements or goals change. We pivot to the next thing.
Responsibilities
There are many ways to create bad spatial computing experiences. My responsibility is to figure out what will resonate with users – while guiding and with my team.
Priorities & vision: Through my years of work before and at Samsung, I’ve developed specific principles for what AR and VR can and should be. These guide my decisions and help to set the team’s vision.
I work with my manager to set priorities for the team, arguing for directions that would have the highest returns on investment.
Design & process: I’m still an individual contributor, so I work on ideation, mockups, prototypes, and user research.
I guide the team to produce high quality work from the design side. This means contributions to what we should continue to explore, what’s ready to prototype, which trade offs should be made so we can meet our deadlines, and working with engineering to figure out implementation details.
Documentation: I lead our documentation efforts (writing, editing, crafting the story, presenting). This also makes me a go to person for any of our historical work. Before our switch to a more collaborative tool, I created a custom website to showcase our documentation, design decisions, and ideations.
Presenting & collaborating: I talk about our work to other teams and external partners. I also present to a number of lab heads, VPs, SVPs, Presidents, and once I presented to a group of Korean Congresspeople.
Patents: I’ve produced four+ spatial design patents that are working their way through the USPTO.
Mentoring: I’m there for my teammates when they are thinking through spatial design problems or figuring out how to navigate Samsung.
Exploration for a memory palace.
Concept render for spatialized sonograms.
Look development for a material.
Modeling practice.
Animating complex materials and microdisplacements.
Another memory palace exploration.
Look development for various materials.
Artistic render, playing with lights and materials.
Siteless is a book full of abstract architectural forms. I use it for modeling inspiration.
Look development for virtual paths.
Hardware VR project for independently controlled eyes, like in Pan's Labyrinth.
Video. When a project needed a tent model and I happened to be learning photogrammetry, I captured and processed my tent for use.
Model. The final processed tent. View in AR on mobile.
Tools & skills for spatial prototyping
UX design
- Maquette, Tvori, Tilt Brush, Blocks, Quill (AR/VR design tools)
- Sketch, Figma, Adobe Creative Suite, Framer Classic, Procreate (2D design)
- User research (design, facilitate, analyze)
AR/VR Prototyping, 3D Modeling
- Blender (modeling, texturing, procedural materials/shaders, 3D VFX compositing)
- Unity (interactions, MRTK)
- Depthkit, Meshroom, Polycam, Record3D (volumetric capture, photogrammetry)
- A-Frame (webVR)
Other
- Identifying key opportunities through design experimentation and prototyping
- Design+prototyping team management
- Video production showcasing XR prototypes and use cases
- HTML, CSS, JS
- Familiar with most AR/VR HMDs and their UX patterns
Patents
All while at Samsung Research America
Electronic device for communicating in augmented reality and method thereof
Interactions for an avatar-based AR chat app.
Automatic representation toggling based on depth camera field of view
Dynamically blend between depth video and an avatar in a AR-HMD and depth capture enabled phone system, based on if the user is within the capture volume.
System and method for augmented reality interaction
Creating a priority system to manage relative content importance.
System and method for transition boundaries and distance responsive interfaces in augmented and virtual reality
Increased legibility and contextual controls based on user distance from UI. Additional system to prevent rapid toggling when user stands on boundary line.
Efficient encoding of depth data across devices
In an AR-HMD and depth-capture-enabled phone system, decrease the depth data that needs to be encoded by using dynamic min-max culling.
Novel interactions to manage multiple open apps/windows, especially for foldable devices.
System and method for head mounted device input
Utilizing head or body movement as a discrete or continuous input. Useful in limited contexts for non-primary input.
Other work
For other examples of my spatial computing work, you can look at Humane Virtuality and Moral Decisions & Haptics in VR as well as my sporadic YouTube uploads.