Apple Design Director
Technically speaking Apple is a self-contained and wirelesr headset that outputs Mixed Reality, high-definition, stereoscopic 3d objects that users control by way of gaze, gesture and voice. Practically speaking HoloLens is a wholly new and more natural way to bring technology into your life. Since 2012 my role as Creative Director and Design Team Leader has been to conceive, design and ship the HoloLens OS and Windows Holographic platform along with the team I’ve both built and managed. My job has largely been about targeting the right ideas at the right time, applying the right processes at the right time and directing people to commit, make and close. It’s also been about inspiring the team each day and focusing them on shipping a paradigm-shifting product that resonates with real people. To date our accomplishments include the HoloLens product concept and story, the Gaze, Gesture and Voice sensory input design language, the holographic interaction and aesthetic design language called "Luminous", and the Windows Holographic UX platform.
____
In January 2015 we demoed Microsoft HoloLens to the world and the reaction since has largely been fierce and curious, even ecstatic. All our suspicions were confirmed, people love holograms and want to see personal computing evolve.
As the HoloLens Design Team Leader and Creative Director charged with directing the User Experience, Interaction Design, Aesthetics Design and Product Concept, and someone involved in shipping my share of highly ambitious products over the last decade, the response has been stronger than even I'd expected.
When we formally began our journey in 2012, HoloLens was comprised of some highly advanced and loosely strung together technologies and a big dream. But it was not a product.
My team and I were tasked with conceiving and designing an entirely new way to engage with computers and the real world through something we call “Mixed Reality.” If Oculus Rift is Virtual Reality - a completely immersive virtual 3d world, and Google Glass is Augmented Reality - a non-immersive HUD-like 2d overlay, HoloLens instead mixes 3d holographic objects into the dynamics of your real environment.
For illustration purposes consider the 2d Spotify app or iTunes app on your phone. HoloLens enables you to place that app as a hologram anywhere you want even alongside your actual speakers. You can also conjure up an interactive equalizer on your wall allowing you to fine tune the sound. And you can splay your playlists across the floor or gather them into stacks of vinyl lps. You can even place a hologram of Kanye or Beck dancing on your coffee table:)
From the very beginning we knew we didn’t want HoloLens to become another science project or a “me-too” effort - things Microsoft has become known for. The greater team of designers, engineers, project managers, scientists and executives focused down on bringing an entirely new consumer product to the world and that’s the standard we’ve held ourselves to. And that’s what we’ve set out to design, build and now ship.
But how?
For me, someone very familiar with a blank canvas with regard to product design, it always starts by listening to the medium and watching the crowd. Where does this technology want to go? What comes naturally to it? What kinds of scenarios flow easily from it and build naturally upon themselves? And provided it actually works, why would we humans care and where would our better angels most want to take it?
Answers to these questions came fairly early. Our initial ideation was intense, prolific and telling. Even beyond the magic of holography (environment locked, stereoscopic 3d rendered projections) and natural user interface (gaze, gesture and voice), HoloLens represents a limitlessness (like Google’s little white box or Apple’s iOs platform) that unlocks a new kind of thinking - in this case a spatial, interactive and object/IOT type one - that our teams became intoxicated by. I realized it’s precisely this kind of limitlessness that we want to productize for others, get them to embrace, run with and own.
Better said, instead of delivering a fait accompli or all-encompassing product solution, we created Windows Holographic, an open, mixed-reality platform for builders and creators and curious people in general. Across the customer spectrum, from developer to consumer, holography is an active phenomena not a passive one. Our platform asks you to open up and interact with your real world.
Our strategy was to invite in partners such as NASA and develop the Windows Holographic Platform and HoloLens interaction model and design language alongside them as they developed their own unique and deeply interactive visions. The result was truly a 1+1=3 dynamic that focused us on building an inspired and stable product and platform as opposed to mostly chasing the heroic expressions.
HoloLens/Windows Holographic is a very large and unique problem/opportunity space. The integrated development between hardware and software, platform and partner ecosystem has required us to forge new processes and new relationships between Engineering, Design, Research, PM, partners and Microsoft all up. There hasn’t been a lot of space for certitude or orthodoxy and a lot of process and tools had to be invented. Like most forms of successful modern development we’ve had to remain agile, alert and together.
But if everything about your process is “new” and agile then you are probably in trouble:) And so, as I have done with other high-flying projects in the past, I've kept our UX process rigorous and fairly traditional.
In my experience, the more advanced and “magical” the product concept the more likely UX is to suffer. With Xbox Kinect we really messed up in this regard. With Kinect the technology and the magic came first and the UX was just a problem to solve - an afterthought instead of at the core of the offering. I wanted the HoloLens UX to be grounded and central and so we invoked User Centered Design (with an emphasis on Research and Prototyping) not only as a design process, but everyone’s process.
My mantra to the team has been “UX flow is the hard human thing and the rare thing. Not tech, not design, but UX flow.” User Centered Design is hardly a panacea for all design problems but wielded strategically I’ve found it’s a great framework to get a lot of high-achieving, multidisciplinary folks to communicate, focus and succeed.
At the same time ours is a very hard core “make” culture. Thanks to a couple of really good interaction designers I hired I was enlightened to the fact that our prototyping efforts could be central to everything (even more so than they already were) and really up our design game. While we’ve tried a lot of methods over the last few years the thing that has really allowed us to move ahead has been aggressive and rapid prototyping across all our sprints. UCD tends to get neck up and can get people mired in meetings-to-nowhere if you aren't careful. For HoloLens I’ve found that balancing UCD rigor with a dominant prototyping culture is a particularly effective and sustainable blend of styles.
Details
My Role
Creative Director, Design Team Leader and Manager
Key Team Disciplines
UX, Interaction Design, Aesthetic Design, Prototyping, 3d Production, Product Storytelling
Team Size
20-30
My Key IC Contributions
Conceptual Development, Design Direction, Project and Program Direction, Art Direction, Team and Leads Management, Budgets and Operations, Recruiting
At the end of each major sprint my team produced a product story video to keep the product vision front and center. Engineering schedules would sometimes take a sensor offline for weeks and people would start to adapt to the “brokenness” of the build. As Creative Director it was on me to keep our vision and our guiding scenarios and principles alive for everyone and sometimes the videos did just that. My team has several in-house VFX artists and a world class art director allowing us to make these videos internally.
The primary HoloLens shell is a GUI that operates in two distinct but interdependent modes. The A mode above appears first as a 2d/3d “app launcher" which remains locked 2 meters in front of you and allows you to select any object by using gesture and/or voice. The B mode is invoked once an "app" is selected and the A mode disappears. In the B mode 3d objects automatically comport to the geometry of your environment (as determined by our Surface Reconstruction process) or instantiate where you intentionally place them.
Regularly updated UX architecture maps are one of the best ways to keep everyone on the greater team current on the design and speaking the same language. I've been religious about maintaining our maps and using them as the primary cross-team communication device. We've always crafted our maps in faux 3d to more accurately mirror the HoloLens UX and keep track of emerging complexities. (Click on image to enlarge. Works on large screens only.)
Many efforts to codify design language (at MS and elsewhere) tend to compartmentalize aesthetics from interaction from brand. For HoloLens I've always sought to express all these aspects together as a single act of design. Most of HoloLens' interactions are entirely new and so it's been critical for us to treat every moment as a heuristic opportunity.
Many things about HoloLens have taken us all by surprise. Surface Reconstruction for instance was first thought to be a background technology the user would never see. One rainy weekend a few designers and engineers decided to visualize the process and they struck gold. SR has since become one of the products most meaningful expressions with huge heuristic value.