HoloLens interactions from the User Experience point of view

Finally I have got the chance to wear a Microsoft HoloLens and see the world from its viewpoint. Holographic objects seemed crystal clear. In fact, the quality and brightness of the holographic world was totally impressing.

On the other hand, as everybody noticed, the FOV (Field of View) was pretty obvious but we all know this is a technical challenge that Microsoft engineers are well aware of. We just hope they can figure it out in the near future. Needless to say, this is the first version, a developer one, not even commercialized yet and nothing can be 100% perfect on its first try.

But on this post I would like to focus more on the User Experience side of the story, particularly the interactions and the inputs.

Mixed reality is a new world in terms of human interactions. In this world, we are not tied anymore to a 2D slate pixel space. Things are in the physical world, in a 3D space, which opens so many new challenges and also so many new possibilities in terms of interactions.

Even though, 3D space interactions seems like a new area, in fact, it is not. We all are living in a 3D space and interacting with objects since the beginning. So we already know well how things work here. So why not utilizing these existing skills when it comes to the mixed reality world as well?

HoloLens provides us with three main inputs to help us navigate in, and interact with the contents; gaze, voice commands and hand gestures.

Gaze indicates where you are looking in the world, where is your vision focus. The point or area could be rendered as a kind of cursor indicator in the world for more clarifications.

Voice commands are powered by Cortana, which is absolutely a great interaction tool if it is integrated well in the apps.

HoloLens is also equipped with a bunch of sensors and cameras which the cameras are used to interpret the hand gestures, besides sensing the world and mapping it.

Even though the first two input tools are easy and straightforward for users, in my view, the hand gestures are not.

To detect the gestures, HoloLens needs to see our hands clearly through its cameras. Better say, our hands need to be almost in front of the face (If the head is straight and looking forward), with a 90-degree tilt upward, which is not a comfortable zone and state for human hands at all. This experience ends soon with a tiredness feeling in our hands after a couple of tries.

But wait why should we do this at all ?

This is not what we do in real life. Our hands are controlled by our mind as it processes a combination of tactile and vision sensations or just the first one. When interacting with objects in the physical world our tactile sensation plays as the main input besides our vision. We can even close our eyes and still interact with the world effectively.

So the question is:

Why should we use cameras to detect the hand inputs?

Why not using something else that could feel more natural and end with a more comfortable user experience?

If we focus more on the natural ways of interactions with the physical world those could be also applied in the mixed reality world as well.

The concept of having a controller/clicker in hand is based on tactile sensation and it could be a great tool if it could be light, small and wearable.

Of course, HoloLens came with a clicker tool to ease the selection, tap, zoom and similar commands. It has also a band to wear it on our fingers but I think, it can do a better job by extending and converting it as two separate normal size rings that can be wear and sit on the middle phalanx of pointing or middle finger on each hand.

Each ring can be equipped with some kind of proximity sensors to calculate the distance between the two, to help us with zooming, rotating, dragging and many other possible interactions. They can also have a front face or crown in cases we need a couple of pointer like indicators in the mixed reality world to interact with the objects besides the gaze one.

In this way, we don't need to rely on cameras to interpret hand gestures and the experience would be more comfortable, natural and real.