Magic Leap adds virtual reality head-tracking and possibly hand-tracking

After more than a year and raising over $1.4 billion, the secretive mega-venture offered a second sneak peek into its VR technology called mixed reality

Magic Leap adds virtual reality head-tracking and possibly hand-tracking
Credit: Magic Leap

A second demonstration of mega-venture Magic Leap’s virtual reality technology indicates that head-tracking has been added and possibly hand-tracking.

Head-tracking, which wasn’t shown in the only other demonstration that happened over a year ago, lets a person move around a hologram to see it from different sides. Hand-tracking, which is a mouse-like metaphor interface that lets people interact with virtual objects using hand movements, also seems to have been added. The report also hints at the principals behind how Magic Leap’s virtual reality works. 

Wired reported on Kevin Kelly’s visit to Magic Leap in Fort Lauderdale, Florida, to experience the second public demonstration of the company’s version of virtual reality that it calls mixed reality (MR). A comparison of the Wired story with one written by Rachel Metz over a year ago for the MIT Technology Review measures Magic Leap’s progress.

There’s only about a page or two about Magic Leap’s technology out of 41 in the Wired story. It’s really an explainer of augmented reality, mixed reality and virtual reality technology and the universe of competitors. Anyone wanting to learn about the market should read this well-executed story. The specifics about the technical progress of this $1.4 billion venture have been extracted from the stories and summarized below.

A year of progress

Since the Tech Review report, Magic Leap has made progress. A year ago, the MR apparatus wasn’t really wearable. This may have been due to the sheer size of the prototype hardware, or the software that adjusts the illusion for head movement and changes in perspective may not have been completed. The complexity of the mathematics of this head-tracking applied to the complexity of the mathematics of Magic Leap’s proprietary light-field chip may have required more development time.

The Wired report of Kelly’s experience indicates the prototype headset he tested is now wearable, perhaps untethered from a high-powered graphics-rendering computer used by the Oculus Rift and HTC Vive. Kelly describes his observation of a realistic 8-in. steampunk drone projected in front of him that he could walk around looking at from every angle. It seems that since Metz checked in on Magic Leap founder and CEO Rony Abovitz and his team last year, the headset has been reduced in size to be wearable and head-tracking has been implemented.

Kelly’s report raises the question: Could Magic Leap have implement hand-tracking, too?  He writes, “When I raise a hand, it approaches and extends a glowing appendage to touch my fingertip.”

If the projection of the drone is programmed to interact with his hand, the relative coordinates in space of the hand need to be understood and movement of the hand tracked to trigger the software rendering the virtual drone to respond by extending the glowing appendage. That one sentence by Kelly leaves so many questions about how this was implemented. Was he wearing a VR glove? Did Magic Leap partner with Leapmotion? Or did the company implement its own proprietary technology? We won’t know until Magic Leap’s next disclosure.

The remainder of what Kevin saw is consistent with what Metz saw, but the subject matter changed a bit. Kevin saw very high-resolution holograms: human-sized robots that walk through walls, and “miniature humans wrestle each other on a real tabletop, almost like a Star Wars holographic chess game.”

The concept of MR is mutually reinforced by Kevin’s account and a new Youtube video released yesterday that depicts holograms of a tabletop Mount Everest, a sneaker and a jellyfish floating across the ceiling.

Most interesting, though, is the Wired story included the first image of the light-field chip ever published.

“Magic Leap’s solution is an optical system that creates the illusion of depth in such a way that your eyes focus far for far things, and near for near, and will converge or diverge at the correct distances,” Kelly writes. And he compares it to the fixed focus length between the eye and the LCD screen of stereoscopic VR headset technology used by almost every competitor.

Stereoscopic VR uses the principals of stereoscopic photography invented by Sir Charles Wheatstone in 1838 in which two photographs taken from slightly different angles are focused through lenses in an apparatus of fixed focal length to create the illusion of three dimensions (3D).

Stereoscopic VR employs the same method, dividing an LCD screen mounted in front of the user’s eyes projecting two slightly different moving images focused into the eyes with lenses to create the illusion of 3D full motion VR. It’s a neat trick that counts on the gullible human brain to generate a consistent sensible world view from what its senses have been given.

Magic Leap contends that it doesn’t trick the brain. Rather it shoots photons into the eye that stimulate the cones and rods as if the hologram were real, or according to founder Rony Abovitz, neurologically true. The image of the light-field chip hints that Magic Leap may be using a very promising technology called silicon photonics.

Bending and switching light

Over the past few decades the need for increased communications speeds attracted huge investments to shift technologies from slow error-prone copper wires to fiber optics. Challenges such as converting electrons used in computers to photons that travel at the speed of light across optical fiber and bending light that would prefer to beam straight ahead around corners were solved. It has required a lot of expensive hardware to make the conversion back and forth between photons and electrons that silicon photonics will change into a commodity.

The idea behind silicon photonics (aka nanophotonics) that has also attracted huge investment from Intel and IBM is simple: take the world’s fastest communication technology (photons), and build it directly into semiconductor chips using well-known and massively scalable production processes. With scale comes multiple orders of magnitude in production cost reduction.

Perhaps Magic Leap has applied all the research of bending and switching light to shoot photons across the light-field chip that forms the headset lenses, turning it into the retina at the point where the photons would be overlaid on reality to create the hologram. It’s just a speculation about how “neurologically true” has been achieved.

To comment on this article and other Network World content, visit our Facebook page or our Twitter stream.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.