Augmented Reality (AR) today has led to the appearance of more than 2,000 apps and over 1.4 billion active iOS devices. Even at this early stage of its evolution, AR is now permeating the consumer products space. In just the next four years, the International Data Corporation (IDC) forecasts AR headset production will grow by 141% annually, reaching 32 million units by 2023. Where will all these AR headsets be used?
Whether through detailed and dynamic anatomical annotations or visualized patient-specific guidance, AR will soon augment the work of every medical practitioner. Today it is already being used as a diagnostic tool. Magic Leap recently hired SyncThink, a company that uses AR for eye-tracking to diagnose concussions and balance disorders. Another startup, XRHealth, has launched ARHealth on Magic Leap, a tool to aid in rehabilitation, pain distraction, and psychological assessment.
Surgeons at the Imperial College London are using Microsoft’s HoloLens 1 in pre-operative reconstructive and plastic surgery where they would normally use CT scans to map blood vessels that supply vital nutrients during procedures. Dr. Philip Pratt, the Imperial College project’s senior researcher, states, “With the HoloLens, we’re now doing the same kind of [scan] and then processing the data captured to make it suitable to look at. That means we end up with a silhouette of a limb, the location of the injury, and the course of the vessels through the area, as opposed to this grayscale image of a scan and a bit more guesswork.” This can dramatically lower surgical risks by helping surgeons visualize the depth of vessels and then choose the optimal incision location. And while HoloLens 1 has only been used for pre-op visualizations, the HoloLens 2 is on track to become part of the operating theatre. The Philips’ Azurion image-guided therapy platform is built specifically for HoloLens 2, providing surgeons with real-time patient data and dynamic 3D imagery as they operate.
AR headsets with their virtual overlays can also be used to improve the sharing of expertise across hospitals and medical practices. In this way, medical specialists with unique skills can be used to offer advice or direct surgeons through complex procedures remotely from anywhere on the planet. Today, Brainlab, a German company, is collaborating with Magic Leap to create a 3D spatial viewer to allow clinicians to work together in surgical procedures across disciplines.
(Editor’s Note: When my brother, a family physician working for the Moravian Mission in Nicaragua many years ago needed to consult the expertise of a surgeon to deal with a complicated procedure, he had to be hooked up through a ham radio operator to an expert in the United States. The remote consult worked but it was fraught with loss of signal from time to time, and the many garbled transmissions that impeded the communication. You’ll be happy to note that the patient lived. AR and the Internet, as my brother would attest, represents a dimensional leap for telemedicine.)
Beyond democratizing medical expertise, AR can even provide instantaneous patient histories, gearing doctors with AI-processed information for more accurate diagnoses in a fraction of the time. By saving time, AR frees doctors to spend a greater percentage of the day engaging in face-to-face contact with patients, establishing trust, compassion, and an opportunity to educate the healthcare consumer rather than merely treating him or her. And when it comes to digital record-keeping, doctors can use the voice control interface to transcribe entire interactions with patients vastly increasing the amount of healthcare delivered daily.
Today, over 3.4 million visually impaired individuals live in the U.S. alone. But with the development of AI-integrated smart glasses the severity of visual impairment may be dramatically eased. As new companies enter the market such as NavCog, Horus, AIServe, and MyEye, they are joining Microsoft and other name-recognized vendors in changing the world for those who are blind. Microsoft has begun development of a “Seeing AI” app which translates the visual world into audio descriptions for the blind using a smartphone camera lens as eyes.
During the Reality Virtual Hackathon in January, hosted by Magic Leap at MIT, two of the top three winners had developed applications for those with disabilities. CleARsite was developing environment reconstruction, haptic feedback, and Soundfield Audio was using sound to enhance a visually impaired individual’s interaction with the world. HeAR, another developer, is using Magic Leap 1 headsets to translate vocalizations and sign language into readable text in speech bubbles that appear in the user’s field of view.
In biometrics, cyclist sunglasses and swimmer goggles have evolved into the perfect medium for AR health metric displays. Smart glasses like the Solos ($499 USD) and Everysight Raptors ($599 USD) provide cyclists with data on speed, power, and heart rate, along with navigation instructions. Meanwhile, Form goggles ($199 USD) display a swimmer’s pace, calories burned, distance, and stroke count in real-time to a depth of 9.75 meters (32 feet) underwater. With AR applications like these, it will shift accessible health data from our wrists into our field of view.
In the year 2030 when we walk into any store it will be retrofitted with AI and sensors to interact with us. Every mannequin will wear a digital design customized to our preference. Forget digging through racks of garments or hunting down your size. With cross-referencing to our personal purchase history, gaze patterns, and current closet inventory, AR-AI displays will be tailor-made for our wardrobe and adjusted to our individual measurements.
Why can I say this? Because Google Lens, an app available on most Android smartphones is already leaping into this marketplace, allowing us to scan QR codes and objects through the smartphone camera. With the Style Match feature, it already can give us some of the capability described in the previous paragraph by identifying pieces of clothing or furniture and viewing similar designs available online and through e-commerce platforms.
AR features are quickly encroaching into ads as well. In July, the New York Times debuted an AR ad for Netflix’s “Stranger Things,” guiding smartphone users to scan the page with their Google Lens app and experience the show’s fictional Starcourt Mall come to life.
Immersive AR advertising in the future won’t all be unsolicited and obtrusive. Many will prove helpful. For example, as you walk down a grocery store aisle, discounts and special deals on your favorite items will populate your AR smart glasses. Or if you find yourself admiring an expensive pair of pants, your headset might suggest similar items at a lower cost, or cheaper distributors with the same product. Passing a stadium on the way to work, you may learn about next weekend’s best concert ticket deals and your AR-AI may remind you that a friend with an upcoming birthday might like to join you at this event. You won’t be bombarded with these messages if you choose not to be by simply toggling them off.
The convergence of today’s AI and AR will give us the ability to create individual learning environments. Students will be empowered to learn as sensors track neural and physiological data and provide instant feedback to nurture the process. It is this convergence of AR with biometric sensors and AI that has the potential to alter education making it delocalized, customizable to every individual, responsive, and a means to accelerated learning achievement.
Already today within classrooms, Magic Leap One’s Lumin operating system allows multiple headset wearers to share a common digital experience, such as viewing a dissection or studying a historical map. The Magic Leap CAD application can assist students in collaborating on design projects.
Learning in the classroom, however, may become a thing of the past. Already, numerous AR mobile apps can be used to identify objects in a user’s visual field, instantaneously presenting relevant information. And as user interface hardware undergoes its own product evolution so will the software which will make gazing out a window at a cloud an interactive information experience in which the wearer will learn about the water cycle and climate science, while walking past an old building, will mean you learn about its history. It is this innovation that will lead to explosive knowledge abundance.
AR will enable on-the-job training at far lower costs in almost any environment, from factories to hospitals. Smart glasses are already beginning to guide manufacturing plant employees to learn to assemble new equipment. Smart glasses in retail will mean new employees will be able to take an AR tour to learn their jobs.
Today, at Jaguar Land Rover, automotive technicians can better understand the internal components of a vehicle without dismantling it using Bosch’s Re’flekt One AR solution giving them X-ray vision to visualize the internal components of a vehicle without removing a dashboard.
In healthcare, AR solutions will train medical students using hyper-realistic artificial cadavers which will facilitate rapid learning for honing for surgical skills in numerous specialties.
And in sports training, simulators with AR headsets will vastly improve athletic performance, while those who practice chess or play the piano will be given a virtual interface to perfect their skills anywhere they can find a flat surface.
The convergence of AR glasses combined with AI will make decision-making on trips much easier. For example what hotel or what restaurant to choose. But perhaps even more exciting is one of AR’s more sophisticated uses, instantaneous translation. Whether you need to decode a menu or access subtitles while conversing across a language barrier, instant translation is about to improve exponentially with the rise of AI-powered AR glasses. Even today, Google Translate in your smartphone is already helping to convert menu text and street signs in real-time. AR will mean you will no longer have to deal with the latency of waiting to figure out what is written in a language you don’t speak.
To start, Nvidia’s Drive platform for Level 2+ autonomous vehicles is already combining sensor fusion and perception with AR dashboard displays to alert drivers to road hazards, to highlight points of interest, and to provide navigation assistance.
In the current transition phase to autonomy, AR integration will allow drivers to better monitor road conditions. That why Volkswagen is already partnering with Nvidia to produce I.D. Buzz electric cars, set to run on the Drive OS by 2020. Nvidia is also partnering with Toyota, Uber, and Mercedes-Benz to make AR displays commonplace in their vehicles in the near future.
We’ve all seen (or been) that someone spinning around with their smartphone to decipher the first few steps of a digital map’s commands. But AR is already making everyday navigation intuitive and efficient. With the Google Maps AR feature already being demonstrated on Pixel phones, instead of staring at a map from a bird’s eye view, you will be able to direct the smartphone camera at the street superimposing directions virtually layered on top of the image. Not only that but as the AI identifies what you are seeing, it can instantaneously communicate with your GPS to pinpoint location and orientation. Although a mainstream rollout date has not yet been announced, this Android feature will likely make it on many smartphone models in the very near future.
We got our first taste of AR’s real-world gamification in 2016, when Nintendo released Pokémon Go. And today, the gaming app has now surpassed 1 billion downloads. But by contrast to VR, AR will increasingly be seen as the medium for bringing gamers together in the physical world, encouraging outdoor exploration, activity, and group collaboration.
AR is expected to make eSports a growing collaborative experience, turning player’s visual displays into live-action stadiums. This year, the market for eSports will exceed $1.1 billion USD.
Many of today’s most popular AR apps allow users to throw dinosaurs into their surroundings, i.e., Monster Park, learn how to dance, i.e., Dance Reality, or try virtual tattoos, i.e., InkHunter. As high-definition rendering becomes more commonplace, art, too, will grow more accessible. Magic Leap aims to construct an entire “Magicverse” of digital layers superimposed on our physical reality. Location-based AR displays, ranging from art installations to gaming hubs, will be viewable in a shared experience across hundreds of headsets. Individuals will simply toggle between modes to access whichever version of the universe they desire. Endless opportunities to design our surroundings will arise.
Apple recently announced the [AR]T initiative consisting of floating digital installations viewable through [AR]T Viewer apps in Apple stores. These installations can also be found in [AR]T City Walks guiding users through popular cities, and [AR]T Labs, which teach participants how to use Swift Playgrounds, an iPad app, for creating AR experiences.
At the recent Siggraph Conference in Los Angeles, Magic Leap introduced an AR-theater hybrid called Mary and the Monster, wherein viewers watched a barren “diorama-like stage” come to life in AR.
While audience members shared the common experience like a traditional play, individuals could also zoom in on specific actors to observe their expressions more closely. So say goodbye to opera glasses and hello to a future of AR headsets when you go to a Broadway show, opera, orchestra or rock concert.
While AR headset manufacturers and mixed reality developers are racing to build enterprise solutions in numerous business sectors, consumer use is following close behind. Magic Leap is leading the way with its “Magicverse” of localized AR displays in shared physical spaces. And as AR-supportive hardware gets integrated into our smartphones, businesses will have an invaluable opportunity to gamify products and immerse millions of consumers in service-related AR experiences. Even beyond the most obvious first-order AR business cases, new industries to support the augmented world of 2030 will soon create a market surge in headset hardware, data storage solutions, sensors, holography and projection technologies.
]]>Americans today spend over nine hours per day looking at screens. That amounts to more than a third of our livelihoods. Because this screen exposure serves as a portal to 90% of the media we consume, it continues to define and constrain how and where we consume content. But soon this may all become obsolete.
With new advancements in hardware and connectivity, augmented reality (AR) is set to replace the two-dimensional interface upon which we have become dependent. Instead, we will see and interact with the world through a digital information layer with AR headsets ultimately immersing us into dynamic stories, learn-everywhere education, and even gamified work tasks.
We got our first taste of AR’s real-world gamification in 2016, when Nintendo released Pokemon Go. Thus began the greatest cartoon character turkey shoot in history. With 5 million daily users, 65 million monthly users, and over $2 billion in revenue, the virtual-overlaid experience remains one for the books.
In the years since similar AR apps have exploded. Once thick and bulky, AR glasses are becoming increasingly lightweight, stylish, and unobtrusive. And over the next 15 years, AR portals will become almost unnoticeable, as hardware rapidly dematerializes.
Companies like Mojo Vision are even rumored to be developing AR contact lenses, slated to offer us heads-up display capabilities — no glasses required.
So let’s look at what else is happening to the technology and its projected growth.
We have already begun to sample the extraordinary functions of AR through smartphone apps. Snap recently announced it is raising $1 billion in short-term debt to invest in media content, acquisitions, and AR features. Both Apple and Google are racing to deploy mobile phones with requisite infrastructure to support hyper-realistic AR. And in the iOS space, developers are using the ARKit in iPhone software to bring high-definition AR experiences to life to a range of iPhone models. Apple’s CEO Tim Cook has repeatedly emphasized his belief that AR will “change the way we use technology forever.”
While recent rumors reveal the company’s AR glasses project has been discontinued, Apple’s foray in AR is far from over. Just recently, the company announced a large number of job postings for AR and VR experts. And although somewhat speculative, Apple is likely waiting for the consumer market to mature before releasing its first-generation AR glasses, or its pivot to an entirely new line of AR hardware products. For now, however, Apple seems to be promoting extensive hardware advancements showcased by its A12 bionic chip, not to mention the variety of AR apps available in its App Store including:
Android users have options just as varied, based on the Android software-compatible ARCore being used by developers. While the recently announced Google Glass Enterprise Edition 2 aims to capture enterprise clients, Android smartphone hardware provides remarkable AR experiences for everyday consumers including:
There is an increasing number of AR headsets in the market. A few of them are pushing the envelope to allow wide field-of-view (FOV) immersion for enterprise applications. The following list highlights key features:
DreamGlass: Connected to a PC or Android-based smartphone, DreamWorld’s headset offers 2.5K resolution in each lens, beating out Full HD resolution screens but in AR. Resolution improvements minimize pixel size, reducing the “screen door effect,” whereby pixel boundaries disrupt the image like a screen’s mesh. Offering unprecedented levels of hand-and-head-tracking precision, the headset even features 6 degrees of freedom (i.e. axes of directional rotation). With a flexible software development kit (SDK) supported by Unity and Android, the device is highly accessible to developers, making it a ready candidate for countless immersive experiences. Currently, at $619 USD the DreamGlass and comparable technology is continuing to fall in price.
Google Glass Enterprise Edition 2 is a major upgrade that includes an 8-megapixel camera, detachable lens, vastly increased battery life, faster connection, and ultra-high-performance Snapdragon XR1 CPU. Already, the Glass has been sold to over 100 businesses, including GE, agricultural machinery manufacturer AGCO, and health record company Dignity Health. Using the Glass, GE has increased productivity by 25%, and DHL has improved its supply chain efficiency by 15%. While currently only available for business customers, the new-and-improved Google Glass at $999 USD is expected to decrease in price as production costs plummet.
Vuzix Blade resembles chunky Oakley sunglasses. These smart glasses are extraordinarily portable, with a built-in Android OS as well as both WiFi and Bluetooth connectivity. Designed for everyday consumer use at a price point of $700 USD, these glasses are slowly chipping away at smartphone functionality. For easy control of an intuitive interface, a touchpad on the device’s temples allows consumers to display everything from social media platforms and user messages to “light AR” experiences. Meanwhile, an 8 Megapixel HD camera makes you no longer need your smartphone camera so that you can remain immersed in the AR experience while digitally capturing it at the same time. The glasses include built-in Alexa capabilities and vibration alerts which extends the wearers’ experience beyond pure visual stimulation.
Microsoft HoloLens 2 is the conpany’s newest headset providing an industry-leading FOV of 43° x 29°. That’s more than double the 2016-released version. FOV enlargement is not the only improvement. The headset’s center of gravity now rests on the top of the head rather than the typical front-loading of other headset models. In addition tiny cameras on the nose bridge verify a user’s identity by scanning the wearer’s eyes and customizing the display based on distance between the pupils. Once accompanied by emotion-deducing AIs which are now under development, this tracking feature could evolve to intuitively predict a user’s desires and emotions. Geared with a Qualcomm 850 mobile processor and Microsoft’s own AI built-in engine, the Hololens appears to have limitless potential.
Magic Leap One weighs 360 grams (less than 0.8 pounds) and provides one of the most lightweight headset experiences available today. Its 40° x 30° FOV is barely eclipsed by Microsoft’s HoloLens 2. A small “Lightpack” attachment connected by cable to the goggles can be placed in a wearer’s pocket to enhance the AR experience with a handheld controller containing a touchpad, haptic feedback, and 6 degrees of freedom motion sensing. Meanwhile, light sensors make the digital renderings even more realistic reflecting physical light into the viewer’s space. Magic Leap even features a virtual human called “Mica,” that provides AI-drivesn responses to the wearer’s emotions through eye-tracking detection.
As headsets plummet in price and size, AR will become ubiquitous in the next decade. Once we have mastered headsets and smart glasses, AR-enabled contact lenses will make our virtually enhanced world second nature. And ultimately, brain-control-interfaces (BCIs) will directly communicate with our neural signals to provide an instantaneous, seamlessly intuitive connection, merging minds with limitless troves of knowledge, and never-before-possible experiences. While only approaching the knee of the curve, these pioneering mobile apps and novel headset technologies as described above will begin a fundamental transformation of our lives.
Recently wearables have been under a cloud of sorts with reports of tepid interest and deteriorating sales – particularly for smartwatches. Granted there have been some exceptions, most notably Fitbit’s range of fitness trackers, but these are few and far between.
The one silver lining that may very well rejuvenate wearables and transform them completely is a renewed interest in Augmented Reality (AR).
Resurgence in AR
Ever since the Google Glass AR headset project was shuttered, AR seemed to go into the doldrums. Most notably AR could be seen in Snapchat filters that overlay images on video message faces. And AR was also to be found in video games such as Niantic’s Pokemon Go.
Due to these applications, fully immersive AR has resurfaced with big names such as Microsoft, Google, Apple and Facebook all invested. Some of these companies will undoubtedly center on applications for devices such as smartphones. But ultimately some form of wearable AR such as a successor to Google Glass will become the preferred platform for delivery.
Role of Wearables in Immersive AR
Why wearable glasses are the preferred AR delivery tool lies in their simplicity for delivering information to the wearer’s field of vision. Wearing AR glasses gives the user the ability to point to an object and receive information about it in an unobtrusive way.
But glasses aren’t the only type of wearable for the budding AR world. Any AR wearable can provide sensory information and feedback. That includes watches, rings, audio devices, footwear, and clothing. AR wearables can measure or track specific data, and provide other functionality such as GPS, video capture, and movement tracking.
Are we there yet?
Not really. Today the focus remains on wearable devices for the eyes with Microsoft’s Hololens, Magic Leap’s AR glasses, Solos AR Glasses, Vuzix M300, Meta AR (seen in image below), and others appearing in the market. Most remain in the experimental phase.
Apple in its typical fashion has adopted a different approach to AR. Initially, the company’s focus is primarily smartphone-based (see image below) with integration from existing wearables such as the new Air Pods and Apple Watch. Ultimately, however, an Apple Glasses is bound to come to market with the typical Apple twist.
What About Fully Immersive AR?
Although AR has made significant strides in the last few years, fully immersive AR remains a distant prospect. Notable challenges remain. For example, the technology needs to miniaturize the components and power sources to make them virtually invisible. A decade from now we may see the integration of AR wearables into our everyday lives. But more likely, AR’s initial success will be in play environments where adoption of wearables is already readily accepted. Just look to VR game technology today to see the short-term direction AR may follow.
]]>