Exploiting the Potential of Augmented Reality
-
Upload
aricent -
Category
Technology
-
view
58 -
download
1
Transcript of Exploiting the Potential of Augmented Reality
-
-
AR is a game changer that is boosting operational efficiency, transforming customer engagements and creating new business opportunities
Exploiting the Potential of Augmented Reality
2
Figure 1: AR offers a modified view of reality using computer-generated sensory inputs
Augmented Reality (AR) is a promising emerging technology that is capturing the imagination of many businesses. This paper aims to present an overview of AR, its applications and underlying technologies, and explore how AR can improve productivity and efficiency and potentially transform the nature of work as we know it.
Augmented Reality is a field of computer science where “a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality, in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality.”1
AR can be viewed as a natural consequence of digital evolution. It’s an example of how technology has the power to change the way we perceive and interact with our environment. The potential of AR is made possible by the mobile
revolution, which brought technology into the hands of people in the forms of smartphones, tablets and now wearable devices. Devices such as cameras are embedded into these devices with programmable interfaces. With touch as a primary input method, mobile devices are very cogent in the experience they provide. In the un-augmented real world, it is not possible to interact physically with every object or parts of objects. This could be due to obstacles in terms of size, location, safety or physical characteristics, such as fragility.
Consider a car showroom. There are several models of cars for display and the customer can be overwhelmed when a salesperson explains the details of each model. AR lets the customer learn about the various models on their own terms, using an immersive experience that can provide the information they need to make the buying decision.
AR technology involves digitally recognizing an object and augmenting it, for example by enhancing it with relevant content including imagery, videos and documents. It allows users to engage with real-world objects by taking advantage of sophisticated visual assimilation techniques. It removes obstacles and delivers a deep level of interaction with the object and the environment. Depending on the type of object, the augmentation can create an immersive experience leading to an improved cognition of the object. With mobile devices, the experience is enhanced with various input techniques such as touch and voice.
Consider again the application of AR in the car showroom. When a customer enters a showroom, they are handed a mobile device such as a tablet with predefined software applications deployed. The customer launches an app that internally launches the device camera. When the customer approaches a particular car, the app intelligently identifies the car and augments the live frame on the device with relevant content. This might include text about the key features of the model, a video demonstrating the innovative engine technology, or an animation of the interior.
AR can be applied in countless scenarios—factories, offices, and commercial and public spaces—with the objective of delivering rich, experiential information, insight, and education unmatched by any other
The Case of AR01
Reality
AugmentationContent
EnhancedPerception
1. Augmented Reality, Wikipedia, https://en.wikipedia.org/wiki/Augmented_reality
3
Figure 2: Information Augmentation for machines using AR helps employees in a manufacturing facility
Business Use Cases
Both large and small businesses are at the forefront when it comes to adopting and applying AR. The technology is being used in a variety of scenarios from employee training to customer engagement. It has the potential to improve operational efficiency and open new avenues to innovate both products and services that enhance customer engagement and loyalty, and drive competitiveness. What follows are four examples of how AR is being applied today in the business world.
Manufacturing
Optimizing operations by increasing efficiency is a constant focus in the manufacturing sector. Many production tasks require quick and easy access to the right data at the right time to perform a specific task. For example, consider a large shop floor with a variety of different machines. The operation of each of these machines includes monitoring them while in use and scheduling regular maintenance. Given the large amount of disparate information available on each machine, it is usually not possible to have instant access to information on a specific machine’s operational settings or maintenance and repair records.
Using AR, all of this information is accessible instantly to the employee. Each machine can be defined in the AR system and all the relevant data associated with it can be filed. Using a mobile device, the employee can learn about the machine, access relevant data, and take necessary actions to operate, maintain or repair
Warehouse Management
Large warehouses and distribution centers are common in many industries, acting as inventory hubs for manufacturers and retailers, including e-commerce retailers. Some double as sales centers accessible by customers. With millions of SKUs in inventory, it can be challenging for workers to find accurate information about the products and manage the picking and replenishment of inventory. Warehouses that service customers directly face the added challenge of accessing detailed information about the products.
AR can be a very effective solution to help warehouse workers/customers manage the products on the shelves so that they can make accurate selections. For example, the company can provide workers with an app to use on their mobile devices that augments the image of a product with the relevant specifications when viewed through the live camera frame. This can increase productivity by decreasing the chance of workers selecting the wrong product to be shipped. In warehouses that allow customers direct access, the app can significantly improve their shopping experience by helping them find,
technology. AR can be a game changer for the business world, given the need for companies to boost productivity, increase the skills of their employees, and create deeper engagements with their suppliers and customers.
the machine. They can take advantage of technologies such as virtual assistants along with AR that collect and analyze data continuously, providing insights and actionable tasks that lead to simpler and more efficient solutions that boost both capital and labor productivity.
4
Figure 3: Warehouse workers can manage products on shelves using AR
Figure 4: Pre-loaded AR on mobile devices enhances training by providing interactive wizards
have to provide detailed instructions and information to workers and educate them on each individual machine and its components.AR is very useful in such scenarios. Trainers are given mobile devices preloaded with AR-enabled training modules, where each module acts as an interactive wizard to guide the trainees through each step. It also expedites the process of learning as the trainees can repeat the wizard until they are proficient in the module. The learning process can take place independent of the trainer.
Training
Hands-on training is an essential part of learning and development process at many companies. The more complex the machines or processes are, the more challenging the training process and the longer it takes. For example, consider a product assembly process in a manufacturing plant that requires workers to manage a variety of complex machines. Trainers
Product Marketing
Product marketers are constantly trying to renew and reinvent the product marketing process to engage and excite the customer. Storytelling is integral in the marketing process. Businesses aim to continuously reorient themselves to adapt
evaluate and select the right products with minimal involvement from staff. Also, the app can provide accurate information about the kind of products customers buy and the factors involved in the decisions they make. This data can be analyzed and used, for example, to manage inventory levels and store popular products closer to the customer service desk.
to changing market conditions and tell compelling stories about their products to existing and potential customers. Successful storytelling is the starting point for increasing sales and market share and driving customer loyalty.
AR helps marketers create a “wow” factor that is integrated into the storytelling through powerful visuals and content. The AR solution enhances the customer experience and can increase the likelihood of a sale. Consider the car showroom example described above. A customer is provided with a mobile device that has a preloaded AR module to use as they tour the showroom floor. As they learn about the features and attributes of the car models, they are also being marketed to through the imagery and the messaging of the content.
Figure 5: AR helps marketers provide immersive experiences for their customers
The opportunity is similar in other industries, such as retailing. AR solutions can help retail marketers present their products in innovative ways. For instance, a retail clothing store that provides shoppers with AR-equipped mobile devices might include the option of selecting the product category the shopper is looking for and directing them to that department. The visual enhancements might include information and visuals about items on sale and the quality of the products they are shopping for, as well as direct them to articles of clothing that are complementary.
6
Figure 7: Object definition in an AR solution
Solutions02Solution Model
There are a variety of models included in the range of AR solutions, from those that are device-only to others that include a client and server. Regardless of the type, the typical solution comprises the following four categories: Object Definition Object Classification Object Detection and Tracking Object Augmentation
Object Classification
Once the objects of interest are identified and defined in the system, they are classified into specific groups based on either predefined or user-defined parameters. The goal of the classification is to enable best possible detection of the object when it is detected by the device. The object classification is done by extracting features from the visuals—still images and videos—that were collected during the definition of the object, either using an automated or user-triggered process. These features can be shape, contour, color, histogram, texture or other characteristics. Further, content creators use algorithms to extract features from digital images, including the size/rotation/illumination invariant. SURF (Speeded Up Robust Features), SIFT (Size Invariant Feature Transform), GLOH (Gradient Location and Orientation Histogram), and ORB (Oriented FAST and Rotated BRIEF) are some of the algorithms widely used to create image descriptors and feature extractions.
Object Definition
Object Definition is a key step in developing an AR solution. Content creators first define the object and collect and save visual representations of the object that may include still images and videos. The visuals are captured in as many dimensions as possible with the best possible resolution and quality and are used in subsequent process steps to classify and detect the object. Content creators also define tags or areas of interest in the object that identify specific parts of the object and associated content. This content is used to augment the live scene that is captured on the device. This content can include plain text,
01 —
02 —
03 —
04 —
documents, videos, images, animations, and other visual constructs.
Figure 6: The four elements of a typical AR solution
Object Definition
Object Augmentation
Object Classification
Object Detectionand Tracking
Visual Elements
Attribute 1Attribute 2
Attribute N
.
.
.
Object
Content
Aa
Figure 8: Object classification in an AR solutionFigure 9: Object detection and tracking in an AR solution
Figure 10: Object Augmentation in an AR solution
7
Object Detection and Tracking
Detection of objects is an end-user triggered process. A device with a camera is established to detect and augment the objects of interest that have been predefined and classified. The camera captures the current live frame and provides it as an input to the detection algorithms that are associated with the classification being performed. Several combinations of features and image descriptors on the live frame are used along with feature-matching algorithms such as FLANN matcher/brute force to detect the object.
Once the object is detected with the movement of the device in the user’s hand and the change in the user’s position, the object is tracked in the live frame.
Object Augmentation
Augmentation of the detected object on a live camera frame involves enhancing the object in the live camera frame with visual elements that are predefined or generated based on the properties defined for the object. The associated tags and content that are defined along with the object can be used to construct these augmented visual elements. Animations and graphics can also be used together with standard elements—such as text, images and video—to enhance the live scene.
Objects
Shape based Classification
Classifier
7
Objects
Shape based Classification
Classifier Objects
Color based Classification
Classifier
Figure 11: A device-only AR solution
Figure 12: A client-server based AR solution
8
Solution Architecture
AR solutions involve understanding reality and converting it into a digital form. The most efficient and widely available transducer to do the job is the camera. A live feed from a digital camera is used to capture and convert the image of an object into an understandable format that AR uses to detect and recognize the same live object when the camera is focused on it again. AR solutions can be implemented differently based on factors such as how and where the solution is being applied and the scale of use. The type of solutions can be classified as following: Device Only Device and Server
Device Only
A device-only solution will have all the components associated with the process steps of AR described in the solution model above but will reside on a single device. A simple, yet robust AR solution can be implemented with a camera that has an acceptable minimum resolution, and good processing and storage capability. The following diagram illustrates a device-only solution:
Device and Server
A device-and-server solution has components associated with the four process steps of the AR solution but they are distributed across a device and a server. This typically is a client/server model in the application world. The following diagram illustrates the device-and-server solution:
The main disadvantage of the device-only solution is the limited capacity—both processing and storage—of the device. The solution can only scale as much as the device capacity can support. For example, on a device with 64 GB storage, the size of the image, video and classifier data will be limited to a maximum of around 56 GB—excluding the operating system—and the processing speeds will be limited by the number of cores on that device.
01 —
02 —
Figure 11: A device-only AR solution
This solution has the advantage of utilizing the power of a server or cluster of servers with higher processing and storage capabilities than the device for feature extraction and training. This enables the solution to scale. The detection process can be initiated either on the server or on the local device depending on the capability of the network available to the device.
As with any client/server application, network latency is a disadvantage for the AR device-and-server solution. When the process of detection is executed on the device, it leads to a solution closer to the device-only solution with the exception that the image classification and training is performed on the server and all the defined objects and their associated attributes are stored on the server. The required data for detection and augmentation can be synchronized with the device, thereby exploiting the capabilities of both the device and the server(s).
Device-only solutions are useful when they need to be executed independently of a network. For example, if a small, packaged product is to be marketed to customers across different locations by augmenting information about the product and creating a compelling AR customer experience, the marketer can use a device-only solution and create this experience without dependency on the network.
Benefits
9
While physical paper catalogs and manuals are
useful, it would be difficult for anyone to
remember every detail of an object that they deal
with, based only on what they have read. AR
solutions help enhance the assimilation of
information by utilizing visual images, sound and
touch. These solutions render complex sensory
information about objects that enhance learning.
03Enhanced Assimilation
In a world where digital systems have redefined
the way we capture and store information about
various objects, being able to render such
information as required plays a significant role in
improving productivity. Conventional interfaces
such as web-based and mobile interfaces that
communicate digital content can enhance
learning and focus attention. However, an
interactive experience has the capability to more
fully capture and hold the user’s attention and
keep them engaged. Interactive experiences are
the core of AR solutions and contribute to
increasing productivity.
Increased Productivity
Information about complex objects and systems
is difficult to visualize and communicate to
others. This could be due in part to the sheer
volume of information about, and the structural
complexity of the object or system. AR solutions
can help manage this complexity by providing
content for augmentation for different part and
Managing Complexity
levels of the object. This layered approach helps
users learn the details of each part and
component while progressively building an
understanding of the overall object or system.
For example, consider a complex office all-in-one
printer that provides capabilities such as image
scanning, messaging and photocopying. The
printer has several components such as paper
trays and toners. Understanding these
components is necessary for maintenance. AR
can help visualizing the attributes of each of the
components of the machine cascading down from
the most complex part to the least complex part.
Users of digital systems in general prefer to
collaborate with other users—family, friends and
co-workers—to shape opinions and make
decisions about the products they purchase. AR
solutions can accommodate collaboration by
including live interaction between multiple users
who share their views and opinions during the
AR experience.
Collaborative experiences are particularly useful
for remote training. For example, consider a
scenario where a trainer is explaining the
functionality of a sophisticated machine to a
group of people in a different location. When the
trainer focuses a live camera device on the
machine and its key components, AR can help
trainees view augmented content such as a video.
With an appropriate solution, the trainees can
also point to an area of interest in the video and
ask the trainer for more information through
augmentation.
Collaborative Experiences
One of the key challenges in developing AR
solutions is creating object-detection algorithms
that can identify objects in all orientations, sizes
and lighting conditions. However, there is no
one-stop, sure-fire algorithm that can
consistently deliver a 100% match during the
object detection. Hence, the aim in detecting
objects is to maximize the probability of correctly
detecting the object. This uncertainty requires
solution designers to make assumptions about
certain scenarios, which may limit how well the
solution can work beyond the scenario they are
designing for.
04
10
Challenges
Accurate Detection and Recognition of Objects
Another challenge with object detection is the
issue of “false positives.” A false positive is a
match to an object that does not exist in the
template/definition within the solution. That is,
while defined objects are correctly detected as
expected, the objects that aren’t valid or not
defined are incorrectly detected to be valid
objects. This issue can be resolved by providing a
larger sample of images and videos to train the
system and by applying customized algorithms
that can filter with greater granularity during
detection.
For example, consider two objects such as an IP
phone and a notebook computer that are the
same color. If there are not enough images for
either of these objects, it is possible that the IP
phone could be detected as a notebook and vice
versa.
False Positives
Mobile devices come in various screen sizes and
dimensions. The amount of information that can be
augmented for any given object on a device depends
on the screen’s available real estate. For example,
when there is a defined amount of information that
can be used to augment the object, it becomes more
challenging to manage the augmentation the smaller
the device’s screen size. A trade off needs to be made
between how generic and pared down the solution
can be made versus how many tailored solutions can
be made for different devices.
For instance, consider an iPad and an iPhone. The
screen sizes of these two devices are quite different.
For the same object, the amount of information that
can be augmented on an iPad will be greater that on
an iPhone. If the same amount of information is
attempted to be shown on the iPhone as the iPad, it
may lead to information clutter that will distort the
view and hamper the user experience.
Size of Mobile Device
Accurate tracking of an object in a live frame as the
user moves the mobile device around the object in
the contextual space is dependent on the quality of
the sensors and GPS in the device. These factors
can limit the deployment of the AR solution on a
limited set of devices based on their hardware
capabilities.
Accurate Object Tracking
Augmented Reality solutions rely on visual
input—images and video—of objects captured and
trained to eventually be detected and recognized. It
is important to ensure the sources of these inputs
are authorized to collect and store the information.
While this is relatively low risk for in-house worker
training and education solutions, it can be an issue
for public-facing solutions.
Security
11
The recent release of the Pokémon Go app is an
evidence of how geo-location is being married
with AR to provide an immersive user experi-
ence.
All the object recognition challenges for AR
solutions can be reduced to some extent by using
the user’s location as one of the dimension for
overlaying the data and identifying the object.
For example, if the user is in front of a building
that is a point of interest (POI) and the AR
application is able to display the building’s floor
plan, then the AR app need not rely on image
recognition. Instead it can use the location of the
user as the primary image recognition input to
identify the building.
05 Location and AR In certain cases, AR can utilize the camera’s depth
sensor, a new component that is starting to be
introduced into tablets. As devices get more sophisti-
cated, and accessories such as glasses and wearables
mature, usage of AR solutions will increase.
One area of location-based AR of interest to develop-
ers is the automotive market. Auto use cases will use
location-based input to stitch the live stream from
multiple cameras in the car with POI data to display
augmented images on the windshield.
Augmented Reality on various visual interfaces is
a promising paradigm providing an opportunity
to create immersive experiences for users. Given
the pervasiveness of mobile devices such as
phones and tablets, creating these experiences
The Way Forward06
12
can help companies grab a larger mind share of
users and increase brand loyalty. As the
Internet-of-Things wave spawns new devices and
enhances existing ones, AR is poised to play a
potentially pivotal role in creating the next
generation of user experiences. With the
technology evolving rapidly, now is the time to
invest resources to develop AR solutions that will
define the future of the user experience.
Aricent is a global design and engineering services provider working with market leaders to
develop best-in-class products and services in the areas of communication infrastructure,
software and internet services, and embedded systems. Aricent has developed a software
framework that provides application programming interfaces (APIs) for developing integrated
Augmented Reality applications and solutions on mobile platforms for different domains.
The key features of Aricent’s Augmented Reality Framework (ARF) include:
Image Processing Engine
Feature Extraction
Multi-Class Object Detection
Gesture Recognition
Proximity Sensing, Ambient Location, Absolute Location (for both indoor and outdoor
locations)
Content/Template Definition
Floor Map Integration
Activity Mapping (Location, Image/Text, Content, Gesture Events, Point of Interest)
Alerts and Notification
Barcode/QR Code/OCR-Bbased Detection of Objects
About AricentAricent is a global design and engineering company innovating for the digital
era. With more than 12,000 design and engineering talent and over 25 years of
experience, we help the world’s leading companies solve their most important
business and technology innovation challenges - from Customer to Chip.
© 2016 Aricent. All rights reserved.
All Aricent brand and product names are service marks, trademarks, or registered marks of Aricent in the United States and other countries.
13
Contacts
Raghu Kishore Vempati, Principal Systems Engineer, InnovationEmail: [email protected]