Research Report: Augmented Reality for Maintenance, Repair ... · In between lies augmented...

Research Report: Augmented Reality for Maintenance, Repair and Overhaul (MRO)

This research has been sponsored by PTC.

2

RESEARCH REPORT: AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL (MRO)

Executive Summary 3

What is Augmented Reality? 4

Benefits of Using AR for MRO 5

Challenges of Using AR in Manufacturing 6

Augmented Reality Options – Hardware 7

Daqri Smart Glasses 8

Google Glass 9

Microsoft Hololens 10

Vuzix M100 & M300 Smart Glasses 11

Use Cases 12

Google Glass on the Audi A8 Production Line 12

Installation and Maintenance of Hauwei Sun2000-KTL Inverters 13

NLR & KLM Royal Dutch Training Concept 14

AR Elevator Maintenance at Thyssenkrupp 15

Augmented Reality for Maintenance, Repair and Overhaul 16

TABLE OF CONTENTS

3


The term Industry 4.0 denotes a cluster of technologies that’s poised to fundamentally reshape manufacturing and bring about a new industrial revolution. These include 3D printing, also known as additive manufacturing (AM), the Industrial Internet of Things (IIoT), artificial intelligence (AI) and mixed reality technologies, more commonly known as virtual reality (VR) and augmented reality (AR). The two most common hurdles to implementing these Industry 4.0 technologies are cost and integration.

However, the combination of increasing computing power and declining hardware costs are rapidly diminishing the severity of the first hurdle. As for meeting the second challenge—integration—that often comes down to finding the right application where the technology truly shines. In the case of AR, maintenance, repair and overhaul (MRO) applications are just that—which is why we wrote this report.

In the following pages, readers will learn the benefits and challenges of using AR for MRO. The positives include streamlining inspections, access to remote expert assistance, and the ability to upskill on demand. These advantages are contrasted with the challenges of using AR in an industrial environment, which include employee reluctance to change, the difficulties of AR training and potential interoperability issues.

Readers will also learn about the available options for augmented reality from some of the world’s leading suppliers, including Google Glass, Microsoft HoloLens and RealWear HMT-1. The report concludes with several case studies of AR being used for MRO applications, all of which point to the idea that augmented reality may offer a technological solution to the impending skills gap in manufacturing.

EXECUTIVE SUMMARY

4


In addition to ‘augmented reality’ and ‘virtual reality’, you may have heard the term ‘mixed reality’, which denotes a broader category. All three concepts can be understood in terms of a continuum, with real environments at one end and wholly virtual environments at the other. In between lies augmented reality, as well as augmented virtuality. This in-between space in its entirety constitutes mixed reality.

RealEnvironment

AugmentedReality (AR)

VirtualEnvironment

AugmentedVirtuality (AV)

Reality-Virtuality (AV) Continuum

Mixed Reality (MR)

The notion of a reality-virtuality continuum was first introduced more than two decades ago by Paul Milgram, a professor of mechanical and industrial engineering at the University of Toronto. To put the continuum in a manufacturing context, think of the far left (Real Environment) as the traditional factory floor: workers operating machines based on information displayed on dials, gauges or screens.

The far right (Virtual Environment) is exemplified by the use of virtual reality in factory floor planning.

It may be difficult to imagine examples of augmented virtuality in manufacturing, but one possible application could be using handheld peripherals to input commands, or testing ergonomics in VR factory-floor planning.

WHAT IS AUGMENTED REALITY?

5


Even a passing familiarity with AR should give any manufacturing professional ideas for implementing the technology on a production line. As a simple example, Mitsubishi Electric has been developing maintenance-support technology using augmented reality based on a 3D model that enables users to confirm the order of inspection on an AR display and then enter inspection results with their voice.

“We think that the most useful application of AR is maintenance in a manufacturing environment,” said a representative from the Mitsubishi team developing this technology. “Currently, field workers confirm a maintenance target using a maintenance manual. The

process is time-consuming and tiring for the workers because the manual and the maintenance target should be confirmed together.”

In addition to streamlining inspections by replacing a pen-and-paper checklist with a voice-based system, augmented reality can enhance a worker’s ability to perform maintenance routines by superimposing simple step-by-step instructions on their field of view. This is the logical next step in being able to see machine status at a glance using AR, which is a benefit that goes beyond MRO applications.

Perhaps the most intriguing benefit of using AR for MRO is the ability for users to invoke remote expert support when completing maintenance or repair tasks. Field service often requires experts to travel to remote worksites, but the telepresence afforded by AR means a single expert can service multiple sites without ever having to leave the office.

Taken together, these benefits constitute a kind of upskilling on-demand, enabling workers to perform tasks that may go well beyond their nonaugmented skillset. That means reduced training time and increased first-time fix rates. While many of these benefits apply to more manufacturing operations than just maintenance, AR is

particularly well-suited to meeting the demands of MRO applications.

BENEFITS OF USING AR FOR MRO

6


It’s easy to get swept up in the hype of new technology and consequently fail to recognize the practical difficulties it presents. This is especially true in the case of Industry 4.0 technologies—such as augmented reality—which tend to be wrapped up with so many marketing buzzwords that one may wonder whether there’s anything of substance to it at all. Now, while AR is certainly more than a buzzword, there are nevertheless challenges for new adopters which can be loosely divided into two categories: technical and organizational.

The technical challenges for AR generally boil down to the user interface, with the need to improve field of view (FOV), brightness, display quality, latency, etc. In the case of FOV, for example, even the best AR HUDs can only offer up to 90 degrees. Compare that to the 190-degree horizontal and 120-degree vertical FOVs for

normal human vision and the gap between where the technology is now and where it needs to go becomes obvious.

The principal organizational challenges for AR are a lack of knowledge about the technology and an insufficient level of digitalization within the company. The first challenge is somewhat ironic, given the new possibilities for training and education that augmented reality brings to the table. However, unlocking that potential requires one

to recognize it in the first place, which is difficult in the absence of first-hand experience. The second challenge—insufficient digitalization—is an issue for many Industry 4.0 technologies, including augmented reality. However, it’s particularly problematic for AR because of how much the technology relies on data collection and analysis. Without a sufficient number of assets sharing information, adding AR capabilities will do little to improve plant efficiency.

CHALLENGES OF USING AR IN MANUFACTURING

7


According to a recent analysis by Research and Markets, the global AR/VR market will reach $94.4 billion USD by 2023. Although VR devices currently make up a larger portion of that market, the analysis predicts that the market for AR devices will grow faster, with a CAGR of 73.8 percent. All this growth entails some uncertainty regarding which AR hardware options will still be supported a decade from now. That’s why this report focuses on surveying the current AR offerings from some of the best-known suppliers. Check out the AR/VR section on engineering.com for the most up-to-date information.

AUGMENTED REALITY OPTIONS – HARDWARE

8


DAQRI SMART GLASSESThe latest iteration of the company’s augmented reality hardware, DAQRI’s Smart Glasses are designed for professional use. The unit is based on a 6th generation Intel Core m7 processor and uses dual LCoS optical displays, which give it a 44° diagonal FOV. It comes equipped with Worksense Standard, DAQRI’s suite of apps for AR tasks. Detailed specs for DAQRI Smart Glasses appear below.

Processor 6th Gen Intel Core m7 (Up to 3.10 GHz)

Optics Dual LCoS Optical DisplaysResolution: 1360 x 768Frame Rate: 90 fps

Connectivity WiFi 802.11 A/B/G/N/AC 2.4/5 GHzBluetooth

Battery Li-ion 5800 mAh

Storage 64 GB SSD

I/O Ports 2 USB 3.1 Type C3.5mm Headphone Jack

Audio 2 Microphones with Active Noise Cancellation

Depth Sensor Camera Range: 0.4mm to 4mResolution: 640 x 480Frame Rates: 30, 60, 90 fps

Color Camera RGB 1080p HD, 30 fps

AR Tracking Camera 166° Diagonal Wide-Angle Fisheye LensResolution: 640 x 480Frame Rate: 30 fps

9


GOOGLE GLASSWhile it failed to find purchase in the consumer market, Google’s foray into augmented reality has seen considerably more success in medical and manufacturing applications. The device features some ambitious technology, including bone conduction for audio output and a prism projector display. The successor to the Explorer Edition, Google Glass Enterprise Edition eschews the bone conduction tech, but also makes several improvements over previous iterations, including an Intel Atom processor and a barometer. Detailed specifications for Google Glass Enterprise Edition appear below.

Processor Intel Atom (32) -model unspecified

Optics 640 x 360 Himax HX7309 LCoS display

Connectivity Dual-band 2.4+5 GHz 802.11 a/b/g/n/acBluetooth

Battery 780 mAh

Storage 32GB

I/O Ports Micro USB

Audio Speaker -details unspecified

Sensors BarometerCapacitive head sensorHinge sensorAssisted GPS & GLONASS

Cameras Supports 5MP stills and 720p video -details unspecified

10


MICROSOFT HOLOLENSBilled as the world’s first full untethered holographic computer, the HoloLens is available in both a Development Edition and a Commercial Suite configuration. Companies based in North America also have the option of renting a HoloLens through a Microsoft partner. The current version of the HoloLens has a 35° FOV, but a recent patent from Microsoft for a MEMS laser scanner could increase the FOV for the next generation of HoloLens to 70°. Detailed specifications for the HoloLens appear below.

Processor Intel 32 bit architecture with TPM 2.0 supportCustom-built Microsoft Holographic Processing Unit (HPU 1.0)

Optics See-through holographic lenses (waveguides)2 HD 16:9 light enginesHolographic Resolution: 2.3M total light pointsHolographic Density: >2.5k radiants (light points per radian)

Connectivity Wi-Fi 802.11acBluetooth

Battery 2-3 hours of active useUp to 2 weeks of standby timeFully functional when charging

Storage 64GB Flash

I/O Ports 3.5mm Headphone JackMicro USB 2.0

Audio Built-in speakers4 microphones

Cameras/Sensors 1 IMU4 environment understanding cameras1 depth camera1 2MP photo/HD video camera1 ambient light sensor

11


VUZIX M100 & M300 SMART GLASSESFounded in 1997, Vuzix has a long history in the augmented reality industry. Its current offerings suitable for industrial applications are the M100 and M300 Smart Glasses. The M300 is marketed as having enhanced functionality, and there are some important differences between the two. For example, the diagonal FOV on the M100 is 15° and 20° on the M300. Detailed specs for both models appear below.

M100 M300

Processor OMAP4430 @ 1GHz Dual Core Intel Atom CPU

Optics WQVGA color displays 16:924-bit color

nHD color display 16:924-bit color

Connectivity Wi-Fi 802.11 b/g/nBluetooth

Wi-Fi b/g/n/acDual-Band 2.4/5 GHzBluetooth

Battery 600mha internal battery3800mha external battery pack

160mAh internal battery860mAh external battery

Storage 4GB internal flash memoryMicro SD support up to 32GB

64GB internal flash memory

I/O Ports MicroUSB MicroUSB 2.0 HS

Audio Ear speaker and noise cancelling microphone

Ear speaker and noise cancelling microphone

Cameras/Sensors 5 MP Stills1080p Video, 16:93 DOF gesture engine (L/R, N/F)Proximity & Ambient Light Sensors

10 MP Stills1080p VideoAuto-FocusOptical Image StabilizationFlash/Scene IlluminationInward & Outward Facing Proximity Sensors

12


Although AR is still a relatively new technology in the manufacturing space, there are already numerous examples of its benefits in a broad range of applications. Many of these are within the space of maintenance, repair and overhaul. This report focuses on four use cases in particular.

GOOGLE GLASS ON THE AUDI A8 PRODUCTION LINE

In 2015, researchers from Heilbronn University in Germany ran a four-month project on the AUDI A8 production line using the Explorer Edition of Google Glass. The target task was the calibration of the drive assistance system testing bay, which fell to workers on the Monday morning shift. All the sensors and actuators on this system have to be checked at various intervals (e.g., weekly, monthly, yearly), further complicating the calibration.

Prior to the study, workers calibrated the testing bay using paper-based documentation. Interestingly, although the intent was to replace this pen-and-paper system with one based on AR, the researchers elected to leave the calibration workflow unchanged after determining

that it was already “adequately optimized.” Normally, the calibration involves consulting data displayed on various screens in the testing bay and documenting performance by stamping a protocol book. The researchers were able to replace this approach with a voice-based system, encouraging users to “think out loud” while using Google Glass.

As a result, according to the researchers, “Any employee independent from his training level now is able to perform the calibration of the driver assistance system testing bay in an acceptable time span.”

USE CASES

13


INSTALLATION AND MAINTENANCE OF HAUWEI SUN2000-KTL INVERTERS

In 2016, the electronics giant Hauwei partnered with Inglobe Technologies to develop an AR solution for the installation and maintenance of the former company’s SUN2000-KTL inverters. According to Hauwei, these units are spread all over the world, potentially requiring technicians to travel significant distances in order to maintain them. Moreover, the variety of issues that a technician might encounter means that they are generally required to be familiar with more procedures than they can possibly remember. Taken together, these factors point to augmented reality as an obvious solution.

For Inglobe Technologies, the primary challenges for this project were to create an application that could rely entirely on natural feature tracking and which could reliably function outdoors, despite varying light levels. On top of that, Inglobe Technologies never had direct

access to a SUN2000-KTL inverter. Nevertheless, the two companies were able to create a simple interface that would work for Chinese and American technicians alike.

Once the user selects a language (English or Mandarin), the application can automatically recognize a nearby inverter and display a list of procedures on screen. These can be assigned by a maintenance manager or initiated by engineers in the field. Procedures can then be followed in a linear fashion or accessed as required.

According to Rhonda Truitt, Director of US R&D Technical Communication Innovation & Best Practices at Hauwei, “Our customers have responded overwhelmingly positive about this experience. I am confident that we will see a sharp decline in installation errors due to this AR experience for post-installation checks.”

14


NLR & KLM ROYAL DUTCH TRAINING CONCEPT

In 2017, the Netherlands Aerospace Centre (NLR) and KLM Royal Dutch Airlines collaborated on a training concept for aircraft maintenance engineers. Groups of students were taught how aircraft air conditioning systems work in a typical classroom setting. However, during the training, students and instructors both wore HoloLens systems in order to view detailed 3D models of A/C systems. This allowed students to walk around and through the system, and also collaborate on tasks and ask “hands-on” questions of the instructor.

According to NLR, this approach to training yielded significantly better results—particularly in terms of student engagement—compared to conventional techniques. “[I]n the virtual world it is easier for an instructor to look over the shoulders of the trainees than in

the real world, enabling him to give feedback,” said Harrie Bohnen, Manager of NLR’s Training, Simulation and Operator Performance Department. “The training can be given anywhere, and practical exercises can be performed in a virtual aircraft instead of a real one. You no longer need the actual aircraft, which increases its availability and reduces wear and tear.”

15


AR ELEVATOR MAINTENANCE AT THYSSENKRUPP

The German elevator manufacturer, thyssenkrupp, was one of the early adopters of AR for MRO when it began using the HoloLens in 2016. The company announced plans to equip over 24,000 service technicians with AR capabilities, enabling them to visualize and identify problems before they arrive at the job site, in addition to giving them remote, hands-free access to technical and expert information on-site.

According to the company, initial field trails saw service maintenance interventions being performed up to four times faster. This is no doubt in part due to the integration of HoloLens with MAX, thyssenkrupp’s IoT-enabled predictive maintenance solution, launch back in 2015.

“By utilizing the out-of-the-box Skype experience without any additional development required, thyssenkrupp’s 24,000 service engineers can now do their jobs safer and more efficiently,” noted Scott Erickson, General Manager for Microsoft HoloLens. “Triaging service requests ahead of the

visit and getting hands-free remote holographic guidance when on site has reduced the average length of thyssenkrupp’s service calls by up to 4x.”

16


It’s easy to dismiss the technologies that fall under the broad heading of the fourth industrial revolution as being overhyped or years’ away from being practical, but as augmented reality in the context of maintenance, repair and overhaul applications demonstrates, there are pragmatic reasons for taking AR and other Industry 4.0 technologies seriously. The decreasing cost of AR hardware combined with the growth of the overall market means that augmented reality solutions are only going to become more accessible over time. As more and more manufacturers recognize the value of AR for MRO, having AR capabilities will become less a competitive advantage and more a necessity.

How long can you afford to wait?

AUGMENTED REALITY FOR MAINTENANCE, REPAIR AND OVERHAUL

This research has been sponsored by PTC.Industrial AR connects users across digital and physical worlds. It will do more than just change products, operations, and services – it’ll transform business. To learn more, visit: https://www.ptc.com/vuforia

Research Report: Augmented Reality for Maintenance, Repair ... · In between lies augmented...

Documents

Transcript of Research Report: Augmented Reality for Maintenance, Repair ... · In between lies augmented...