Offshore Wind Farms failures, e-maintenance using Augmented Reality

Summary

• E-maintenance • Definitions

• E-maintenance for offshore Wind farms

• Augmented Reality • Definition of Augmented Reality

• Display technologies

• Maintenance configurations

• Experiences made

• Future works

2

• e-maintenance → establish a real time Heath report • Anticipation of degradation • dynamic estimation of the time to failure

• e-maintenance integrate : • A distance access to information of maintenance • high level of proactivity (Intelligent Maintenance

System)

• e-maintenance allows : • cooperation and sharing of knowledge throughout the

maintenance process

• e-maintenance is based on : • ITC • Web service • Teleopration

3

Maintenance Remote

management

ICT

e-maintenance

What is e-maintenance

Definitions

• E-maintenance: (the “e” in e-maintenance)[Baldwin]

• = + +

4

Excellence

maintenance

Efficient

maintenance

Effective

maintenance

Enterprise

maintenance

do more with

fewer people

and less money

improve

equipment

performances

contribute directly

to enterprise

performance

E-maintenance

E-manufacturing

Conceptual definition

5

E-business

Maintenance

Applications

Reliability

Applications

Synchronization Integration

Conceptual point of view [Baldwin]

The network that integrates

and synchronises the various

maintenance and reliability

applications to gather and

deliver asset information

where it is needed. E-

maintenance is a subset of

e-manufacturing and e-

business [Baldwin].

E-maintenance

E-manufacturing

Conceptual definition

6

E-business

Maintenance

Applications

Reliability

Applications

Synchronization Integration

Conceptual point of view [Baldwin]

Production Monitor links

Costumer

feedback

The ability to monitor plant

floor assets, link the

production and

maintenance operations

systems, collect feedbacks

from remote customer sites,

and integrate it to upper

level enterprise applications

[Intelligent Maintenance Center].

Bu

sine

ss syste

m

E-maintenance

E-manufacturing

Conceptual definition

7

E-business

Maintenance

Applications

Reliability

Applications

Synchronization Integration

Production Monitor links

Costumer

feedback

Transformation system that enables the manufacturing

operations to achieve

predictive near-zero-downtime

performance as well as to synchronize with the business

systems through the use of

web-enabled and tether-free

(i.e. wireless, web...) infotronics

technologies [Lee and Ni 2004].

Near –zero-downtime

Usin

g IC

T

Bu

sine

ss syste

m

E-maintenance

E-manufacturing

Conceptual definition

8

E-business

Maintenance

Applications

Reliability

Applications

Synchronization Integration

Production Monitor links

Costumer

feedback

a maintenance management concept where maintenance

tasks are managed

electronically using real time

equipment data obtained over the Internet. It introduces an

unprecedented level of

transparency and efficiency

into the entire industry.

Near –zero-downtime

Usin

g IC

T

Management

concept

Transparency Efficiency

Definition of e-maintenance [http://www.devicesworld.net (iSCADA)]

9

Assets/information sources

Maintenance Dept

Middle Management

Top Management

Inspection/ Complaints

Reports

Reports

Assets/information sources

Maintenance Dept

Middle Management

Top Management

Precise &

Concise

Information

Order and act

E-maintenance for Offshore Wind Farms (OWF)

10

• E-maintenance concept is a very interesting way to the maintenance of offshore wind turbines because:

• Assessment of data flow

• Management on personal regarding experience and efficiency

• Management of team work using (information and communication technologies) ICT

• Reduce the Corrective Maintenance actions (preventive maintenance, condition maintenance, etc)

• Improve collaboration of different part of the systems

• Access to OWF is not always possible for experts

• Possibility to use the web technologies and the collaboration between technicians and experts

Context and problematic

• Wind turbine are off-shore (Mer Innovate)

• ~1 hour for accessing a wind farm.

• Accessibility depends on weather conditions.

=> Few time to perform maintenance.

• Expert can’t always be physically off-shore.

12

Need to

improve communication

Definition of Augmented Reality

Augmented reality (AR) is defined by (Azuma 1997) as: 1. Combines real and virtual

2. Interactive in real time

3. Registered in 3-D

Real

environment

Virtual

environment

Augmented

reality

Augmented

virtuality

Reality-virtuality continuum

Mixed reality

13

Definition of Augmented Reality : how it works?

• Principle for augmented reality vision-based tracking (Azuma 1997) • A camera gets the scene

• Based on the scene, camera pose is computed (position and orientation).

• When cam pose is known, the virtual object is added in the scene.

• Real and virtual objects are mixed before displaying the result.

14

scene acquisition

Compute camera pose

Add virtual object

Mix virtual object and real scene

Display Augmented

Reality

Definition of Augmented Reality

Augmented reality application examples

Volskwagen MARTA project Maintenance

Ikea Marketing

Metro Paris application Entertainment

15

Weather cast

Not AR, this is Augmented Virtuality

Projects with Augmented Reality and Maintenance

MARTA Volkswagen Mobile Augmented Reality Technical Assistance (with metaio framework) Sept. 2013 [MARTA 1] / [MARTA 2]

eKurzinfo Audi (with metaio framework) Oct. 2013 [eKurzinfo]

Festo and Industry 4.0 – production systems of the future [Festo]

Mitsubishi Electric meViewAR on Epson Moverio display (with metaio framework) [Mitsubishi 1] / [Mitsubishi 2]

ARMRO bitstars Augmented Reality Maintenance Repair Operation [ARMRO] / [bitstars]

KARMA (Knowledge-based Augmented Reality for Maintenance Assistance) First project with AR and maintenance. [KARMA]

16

Definition of Augmented Reality (Azuma 1997) (Krevelen & Poelman, 2010)

Topics (on left) around Augmented Reality and influences parameters (on right)

17

Robot Vision

Registration

Tracking

3D

Rendering

Camera pose

Brightness

Camera resolution

Display technologies

Marker type Marker size

Outdoor / Indoor usage

Display technologies (Krevelen & Poelman, 2010) (Roussel, 2014)

Several display methods in AR Retinal display

Head mounted display VST Video See-Through

OST Optical See-Through

Hand-held display

Projection-based display

18

Maintenance configurations

• We define 3 mains Augmented Reality maintenance configurations

• Live maintenance Operator and expert are interacting directly with video, speaking (1) and Augmented Reality (2).

20

1

2

Operator Expert

Maintenance configurations

• We define 3 mains Augmented Reality maintenance configurations

• Intended or known maintenance Expert prepares the maintenance tasks in AR (1) Operator knows the task he must perform, loads the AR from the database (2) then perform the maintenance steps.

21

1

2

Operator

Expert

Maintenance configurations

• We define 3 mains Augmented Reality maintenance configurations

• Delayed maintenance Operator tells his issue with video attached (1). Expert is answering (2) and sends him back the answer in Augmented Reality (3). Operator gets the answer (4).

22

1

2

4

Operator Expert 3

Exp1: Exploring the benefits of augmented reality documentation for maintenance and repair (Henderson & Feiner, 2011)

Comparison of 3 maintenance display methods in intended maintenance.

(armar video))0’50” / 1’22” 23

Display Devices

AR Video See Through Head Up Display HUD

LCD screen

Exp1: Exploring the benefits of augmented reality documentation for maintenance and repair (Henderson & Feiner, 2011)

Quantitative Results

AR increases task localization (50% better) BUT not task completion (20% longer).

Qualitative Results (based on a questionnaire)

LCD remains the preferred mean

AR is appreciated but it remains some drawbacks. • Restrictive vision.

• Head Worn display.

• Vision occlusion.

Drawbacks can be improved with Optical See Through.

HUD cumulate both drawbacks • No augmented Reality

• Head-worn display

27

AR T.A.C: Augmented Reality with Monocular Orthoscopic Video See Through

VISIO: AR system. Operator has a camera tie and the augmented scene is on a PC.

TEL: a hands-free phone.

Exp 2: Collaborative Tele-Assistance System (T.A.C) (Bottecchia, 2010)

Comparison of 3 maintenance display methods in live maintenance.

28

Communication mode Display Methods

Exp 2: Collaborative Tele-Assistance System (T.A.C) (Bottecchia, 2010)

• With the T.A.C interface, expert can augment the Operator view.

• Picking, Outlining, Adding concepts.

• Therefore, they can communicate with terms like “the, this, here”. = copresence feeling

29

(e): " Remove the cap for making the shaft turn afterward. " (f): « Put the the tool in hole 1, it may be this one » (g): « Look here, the indicator shows 0.2, it’s ok."

Expert IHM T.A.C Interface

Exp 2: Collaborative Tele-Assistance System (T.A.C) (Bottecchia, 2010)

Quantitative Result

• AR T.A.C system allows performing tasks 10% faster than 2 others systems.

Qualitative results (based on a questionnaire)

With AR T.A.C

• Conversation is better.

• Expert can instantaneously correct errors.

• Operator is less stressed during task execution thanks to the expert validation.

VISIO is uncomfortable due to

• constant head movement

• cognitive effort for projecting the AR from the screen to the reality.

32

Works done

• State of the art of augmented reality.

• State of the art of Augmented Reality frameworks and devices.

• 2 demonstrators of augmented reality (with our without marker).

33

AR Demonstrator with marker

created with ARToolKit

AR Demonstrator markerless

created with Vuforia and Unity3D

Future works

• Improve AR registration with 3D vision.

• Improve AR registration with 3D model of the tracked object.

• Qualitative comparison of display technologies in Augmented Reality for e-maintenance (Optical See Through monocular, binocular)

34

Google glass monocular Space glasses binocular

META

35

Questions ?

Questions ?

References

Azuma, R. T. (1997). A Survey of Augmented Reality, 4(August), 355–385.

Bottecchia, S. (2010). T . A . C : Augmented Reality System for Collaborative Tele- Assistance in the Field of Maintenance through Internet ., 2010.

Henderson, S., & Feiner, S. (2011). Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Transactions on Visualization and Computer Graphics, 17(10), 1355–68. doi:10.1109/TVCG.2010.245

Henderson, S., & Feiner, S. (2011). Exploring the benefits of augmented reality documentation for maintenance and repair. URL: http://graphics.cs.columbia.edu/projects/armar/

Krevelen, D. W. F. Van, & Poelman, R. (2010). A Survey of Augmented Reality Technologies , Applications and Limitations, 9(2).

Roussel, D. (2014). Réalité augmentée Principes , technologies et applications, 33(0).

36