Eliciting Expert Judgement 09 - Welcome to the University ... · Cathay Pacific Operating Airbus...

1

Eliciting Expert JudgementEliciting Expert JudgementEliciting Expert JudgementEliciting Expert Judgement

Ian James

Chief Engineer Electronics & Software Technology

Aero Engine Controls

February 2009A Rolls-Royce and Goodrich Corporation joint venture

Eliciting Expert JudgementEliciting Expert Judgement

Contents of Lecture

Introduction - Speaker & Company Background

Elicitation Overview

The Expert Elicitation Process

Using Expert Information

Elicitation Updates

Summary

2

Engine Control Systems

Take an Aircraft view of an Engine Fuel Control System ....

Fuel Tanksand

Fuel Pumps

Cockpit Controls

Engine and Fuel Control

System

Cathay Pacific Operating Airbus A330 powered by Rolls-Royce Trent 700 Engines


Engine Control System

Left View

Rolls-Royce Trent 700 Engine

3


Electronic Engine Control - E.E.C.

Expert Elicitation Overview

Data Characteristics

Expert knowledge is defined as a general term encompassing what qualified individuals know with respect to their:

Technical Practices - What they doTraining - What they have been taughtExperience - What they have experienced

Expert knowledge can be categorized into

ExpertiseExpert Judgement

4

Expert Judgement


Expertise refers to the tacit thinking processes used in everyday decision making and is implicitly used in problem solving and decision making

Expert judgment, expert estimates, or expert opinion are terms that refer to the contents of the problem—estimates that populate the structure of the problem

Estimates, outcomes, predictions, uncertainties, utilities and their corresponding assumptions, heuristics, and conditions are all examples of expert judgment.

Expert Judgement


Is affected by the process used to gather it

Has uncertainty (which must be characterized and subsequently analyzed)

Is conditioned on various factors (such as question phrasing, information considered, assumptions and problem solving)problem solving)

Can be combined with other data and information.

5

History of Elicitation of Expert Judgement


The formal elicitation and analysis of expert knowledge dates back to the early 1980s

It is rooted in early human cognition studies and the emergence of Probabilistic Risk Assessment (PRA).

PRA relied heavily on expert judgment because, for most early problems, ‘hard’ (test, experimental, observational) data were sparse or nonexistentdata were sparse or nonexistent

Just as the ‘P’ in PRA implies, these early efforts were based in probability theory.

As a construct of risk, probability theory offered a logical mechanism for handling uncertainties in these analyses and studies.

Eliciting Expert Judgement Data


Extracting knowledge in as raw and unbiased a form as possible, according to the way experts think and problem solve has been shown to be most successful

An analyst who excludes expert opinion as a source of data denies the subtle uses of expert tacit knowledge in everyday decision making

Any analyst who aims to include expert opinion as a sourceAny analyst who aims to include expert opinion as a source of data faces the difficult task of extracting tacit knowledge from one or more members of the subject area under study

The approach is interdisciplinary, there is a shared emphasis on frequent, direct interactions of the expert and interviewer/ analyst, often in face-to-face sessions.

6

Example - Design Changes

Data AnalysisAirline Operator RecordsWarranty ClaimsWarranty ClaimsRepair RecordsProduct Support Database

Design ImprovementsCommon Manufacturing Process Similar Component Types

Reduced Interconnections Reduced Number of AssembliesIncreased Circuit Integration

Balancing the Payoff Versus The Risk

Managing Design Decisions

Where there is Change you will always find Concern

How to weigh up the Benefit/Concern for each changeConsequence to the end user Cost saving .. Cost of failureWhen will we know if its ok ?

How to affect the decision making processg pAbsence of hard data necessitates subjective decision makingAre the views: Personal, Political, Blinkered, Ill-advised ? We need expert knowledge that can be relied upon and ….De-risk test & analysis with defined decision points

A PROCESS to systematically elicit expert concerns

7



Initial Launch MeetingDescribe elicitation objectives to interested partiesDefine Base product and changes from Base to new designSelect Experts / Interviewees

Interview PlanningReview relevant Service DataDevelop proformas and facilitation aids

Carry Out InterviewsCarry Out InterviewsIndividual interviews up to 2 hours maximum Foster an informal, open, no blame environment

Collate ConcernsClassify Concerns into Root Cause CategoriesDistribute Processed concerns for comment

Initial Launch Meeting

f


Define Base Product

Use the group to brainstorm any similarities between the new design and existing products

Agree the most appropriate product (s) for comparison

Break down the Base product (s) into major functional parts

Define Changes from Base to New Design

Are there any new functions ?

Are any functions no longer required ?

Which functions remain similar between designs ?

Are there any major technology changes ?

Any there any manufacturing process changes ?

8

Example – Major Changes Brainstorm


Element Changes from Product X Changes from Product YGeneral Higher Operating temperature

Mechanical Design 3 large modules versus 12 smaller modulesAll Surface Mounted Component versus mixed type on Product X and YThermal Design via pillars versus cold wallAs Product X Shape of bonding StrapConnector Types not used before

Installation angle achieved via mounting bracketChange of Casting Material

Aircraft Interface Similar to Product X but radically different communications Nothing like Product Y

Lightning Strike / HIRF New design based on product X.. Uses SILS used in place of hybrids

F l Fl Si il t P d t X b t l it hi t h b i l t dFuel Flow Similar to Product X but a novel switching system has been implemented

Torque Motor Drives As Product X redesign

Pressure Measurement Mechanical installation change, but electrical similar Similar installation

Multi Chip Module None used. Use of Simple Hybrid Technology

Housing Similar installation to Product X but Mounted directly onto gear box

Linear Circuits Common Circuit blocks comprising discrete components versus of the shelf functionality

Initial Launch Meeting


Candidates For Elicitation

Engineering ConsultantsProject DesignersSystem Design EngineersQuality EngineersManufacturing StaffPurchasing StaffgProduct Support RepresentativesTest EngineersCustomer RepresentativeSupplier Representative

9

Interview Planning


Familiarisation with Base Product Service DataCurrent Issues, Major Trends, Top 10 Reasons for failure, etc

Confirmed Fault [140]

Surface MountWiring / InterconnectPowerLighting StrikePressurePlated Thru Hole

[27]A/C Power

[9]Dedicated Generator

[70]Filters

[3]Connector Pins Other [31]

Hybrid Failure [4]

Solder Joints [5]

Capacitors [14]

Hybrids [12]

PCB [1]

Capacitors [26]

Solder Joint [2]

Ignition Relay [3]

Interview Planning


How to promote useful discussion during interviews

EnvironmentComfortable Surroundings / Room LayoutAvailability of Flipchart / WhiteboardAccess to Computer Network

ToolsSummary of Base Product InformationProformas to expedite capture expert’s knowledgeGauge to aid selection of Likelihood

10

Example – Elicitation Concern Sheet


Product Design StageFunctional Block DescriptionChange Concern Mitigating action Likelihood of

Occurrence

Example – Concern Likelihood Scale


Likelihood Scale

Concern: ………………………….

Engineer: …………………………

Very Unlikely Very LikelyFairly

Never Certain

Lik lUnlikely Fairly Unlikelyy y y yyLikely LikelyUnlikely y y

What is the chance the concern may lead to a fault in use if no corrective action is taken?

0 0.5 1

11

Carry Out Interviews


Focus on difference between new design and base productDevelop the list brainstormed during Launch MeetingUnderstand motivation for each design change

For each difference discuss the impact upon reliabilityPerformance improvement or concern

Can Test or Analysis evaluate the concern ?What is the ultimate mitigation action ?What is the ultimate mitigation action ? Is the work planned ?

What is the probability of a service incident?

Example – Elicitation Summary: Individual Expert


Concern Sub-system Change Concern Mitigating action Likelihood1 General Vibration environment Not on Pump so potential to

be better or worse depending on mounting bracket arrangement

Ensure adequate bracket design

0.1

less corrosion IMPROVEMENT

2 tapping into harder more brittle material resulting in cracking and subsequent leaking or jamming

RET and Qual Test 0.1

3 Data Sheet does not Specify max.

0.5

Case Material now through hardened xyz was 1234

Higher Operating Temperature Supplier AuditsDiodesmax.

4 New Component Supplier 0.55 No experience with new

package typeDesign Evaluation Test 0.5

6 EIDs Reduced life so higher fail rate of EIDs


7 SCM New Package Type Little experience with soldering parts; relies on new manufacturing process

Prototype runs, SPC, RET and design evaluation

0.5

8 Seal Location of Fire Seal Increased chance of Seal compression set causing leakage


12

Collate Concerns


Combine Identical ConcernsIt is likely that different experts will raise identical concernsEnsure there is no double counting

Categorise each Concern into a generic fault class 5-10 Classes will ensure a reasonable root cause levelSee next page

Review Likelihood of OccurrenceReview Likelihood of OccurrenceReview range of likelihood for concerns from multiple experts Review the use Max, Min or Average, or combinations

Distribute Collated Comments for ReviewSeek agreement / approval from individual experts Configuration Control: Formally issue concerns & comments

Root Cause Classification


Manufacture and AssemblyBuild FaultSoldering Process

Component PartsWear-Out Supplier Quality

EnvironmentHandling Damage

Design MarginTolerancesComponent Selection

Extreme Operation

13

Example – Elicitation Summary: Collated


Concern Eng. Ref Sub-system Change Concern Mitigating action Likelihood Fault category1 7.11, 3.3 General Vibration environment Not on Pump so potentail to

be better or worse depending on mounting bracket arrangement

Ensure adequate bracket design

0.1, 0.2 Environment

3.4, 6.2 less corrosion IMPROVEMENT

2 3.5, 6.3, 8.3 tapping into harder more brittle material resulting in cracking and subsequent leaking or jamming

RET and Qual Test 0.1, 0.25, 0.2 Build Fault

3 3.6, 1.7, 13.7 Data Sheet does not Specify max.

0.5, 0.5, 0.5 Design Margin

4 3 7 7 1 New Component Supplier 0 5 0 75 Supplier Quality

Case Material now through hardened xyz was 1234

Higher Operating Temperature Supplier AuditsDiodes

4 3.7, 7.1 New Component Supplier 0.5, 0.75 Supplier Quality5 13.6 No experience with new

package typeDesign Evaluation Test 0.5 Wear-Out

6 3.8, 7.2 EIDs Reduced life so higher fail rate of EIDs

RET and Qual Test 0.5, 0.3 Environment

7 3.9, 13.4, 10.7 SCM New Package Type Little experience with soldering parts; relies on new manufacturing process

Prototype runs, SPC, RET and design evaluation

0.5, 0.5, 0.25 Soldering Process

8 13.8, 7.3 Seal Location of Fire Seal Increased chance of Seal compression set causing leakage

RET and Qual Test 0.5, 0.75 Design Margin

Reliability Assessment


Estimates reliability of a variant design at specified times in product lifecycle using appropriate mathematical models

Aid engineering understanding of performance and to help inform downstream processes that will support enhancement

Alternative scenarios can be investigated through ‘what if’ analysis so that the impact on reliability can be estimated

Provides a tracking system to analyse / record how reliability evolves.

14

New design


Event history datafor relevant

failures

Elicit engineering judgement about failure classes of

concern

Similar design

Changes

Model

Flag failure classes of concern

Estimate reliability

of new design


Event History Data

1.00

Nonparametric Survival Plot for Wear OutKaplan-Meier Method

Censoring Column in Wear Out Censor

MTTFMedianIQR

19318 *0.000000

1.00

0.99

Nonparametric Survival Plot for Supplier QualityKaplan-Meier Method

Censoring Column in Supplier Censor

MTTFMedianIQR

20068 *0.000000

Group Service failure data into Root Cause Categories Generate a Survival Function, R(t), for each Fault CategoryThe distribution then represents all concerns in this category

20000100000

0.95

0.90

Time to Failure

Pro

babi

lity

20000100000

0.98

0.97

0.96

0.95

0.94

0.93

0.92

0.91

0.90

Time to Failure

Pro

babi

lity

15

Reliability Estimate for New System Design


BuildComponentDesign

Expert Judgement

Failure Class

NBNCNDJudgement

Event Data

BCD

RB(t)RC(t)RD(t)

Supplier QualitySoldering ProcessBuild FaultEnvironment

Concern ElicitationInterviews with engineering experts generates a list of potential concerns with the new design. For each concern a likelihood of failure


Service Data

Wear OutDesign Marginality

each concern a likelihood of failure is recorded. Concerns are then categorised into root-cause classes.

P P P P P P

Statistical Model

Failure Distribution

ReviewDevelop Reliability

distribution for each major category of

concern.

RSystem(t)

P1 P2 P3 P4 P5 P6

R2(t)

R3(t)

R4(t)

R5(t)

R6(t)

R1(t)

Wear Out

Environment

Build Fault

Soldering Process

Supplier Quality

Design Marginality

16

0.9

1


0.4

0.5

0.6

0.7

0.8

Soldering ProcessBuild Fault

0

0.1

0.2

0.3

0 5000 10000 15000 20000 25000 30000

Supplier QualityWear OutEnvironment Factored InherentDesign MarginalitySystem Reliability

Elicitation Updating

Concern Elicitation Test & Analysis

Statistical ModelFailure Distribution

Review

RSystem(t)

17

Updating Expert Judgment

f ff


Following a period of action which may affect Concerns

Arrange interviews with a subset of experts

Review Concerns recorded during initial interviews

Has test or analysis activity successfully mitigated any issues ?To what extent has the likelihood of occurrence changed ?Are any new test or analysis activities planned ?

Capture changes not recorded during initial interviews

Develop action plan to minimise likelihood of occurrence


8

10

12

14

16Soldering Process

Build FaultsDesign Marginality

Updating Expert Judgment

0

2

4

6

Supplier Quality

Wear Out

Environment

Initial Assessment

Assessment #2

Review changes in OpinionPrioritise ActionsFocus Attention

0

2

4

6

8

10

12

14

16

18

SolderingProcess

Build Faults Supplier Quality Wear Out Environment DesignMarginality

R oo t C ause

Initial AssessmentAssessment #2

18

Reliability Assessment

Product X Reliability Assessment

0.4

0.5

0.6

0.7

0.8

0.9

1

Pro

babi

lity

0

0.1

0.2

0.3

0.4

0 5000 10000 15000 20000 25000 30000

Operating Hours

P

Worst Case Assessment

Target [100k]

Product X Reliability Assessment

Reliability Assessment - Update

0 4

0.5

0.6

0.7

0.8

0.9

1

roba

bilit

y

0

0.1

0.2

0.3

0.4

0 5000 10000 15000 20000 25000 30000

Operating Hours

Pr

Worst Case Assessment

Target [100k]

19

Summary

Summary

Expert Judgment is a valuable piece of informationThe extraction of Expert knowledge may be very difficultBut …. its exclusion can undermine the decision making process

Thorough Planning is Required for Expert ElicitationGeneral Logistics:- Room, Time, Expert Selection, Facilitation AidsScope of Elicitation:- Selection of Prior Products, Design Changes

Processing the Elicitation OutputCorrelation of Concerns and Root Cause Classification

Practical UsageReliability ModellingTest DevelopmentContinuous Updating

Eliciting Expert Judgement 09 - Welcome to the University ... · Cathay Pacific Operating Airbus...

Documents

Transcript of Eliciting Expert Judgement 09 - Welcome to the University ... · Cathay Pacific Operating Airbus...