Process Operability Class MaterialsRefining Trouble Shooting Skills

Copyright © Thomas Marlin 2013The copyright holder provides a royalty-free license for use of this material at non-profit

educational institutions

FC1

LC1

FC

1

TC

1

TC

2

T

10

T

12

T

11

T

13

fuel

LC

1

L2

LAHLAL

F

4

Basic flowsheet Design with Operability

Refining our Trouble Shooting Skills

Now that we have learned the basics,let’s review some issues that have challenged students

• What is a Fact?

• Multiple Root Causes

• Looking for the Root Cause: “Changes” vs. “Usual”

• Iteratively finding the Root Cause through intermediate causes (Fishbone Diagrams)

• Priorities for the Diagnostic Actions

• Diagnostic Actions – Disturbing the process

PROCESS TROUBLESHOOTINGWhat is a fact?

What is the best statement – the most representative of the situation?

• The temperature is 55 °C.

• The sensor measures 55 °C.

• The meter shows 55 °C.

• We do not have any idea what is going on – I’m out of here!

We need to make clear statements of the evidence so that we do not confuse others (or ourselves).

PROCESS TROUBLESHOOTINGWhat is a fact?

What is the best statement – the most representative of the situation?

• Sensor T7 uses a thermocouple to measure temperature.

• SensorT7 measures the temperature after the flash valve.

• Sensor T7 is shown to be located after the flash valve on the drawing.

• I do not trust T7-I’m going to feel the pipe!

v1

HotOil

v2

v3

L1

v7

v5 v6

HotOil

F1

T1

T3

T2

F2

T4

T5 F

3T6

T8

F4

L2

v8

T7

P1

F5

F6

T9

PROCESS TROUBLESHOOTINGMultiple root causes

Situations with several independent root causes are much more challenging to trouble shoot.

Many of the recent major industrial catastrophes involved multiple faults. The plant operating personnel were not able to consider all of the possibilities.

Often, one of the faults (e.g., a faulty sensor) leads us to incorrectly eliminate the true root cause of the problem.

We are sure that the cooling valve is fully open.

Let’s look elsewhere for the problem.

PROCESS TROUBLESHOOTINGMultiple root causes

Situations with several independent root causes are much more challenging to trouble shoot.

We must try to identify critical situations during design and provide sufficient redundant equipment (e.g., sensors) to ensure that people can trouble shoot them.

Training is also important.

We have inconsistent information about the opening

of the cooling valve..

We better look into this immediately!

PROCESS TROUBLESHOOTING“Change”

vs “Usual”

Two major categories of situations exist for the trouble shooter

• “Change” when significant changes have occurred that likely (but not certainly) have caused the problem. Examples are new plant startup, restart after maintenance, & changes to operating conditions.

Since the equipment has not been tested completely, major faults might exist, such as hand valves being improperly open/closed, pipes being blocked, motors failing, sensors incorrectly calibrated, etc.

PROCESS TROUBLESHOOTING“Change”

vs “Usual”

Two major categories of situations exist for the trouble shooter

• “Usual” when no significant, obvious change has occurred. The root cause must be determined from evidence and applying our knowledge of process principles.

Since the equipment has operated well up to the problem, the change has occurred because of a recent change to equipment or inadvertent action by a person.

The likelihood of “change” faults are very low.

Oh, for heavens sake, when does this stop?

What is the cause?

What is the cause?

The temperature is high

What is the cause?

The cooling water flow is low

Professor Student

PROCESS TROUBLESHOOTINGIteratively find root cause

Symptom(s) Many potential root causes

Real world

Cause to effect

Trouble shooter must work “in reverse”, from effect to cause. It is difficult to jump all the way to

each root cause in one step.

PROCESS TROUBLESHOOTINGIteratively find root cause

Symptom(s) Many potential root causes

Most trouble shooters benefit from breaking problem into smaller segments, each with its cause-

effect relationship.

Heat transferRxn Engineering

Fluid mechanics

The pump head is too small to achieve the needed flow.

What is the cause?

What is the cause?

The temperature is high

What is the cause?

The cooling water flow is low

PROCESS TROUBLESHOOTINGIteratively find root cause

V-30

CW

Dep

rop

an

izer,

C-

8

By-product to fuel, mostly C1 and C2

Product mostly C3

Tower bottoms stream

mostly C4+

reflux

FC-4

FC-4

LC-3

PC-10

This is a simplified drawing of one part of Figure PID-2A from Wood’s Process Design and Engineering Practice

Let’s learn through a workshop: The symptom is “High level in V-30”. The process is a distillation tower. Please find all possible root causes and document in a fishbone diagram.

LAH

LAL

PAH

High level in V-30

???

???

???

Building a Fishbone Diagram: What could cause the symptom of “high liquid level in V-30”?

Start the fishbone diagram with the most fundamental causes of the symptom; do not try to jump to the root cause (we’ll get there).

Some ???

• Sensor error

• Too much liquid into tank

• Too little liquid leaving tank

PROCESS TROUBLESHOOTINGIteratively find root cause

High level in V-30

Too much liquid into tank

Sensor Error

For each of the three intermediate causes in the diagram, find one cause, which could be, but does not have to be, a root cause.

Too little liquid leaving tank

Connection between sensor and vessel is plugged

Tower feed rate has been increased

Pump motor has stopped

PROCESS TROUBLESHOOTINGIteratively find root cause

High level in V-30

Too much liquid into tank

Sensor Error

What could cause the symptom of “high liquid level in V-30”?

Complete all paths to root causes for the three initial causes in the diagram.

Too little liquid leaving tank

Connect between sensor and vessel is plugged

Tower feed rate has been increased

Pump motor has stopped

PROCESS TROUBLESHOOTINGIteratively find root cause

High level in V-30

Controller sensor error

Delta pressure sensor calibrated incorrectly (reading higher level than actually exists)

Connection point (tap) blocked/corroded (level measurement is constant causing controller to make an incorrect action)

Too much liquid into the tank

Steam valve fails open (unsafe)

Too little liquid leaving the tank

Poor feedback control

Magnitude of feedback controller gain (Kc) is too small

Valve malfunction

Reflux or product flow valve failed closed (safe)

Pump malfunction

Vortex (unlikely with high level)

Cavitation

Power loss (motor failure or coupling break)

Increased feed rate (level controller will lower level in time)

Increased % propane in feed (level controller will lower level in time)

Distillate liquid product valve saturation

Reflux or product flow valve stuck , not responding to control signal

Extra vapor overhead

Extra condensation

Cooling water temperature becomes much colder

Building a Fishbone Diagram

Class Exercise: The fishbone diagram on the previous page has many root causes. Add at least one additional root cause to each of the major branches (sensor error, too much liquid in, and too little liquid out).

PROCESS TROUBLESHOOTINGIteratively find root cause

Building a Fishbone Diagram

Class Exercise: The fishbone diagram on the previous page has many root causes. Add at least one additional root cause to each of the major branches (sensor error, too much liquid in, and too little liquid out).

One answer for “sensor failure”:

The alarm level sensor is improperly calibrated. An alarm occurs (audible signal and blinking light) when the actual level is not above the level high alarm value (which might be ~85%).

No process excursion has occurred. Solution is to calibrate the alarm sensor, which is different from the level sensor used for control.

Building a Fishbone Diagram

Class Exercise: The fishbone diagram on the previous page has many root causes. Add at least one additional root cause to each of the major branches (sensor error, too much liquid in, and too little liquid out).

One answer for “too much liquid in”:

The cooling water flow rate has increased, resulting in more liquid being condensed; perhaps, an additional cooling water pump has been turned on. The level controller will ultimately increase the product flow rate.

No process excursion has occurred. The solution is to wait, while monitoring that the level control functions properly.

Building a Fishbone Diagram

Class Exercise: The fishbone diagram on the previous page has many root causes. Add at least one additional root cause to each of the major branches (sensor error, too much liquid in, and too little liquid out).

One answer for “too little liquid out”:

One or more of the block (manual) valves in line with the product control valve could be partially closed. This would reduce the maximum flow rate. This cause would yield the symptom that the product flow valve is 100% open, while the flow rate is much lower than expected.

The solution is to check the block valves and open those in line with the product line to 100% open.

Advantages of Fishbone Diagrams

1. Picture of thought process for learning and avoiding confusion.

2. Documenting all causes for fast trouble shooting in plant.

3. Using information to quickly “trim the tree”, without checking every root cause.When we disprove this cause,

we eliminate all subsequent root causes connected to the branch!

PROCESS TROUBLESHOOTINGIteratively find root cause

Advantages of Fishbone Diagrams

Symptom

Factoid: The blue (solid) root causes yield safe conditions; the red (stripped) root causes yield hazardous conditions.

Class exercise: Where will you concentrate your efforts when collecting additional information?

Collect information about this branch; try to confirm or disprove.

1. Engage

2. Define

3. Explore

4. Plan

5. Implement

6. Evaluate

Where do Fishbone Diagrams fit in the Trouble Shooting Method?

Concept will help you pose good questions

You will use FB diagrams to generate a complete set of hypotheses and evaluate how the information (initial and collected as part of TS) affects the truth of each hypothesis.

The pump head is too small. A root cause!

What is the cause?

What is the cause?

The temperature is high

What is the cause?

The cooling water flow is low

Professor Student

This scenario is GOOD, if we manage the process and achieve the root causes as candidates.

PROCESS TROUBLESHOOTINGIteratively find root cause

PROCESS TROUBLESHOOTINGDiagnostic Priorities

We need to tailor our diagnostic actions to the process priorities.

Safety concerns?

Equipment protection?

Product quality degraded?

Production rate reduction?

Start

Follow defined procedures

Follow defined procedures

Large $ impact? Follow defined procedures

Large $ impact? Follow defined procedures

Y

Y

Y Y

Y Y

N

N

1

N

N

N

N

These could be extreme, including shutting down the process!

PROCESS TROUBLESHOOTINGDiagnostic Priorities

1

Consider many factors in setting the sequence of actions

• Time required for an action

• Cost for delaying the solution of the problem

• Cost for the action

• Probability of each hypothesis being correct

• Ability of a single action to eliminate many root causes

We consider these factors when determining the sequence that has the best expected outcome – solving the problem with the lowest cost.

We recognize that our decisions are based on imperfect knowledge, but we must use the best information and experience available when trouble shooting.

PROCESS TROUBLESHOOTINGDiagnostic Actions

In some cases, a small change to the process will help to test hypotheses.

v1

HotOil

v2

v3

L1

v7

v5 v6

HotOil

F1

T1

T3

T2

F2

T4

T5 F

3T6

T8

F4

L2

v8

T7

P1

F5

F6

T9

For example, many variables are oscillating; what is the cause?

PROCESS TROUBLESHOOTINGDiagnostic Actions

In some cases, a small change to the process will help to test hypotheses.

For example, many variables are oscillating; what is the cause?

Place one controller in manual

Several variables are oscillating

• Cycling in all variables stops

• Cycling is unchanged

• Cycling changes, but continues

PROCESS TROUBLESHOOTINGDiagnostic Actions

In some cases, a small change to the process will help to test hypotheses.

We try to avoid shutting down a process and opening equipment for inspection. This is the last action.

Non-invasive testing can be used some times.

We shutdown for safety and to protect equipment, even though it is expensive!

Refining our Trouble Shooting Skills

These are issues that the instructor is aware of

• What is a Fact?

• Multiple Root Causes

• Looking for the Root Cause: “Changes” vs. “Usual”

• Iteratively finding the Root Cause through intermediate causes (Fishbone Diagrams)

• Priorities for the Diagnostic Actions

• Diagnostic Actions – Disturbing the process

What additional issues are you finding challenging?

How can we confirm that everyone in the class has mastered this important skill?