OIL MOVEMENT TDC3000 TO EXPERION MIGRATION WITH FULL ... · AO FTA in First HPM Cabinet Not...

Americas HUG

OIL MOVEMENT TDC3000 TO EXPERION MIGRATION

WITH FULL PROFIT BLEND & MOVEMENT (PBM) SUITEBrittany Cutrona

June 12, 2019

© 2019 by Honeywell International Inc. All rights reserved.

1

Brittany Cutrona

• Chemical Engineering BS from Lehigh University ’ 6

• 2016 – Phillips 66 Advanced Process Controls New Hire program in Houston, TX

• 2017 – Process Controls Engineer at Bayway Refinery supporting DCS Upgrade of the Oil Movements Control Center

• 2018 – Phillips 66 Technical Lead of Oil Movements Upgrade

• Contact: [email protected]


2

Overview

• Background

• Hardware upgrade

• Cutover plan

• Evolution of hardware cutover plan

• Hurdles during implementation

• Bayway success story


3

Background

• Phillips 66 Bayway facility control centers will be upgraded from TDC 3000 to Honeywell Experion to be off coax by 2025

• Oil Movements Control Center (OMCC) upgraded to Experion R500.2 in November 2018

• Oil Movements & Storage (OM&S) software suite upgraded to Profit Blending & Movement (PBM) R500.2 to maintain Honeywell support for blending and movements package

– 7 programs (Profit Blend Optimizer, Profit Blend Controller, Profit Inventory Manager, Profit Movement Management, Blend Performance Monitor, Blend Instructions, LIMS Viewer)

• 6 HPM pairs, 3500 I/O, 100 controllers

• 8, HPM/AM point & logic configuration, CL programs converted to CM’s, SCM’s and CAB programs, 120 HMI displays

• 2 Gasoline blenders, unit feed controllers & pumps, cavern pressure control, tank storage, remote valves, waste water treatment

Overview


4

Background

• Oil Movements PMM & PBO

PBM Suite and Oil Movements Operations


5

Hardware Upgrade

• Blender I/O migrated to 2 redundant C300s with Series C I/O

– Required for PBM R500 compatibility

• 6 HPM pairs migrated to 6 redundant C300s

– Hybrid hardware utilizing IOLinks preserving PM I/O

– No I/O rewired, “bulk” conversion

– Serial interface cards removed, third party converted to ethernet as necessary

• 5 Dell FX2 servers contain completely virtualized system, console stations on WYSE Thin Clients

– Experion servers, PBM servers, console stations, flex stations, etc.

Dell FX2 Chassis


6

Hardware Upgrade

Hardware diagrams available in Honeywell Experion PKS Control Hardware Planning Guide Release 500 Nov 2016. Series C I/O utilized for blenders. Remaining hardware utilized C300 PM I/O Hybrid system.


7

Cutover Plan

• Preparation

• Blender cutover – 1 week

– Cutover I/O of 1 blender (17 loops, most pulse inputs) to C300 Series C I/O and test blending with PBM suite

– Upon successful cutover of 1st blender, cutover 2nd blender (27 loops, includes shared equipment)

• HPM cutovers – 2 weeks

– 6 HPM pairs; 48 hours each, 24 hour cutover + 12 hours testing

– Operations business team decided to start with “worst” HPM first

▪ 3 AO modules containing 33 controllers – multiple unit feed, shipping controllers, blend component recycle controllers, waste water plant, tank mixers, area hydrocarbon alarming

– Remaining order of HPMs decided based on operational impact

▪ MOVs and pumps prioritized to be able to use PMM to track movements

▪ Peer-to-peer HPMs cutover consecutively

Overview


8

Cutover Plan

Cutover Responsibilities

TPS Console Operator Control the process on the old system

Experion Console Operator Control the process as points are cutover

Cutover OperatorDirecting the Cutover, coordinating between Cutover

Technical Team and Console Operators

Phillips 66 Field Teams (Instrument

Technician/Electrician/Operator)Securing control valves for cutover, additional field rounds

Phillips 66 Control EngineersInactivating points on the TPS system as cutover

progresses. Activating Experion points.

Honeywell Cutover EngineersVerifying points transfer properly and assisting in the cutover

and tuning

Hot Cutover CoordinatorEnsure all personnel are following the cutover plan for the

day

DCS Technicians/Honeywell SSSWork with operations to transfer I/O link cables from TPS to

PKS

24 / 7Staffing


9

Cutover Plan

• Experion system hardware and configuration complete and tested

– 14 weeks of factory and site testing

• Tested pulse input blender flow meters on C300 – reported issues of compatibility at other sites

– 3 wires – identifying power vs signal wires and resistance ahead of time sped up actual cutover

• Tank levels and temperatures brought into Experion and TDC – third party remote terminal unit has 2 outputs

– Other 3rd party communication brought into both systems – NIR analyzer

• Prepared detailed plan for operations listing I/O impacted each day

– Console operators, engineering, and planning & economics all were provided with DCS graphic markups showing what information would be impacted

– Operations developed field plans based on this information

– Instrument techs confirmed locations ahead of time for anything requiring loop checks

• Run through Hot Cutover plan procedure multiple times on test equipment

Preparation


10

Evolution of Hardware Cutover Plan

Total loss of view and control of ALL I/O on HPM

1. Operators bypass ALL control valves on HPM

– What if no bypass?

2. Continuous operator rounds in field to monitor control and indication

3. Shutdown one HPM pair

4. Remove both HPM cards and IOLinks, replace with Series C IOLinks, connect fiber extender to C300

5. Load C300 configuration, activate points/programs, etc.

6. Operators un-bypass all control valves on HPM

Risk of unit upset: High

Rev 0 – HPM Cold Cutover


11


Maintain view and manual control of I/O & controllers on redundant IOP using AO Standby Manual (AOSM)

Lose view of I/O on non-redundant IOP

1. Operators in field by essential controllers and monitoring non-redundant indication

2. Secondary HPM and associated/non-redundant card files converted to Experion

3. Primary HPM and associated card files converted to Experion

– Controllers on one AO IOP placed in manual, connected to AOSM

4. Confirm process values, verify control with Ops

Risk of unit upset: Lower

Rev 1 – HPM Hot Cutover (Shorten Loop Cutover to Experion from Legacy TPS - HUG Presentation 2015)


12


• Please contact Honeywell for detailed HPM to C300 Hot Cutover procedure

– The following is a condensed version of the procedure Bayway Refinery used provided by Joe Yuanzhu Zhang (Honeywell)

Pre-Migration Requirements

– IOLink fiber cables run, IOLink Extender kits ready to install

– Confirm compatibility of PM IOP with C300s. May need to upgrade firmware (can do online if redundant)

– Create graphic or group to monitor AO points during cutover (TDC and Experion)

– Confirm serial interfaces already migrated/removed from HPM cabinet

– Set DO and AO IOP Modules to FailOpt “Hold”

– Evaluate DO interlocks since DO will power off, evaluate peer-to-peer, evaluate programs writing to AO etc.

– Confirm no IOP/HPM or other system errors on TDC. Perform ERDB consistency check.

– Label hardware: IOPs, FTA cables, etc

HPM Hot Cutover Condensed Procedure

This presentation is not intended to used as a procedure. Please contact Honeywell for full procedure.


13

Evolution of Hardware Cutover PlanHPM Hot Cutover Condensed Procedure

1. Make HPM and IOP on File #2 Primary

– Ensure healthy control of AO

2. Unplug all AO IOP on File #1 (prevents write out by EPKS)

3. Remove HPM & IOLink board from File #1 and unplug HPM redundancy cable from Primary file to Backup file

4. Reposition UCN address jumpers on File #1 so there is no addressing

5. Power off File #1 & #3 by removing power cables (A&B); Unplug IOL plugs from File #2 backplane

6. Install IOLink Extender Card, Fiber Adapter on slot 2 of File #1

7. Plug power cables (A&B) back to File #1 & #3

8. In Control Builder, re-load IOPs on File #1 & #3 as Inactive

9. Confirm all loaded DO IOP are inactive

10. Activate with contents AI and DI IOP

11. Check status of loaded IOP and IOLink Channel B

File 3

File 2

File 1

Primary

HPM

Secondary

HPM

Non-

redundant

IOP

Redundant

IOP (mostly)

At this point, view of I/O on File #1 & #3 in EPKS. View and

control of I/O on File #2 in TDC.

This presentation is not intended to be used as a procedure. Please contact Honeywell for full procedure.


14


12. From Control Builder & TDC, place all controllers associated with AO FTA in Manual

13. Perform the following (steps 13-27) ONE AO IOP at a time. Repeat for all AO IOP.

14. Ensure switches for all channels on the AOSMs are NOT set to MAN

15. Plug one AOSM into J3 on AO FTA (slots 1-8) and another into J4 (slots 9-16)

16. Verify AO points in TDC show “STDBYMAN”

17. Press “BAL” button for ch and compare to HPM OP value

– OPTDIR Important: mA will be inverted if reverse

18. Release button and adjust “valve” knob to set AOSM output to match IOP output (again considering OPTDIR)

19. Switch ch switch to “MAN”. Confirm mA on channel

20. Repeat for steps 17-19 for all 8 or 16 channels

21. Plug AO on File #1 back in

22. In Control Builder, reload the AO IOP and activate with contents

23. In Control Builder, verify output blocks for AO loops shows “Standby Man” and set outputs to match HPM output

24. Unplug AO IOP from File #2

25. Press “BAL” button for ch and adjust C3 output to match AOSM value. Repeat for all channels

26. Switch ch switch off “MAN” position. Confirm no mA or process change. Repeat for all channels

27. Disconnect AOSM cables from AO FTA J3 & J4, verify controllers are healthy on C300 side

HPM Hot Cutover Condensed Procedure

Manual switch

J3

J4

“Valve” knob

Balance button

This presentation is not intended to be used as a procedure. Please contact Honeywell for full procedure.


15

Evolution of Hardware Cutover PlanHPM Hot Cutover Condensed Procedure

28. Remove HPM and IOLink boards from File #2

29. Reposition UCN address jumpers on File #2 so there is no addressing

30. Power off File #2 by removing power cables (A&B)

31. Install IOLink Extender Card, Fiber Adapter on slot 2 of File #2

32. Plug both IOL cables back to File #2

33. Plug power cables (A&B) to File #2.

34. In Control Builder, re-load non-redundant IOPs on File #2

35. Confirm all DO IOP are inactive

36. Insert back AO IOPs to File #2

37. Reset IOL errors, enable IOL swap

38. Activate with contents AI and DI IOP on File #2

39. Activate all DO IOPs only (DO NOT ACTIVATE CHANNELS)

40. Activate CMs with DO channels one at a time, verifying input state is healthy to prevent pulse out

41. Verify status of all IOPs

42. Return basic control to console supervisor

43. Begin activating complex programs

File 3

File 2

File 1

Primary

HPM

Secondary

HPM

Non-

redundant

IOP

Redundant

IOP (mostly)

5 HPM pairs to go!This presentation is not intended to be used as a procedure. Please contact Honeywell for full procedure.


16

Hurdles During ImplementationAO FTA in First HPM Cabinet Not Compatible with AOSM

Excerpts and AO-16 FTA diagram from Honeywell Process Manager I/O Installation Guide PM20-620. The installation guide describes AO FTA being compatible with AOSM as long as there

are device connectors.


17


Excerpts from Honeywell Customer Priority Notification #9816 describe AO-16 FTA assembly incompatibility with AOSM due to incorrect circuit board lay-out.


18


Phillips 66 found AO Standby Manual Adapter for use with incompatible models described by notification #9816

• Phillips 66 did not use or test this adapter

– Bypassed all controllers in the field and replaced incompatible FTA

• If using AOSM cutover method, review FTA compatibility with Honeywell project team and test compatibility of all AO FTA before cutover!


19

Hurdles During Implementation

• Valves, controllers, and calculated inputs on node 13; analog inputs on node 5

• Converted node 13 first

• HPM logic block to push node 5 AI value to AO. Daisy chained AO to AI of node 13 temporarily.

Peer to Peer HPM Control – C3/C4 Cavern Pressure Control

C4 Cavern

C3 Cavern

P

P

(Calc)

DP

Node 5

Node 13

N2 Vent/Flare

Vent/Flare

(Avg)

P


20

Hurdles During ImplementationOM&S Pathfinder Would Not Store Manual Valve Position After HPM Migration

• Expected console operators to be able to continue to mark manual valves (circle with x) open/closed during migration

– Anticipated losing remote valves (represented by squares)

• Bayway had custom configuration in OM&S package to prevent loss of valve positions upon AM failure

– Manual with Flags Valves

– AM valve tags had input of front end HPM tag

– Once HPM was removed from TDC, valves showed yellow (moving state)

• Confusion caused for the console operators

– Moved conversion of HPM up to get valves over to Experion as soon as possible

– May have investigated making front end AM tags for conversion

Portion of PBM Tank Farm Graphic showing manual and remote valves


21

Hurdles During ImplementationImpacts of keeping tags active on both systems

• Data from NIR analyzer for gasoline blending dropping out every 10-15 minutes

– Could not optimize

• NIR analyzer connected to TDC (serial interface) and EPKS (PCDI)

– Inactivated TDC arrays

• Complex data validation code in DCS

• Once validated, DCS handshakes with analyzer via read/write flag

• Spent ~5 days troubleshooting

• Read/write array was left active in TDC

Data Dropouts


22

Hurdles During ImplementationEnsure Alarm Sounds & Channel Range Updated from Default

1. Started up the first blender, and component valves went wide open but had no flow

2. Back pressure controller was in alarm, but did not hear audible alarm

3. Pressure AI was Bad PV

4. Kicked controller into manual

5. High pressure in line due to controller in manual caused components to backout

No operational impact due to this event

Causes:

• Sounds not configured on console station for High and Low priority alarms

– Tested triggering alarms during FAT and SAT

• Default Experion channel range used, were not wide enough causing AI to Bad PV

– TDC ranges were carried over on CM DACA block, but not on channel


23

Hurdles During ImplementationPoint References in Graphics

• During our SAT and just before cutover many incorrect references were identified in our HMI displays

• Did not have a dynamic FAT of our DCS graphics which would have caught many of these errors

• Moving forward for converting other control centers:

– Utilizing OpenVEP for Phillips 66 to check references in displays during front-end engineering

– Stringent HMI checklist for Honeywell before handing over


2424

Bayway Success Story

Complete ahead of schedule No production impacts Project on budget

Build flexibility into your cutover plan

• Prepared with valve fail positions

• Organized documentation with equipment on each cabinet

Success due to having a cross-functional team and knowledge sharing with Wood River refinery

Stay very involved with Honeywell in the planning process

• Honeywell has the technical expertise, but you have the site expertise

ACKNOWLEDGEMENTS

Emad Abraham

Kevin Arnold

Dennis Brown

Yiliang Chi

Charlie Dyer

Phil Evans

Scott Gallagher

Jillianne Heckman

Luis Irala

Virgil Loudermilk

John Mackay

Luke Memet

Raghuvaran Aakula

Annie Alpin

Tom Ayral

Anita Barawkar

Mike Bonczek

John Chauvin

Van Cockerham

Shriram Deshpande

Amit Gaikaiwari

Puneet Gupta

Jagdish Lagisetty

Phill O’Connors

Angelo Petronico

John Petruska

Jerry Lin

Aravind Mohan

Diana Pee

John Petruska

Karyn Simon

Joe Yuanzhu Zhang

Thank you to everyone involved in a safe and successful DCS cutover

Yuk-See Ng

Don Reulbach

Patrick Robinson

Piyush Patel

Amanda Saavedra

Johnny Serafin

Michael Stever

Jeff Swope

Paul Thibault

Sandeep Verma

Bob Vitt

Phillips 66 Honeywell

OIL MOVEMENT TDC3000 TO EXPERION MIGRATION WITH FULL ... · AO FTA in First HPM Cabinet Not...

Documents

Transcript of OIL MOVEMENT TDC3000 TO EXPERION MIGRATION WITH FULL ... · AO FTA in First HPM Cabinet Not...