SCADA System Failure Impact Study at Lorne Park and ... · SCADA System Failure Impact Study at...
-
Upload
trinhtuyen -
Category
Documents
-
view
225 -
download
0
Transcript of SCADA System Failure Impact Study at Lorne Park and ... · SCADA System Failure Impact Study at...
SCADA System Failure Impact Study at Lorne Park and Lakeview
WTPs
Shawn Xiong - Cole Engineering Jeff Hennings, Jason Little, Claudio Cuffolo - Region of Peel Dale Barker, Eric Zhang - OCWA
OWWA Conference 2014 May 5th
London, ON
Project Background
Project Objectives
Analysis Approach
Recommendations
Presentation Outline
Background
Lorne Park WTP and Lakeview WTP
Supplies water to South Peel and York Region
Owned by Peel Region and operated by OCWA
Background
Employs advanced treatment process: membranes, UV, Ozone, BACC, GACC, and conventional filters
Advanced SCADA system for automation
Background
Failures have been experienced at both plants
Adverse impact on the plants’ ability to maintain water production
Maintain three objectives during a SCADA failure event:
Water quality
Water production
Compliance reporting
Project Objectives
Establish critical process areas at each plant that are essential to the plant’s ability to meet water quality and quantity requirements.
Identify critical instruments for MOE compliance reporting.
Review each critical process area to identify operational and data reporting problems that could occur if SCADA was disrupted.
Project Objectives
Recommend operation procedures to follow during a SCADA failure for each of the problems identified above in order to minimize risks to water quality or quantity, or compliance reporting.
Conduct network or SCADA failure modes and effect analysis (FMEA).
Document the evaluation and recommendation in an organized format.
Project Objectives
A systematic analysis approach is required to analyze such a complex system with a multitude of subjects
Analysis Approach
CAUSES
FAILURE
IMPACT
DESIGN IMPROVEMENT
EMERGENCY PREPAREDNESS PLAN
AWWA G440 - Emergency Preparedness Practices AWWA M19 - Emergency Planning for Water Utilities AWWA J100 - Risk and Resilience Management of
Water and Wastewater Systems
Analysis Approach
Risk Management Framework ∗ Identify Risks ∗ Analyze Risks ∗ Evaluate Risks ∗ Treat Risks
Analysis Approach
Risk Analysis Processes and Tools ∗ RAMCAP - Risk Analysis and Management for Critical
Asset Protection ∗ HAZOP - HAZard and OPerability study ∗ FMEA - Failure Modes and Effect Analysis
Analysis Approach
Combination of RAMCAP and FMEA
Analysis Approach
*From AWWA, J-100
FMEA
Project Methodology ∗ Step 1 – Asset Characterisation (RAMCAP Step 1) ∗ Step 2 – Threat Characterization (RAMCAP Step 2) ∗ Step 3 – FMEA Analysis (Consequence, Risk Analysis
and Management, RAMCAP Step 3-7)
Analysis Approach
Asset Characterization – Lorne Park
LOW LIFT
MEMBRANE AOUV GAC
MUG
FILTER 9-12 UV
CONVENTIONAL FILTERS
RESERVOIR HIGH LIFT
Low Lift Pumping Plant 1 - MUG - Membranes Plant 1 – MUG - AOUV System Plant 1 – MUG - GAC Contactors Plant 2 - Filters No. 9-12 Plant 2 - Filters No. 9-12 UV High Lift Pumping Chemical Systems – Coagulant, Bisulphite for WW
supernatant dechlorination, Hypochlorite for disinfection
Critical Process Areas - Lorne Park
Compliance Intruments – Lorne Park
MEMBRANE/UV/GACC
CONVENTIONAL FILTERS 9-12 & RESIDUAL MANAGEMENT
CHEMICAL SYSTEM HIGH/LOW LIFT PUMP STATION
AIT39401
TURB
AIT39301
TURB
AIT39201
TURB
AIT39101
TURB
AIT34801
TURB
AIT34701
TURB
AIT34601
TURB
AIT34501
TURB
AIT34401
TURB
AIT34301
TURB
AIT34201
TURB
AIT34101
TURB
AIT35601
TURB
AIT35501
TURB
AIT35401
TURB
AIT35301
TURB
AIT35201
TURB
AIT35101
TURB
AIT35001
TURB
AIT34901
TURB
AIT87501
AIT87502
TURB
CL2/pH
AIT87504
FL
AIT42002
CL2
FIT64421
COAGFIT
16301
FIT17301
FIT17311
FIT39401
FIT39301
FIT39201
FIT39101
FIT39801
FIT39001
BW
GACBW
FIT42001
SUPERNATANT
CONTROL ROOM
RESERVOIR
LIT87501
FIT55101
FIT55201
FIT55301
FIT55401
FIT55501
FIT55601
FIT55701
FIT55801
GAC FLOW
FIT38801
FIT38701
FIT38601
FIT38501
FIT38401
FIT38301
FIT38201
FIT38101
LIT87511
TIT87501
TIT87511
PLC & CCP – Lorne Park
SCADA System Failure Modes
Threat Characterization
SCADA SYSTEMFAILURE
SCADANETWORKFAILURE
SCADA SERVERFAILURE
PLCFAILURE
RING SWITCHFAILURE
CABLE FAILURE- FIBRE RING
PANEL SWITCH FAILURE
POWER SUPPLYFAILURE
RACK FAILURE
CPU FAILURE
I/O MODULEFAILURE
BOTH SERVERSFAIL
VIRTUAL MACHINECRASH
PRIMARY SERVERFAILURE
SECONDARY SERVER FAILURE
CABLE FAILURE- CAT6 CABLE
BUS NETWORKFAILURE
CONTROLNETFAILURE
MODBUSFAILURE
DEVICENETFAILURE
SECURITYBREACH
POWERFAILURE
INSTRUMENTFAILURE
Threat Characterization
Failure Modes
Failure Causes Detection Impacts Response /Remediation
Criticality
SCADA Network Failure
Ring Switch Software Failure (RSTP) - Unidirectional link - Duplex Mismatch - Software bug - etc
SCADA freezing - Shutdown of the plant or process areas - Loss of compliance reporting - Loss of control of the Plant
Shutdown network and reboot switches one by one
High
PLC Hardware Failure - Major
Power Supply Failure - Remote Rack
All lights off at rack Loss of control of some equipment
Replace power supply
Medium
Remote Rack Failure All lights off at rack Loss of control of some equipment
Replace rack Medium
ControlNet Failure
ControlNet Module Failure - OK Status LED steady red, display "FAIL"
Loss of control of the affected process area
Test, reboot, or replace module
High
ControlNet Media Failure - Module indicates "NET ERR" - Network Channel Status A/B red or flashing red
Loss of control of the affected process area
Reconnect media High
Instrument Failure
Compliance Instrument Failure
- Fault alarm - Out of range alarm - Loss of Echo
- Loss of compliance data - Manual sampling and recording
High
Select Process/System for Evaluation Select Team Diagram system blocks Identify failure modes Identify failure causes Evaluate failure severity Evaluate failure probability Evaluate failure detection & rectification Calculate risk priority number (RPN) Recommend actions to reduce RPN
FMEA Analysis
Failure impacts ∗ Water production ∗ Water quality ∗ Compliance reporting ∗ Operational resources
FMEA Analysis
FMEA Analysis
FMEA Analysis
FMEA Analysis
FMEA Analysis
Scoring of Failure Severity
Score Severity Rationale
1 not perceptible No Effect
2 small This failure is not significant. Treatment process can continue with no interruption.
3 medium
This failure is significant enough to affect the treatment process system operation, compliance, water production, and water quality. Additional resources for plant operation are required.
4 significant
Inoperability of a significant process in the treatment plant. Significant requirement of additional resources to operate the process in local-manual mode.
5 very significant Total inoperability of the treatment plant
FMEA Analysis
Scoring of Failure Probability Score Probability Rationale
1 improbable MTBF > 25 years 2 very small MTBF [ 10-25] years 3 Small MTBF [ 3-10] years 4 Medium MTBF [ 1-3] years
5 High MTBF < 1 years, or has occurred previously
FMEA Analysis
Scoring of Failure Detection and Rectification Score Detection
/Rectification Rationale
1 very probable Very probable and immediate detection and rectification of failure (<1 hr)
2 high Chance of discovery high, time to detect and rectify failure is short. (<4 hr)
3 medium Time to detect and rectify the failure within a day. (<24 hr)
4 small
Failure may be undetected for extended period of time or it may take days to troubleshoot and rectify the failure. (< 1 week)
5 very small Very small chance of discovery or it will take a long time to rectify the failure (> 1 week)
Design Improvement ∗ I/O distribution ∗ Redundant network cabling ∗ Environmental control for network equipment ∗ Local data recording ∗ Local operation of critical process equipment ∗ Signal communication for process control
Recommendations
Operations Improvement ∗ Treatment process response to SCADA system failure ∗ Instrumentation response to SCADA system failure ∗ SCADA system maintenance
Recommendations
Recommendations
Process Criticality: High
Rationale: Supplies raw water to Plant 2.
SCADA Communication Dependency:
None.
Local-Manual Operation: Low Lift Pump Operation – Operate pump in Local-Manual mode from pump Control Gallery or Local Control Station. Start the required number of pumps based on filters feed flow requirement. Adjust flow control valve FCV16301/16302 to meet the flow setpoint and also to maintain a target level in Settled Water Conduit.
SCADA Failure Response: (1)If SCADA network fails a.Low Lift Pumps for Plant 2 can stay under SCADA control as there is no communication requirement between PLC-116 and PLC-398 to allow the operation of Low Lift Pumps
(2)If PLC-116 fails a.Check the feed flow demand for Plant 2 and start the appropriate number of low lift pumps. b.Manually adjust the position of flow control valves FCV16301/16302 to achieve the target feed flow demand and also to maintain a target level in Settled Water Conduit.
Additional Operational Resources Requirements:
One operator is required to monitor and control the operation of the low lift pumps to maintain water supply to Plant 2 when operating the pumps in Local Manual mode.
Low Lift Pumps for Conventional Filters 9-12
Emergency Preparedness Plan (EPP) ∗ Objectives: ∗ Emergency Response Team ∗ Communications Chart ∗ Plan Activation ∗ Component Specific Plans (process & instrumentation) ∗ Emergency Response and Recovery ∗ Training, Testing & Drilling
Recommendations
Recommendations
Abnormal Event SCADA Problem?
Actual/Probable Adverse Impact to
Water Supply?
OCWA designate contact
Peel On-call SCADA Tech
Peel SCADA follow with OCWA on action taken/
required
NO
OCWA ORO/Designate Contacts Peel On-call
Compliance Rep. Activate EPP
YES
Determine Failure Impact and
Affected Process Areas
OCWA Follows Appropriate Treatment
Process and Instrumentation Plans
OCWA/Peel SCADA Troubleshoots the SCADA
System
Calls in additional operations resources if
required
Communicates with internal/external contacts per Communication Chart
Isolate event causes and apply remedial actions
Treatment Process Recovery
Event Documentation
Engage the Emergency
Response Team
Regional Municipality of Peel Ontario Clean Water Agency AECOM CH2M Hill
Acknowledgement