Control System Studio - CSS - Alarm Handling

Managed by UT-Battellefor the Department of Energy

Kay Kasemir, Ph.D.ORNL/[email protected]

June 2011 at KEK

Control System Studio- CSS -

Alarm Handling

mailto:[email protected]

2 Managed by UT-Battellefor the Department of Energy

Previous Attempts at SNSALH; manual “summary” displays; generated soft-IOCs + displaysIssues

– GUI· Static Layouts· N clicks to see active alarms

– Configuration· .. was bad Always too many alarms· Changes required contacting one of the 2 experts, wait

~days, restart CA gateway, hope that nothing else broke– Information

· Operator guidance?· Related displays?· Most frequent alarm?· Timeline of alarm?


Now: Best Ever Alarm System Tool

Yes, alarms are always a little scary…


Alarm System Components

Control SystemAlarm Server

Cool UIConfiguration

B. Hollifield, E. Habibi, "Alarm Management: Seven Effective Methods for Optimum Performance", ISA, 2007

1. What you see2. Technical details3. How to use it


1. What you see

Alarm GUI used by Operators


What you see: Alarm Table· All current

alarms– new, ack’ed

· Sort by PV,Descr., Time, Severity, …

· Optional: Annunciate

· Acknowledge one or multiple alarms– Select by PV or description– BNL/RHIC type un-ack’


Another View: Alarm Tree

· All alarms– Disabled, inactive, new, ack’ed

· Hierarchical– Optionally only show

active alarms– Ack’/Un-ack’ PVs or sub-tree


Guidance, Related Displays, Commands

Basic Text Open EDM/OPI screen Open web page Run ext. commandHierarchical:

Including info of parent entries

Merges Guidance etc. from all selected alarms


Integrated with other CSS Tools

AlarmsHistory of PVEPICS Config.


CSS Context Menus Connect the Tools

Send alarmPV to anyother CSSPV tool


E-Log Entries

· “Logbook”from context menucreates text w/basic info aboutselected alarms.Edit, submit.

· Pluggable implementation, not limited to Oracle-based SNS ELog


.. may require Authentication/AuthorizationLog in/out while CSS is running

Online Configuration Changes


Add PV or Subsystem1. Right-click on ‘parent’2. “Add …”3. Enter nameOnline. No search for config files, no restarts.


Configure PV· Again online· Especially useful

for operators toupdate guidanceand relatedscreens.


2. Technical details

Behind the GUI;Tools to monitor performance


Technical View

Alarm Cfg & StateRDB

IOCs

Alarm ServerCurrent Alarms: Acknowledged? Transient? Annunciated?

LOG

MessageRDB

JMS2

Speech

JMS2

RDB

Tomcat- Report

s

CSS Applications

Alarm Client GUI

JMS

Alarm Updates Ack’; Config UpdatesAnnunciationsLog Messages

TALK ALARM_CLIENTALARM_SERVER

PV Updates (Channel Access, …)


General Alarm Server Behavior· Latch highest severity, or non-latching

– like ALH “ack. transient”

· Annunciate· Chatter filter ala ALH

· Alarm only if severity persists some minimum time· .. or alarm happens >=N times within period

· Optional formula-based alarm enablement:– Enable if “(pv_x > 5 && pv_y < 7) || pv_z==1”– … but we prefer to move that logic into IOC

· When acknowledging MAJOR alarm, subsequent MINOR alarms not annunciated– ALH would again blink/require ack’


Logging

· ..into generic CSS log also used for error/warn/info/debug messages

· Alarm Server: State transitions, Annunciations· Alarm GUI: Ack/Un-Ack requests, Config changes· Generic Message History Viewer

– Example w/ Filter on TEXT=CONFIG


Logging: Get timeline· Example: Filter on TYPE, PV

1. PV triggers,clears, triggers again

2. Alarm Server latches alarm

4. Problem fixed

3. Alarm Server annunciates

5. Ack’ed by operator

6. All OK


All Sorts of Web Reports


3. How to use it

This may be more importantthan the tools!


Best Ever Alarm System Tools, Indeed

.. but Tools are only half the issue

Good configuration requires plan & follow-up.

B. Hollifield, E. Habibi,"Alarm Management: Seven (??) Effective Methods for Optimum Performance", ISA, 2007


Alarm Philosophy

Goal:

Help operators take correct actions

– Alarms with guidance, related displays– Manageable alarm rate (<150/day)– Operators will respond to every alarm

(corollary to manageable rate)


· DOES IT REQUIRE IMMEDIATE OPERATOR ACTION?– What action? Alarm guidance!

· Not “make elog entry”, “tell next shift”, …· Consider consequence of no action

· Is it the best alarm?– Would other subsystems, with better PVs, alarm at the

same time?

What’s a valid alarm?


How are alarms added?· Alarm triggers: PVs on IOCs

– But more than just setting HIGH, HIHI, HSV, HHSV– HYST is good idea– Dynamic limits, enable based on machine state,...

Requires thought, communication, documentation· Added to alarm server with

– Guidance: How to respond– Related screen: Reason for alarm (limits, …), link

to screens mentioned in guidance– Link to rationalization info (wiki)


Impact/Consequence GridCategory So What Minor Consequence Major Consequence

Personnel Safety PPS independent from EPICS?

Environment, Public

Can EPICS cause contained spill of mercury?

Uncontained spill??

Cost:Beam Production, Downtime,Beam Quality

No effect

Beam off < 1 sec?

Beam off <10 min

<$10000

Beam off >10min

>$10000

· Mostly: How long will beam be off?


.. combined with Response TimeTime to Respond Minor Consequence Major Consequence

>30 Minutes NO_ALARM MINOR

10..30 minutes MINOR MAJOR

<10 minutes MAJOR MAJOR + Annunciate

– This part is still evolving…


Example: Elevated Temp/Press/Res.Err./…· Immediate action required?

– Do something to prevent interlock trip

· Impact, Consequence?– Beam off: Reset & OK, 5 minutes? – Cryo cold box trip: Off for a day?

· Time to respond?– 10 minutes to prevent interlock?

· MINOR? MAJOR?· Guidance: “Open Valve 47 a bit, …”· Related Displays: Screen that shows Temp, Valve, …


“Safety System” Alarms· Protection Systems not per se high priority

– Action is required, but we’re safe for now, it won’t get worse if we wait

· Pick One“Mommy, I need to gooo!”“Mommy, I went”

(Does it require operator action? How much time is there?)


Avoid Multiple Alarm Levels· Analog PVs for Temp/Press/Res.Err./…:

– Easy to set LOLO, LOW, HIGH, HIHI· Consider:

– Do they require significantly different operator actions?

– Will there be a lot of time after the HIGH to react before a follow-up HIHI alarm?

· In most cases, HIGH & HIHI only double the alarm traffic– Set only HSV to generate single, early alarm– Adding HHSV alarm assuming that the first one is

ignored only worsens the problem


Bad Example: Old SNS ‘MEBT’ Alarms

· Each amplifier trip:≥ 3 ~identicalalarms, no guidance

· Rethought w/ subsystemengineer, IOC programmerand operators: 1 better alarm


Alarms for Redundant Pumps


Alarm Generation: Redundant Pumps the wrong way· Control System

– Pump1 on/off status– Pump2 on/off status

· Simple Config setting: Pump Off => Alarm:– It’s normal for the ‘backup’ to be off– Both running is usually bad as well

· Except during tests or switchover– During maintenance, both can be off


Redundant Pumps

· Control System– Pump1 on/off status– Pump2 on/off status– Number of running pumps– Configurable number of desired pumps

· Alarm System: Running == Desired?– … with delay to handle tests, switchover

· Same applies to devices that are only needed on-demand

1Required Pumps:


Weekly Review: How Many? Top 10?


A lot of information available· How often did PV trigger?· For how long?· When?

· Temporary issue?Or need HYST,alarm delay,fix to hardware?


Weekly Check: Stale, Forgotten?


Summary

· BEAST operational since Feb’09– Needs a logo– For now without BEAUtY– DESY AMS is similar and has been

operational for longer

· Pick either, but good configuration requires work in any case– Started with previous “annunciated” alarms

· ~300, no guidance, no related displays· Now ~330, all with guidance, rel. displays

– “Philosophy” helps decide what gets added and how· Immediate Operator Action? Consequence?

Response Time? – Weekly review spots troubles and tries to improve

configuration

Control System Studio - CSS - Alarm Handling

Documents

Transcript of Control System Studio - CSS - Alarm Handling