Common Issues in the NOC - ABB GroupFILE/ATTBK4D0.pdf/05.… · Fire Protection . Integration...
Transcript of Common Issues in the NOC - ABB GroupFILE/ATTBK4D0.pdf/05.… · Fire Protection . Integration...
Operational Excellence Data Center Solutions Day, 5 July 2013, Martin Hogberg
Increasing Uptime in Data Centers by
Addressing Human Factors
Human Factors It’s not About Pointing Fingers
Mike, are you
racking the new
server?
Yea, some guy
installed more
power last night.
Ok put it in. Right, here
we go!
Common Issues in the NOC
Why is it important to focus on the operator?
1. Rick Schuknecht, The 451 Group Uptime Institute, June 2011
Availability/Downtime 75% of downtime due to human error1
• Over the total life cycle, most important decisions are made by the
operator.
• The amount of information from the process has increased dramatically
with risk for stress in critical situations.
• Experience and operator skills are invaluable assets.
• It is difficult and expensive to recruit and train new operators among the
young generation.
Traditional NOC environments Issues we are seeing across every industry
• A complex environment with a multitude of different systems and a low level of
integration.
• Very difficult to get an overview of what’s going on in the data center and what’s
about to happen.
• Lack of attention to human factors result in an unhealthy working environment.
Traditional NOC environments Issues that degrade reliability and profitability
• Poorly integrated CCTV and telecom equipment.
• Too many keyboards and other input devices.
• None ergonomic operator desks with built-in computers that generate heat,
noise and are difficult to keep clean.
Newer control rooms with traditional approach Issues that degrade safety, quality and profitability
• A newer centralized control
room built according to
traditional principles.
• Very low level of
integration results in total
screen and keyboard
overload.
• Overview information
screens very difficult to
view.
Operations Environment
University evaluating the positive alertness effect by working with Modern Concepts In the spring of
2012 Chalmers
has been testing
the
effectiveness of
the EOW, and
compared it to
the traditional
way of working
with control
room design. To
facilitate the test
Chalmers has I
corporation with
ABB created a
simulation of a
“color factory”
State of the art ergonomic and high
level of integration environment
Research Trials Traditional Operations
Basic level of integration that fulfill
standard traditional requirements
How to measure the alertness effect ?
Chalmers testning the Eow Concept
The tests has been conducted but using subjective and
Objective measuring tools as well as interviews
The subjective Tools measure
the perceived comfort and stress
level of the operator.
The SAM form one of several different
forms that use to evaluate the result.
The objective tools measure
the reaction time, cursor path
and number of problems
resolved
The specially developed facility
How to measure the alertness effect?
Chalmers evaluating the effect of the EOW Concept
Basic level of integration
that fulfill standard and
traditional requirements
Subjective Objective Interviews
State of the art
ergonomic and high
level of integration
environment
Subjective Objective Interviews Analysis and evaluations of
operators
environment and
technology solutions
resulting in expected
findings of
5-10 % increased operator
effectiveness
Integrated operations High information density and ergonomic workplaces
Interactive, close large overview ”owned” by the NOC operators
Easy to use individual ergonomics
Built-in communication tools simplifies collaboration
Saves up to 30% floor space compared to traditional solutions
Adaptive NOC Environment
Extended Operator Workplace as a centerpiece
of control room.
Cover all operator needs from one
console.
Futuristic NOC equipment.
Desk raises up on critical alarms so
Operator has to stand to deal with
situation.
Blows cold air to maintain Operator
alertness.
Sound shower directs operator through
audible alarms.
Can link to lights & window-blinds.
Going beyond the operator workplace
Overall control room environment is critical
Dedicated space for control, removes distractions and focus operators
on task of controlling the facility.
Visitor area to keep non-essential personnel out of the control area
Collaboration space. Meetings, troubleshooting, problem solving with
A/V tied to control center visualization.
Relaxation area for extended shift recharging, separate eating area to
avoid noise, segregated printers to remove noise
Helping the Teams and Communication - Integration
Data Centers – Management Today
Isolated Silos with Individual Planning,
Applications and Reporting
System and Network Management, Servers, Network and Storage
Building, Power and Environmental Management
Change Management, Help Desks and Trouble
Tickets
Integrated Reporting: Monthly Spreadsheets and Slides
IT Facility Operations
Utility Services
ABB Decathlon™ System
Security
Switchgear RDU CRAC VESDA/NOVEC
Chiller
Control
Fire
Protection
Integration Command Center – Single Pane of Glass
Alarm List
Trend Display Documentation
Faceplate
Navigation
Display Operator Note
One display management system
Consistent methods for navigation and display access
Supports power, cooling, and IT
Provides local and remote display access
Improves operational efficiency with better visibility
Creating a Collaboration Culture
CCTV
IT, planning the moves adds
changes.
Utility & Facility, verifying the
necessary space, power and
cooling is available.
Possible though an integrated
system.
Integration of Facility and IT Work Faster and Avoid the Mistakes
Creating a Collaboration Culture
Effective Software which Supports the Operator
Reduce the Isolated Systems Information Kept With the Asset
Data center based information
access
One Click to all information
Navigation based on job function
Real-time decisions and action
Control
Faceplates
Trend
Graphics
Video
Reports
Documentation ERP/ CMMS
Asset
Health
Integrated Information – CRAC Example
Trends
Alarm Instructions
Documentation
Operator Note
Overview of Effective Data Center HMI
High Performance Design
User Interface - The human performance improvement is
supported by selection of color scheme, shape of elements
and icons and tab based navigation.
Addressing Alarm Flooding - The effectiveness of new
operator interface concepts for improving operators’ ability
to handle data center disturbances that generate alarm
floods.
Integrated Alarm Management Alarm Management - A key concern for data centers
Too Many Alarms! Operators acknowledge/silence
alarms without looking at or
acting on them.
Incidents or near-incidents
where operators missed alarms.
Acoustic alarms turned off.
Operators don't know what
particular alarms mean.
Alarms disabled/suppressed for
long periods without review.
© ABB / PA / Control Systems
July 8, 2013 | Slide 25
Integrated Alarm Management Mission-Critical performance
View alarms from cooling, power, and IT in a
single list master list as well as individual lists
sorted chronologically, by priority, or by equipment
and filtered by location, functionality, or network
connectivity
Embedded analysis tools for determining the most
frequent offenders, the ones that take the longest
to remedy, or those that result in the greatest
downtime
Integrated Alarm Management
Enables a starting point for alarm rationalization
Analysis of ”worst cases”
Allows for continuous analysis and optimization
View alarms from cooling, power, and IT in a single list
sorted chronologically, by priority, or by equipment and
filtered by location, functionality, or network connectivity
Consistent alarm-response navigation
Fast navigation from an alarm reveals detailed information
needed to handle it correctly.
© ABB / PA / Control Technologies July 8, 2013 | Slide 28 3BSE068040 en A
Alarm grouping replaces long lists
Damage limitation no longer depends on an individual’s
ability to ‘piece together’ information from several
simultaneous alarms
© ABB / PA / Control Technologies July 8, 2013 | Slide 29 3BSE068040 en A
Condition Based Maintenance
The System Guides the User
Common format to determine faults in real time including Description of fault Possible cause Suggested action Priority / Severity of fault
SMS / e-mail messaging
Online snapshot reports for better change management Reduces time for shift handovers and maintenance stops
Reduced time to get right information Thanks to the integration capabilities of the system
the ”snapshot” covers all DDC, PLC’s etc regardless of brand
Enables real-time shift reports for example Navigation with the context menu that’s accessible from the
report
Decathlon Users Future Operator
Future Operator - Possibilities
Augmented reality
3D visualization
Gestures
Social media
Sensor technologies
Mobile devices
Touch
Depth cameras
Eye-tracking
© ABB Inc
3BSE074371 en.
July 8, 2013 | Slide 33
Future Operator – Visualizing in 3D When it makes sense
Before After
Future Operator - Controlling by Eye-Tracking Gesture Interaction
Future Maintenance Engineer Augmented Reality
A coffee cup tag shows
information related to the
coffee cup
An ABB logo shows the
latest news related to
Decathlon
A factory logo brings up a
3D plant