Ensuring Operations are (Cyber) Secure

Kevin Manderson

Hydro Tasmania

Hydro Tasmania

� Hydro Tasmania is celebrating its 100 birthday this year� Tasmania had the first hydro electric power station in

southern hemisphere at Duck Reach, Launceston;� Australia's largest renewables generator and water

manager;� 30 power generating stations (hydro, gas and wind); and� Dams, tunnels, weirs, flumes.

CC BY-NC-SA

Cyber Security � My Definition

Cyber is taken as computer

so

�How do I keep my computer systems operational, regardless of the threat�.

CC BY-NC-SA

Recent �Moment�

� Jan 23 2014.

� A major London Underground control room was flooded by a sea of rapid setting cement.

� Back in operation after 8 hours.

CC BY-NC-SA

Holistic Control Environment

� �Public� control centre is above the water

� Control Infrastructure and remote sites are below the water.

� Mostly today is below the water aimed at keeping above the water operating

Image: Attribution in appendix

CC BY-NC-SA

Hydro�s Control Environment

� Hydro operates several control environments and today I will discuss our primary environment;

� Dual purpose control centre:� Dispatch management using the SCADA system� Bidding management using the corporate systems

� The SCADA environment is secure, redundant and has a dedicated support team with 20 minute maximum callout to on-site availability; and

� The bidding environment relies on virtualisation and SAN capability for redundancy and has a more informal call out capability

CC BY-NC-SA

So What am I Keeping Operational

� The core dispatch process:� Sliding windows of time with tightly coupled processes

occurring across a distributed/redundant group of closely and loosely coupled systems� Four seconds � data exchange with power stations,� Eight seconds � control data exchange with AEMO,� Five minutes � dispatch interval, some data exchanged� 30 minutes � market interval,� Daily � reporting and other processes,� A number of ancillary services, and� Other contracted and mandated services.

� I am the custodian of data and control in part of the chain of the complete process.

CC BY-NC-SA

Security List

� ASD Top 35, top few:� Whitelisting� Patching � App and OS� Restrict admin level access� Then the other 30 or so controls, including monitoring

� Whitelisting, in SCADA system, � Adding signature checks to all systems

� Patching is a common issue in a most SCADA environment; and

� Match, patch, patch, watch.

CC BY-NC-SA

SCADA Good Practice Guide (GPG)

CC BY-NC-SA

Hydro Architecture

� The GPG is a baseline, � Additional tiers of access control, some as

processes/layers, others as diversity in the boundary traversal and monitoring and alerting,

� Hydro�s production environment has over 40 servers/systems �integrating� the production (dispatch) process. Redundant over multiple sites and communications paths. Builds on the GPG, and

� Logging, monitoring and alerting.

CC BY-NC-SA

Vulnerability Analysis Approach

� Perform a vulnerability analysis for each process flow, then each segment of the system, and

� Brainstorm possibilities:� People,

� Services,

� Black swans, and

� All hazards approach � that is, all or none?

� People/Technology (aka physical/virtual) boundaries need the most attention

CC BY-NC-SA

Deciding on Security Barriers

� After analysis identify the process change, ownership transfer and different security groups. Consider the location of the control room and physical security issues. The change points are highly likely to be the vulnerability points,

� What user groups are involved� What is the vulnerability (confidentiality/integrity/availability):

� Denial of Service/interruption, � data or control injection/corruption, and� data or physical access.

� What consequences, and� What cost to control (inputs or consequence).

CC BY-NC-SA

Physical Security

� What if the control centre is not physically secure and easily accessible?

� What if parts of the control centre are only occasionally visited?

� What if parts of the control centre are physically close to unsecured corporate infrastructure?

� What if people wander in and out of secured facilities?

� Physically remote sites compound the physical security issues

CC BY-NC-SA

Example - Corporate Data Interaction

� Periodic market data,

� �Highish� availability.

Data to and from AEMO

SCADA and Dispatch

processing

Power stations

Data to and from Corporate.

CC BY-NC-SA

Corporate Data Interaction Analysis� Periodic market data,

� �Highish� availability

Data to and from AEMO.

SCADA and Dispatch

processing

Power stations

Data to and from Corporate.

� Multiple firewalls and proxies

� Secure protocol

Monitoring

CC BY-NC-SA

Security Controls

� No path from corporate to SCADA� SCADA requests/sends data from or to corporate,

� Secure,

� Protocol is further protected by multiple buffering and proxies,

� Multiple barriers,� Multiple diverse firewalls

� Non addressable

CC BY-NC-SA

A Security `Something� is What

� Any change which causes the overall process to move to an undesired or unknown state,

� So what can have an impact?� Malicious hacker,

� A �Snowden� or admin event,

� Physical intervention,

� Negligence/Accident,

� Equipment/software malfunction, and

� Loss of services.CC BY-NC-SA

What Touches my Systems

� The systems sit in locked racks, relatively inert,

� What can have an impact?� People,

� Data,

� Services, and

� Physical environment.

CC BY-NC-SA

What has Caused Problems

� Contracted services:� Power,� Air conditioning/cooling,� Building access,

� When things go wrong expect more than one event:� Example when testing generators, � power issues and � air-conditioning

CC BY-NC-SA

High Availability (Power)

� Redundant sites; � Redundant UPSs;� Redundant power sources and phases;� In rack: � UPSs,� Power transfer switches,� Dual power supply equipment,

� Equipment diversity; and� Monitoring/knowledge of state of power sources� Good documentation of exactly what is used and

where.CC BY-NC-SA

Other devices

Rack Power Connectivity

Server

Other devices

Server

In rack Transfer Switch

Critical Services

(UPS)Power

In rack UPS

Non Essential

(raw) power

Monitor

EssentialPower

AirCondetc

CC BY-NC-SA

Other Power Examples

� Smart Power Distribution Units (PDU); and

� After a series or short power failures will the outlets power on?

CC BY-NC-SA

Futures

� Mobile and BYOD devices;� Operators working from outside control centre

� Operators using own devices

� Visualisation:� Traditional SCADA is one line screens, data lists,

alarms

� Expect specific users receiving visual representation of issues, trends, displays and information

CC BY-NC-SA

Risks

� Malware/viruses;� Low � have to jump into controlled environments

but Stuxnet proved it can happen�

� Discontent;� Snowden effect

� Social engineering; and� Always present

� User mistakes:� Happen

CC BY-NC-SA

Security - Implications

� Current model is typically segmented systems with serial links to remote sites. Tightly controlled.

� Future: � �IP� based,� Users will expect open access, � Ability to share information easily,� Operate on non-specialised devices and systems, � Immersive, trendy term but will happen.

CC BY-NC-SA

High Availability � Workstation Controls

� Multiple disks (raided to survive disk failure);� Multiple communications paths to servers ;� Multiple monitors per workstation (+spares);� Adjacent workstations powered from alternate

supplies;� Considering one workstation to have inline UPS;

and� Workstation resource usage trend monitoring,

SMS to the on call engineer.

CC BY-NC-SA

Think of Security Holistically

� Who went to Ruxcon?� Other white/grey/black hat

conferences,� Use a range of tools and

test your systems,� Do pen testing, � Think touch==own,

physical security is critical, � Be aware of what's

happening, and� xkcd is good� � Think black swans� Keep �simple� involved,

CC BY-NC-SA

Comments or Questions

CC BY-NC-SA

Attribution

� Concrete images � multiple press outlets, credited to UsVsth3m.com

� Iceberg image

� SCADA GPG, � Australian Government material

� Roman Empire image� https://en.wikipedia.org/wiki/Roman_Empire

� Others by Hydro Tasmania or me.

http://upload.wikimedia.org/wikipedia/commons/a/ac/Iceberg.jpgBy Created by Uwe Kils (iceberg) and User:Wiska Bodo (sky). [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons

CC BY-NC-SA