Network Aspects of the SLAC Cyber Security Improvement

ProgramNLIT Summit,

San Francisco

June 29-July 2, 2014

Dr. R. Les Cottrell,Hadas Niv,

& Ben CalvertSLAC

2

SLAC Cyber Security Improvement Program (CSIP)

• Secure Mobility

• Secure wireless

• Mobile Device Management

• IP Address Management (IPAM)

• Network Security (Firewall)

• Network intelligence

Collaboration between Network Engineering &Cyber Security

Network Part of an overall SLAC CSIP includes:

Technical OverviewProgram Area Risk/Gap Business Benefit Solutions

Secure Wireless

• Open “guest” network creates vulnerability for SLAC assets

• Less exposure to wi-fi attacks• Better user experience• Improved productivity

Cisco

Secure Mobility

• No ability to secure smartphones & tablets

• Protect data on mobile devices• Enable future mobility

Airwatch

IP Address Management

• Poor asset data impedes incident response

• High cost of MAC’s due to static IP addresses

• Faster resolution of incidents• Lower overhead to network

management

NetDB

Border Firewall

• No stateful inspection• No app awareness• No intrusion prevention

• Attacks stopped at perimeter Palo Alto Networks

Network Intelligence

• Little network security instrumentation

• Limited FW coverage

• Provide network traffic information to aid with policy design & implementation

• Defense in depth

Bro

3

Secure Mobility – Secure Wireless

Guillaume Cessieux Network Engineer

5

Current state of Wireless

4K unique devices/monBusiness day: ~1500 devices

Weekend: ~ 200 devices

• 271 Access points, • 65% not 802.11n capable• Only uses 2.4GHz band

• Access via open visitor net• No authentication, no data encryption• “Starbucks” model

• Grew from “nice to have” (best effort) to critical

• Very popular

0200400600800

10001200140016001800

• Not scalable/manageable

6

What do we need?

A corporate like network for SLAC employees

• Providing authentication & data encryption

• Grant more privileges to wireless users

• User impact – Ease of use critical• Visitor wi-fi? Yes, it will still be there.• Add always-on secure network connection to SLAC internal• No sponsors• No manual login

• Seamless access to SLAC services (intranet, Lync…)• Same privileges as in office for mobile users • Reduce VPN needs • Improved roaming, robustness, availability, coverage, speed• Move towards strategic vision, reduce need for wired copper

• Forrester predicts 59% of all data traffic will move from wired to wireless by 2017

What does it provide? – Scope

• Value Add: Grant privileges to SLAC wireless users to

seamlessly access internal services - Secure roaming

• Visitor & secure wireless deliver by same infrastructure

• migrate to central/robust/easier to manage solution

• Speed

• Replace old, slower 802.11a/b/g Access Points with 802.11n

• Add capacity in conference rooms and dense buildings

• Using 802.11ac, get experience

7

8

Encryption/Authentication

• Encryption: WPA2 with AES

• Using EAP-TLS as the authentication mechanism

• Certificates are attested by the Microsoft Certification Authority

(CA)

• Certificates are delivered to clients via Windows Active

Directory (AD) Group Policy Objects (GPO)

Client

Access Points

2 WirelessControllersCisco 5508

AD

Radius

Kerberos

9

Current State of Secure Wireless

Only supports managed Windows devices• Working on smartphones tablets (see MDM next section)

• Add MacOS support March 2015

Controllers are logging to Splunk.• Next mine for analytics

Rolled out to 2 major buildings

Already popular:• ~ 40 users / work day using secure wireless

• Simplifies sign on,

• Easy roaming from office

Site rollout July 9th, 2014

Secure Mobility - Mobile Device Management

Edgar Estabanez Cyber AnalystGuillaume Cessieux Network EngineerRodney Wong Windows Engineer

11

What is the Mobile Device Management Project (MDM/EMM)

• AirWatch chosen• Cloud solution did not require a large infrastructure by any one entity

• Scalability is very flexible by having the infrastructure hosted by AirWatch

• Secure & support smart phones and tablets• Current gap in our security posture

• Audit finding from Stanford Information Security

• A requirement of the Stanford Mandate (2014)

• SLAC joined Stanford MDM Affiliates• Consortium of four Stanford entities

- Stanford University (SU)

- Stanford Hospital and Clinics (SHC)

- Lucile Packard Children's Hospital (LPCH)

- SLAC National Accelerator Laboratory

• Chartered to deploy a solution with a feature set and platform coverage acceptable to

all members

12

What is Mobile Device Management

• Device set up

• Policy acceptance

• Profile distribution

• Device compliance

• Applies to company owned and employee owned devices (BYOD)

Enforce

• Device tracking

• Applications catalog

• Monitoring & reporting

Manage Assets

• Passcode enforcement

• Remote lock

• Remote wipe

Secure information

13

Airwatch: lifecycle example

13

J. DoeRole: Account Manager

Corporate ServicesNone

J. DoeRole: Account Manager

Corporate ServicesVPNWi-FiExchange 2007

J. DoeRole: Account Manager

Corporate ServicesVPNWi-Fi (New)Exchange 2010

J. DoeRole: Business Director

Corporate ServicesVPNWi-Fi Exchange 2010Corporate Apps

Stolen Device

All Access Denied

Corporate Resources

Certificate Services Directory Services Mail Services Wi-Fi VPN Content

Securely enroll device

using AirWatch1

3

AirWatch configures device to access corporate services

Company upgrades Exchange and rotates Wi-Fi certificates

AirWatch automatically upgrades device configuration

J. Doe is promoted to Business Director

AirWatch configures device based on new role for higher access to corporate services

Device stolen!

AirWatch removes corporate data, apps, access to corporate services and

remotely wipes the device

5

2

4

AirWatch

Applications

14

Why at SLAC

• Security• 1300+ mobile devices connecting to Exchange using Active

Sync- Currently unsecured and unmanaged

- Stanford mandate requires encryption & passcode for PII & health data

• iOS and Android are the widely used platforms at SLAC

• MDM is a basic foundation for mobility program• Computing initiatives

- Security guards checking in, emergencies, construction

managers

• SLAC departments are looking at developing apps in-house

71%

24%

5%

Mobile Platforms at SLAC

Apple iOS

Google Android

Other

15

Policies being worked out

• SLAC does not provide smart phones or carrier service

• Enrollment requirements• All SLAC-owned devices must be installed with AirWatch

• Installation of an agent on device

• Will be a Passcode requirement

• Wipe of lost devices

• Wipe of enterprise data on employment termination

• Careful consideration needed when affecting “birthright”• How do rules apply to employee owned devices (BYOD)?

- Consider enforcing MDM as a requirement to access SLAC email?

- Alternately, enforce a similar security policy on unmanaged devices

using ActiveSync?

16

Communication Plan is Critical

• Road show: Stanford enrolled 7,000 devices by selling

the personal protection aspects

• Engagement with key groups (Business Managers, Sys

Admin Forum, Governance group)

• Web and email communications

• Marketing message

- Access to internal resources via Secure Wireless

(limited to managed devices)

- Personal benefits of device security• Convenience & productivity (one step setup for Secure

Wireless, VPN, Lync, etc.)

- Future benefit of enterprise applications

- Foundation for future

Network Security – IP Address Management (IPAM)

Open source: From Stanford

Yee Ting Li, network engineerKent Reuber, network engineer

18

What is “IP Address Management”? – Scope, assumptions

• System of record for DNS names to IP addresses for all IP

assets on SLAC internal networks• Contains information of physical locations

• Map IP to physical interface, host type and user/administrator

• Integrates with DNS servers (BIND, Windows) and DHCP servers

• Related to asset management

• Current system (1980’s vintage) not meeting current needs• E.g. does not support IPv6, Media Access Controller (MAC)

addresses, no granularity of authorization

• Replace the IPAM part of current system with NetDB

from Stanford

What is NetDB?

• Stanford University Open Source IP address management

system. In use for over 20 years.

• Web GUI interface. Also includes a Command line

interface & interfaces for power users and scripting.

• Over 1,000 Stanford NetDB registered admins.

19

What does NetDB provide?

• Authorization of admin allowing granular per-admin and per-

group access privileges.

• “Full Search” capabilities: find all devices in a certain building,

maintained by a certain administrator, …

• Automatically feeds DNS and DHCP services

• IPv4 and IPv6

• Supports static (Media Access Control (MAC) address -> fixed

IP) and dynamic (“Roaming”) addresses or a mix of the two.

• Easy to learn. Stanford estimates that the GUI can be taught in

15 minutes.

• Extensive Help that is actually useful.

20

21

Migration of data: Current Record of RecordsOld

IPAMLANMON DHCP DB GOLIATH TAYLOR RACKWISE SLAC PC

Primary Data IPAM ARPs, Location

Mac Addresses

Windows Unix Data Center Purchasing

Data Store Oracle Postgres Oracle SQLServer Flat Files Excel / SQLServer

Oracle

IP Y Y Y Y

MAC Y Y Y Y

DNS Y Y Y

Administrator(s) Y Y Y Y Y

User(s) Y Y Y Y Y

Custodian(s) Y

Location Y Y Y Y

Make Y Y Y Y Y

Model Y Y Y Y Y

Serial Number(s) Y Y Y Y

PC Number Y

Purchase Info Y Y

Warranty Info Y Y

• Example:Get ARPs from LANMONCorrelate to IP address in the old IPAMMatch IP addresses in Goliath and Taylor (get user data, serial

numbers)Match Serial Numbers to SLAC PC (get purchase information)All non-matching entries in old IPAM can be deemed ‘stale’

• But still requires human to validate each entry…

22

Cleaning /prune data

• ‘bad’ CANDO entries: ~9,000 out of ~25,000• did not match to a network ARP within last 6 months

• Mostly old clusters (decommisioned equipment) or temporary DHCP

records

• ‘non-matching’ records: ~2,000 out of 16,000• Entries with conflicting information about IP or MAC addresses

• Stale data on ‘people’• Map to groups and or (company) hierarchy.

• Import into NetDB, coordinate with ServiceNow Assets

• Part of a federated set of databases

Conclusion

• SLAC is replacing its old IPAM with Stanford’s NetDB.

• Goals:• Better ability to find & track down systems & contacts

• Enable self service: Distributed administration with limited privileges

- Create permission groups and user records

• Support for IPv6

• Increased ability to use DHCP, automate address assignment etc.

• Feed SLAC DNS & DHCP

• Ongoing work to clean & convert old data into NetDB.

• Train admins on NetDB.23

Network Security – Border firewall

Antonio Ceseracciu, network engineer

25

Objective

• Create a Next Generation SLAC Border Firewall - a

security device that sits logically between the

"SLAC Business network" and outside networks.

26

Proposed StateThe basic changes involved with this project

Impact:• Fewer viruses &

attacks get through

• Mostly transparent to user

• Addition of Next Generation border firewalls with content inspection

• High speed bypass to support Scientific Computing data transfers

SLACBorder

SLACCore

• Retirement of existingweb proxies

Visitor

Enterprise

Buildings Controls

Internet

Science DMZ

27

Firewall Features

Feature Description

Application Level Firewall Application level firewalls inspect the content of each packet and analyze the protocols.

Application level inspection Enforce policies based on web applications

SSL Interception, incl. HTTPS

SSL Interception allows the firewalls to act as Man In The Middle and terminate and inspect the content of SSL connections.

SSH Interception SSH Interception allows the firewalls to act as Man In The Middle and terminate and inspect the content of SSL connections.

Signature-based Malware Detection

Allows the firewalls to identify Malware by matching network packets against a database of known malware signatures.

28

Firewall Features

Feature Description

Sandbox type Malware Detection

Allows the firewalls to identify zero-day Malware by opening files within a sandbox environment and analyzing their behavior.

URL Filtering, content-based Allows the firewalls to selectively block access to certain categories of web sites.

URL Filtering, white/black list Allows the firewalls to block and unblock specific web sites.

User-based policy Allows application of policy based on users privileges & application usage. E.g. Only allow marketing to use Facebook posting.

Selective Bypass capability Allows the engineer to selectively bypass the firewalls to handle exceptions or facilitate troubleshooting.

29

Deployment Strategy and Approach

• The firewalls shall not constrain or disrupt legitimate

science activities and operations related to the SLAC

mission.

• The firewall deployment introduces no Single Points of

Failure in SLAC Core Network.

• Enable the firewall protection without adverse impact

on users' computing experience.• As much testing as possible will be done in the staging phase

• Rollout will be gradual: a small group of testers first, then the Computing

Division building, then the entire site

• Each iteration includes time to correct issues and provides assurance

• This approach is reflected in the project deployment schedule

30

Firewall Project Status

• The firewall vendor has been selected: Palo Alto

Networks

• Purchased:• A pair of PAN 5060 firewalls

• Threat Protection

• Wildfire (sandboxing)

• URL filtering

• The firewalls are being configured and tested in a test

bed network

Network Security – Network Intelligence and Forensics

Dr. Yee-Ting Li, network engineer

Bro

32

Goals

• Design and implement scalable cyber infrastructure to

monitor and alert cyber incidents (network taps etc.)• Complement Next-Gen Firewall activities

• Firewall will not protect scientific computing network

• Implement Bro cluster to monitor science traffic

• Evaluate and scope the need for full/partial packet captures

• Help drive meaningful policies based on empirical

information from Bro and network monitoring system(s)

33

Where does Bro come from?

• International Computer Science Institute (ICSI) / UC

Berkeley / LBNL• Developed >15 years ago at LBL

• Open Source (BSD)• Funded mostly by R&D grants (NSF, DoE)

• Large online community and annual conference

• Used at major universities, labs, supercomputing

centers, corporations, researchers

Sponsors

34

What is Bro?

• Clustered software that runs on *nix

• Takes in a network ‘feed’• Does ‘super’ tcpdump

• Can feed in syslogs (or any other streaming log)

• Analyses the packets ‘live’ using ‘bro scripts’ • Uses a security domain specific scripting language

• Bro scripts effectively notify Bro that:- should there be an event of a type we define,

- then let us have the information about the connection

- so we can perform some function on it.

• Can write our own!

• Creates log files from `bro scripts’

35

Why do we need Bro?

• Many cyber attacks per day, constant change• Spam, phishing, brute force, DoS, virus, Trojans, worms etc.

• Traditional IDS/IPS very costly (or do not exist) at high network

speeds/volumes• SLAC plans to deploy 100Gbps network for science

• Commercial IDS/IPS products cost prohibitive at such speeds

• Bro scales horizontally by adding more commodity servers• Requires frontend ‘load-balancing’ solution

• Out-of-the-box will only monitor traffic• Transparent to users

• Necessary to help protect SLAC science• Not protected by Next Gen Firewall

36

What can Bro do?

• Provide audit trail for forensic analysis• Who else did a host infect?

• When was a machine last seen on the network?

• Who’s trying (and succeeding?) to attack us?

• What types of attacks are we seeing?

• Provides insight into what’s happening on the net• Ensure only science applications are installed on science networks

- Top 10’s of application traffic volume, top talkers

• Plus anything you can think of via Bro Scripts• Take action based on Bro alerts

• e.g. send email; add a null route for suspicious IPs etc.

37

Value Proposition of Bro

• Monitor and alert on network activity ‘as it happens’• Most of our systems are ‘near real-time’ (scans etc.)

• Provide insight into application level data• Netflow only provides ‘meta’ data

• Customizable to internal procedures and processes• e.g. use router block as opposed to having inline IDS

• $ per GB of traffic analysis very cost effective

• Plenty of community support and direct application

to our (lab) environment

• Easy to create reports:• Top application traffic

• Unpatched software

38

Bro at SLAC Implementation

SPAN or Optical split

High performance compute & data clusters

for science

Front end Bro switche.g. Cisco 3172TQ or Arista 7150

Bro Clustere.g. Del R620

Packet CaptureNetwork: 1x40Gbpsor 2* 10Gbps

Monitoring network2x 10Gbps Cu/host

Time machine, e.g. Dell R710 + 16TByte storage

1 dedicated manager/proxy

4TB storage

Currently 6 Bro instances

(scalable)

Gen purpose net:Patching, ssh etc1Gbp/host

39

Time Machine Packet Capture

• Keep copies of data packets on SLAC network• tcpdump with snaplen of the entire packet

• Exclude storing ‘uninteresting’ data• ISO images, encrypted traffic etc.

• Useful for forensics and replay

• Full system with decent retention very expensive

• Will repurpose existing hardware as trial• 16TB storage

• Target important networks

• Defense in depth

Summary

41

Summary: Mobility

• Secure Wireless

• MDM – AirWatch design

APs

2 WirelessControllersCisco 5508

AD

Radius

Kerberos

42

IP Adress management, DHCP

• IPAM –NetDB features and

Capabilities

• IPAM –NetDB data aggregation

43

NGFW & network intelligence

• Border Firewall – features and

deployment strategy

• Network Intelligence - Bro

design

44

The end: Questions?

Contacts:Network manager:

cottrell@slac.stanford.edu,

PM:

hadas@slac.stanford.edu

CSIO:

ben.calvert@slac.stanford.edu

SLAC National Accelerator Lab,

2575 Sand Hill Road

Menlo Park, CA 94025

Slides available at:https://confluence.slac.stanford.edu/download/attachments/17164/nlit-2014-net-csip.pptx