Cisco IT – Identity Services Engine (ISE) Deployment

and Best PracticesBassem Khalifé, Cisco IT

CCSSEC-2002

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

Cisco IT ISE Global Deployment

Cisco IT ISE Global Deployment (WLAN, LAN)

ISE PSNs Data Center (8) Network Devices Cluster (800+) Auth traffic to ISE PSNs

6,379 3,583

2,232

2,107

Products

•ISE• CITEIS

• Cisco Prime Infrastructure

• Webex

• Jabber

• Splunk

Practice

•Motivation

•Attitude

•Knowledge

•Experience

•Skills

Process

• Product Life Cycle

• Operational Excellence

• Fast IT, Continuous Delivery

• Change Management

• Agile

People

The four P’s

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

Capability Major Technical

Outcome

Major Business

Outcome

Secure Guest Network

ION (Internet Only Network)

Simplified single secure

platform (reduce device/server

footprint from 28 to 8)

• High availability

• Secure offering for guests,

partners, and employees

802.1x Auth: WLAN, CVO*,

LAN, and VPN

Complete visibility and control of

devices connecting to the

network

• One scalable policy

enforcement environment

Consistent Assured Network

Access

Scalable enterprise secure

network

• Enhanced Risk Management

• Consistent User Experience

• Improved Operations

*CVO is Cisco Virtual Office, for small office/home office

IT RequirementsIT set out to deliver multiple capabilities with ISE

Access Control

Authentication on

wired & wireless

networks

BYOD

Support Trusted

Device Standard

and enable BYOD

Profiling

Ability to

identify users

and devices on

our network

Endpoint

Protection

Protect the

network

from

infected

devices

Guest Access

Restrict

unauthorized

devices & users to

Internet access

only

Device Control

Secure network

while allowing

mobile device

access*

*Cisco uses a 3rd party MDM and connects to ISE

• Identity of a device on the network

• Quantify the risk

1. Profiling

• User and end device attribution

• Identification of end points on Wireless connections

2. Authentication• Device security

posture identification

• Allows for better policy & security decisions

3. Posture

• Ability to enforce policy decisions based on context

• Untrusted devices have restricted access

4. Enforcement

The Four Stages of a Secure Network

ISE 1.2 Profiling

ISE 1.2 802.1X Auth

WLAN, CVO

ISE 1.3/1.4 802.1X Auth

CVO, Wired, VPN, MDM

ISE 1.4/2.0802.1X Wired Auth Mode

MDM√ √

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

“However beautiful the strategy, you should occasionally look at the results.”

Sir Winston Churchill

• Avoid the “Big Bang”• Too many new capabilities to enable in a single deployment.

• “ISE Deployment Bundle” model• Capabilities have been grouped into bundles to enable targeted & manageable deployments

• Multiple clusters consolidated• Pros and cons of single vs. distributed: ISE Limits, Scalability, # EP, Auth, Latency, AD…

• “Start with one cluster and add more if necessary”

• Global Infrastructure Foundation• Deploy global VM infrastructure and ISE servers first

• Guest Network (ION) enabled on a separate deployment to reduce risk

• Enable features (based on “ISE Deployment Bundles”) theatre by theatre

• Use different Virtual IPs by service (e.g., WLAN, LAN, CVO, VPN) for better manageability and ease/speed of control

Cisco IT Deployment Strategy

Cross Functional Ownership for Execution

COO

CTO

SVP IT

VP IT

Any Device Team

SVP

Infra Services

Sr. Dir

Network Services

VP

Ops/Impl

Sr. Dir

Strategy & Security

Security Services

Directory Services

Sr. Dir

Data Centers

Sr. Dir

Arch/Design

SVP

Security & Trust

VP

InfoSec

Security

Requirements

and Policy

Owns Mobile

Devices,

Responsible for

Posture

Enforcement

Provides DC

and Virtual

Infrastructure

Owns and

Manages the

Deployment of

NW Services

Owns and

Operates the

NW

Infrastructure

My team, owns

the ISE infra

and enables

security

services

Owns Active

Directory Infra

and Services

High Level

Architecture

and Design

Operational

Excellence:

99.999%

Availability

Resources for Operations + Resources for Deployment

2 Sr. Engineers + 2 Support Engineers (multi-zone)

• Performance monitoring and tuning, scalability for growth

• HW/SW issues

• Troubleshooting, field issues

• Provisioning of Network Devices, and Users support

• Monitoring and Reporting (ISE reports and Splunk)

• Patch/Upgrade

• Infra/VM issues, Change Request support

• Policy Management

2 Sr. Engineers, 1 Analyst, 1 PM

• Learning and Testing new capabilities

• Service Verification Lab testing and certification support

• Automation of new operational activities

• Limited Availability validation of new features

• Product and platform bug identification and validation

• Data analysis and scalability for new capabilities

• Acquiring knowledge of new capabilities & cross functional environment support

• Documentation

Sample ISE Basic Deployment Roadmap (6 quarters)CY14 Q3 CY14 Q4 CY15 Q1 CY15 Q2 CY15 Q3 CY15 Q4

Fine tune Optimize

Foundation ISE 1.2

Install

ISE 1.3

Upgrade

ISE 1.4

Upgrade

Infra

Design, Proof of Concepts, Data Analysis

Apply

patches

Fine tune Optimize

Network

Guest

Wireless

Monitor

Endpoint Analysis: Wired dot1x MM & Profiling

VPN

Wired

802.1x Authentication

Guest Access

Wireless (WLAN) Auth Deployment

CVO (Home Office) Wireless Auth

VPN Auth

CVO Wired Auth

Limited Sites Wired Auth

Global Wired Auth Deployment

Quarantine/RemediationPosture Assessment & Enforcement (MDM)

Security Group Tagging (SGT)

Advanced Capabilities

Deployment Readiness

Design Engineer Personal Lab

Service Verification Lab

Stage & Pilot

Deploy!

• Production Infrastructure• Network Access ISE 1.3, P2 24 VM servers in one global deployment

• Guest Services ISE 1.2, P13 8 VM servers in one dedicated deployment

• Services• Guest services (ION) (400+ sites, potential 130K+ users & 14K guests per week)

• Internet Only Network access requires pre-registration via ISE guest portal for all users; CWA (central web auth)

• 802.1X Wireless Auth Mode (400+ wlan sites, 90K+ users, ~150K endpoints)

• 802.1X Auth CVO (wireless/wired) (27K Network Devices for home access; ~60K endpoints)

• 802.1X Wired Monitor Mode* (3.5K LAN Switches and Gateways, ~200K endpoints)

• Wireless Policy Enforcement (2 Extranet Partner sites in pilot mode)

• To-date: ~650K Profiled Endpoints; Max of 60K+ Concurrent

Cisco IT ISE Production Deployment Metrics

* Wired Auth deployment currently at 600 NADs

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

Original Design for Multiple ISE Deployments

Guest Global

Services

TYO

BGL

AER

RTPALNMTV

HKG

SNG

Secondary ISE PAN/M&T

ISE PSN

Primary ISE PAN/M&T

Primary Secondary Admin Nodes

EIC (6)

MTV AERAll-in-one

Single Global ISE 1.3 Deployment (WLAN, CVO, LAN, VPN)

AER

RTP

ALN

MTV

SNG

Secondary ISE PAN/M&T

ISE PSN

Primary ISE PAN/M&T

Global Deployment: 24 ISE Nodes

20 PSNs; 8 DC (Node Groups)

TYO

HKG

BGL

18,362

9,961

23,969 26,070

32,651

28,124

12,870

5,317

32,856

14,765

40,995

37,481

58,846

51,878

21,384

9,445

-

10,000

20,000

30,000

40,000

50,000

60,000

70,000

AER ALLN BGL HKG MTV RTP SNG TYO

Users

Endpoints/MAC

Users/Endpoints by Node Group

Guestnet Original Deployment

MTV

Sponsor

Portal

(GSS)

internet.cisco.com

Guest Account

Creation

Wireless access

Wired access

NADs AMER

Guest Portal Auth

VMS

Tool

Lobby

Ambassadors

Guest Account

Creation

OEAP

OEAP Device

Provisioning

Script + Store

Wireless access

Wired access

NADs EMEA/APJC

Guest Portal Auth

AER

Before MTV AERAll-in-one

Primary

SecondarySingle Point of Failure

All services will be affected, and

likely to also impact the

secondary node.

Guestnet Redesigned Deployment

MTV

Sponsor

Portal

(GSS)

internet.cisco.com

Guest Account

Creation

Wireless access

Wired access

NADs AMER

Guest Portal Auth

VMS

Tool

Lobby

Ambassadors

Guest Account

Creation

OEAP

OEAP Device

Provisioning

Script + Store

NAD

Configuration

And GSS By

Geo Proximity

Pri

mary

ion-mtv-guest ion-aer-guest

ion-aer-sponsorion-mtv-sponsor

Wireless access

Wired access

NADs EMEA/APJC

Guest Portal Auth

Pri

mary

AER

PPAN Alias

PAN

PAN MnT

MnT

PSN PSN PSN PSN

Primary

Secondary

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

ISE Deployment Ecosystem: Building Blocks

ISE(Logical Layer)

ISE (Physical Layer) : ISE Appliance OR VM (Fabric, Compute, Storage)

Network: DNS, NTP, SFTP, UDP, TCP, (& Load Balancers)

Network Access

Devices

Endpoints: Devices,

Users & Supplicants

Enterprise Monitoring: HTTP(S), RADIUS, PEAP, EAP-FAST

User

Provisioning

Mobile Device

Management

Network

Device

Provisioning

ISE Policy

Management

Active

Directory

Call Manager100’s K

10’s K

1 PAN

X PSN

Data

Analysis

(Syslog)

• Trained support team with broad knowledge of environment, across multi time zones

• Troubleshooting using both ISE and Splunk

• Enterprise monitors, load balancer probes, plus other monitors

• Reporting and/or Alerting, covering the ecosystem:• Number of Devices, Endpoints, Auth, failed/passed, by region, device type…

• Profiled devices by group, analysis and validation of profiling results (if used in policy)

• Measure progress of deployment based on pre-migration data

• Splunk ISE app, dashboards providing detailed usage

• Infra utilization, and alerting on CPU, Disk, Memory

Operations Maintenance & Monitoring

Scaling ISE for large scale distributed deployments

• Don’t let replication or misconfiguration become an issue for authentication:

• Tuning the “deployment” (ISE, NADs, and Endpoints)

• RADIUS Accounting

• Profiling

• Authentication(s)

• Latency & Distributed Replication

• Failover & Redundancy

• Tuning the “environment”

• Load Balancers

• Active Directory

Cisco IT Deployment Challenges

Replication Challenges

Iceberg (īs’bûrg’)

(Former state)

ISE Burj

(Current state)

Replication

Issues

Profiling

Error notification

Load Balancer misbehavior

NAD misbehavior

Endpoint misbehavior

Latency

Radius accounting

SNMP errorsMisconfiguration

One change can have huge implications (+ve/-ve):

1. Enabling SNMP and RADIUS Profiling while in an “Accounting Storm”

2. Increasing or decreasing the value of the RADIUS Accounting Suppression

3. Upgrading NADs to recommended OS version

4. Synchronizing timeout settings between NADs and Load Balancers

5. Introducing new “settings” in the product

Snapshot of Key Challenges & Enhancements

Item Owner Impact

Configure ACE for

accounting “stickiness”

Cisco IT High – reduced

accounting traffic

from 6M to 3M

txns per day

Implement eng fix to

enable accounting

suppression

SAMPG High – further

reduction in

accounting traffic

Remove “IP” as a

significant attribute

SAMPG(design

change)

High – removed

traffic from “noisy”

endpoints

Implement WLC OS

updates to fix duplicate

accounting issue

Cisco IT High – reduce

traffic from

wireless network

accounting txns

Implement eng fix for

SNMP polling

SAMPG High – reduce # of

SNMP traffic to

enable CVO

Octo

be

r

September November

Tune the Deployment and the Environment

Configuration

Changes:

NAD and ACE

(LB)

Accounting

Suppression

Fix:

CSCur42723

Removing IP

as Significant

Attribute Fix:

CSCur44879

ISE Global Deployment Profiling Setting PSN Configuration

27K CVO Network Devices configured under 29 subnets

in ISE. SNMP polling is disabled using the new option

“zero” as the devices were not always connected,

resulting in timeout errors (600K), affecting replication.

CSCur95329

Simple fix; Great

value!

Explicitly choose

the polling PSN

Misbehaving Supplicants

EndPoint Profile Auths Per DayCount of

EndPoints% of Total EndPoints

Avg Auths Per Endpoint

Windows7-Workstation 98,394 25,918 20.99 3.8

Apple-iPhone 745,807 17,820 14.43 41.85

Microsoft-Workstation 69,216 16,469 13.34 4.2

Apple-Device 67,167 8,720 7.06 7.7

Workstation 49,834 8,408 6.81 5.93

Android 115,839 5,160 4.18 22.45

OS_X_Mavericks-Workstation 17,529 4,644 3.76 3.77

OS_X_Yosemite-Workstation 17,718 4,276 3.46 4.14

Apple-iDevice 97,862 3,813 3.09 25.67

Android-Samsung-Galaxy-Phone 78,539 3,146 2.55 24.96

Android-Samsung 39,250 3,132 2.54 12.53

Apple-MacBook 14,014 2,883 2.34 4.86

Android-Motorola 70,695 2,226 1.8 31.76

Android-Google 44,835 1,761 1.43 25.46

Wireless auth over 24 hours.

1

3

2

4

ISE Authentication Storm/Meltdown From a “Simple” Change

Network Access

Devices

Endpoints: Devices,

Users & Supplicants

Data

Analysis

(Syslog)

Call Manager

IP Phones

accidentally

enabled for

802.1x auth

1,600 IP Phones

started MAB/dot1x

auth frenzy: 2,600

attempts per day,

per phone= ~4M

Luckily only ~4M auth requests per day due to

limited deployment of a dozen switches. Target scope is 100K IP Phones = 250M Auth

Detection &

troubleshooting

ISE

(Logical Layer)

ISE

(Logical Layer)

Large Deployments – Bandwidth and Latency

PSN PSN

PAN MnT MnTPAN

PSN PSN

PSN

PSN PSN

PSN

PSN PSN

PSN PSN

PSN PSN

PSN PSN

PSN PSN

PSN PSN

200ms

Max round-trip

(RT) latency

between any two

nodes in ISE

1.2/1.3

• Bandwidth most critical between:

• PSNs and Primary PAN (DB Replication)

• PSNs and MnT (Audit Logging)

• Latency most critical between PSNs and Primary PAN.

• RADIUS generally requires much less bandwidth and is more

tolerant of higher latencies – Actual requirements based on

many factors including # endpoints, auth rate and protocols

WLC Switch

RADIUS

Latency Resolution Options

AER

169msALN RTP

TYO

134ms

SNG

186ms

HKG

154ms

MTV

BGL

219ms

45ms

AER

169msALN RTP

TYO

134ms

SNG

186ms

HKG

154ms

MTV

BGL

219m

sX

Option 1

Option 2

+ 45ms

- 45ms

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

• Acquire ISE expertise upfront; invest in design

• Fine tune deployment and environment, they must work in tandem

• People, Process, Practice, and Products will derive success – or not

• Follow BU guidelines; will cover 80% of the known challenges• ISE, NADs, Load Balancer configuration

• Requirements from AD, and DNS,

• Profiling configuration

• Bandwidth Calculator

• Listen to the data, and the alerts/alarms

• Do not take the network for granted

• Plan for the unexpected!

Lessons Learned

Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

• Your favorite speaker’s Twitter handle @BJKhalife

• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could Be a Winner

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

• Introduction

• Security Requirements

• Deployment Strategy

• Design Changes

• Key Challenges

• Learning

• Q&A

Agenda

Hardware CPUs Memory Storage TypeDisk Space

(PAN)

Disk Space

(MnT)

Disk Space

(PSN)

NIC

Speed/Count

Recommended

Min 4 x 2.4 GHz 16 GBDedicated

SAN200 GB 200 GB 100 GB

4 x Integrated

Gigabit NICs

Cisco IT PROD

8 x 2.032 GHz

(Gold

Reservation)

32 GB

Dedicated

SAN (except

HKG/TYO)

600 GB600 GB (1 TB

for tftp)200 GB 1 x 1 GB NIC

NAD Type Profiling Probes

WAN-GW, BB-GW’s DHCP Probe

- ip helpers on VLANs

3750 – LAN Switch

3850 (3.6.2a IOS XE) – LAN Switch

4510

RADIUS Probe

- MAC OUI

- Device Sensor

- ip dhcp snooping

- ip device tracking

SNMP Query Probe

- CDP information

3850 (3.3.4 IOS XE) – LAN Switch

6500

CVO

RADIUS Probe

- MAC OUI

- ip dhcp snooping

- ip device tracking

SNMP Query Probe

- CDP information

WLC/WiSM - Device sensor (DHCP only in 7.2.110 and HTTP in 7.3)

- Disable DHCP proxy (until device sensor available)

WLC

Switch

PSN

PSN

PSN

ALLN

isealln-prd-wlan

isealln-prd-lan

PSN

PSN

PSN

MTVisemtv-prd-wlan

isemtv-prd-lan

ACE Load

Balancer

ACE Load

Balancer

WLC

PSN

PSN

PSN

isemtv-prd-wlan

MTV

isertp-prd-wlanPSN

PSN

PSNRTP

isealln-prd-wlan PSN

PSN

PSNALLN

acs-mtv-p1-1-l

acs-mtv-p1-3-l

acs-mtv-prd

MTV

acs-aln-p1-2-l

acs-aln-p1-5-l

acs-alln-prd

ALLN

Testing High Availability When 1 DC Fails (AER => RTP)

Sample Endpoint Tracking For WLC Migration From ACS to ISE

Thank you