Inside Cisco ITIPv4 Address Exhaustion &IPv6 Progress across Cisco

Khalid Jawaid – Member of Technical Staff

CCIE 6765

Inside Cisco IT Sessions

12 Sessions in Total Plus Lunch & Learn

COCEWN-2014 : Inside Cisco IT : Next Gen Enterprise

Network (Tuesday, Feb 16th @ 1645 Hrs)

LALCOC-0002 : Inside Cisco IT : Next Gen Enterprise Network

(Wednesday, Feb 17th @ 1300 Hrs – Catering Hall 4.1)

3

• Booth Number: C3 (just to the left of the Cisco Campus in Hall 4.2)

• Speak with our subject matter experts, sharing their real-world experience using and deploying Cisco technologies in our own environment.

The Cisco on Cisco Booth

Live Demos…

• Internet of Things: In the Workplace

• Network Infrastructure Security

• Application Centric Infrastructure (ACI)

• Collaboration & Video

Session Abstract

This session is about how Cisco has dealt with the IPv4 addressexhaustion problem and coupled that with IPv6 deployment across ourglobal network. We will talk about the challenges and solutions, thecompromises and frustrations we faced and hopefully benefit yourplans in some way.

This is a developing Cisco on Cisco story of IP continuity and growth.

5

• Cisco Network Overview

• IPv4 Address Exhaustion

• Defining Address Exhaustion

• Planning

• Implementation

• Challenges and Expectations

• Lessons Learnt

Agenda

• IPv6 Deployment & Plans

• Introduction & History

• Preparation and Planning

• Use Cases and Deployment Status

• Lessons Learnt

• Future Plans

• Hindsight……What if I started over now!

Network Overview

7

Cisco IT Network – People and Places

The Extended Cisco Family• 369 locations in 90 countries

• 450+ buildings

• 200,000 Sq Ft of DC space

• 1500+ labs worldwide (500+ in San Jose)

• 66,000+ employees

• Channel Partners, ASPs, Business Dev

Partners

8

5 Million IP Endpoints

Network Tiers & Services

Tier 4

Tier 3

Tier 2

Tier 1

Tier 0

Tier 3 Home & MobileAnyConnect

3C

Office

Extend AP

3B

Cisco Virtual

Office

3A25,000+ users

Tier 2 Branch Offices Business

Essential

2C

Business

Ready

2B

Business

Professional

2A

Ultra-

Resilient

Branch 2A+369 branch offices

Tier 1 Global Backbone Tier 1A

Agg+ Internet

+ DC

Tier 1B

Agg +

Internet20 hubs

Tier 1C

Aggregation

only

Tier 0 Data CentresProd

5 data centres/public SaaSNonProd Engineering Cloud

Tier 4 Extranet Partners

500+Manufacturing Off-Shore DC TAC

9

COCEWN-2014 : Inside Cisco IT : Next Gen

Enterprise Network (Tuesday, Feb 16th @ 1645 Hrs)

Tier 1 WAN & Internet PoPs

NY

RichardsonHawthorne

San Jose/

Redwood CtyDenver

Kanata

Chicago

RTP

LondonAmsterdam

Bangalore/

Chennai

Hong Kong

ShanghaiTokyo

Singapore

Sydney

Orlando

Sao Paulo

Lawrenceville Hawthorne

Hawthorne

San Jose

San Jose

Hong Kong

Bangalore

Tokyo

Sydney

10 Gb/s

2.5Gb/s (OC48)

622Mb/s (OC12)

155Mb/s (OC3)

Tier 1a Transit Node

Production DC

Internet

Tier 1b Non-Transit Node10

EuropeN. AmericaASIAPAC S. America Middle East

TOKYO

SYDNEY

SHANGHAI

BANGALORE

SINGAPORE

HONG KONG

SAN JOSE

UKI

BXB

RTP

RCDN

AMS

11

Cisco Global Internet Presence

IPv4 Address Exhaustion

12

• Defining Address Exhaustion

• Planning

• Implementation

• Challenges and Expectations

• Lessons Learnt

Address Exhaustion Agenda

13

What is Address Exhaustion?

14

IPv4 address exhaustion is the depletion of the pool of unallocated Internet Protocol Version 4 (IPv4) addresses, which has been anticipated since the late 1980s. This depletion is the reason for the development and deployment of its successor protocol, IPv6

Source …. Wikipedia

"Exhaustion" is defined here as the time when the pool of available addresses in each RIR reaches the threshold of no more general use allocations of IPv4 addresses.

Source …. www.potaroo.net

Unadvertised Space – Address Grey Market?

15

Source …. www.potaroo.net

@ $8-$11 an address

Maybe not all available to the market?

It boils down to this …..

16

vs

Doesn’t work for

RFC1918 though

• Short lived investment

• Delays resolution of the core issue (address exhaustion)

• Does not solve RFC1918 exhaustion

• Internal recovery is more viable and cost effective (approx. $3/address)

• We had enough to re-position

• Cost of buying IPv4 will fund IPv6 deployment

Our Reasons for not buying IPv4

17

理由

Planning

18

The Unified Long Term Plan

19

1. Capacity (Public address space available)

2. Recovery / Reclamation

3. 5 Year Demand + 10%

4. Policy & IPv4 Life Span

5. Feed into IPv6 Roadmap

Capacity (Public Address Space)

• Total Capacity 1.12M Addresses

• Fragmented 21%

• Free (Usable) 18%

• Subnet Efficiency 45%

• Desktops 22%

20

DC/iPoP

39%

Free

18%

Fragmented

21%Desktop

22%

Recovery / Reclamation

Fragmented 21%

Free (Usable) 18%

Desktops 22%

21

Recover

Defragment

336K Addresses

3 MillionBUY @ 10

1 MillionDIY @ 3

2 Million

5 Year Demand + 10%

22

YoY Requirements

DC

iPoP/DMZ

Public Cloud

Approximately /16 per Year (Total 320K )

Policy & IPv4 Life Span

23

Recovery IPv4 Lifespan

Estimated 5 Years

Addressing Policy Changes

Extend IPv4 Lifespan

Improve process

2018 201920172016

IPv6 Roadmap (IPv6 only and IPv4aaS)

2015

Program development

(Identify

opportunities, Gaps,

BU Relationship, IPv6

HIP team)

IPv6 Translation Technologies

Design Development and

Deployment

( SLB46, SLB66, NAT464)

Training (Application Teams)

Infrastructure Regional Extranet Tunnel Head

ends

IPv6 Only Pilot

(IPv4aaS Offering)

App Testing / Development Start. Migration ongoing

ACI IPv6 Only Mandate for Greenfield Workloads

ACI IPv6 Only

Mandate for

Migrations (CITEIS to

ACI)

IPv6 Only (Remove Dual Stack – Pilot)

IPv4 Public Address recovery

(or reduced demand)

Critical Dependencies Review

24

Implementation

25

• RFC1918 for Desktops and Infrastructure links

• RIR style scrutiny of new requests (internal)

• IPv4 resources tied to the PLC (Project Life Cycle)

• Non-Routable (internally) RFC1918 Lab block (/12)

• Garbage Collection (actively seeking out unused blocks)

IPv4 Addressing Policy Changes

26

Post Policy Change TrendIPv4 Addressing Capacity (Policy changed - Jan 2010)

0

20

40

60

80

100

120

140

160

180

200

Jan-10 Jan-11 Jan-12 Jan-13 Jan-14 Jan-15 Jan-16

Perc

enta

ge IP

v4 U

tilized

1

1 RFC1918 on Desktop

2

2Reclamation Phase 1

Stricter Policy

3

3 Reclamation Phase 2

Challenges and Expectations

28

Address Consumption Challenges

• Acquisitions

• No RFC1918 for DCs (overlap with acquisitions)

• Virtualization

• Migrations / New Technologies

• RFC1918 for Labs (Insatiable!)

• No process to supervise or ration IPv4 resources

• Garbage collection / Permissions / Management

Data Centres and

Labs are the biggest

consumers of IPv4

Address Space

• Organic growth

• Lack of Knowledge (Support staff)

• Tooling

• Enough contiguous RFC1918 to re-address publicly addressed desktops

• Re-addressing

• De-fragmentation

Support Challenges

30

• Enough Public IPv4 Address Space to see us to IPv4 Sunset / IPv4aaS

• Overtime and with IPv6 only in place, IPv4 demands will be reversed utilisation trending downwards

• Reduced OPEX as we move towards IPv6 only

• Increased stability across core and distribution (IPv6 Summaries)

Expectations

31

Lessons

32

• Organic growth – Addressing upkeep must be an operational task to keep summaries clean due to organic growth.

• Knowledge Transfer – Constant ongoing training through support staff churn to ensure addressing is understood and carved properly

Lessons Learnt

33

• Address Management Tools – Manage permissions at the right hierarchy for the right staff, Integrated automation for garbage collection and clean up, subnet efficiency monitoring, rules based assignment (policy compliance checking)

• Automated Capacity Planning – A must have on a recurring basis to highlight trending and identify areas of concern in advance

Lessons Learnt

34

Re-addressing IPv4

35

By Jonas Ekman Licensed Creative Commons Attribution.

IPv6 Deployment and Plans

37

• Introduction & History

• Preparation and Planning

• Use Cases and Deployment Status

• Lessons Learnt

• Future Plans

IPv6 Agenda

38

Cisco IT’s IPv6 Target State

IPv6 Internet Presence• Internet Evolution

• Business Continuity

• Customers, partners,

employees IPv6 Internet

Dual-Stack Enterprise

Ubiquitous IPv6 Access• Globalization

• Technology Leadership

• Product Development

34

The IPv6 Journey – A High Level ViewIPv4-only IPv4 and IPv6 co-exist IPv6-only

2015201420132012201120102002-2009

Ubiquitous IPv6 Access (Inside-Out)On-demand tunnel services

Dual stack “alpha” networks

Dual stack global core

Resilient tunnel services

Dual stack user

access (pilot)

Dual stack user access (prod)

Dual stack internal DC and apps

IPv6 Internet Presence (Outside-In)www.ipv6.cisco.com www.cisco.com

accessible over IPv6

Entire cisco.com platform

accessible over IPv6

35

IPv6

Making the Case for IPv6

Business DriversLeadership and Mindshare

Product Readiness

Internet Evolution

IT DriversProduct Development and Testing

Continuity and Growth

Cisco On Cisco

ConstraintsMaintain IPv4 SLA & Security Posture

Funding & Resourcing

Product & Service Gaps

GoalsIPv6 Internet Presence

Ubiquitous IPv6 Access

36

Preparation and Planning

42

Preparation

Cross

Functional

Collaboration

Assessment Implementation

Strategy &

Plan

38

Architect & Design

Address Planning

Address Planning

• Establish IPv6 Addressing policy

• Hierarchical Model – Global, Regional, Sub-Regional and Site levels

• 50% Sparing at all levels of the Hierarchy

• Functional and Regional Assignment

• Template-based addressing - easy for Implementation and Operations Teams

• Address management tool support for IPv6

39

/34 Global Level

(50% spares)

/35 - /36

per Region/37 - /39

per Sub-Region

/40 per Campus

(256 Buildings)

/48 per Building/Branch

(16 PINs per Building/Branch)

* PIN = Place In the NetworkA framework to classify functional areas of the network

eg, Lab, Desktop, DC, DMZ etc

/52 per PIN *

(4096 Subnets / PIN)

Address PlanningHierarchy (Breaking up our /32)

FIXED TEMPLATE VARIABLE

45

/52/48 /64

Subnets / PIN

(4096)

2001:0420:028C:1000::/52 - Desktop PIN

2001:0420:028C:1300::/64 – Desktop VLAN 300

2001:0420:028C:1301::/64 – Desktop VLAN 301

2001:0420:028C:2000::/52 - Lab PIN2001:0420:028C:2001::/64 – Lab Subnet 1

2001:0420:028C:2002::/64 – Lab Subnet 2

Address PlanningTemplate Addressing

41

PIN

(16)

0 = Infra

1 = Desktop / Wireless

2 = Lab

3 = Guest

4 = DMZ

D = Building DC

... etc

(13th Nibble) Functional Identifier

Building/Branch

Regional Identifier

Address Management

Address Planning

42

IPv6 Address Plan (Top Level)

Global 2001:420::/32

Americas 2001:0420::/34

EMEA and Asia Pacific 2001:0420:4000::/34

Global Spare1 2001:0420:8000::/34

Global Spare2 2001:0420:C000::/34

Global Infrastructure 2001:0420:C000::/42

Global Mobility 2001:0420:C040::/42

EuropeN. AmericaASIAPAC S. America Middle East

TOKYO

SYDNEY

SHANGHAI

BANGALORE

SINGAPORE

HONG KONG

San Jose

UKI

Boxborough

RTP

Richardson

Amsterdam

49

Cisco Global Internet Presence IPv6 Advertisements(2001:420::/32) 1000::/40

1100::/41

C0CC::/46

::/32

2000::/35

2C48::/45

C0E4::/46

4000::/36

C0C0::/46

4000::/36

4000::/38

C0F0::/46

4000::/34

5E00::/39

C0DC::/46

4000::/34

5800::/39

5A00::/39

C0D8::/46

4000::/34

5C00::/39

C0D4::/46

4000::/34

5200::/39

5400::/39

C0E0::/46

4000::/34

5000::/39

C0D0::/46

Use Cases & Deployment Status

50

Building / Lab = Manual 6in4 tunnels (Retired)

User = Anycast ISATAP (Retired)

Global tunnel infrastructure (Retired)

IPv6 Tunnel

Overlay

Ubiquitous IPv6 AccessShort Term Plan – Tunnel Infrastructure (Retired)

Long Term Plan – Native IPv6 Everywhere

44

SLA same as IPv4

Dual stacked core

Ubiquitous IPv6 Access• Dual Stack Deployment Status

100%

Offices/Labs/ Anyconnect, External Email, DC**

Complete

Future

* Pilot

** Upto DCC GW (Not on VLANs)

92%

DMZ

95%

DNS

8%

CVO*

0%

Extranet

0%

CITEIS

Migrate to ACI

47

53

Cisco’s IPv6 Web Presence• Architecture for www.cisco.com

Model 3 – Dual Stack

Web Servers

IPv6 IPv4

Cisco.com Web Servers

Server Load Balancer

DMZ Network, Security

Database

App Platforms

Data Centre Network

Internet

Svc A

ssu

ran

ce

Middleware

ContentIdM, Authz

AKAMAI

ww

w.c

isco.c

om

ww

w.c

isco.c

om

Model 2 – SLB64

Cisco.com Web Servers

Server Load Balancer (ACE)

DMZ Network, Security

Database

App Platforms

Data Centre NetworkS

vc

Assu

ran

ce

Svc

Assu

ran

ce

Middleware

ContentIdM, Authz

AKAMAI

IPv6 IPv4Internet

ww

w.c

isco.c

om

ww

w.c

isco.c

om

Model 1 – 6to4 Proxy

at Internet Edge

Cisco.com Web Servers

Server Load Balancer (ACE)

DMZ Network, Security, Proxy

Database

App Platforms

Data Centre Network

Svc

Assu

ran

ce

Middleware

ContentIdM, Authz

AKAMAI

IPv6 IPv4Internet

ww

w.c

isco.c

om

ww

w.c

isco.c

om

IPv6 Web Presence

• External - Cisco.com, apps behind and subdomains

• Internal – Intranet, apps and Intranet subdomains

80%

External

0%

Internal

Complete

Future

46

Lessons Learnt

55

Lessons Learned – Creating the IPv6 Program

• Making the case

• Business case for IPv6 internet presence is simpler to articulate

• Business case for IPv6 on internal corporate network may be more difficult to justify

• Cross functional effort across the IT Stack

• Starts with networking team taking the lead

• Early engagement of security team, infrastructure and application teams follow

• Early planning is key

• Absorb the IPv6 effort into existing network lifecycle management process

• Hardware upgrades

• Software image upgrades

• Configuration (automate where you can)

49

Lessons Learned – Product Support

• Network hardware, software, functionality

• Routers, server load balancers

• Wireless, switches

• Network management and service assurance

• External and internal availability and performance monitoring

• Security

• Firewalls, IDS/IPS, security event management and forensics logging

• 3rd Party

• Tooling

50

Lessons Learned - Security

• The goal is security parity with IPv4

• User attribution (IPv6-to-MAC binding), custom Internal tools, third party vendors, incident response playbook, firewalls, anomaly detection, netflow, IDS, log data, pen testing, transparent proxy with anti-malware

• Opportunities to improve security as IPv6 is introduced

• First hop security in our access networks

• Unique security considerations with IPv6

• ICMPv6

• DHCPv6 or SLAAC

• Privacy extensions for SLAAC

• Hop by hop extension header

51

Lessons Learned - Netflow

• IPv6 requires NetFlow v9• Some collectors cannot receive/process NetFlow v9

• Some routing platforms don’t support for both NetFlow v5 and NetFlow v9

• Some routing platforms are constrained to two export destinations

• Netflow analysis tools must be IPv6 capable

• We had to shift NetFlow collection to our DMZ devices to deal with the constraints above

52

Lessons Learned – ISP Concerns

• Will the same SLA apply for IPv6?

• Can the circuit that services the existing IPv4 connection be converted to dual-stack without the physical changes?

• Are full IPv6 global routes available to end customers?

• Is there an IPv6 “looking glass”?

• Are there any restrictions on prefix advertisements?

• What percentage of your IPv4 peers to you currently peer with for IPv6

• Global reachability

• BOGON List update

53

Lessons Learned – SP Concerns

• ISPs

• IP WAN providers (Layer 3 MPLS)

• External content monitoring providers

• Content distribution providers

• Geo Location providers

54

Lessons Learned – Application Impact

• Geo-location and web analyticsClient_IpAddress := X-forwarded-for address first address;

If null then

Client_IpAddress := remoteAddress

end if;

use Client_IpAddress for IPCheck

• Development, testing, and QA teams require IPv6 access

• How will they get IPv6 access from within the corporate network?

• Supports the business case for an internal corporate network IPv6 deployment

55

Lessons Learned – Path MTU

• Allow PMTUD across the network

• PMTUD allows devices to negotiate the MTU size between hosts

• PTB (Packet Too Big) messages must be permitted

• PTB for hosts behind Tunnels (IPSec/GRE) with reduced MTU

• PMTUD works between hosts for end-to-end communication. If this is broken, hosts may not be able to communicate over IPv6

56

Lessons Learned – End Devices

• Many of our end devices are already IPv6 enabled

• From Microsoft Vista and Server 2008

• From OS X Lion (10.7)

• “Happy Eyeballs” can mask IPv6 connectivity issues

• End Devices

Source: http://www.worldipv6launch.org/measurements/

57

Future Plans

65

IPv4 / IPv6 Co-existence

IPv6 Transition Technologies

IPv4 Prevalence Dual Stack

IPv6 Prevalence

IPv4 as a Service

IPv6-Only

IPv4-Only

IPv6 Multi-Year Strategy

2010-2015 2016-2022

Address Exhaustion / Reclamation Plan59

2018 201920172016

IPv6 Roadmap (IPv6 only and IPv4aaS)

2015

Program development

(Identify

opportunities, Gaps,

Relationships)

IPv6 Translation Technologies

Design Development and

Deployment

(DC Edge, Internet Edge)

Training (Application Teams)

Infrastructure Regional Extranet Tunnel Head

ends

IPv6 Only Pilot

(IPv4aaS Offering)

* All dates and efforts tentative subject to approval

App Testing / Development. Migration ongoing

ACI IPv6 Only Mandate for Greenfield Workloads

ACI IPv6 Only

Mandate for

Migrations (CITEIS to

ACI)

IPv6 Only (Remove Dual Stack – Pilot)

IPv4 Public Address recovery

(or reduced demand)

Critical Dependencies Review

67

What if I had to start over now?

68

What if I had to start over now?

• I would get my translation service in place first (DC Edge, Internet Edge)

• Dual stack is bad .. IPv6 only is the way (Dual stack only where necessary)

• Masks connectivity problems with IPv6

• Uses up TCAM resources on devices

• Requires operational support twice

• Potential of conflicting policies between v4 and v6

• ACLs are a nightmare!

• Gives me more of the scarce IPv4 addresses to use only where critical

• Gets app owners working on their applications in good time

62

Learning more about IPv6BRKRST-2616 Addressing Networking challenges with latest Innovations in IPv6 Tue 16 11:15:00

COCIP6-1013 IPv4 Address Exhaustion and IPv6 Progress across Cisco IT Tue 16 11:15:00

BRKRST-2116 Intermediate - IPv6 from Intro to Intermediate Tue 16 14:15:00

DevNet-1275 Developing Better Applications with IPv6 Tue 16 16:30:00

BRKRST-2022 IPv6 Routing Protocols Update Tue 16 16:45:00

BRKSPG-2603 Intermediate - How to Securely Operate an IPv6 Network Tue 16 16:45:00

LABIPM-2007 Intermediate - IPv6 Hands on Lab Wed 17 09:00:00

CCSIP6-2006 BMW: Enterprise IPv6 adoption Wed 17 11:30:00

LABSPG-7122 Advanced IPv6 Routing and services lab Wed 17 14:00:00

BRKIP6-2100 IPv6-centric application development Wed 17 14:30:00

BRKRST-2667 How to write an IPv6 Addressing Plan Wed 17 14:30:00

BRKSPG-2300 Service Provider IPv6 Deployment Wed 17 16:30:00

PNLCRS-2307Don't Be Left Behind: Consumer Internet Traffic is Shifting to IPv6, Will

your Organization Follow?Wed 17 16:30:00

BRKRST-2312 Intermediate - IPv6 Planning, Deployment and Operation Considerations Thu 18 09:00:00

BRKSPG-2061 IPv6 Deployment Best Practices for the Cable Access Network Thu 18 09:00:00

BRKCOL-2020 IPv6 in Enterprise Unified Communications Networks Thu 18 11:30:00

BRKSEC-3003 Advanced IPv6 Security in the LAN Thu 18 11:30:00

BRKRST-3123 Segment Routing for IPv6 Networks Thu 18 14:30:00

BRKSEC-3200 Advanced IPv6 Security Threats and Mitigation Thu 18 14:30:00

BRKRST-2301 Intermediate - Enterprise IPv6 Deployment Fri 19 09:00:00

Lunch and Learn:

• Service Provider IPv6: Tue 16 12:45

• IPv6 in the Enterprise: Thu 18 13:00

Walk-in Self-Paced Lab:

LABCRS-1000 Intro IPv6 Addressing and Routing Lab

Experiment with IPv6-only WiFi:

SSID: IPV6ONLYEXP

WPA passphrase: iknowbesteffort

SLAAC + stateless DHCP

NAT64 included to access legacy

Ask all World of Solutions exhibitors fortheir IPv6 support

Complete Your Online Session Evaluation

• Please complete your online sessionevaluations after each session.Complete 4 session evaluations& the Overall Conference Evaluation(available from Thursday)to receive your Cisco Live T-shirt.

• All surveys can be completed viathe Cisco Live Mobile App or theCommunication Stations

Thank you