Post on 30-Aug-2018
NAT Deployment in Cloud Networks
BRKDCT-2448
Jason Yang – CCIE #10467
Technical Marketing Engineer
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Session Goals
• NAT is becoming the critical component of the Cloud Gateway, customers are thirsty for recommendations and best practices to design NAT with high scalability and high availability in the Hosted Cloud Networks.
• This session will share
1. How VRF Aware Network Address Translation (NAT) enables Cloud Gateway Architecture
2. Cloud Gateway High Availability Design
3. Performance, Scalability & Operation Best Practice* *This section will focus on ASR 1000 as the Cloud Gateway platform
3
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Agenda
• Cloud Gateway Architecture enabled by VRF-Aware NAT
• Cloud Gateway HA Design
• Perf/Scale & Operation Best Practice
• Summary and Take Away
4
Cloud Gateway Architecture enabled by VRF-Aware NAT
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
MPLS VPN
Cloud Gateway Architecture
Apps
Internet
Partners
AAA
Location
Hosted Cloud Services
Multi-tenant
• VRF Aware
• VRF Scale
Private/Overlapping
Addressing access
Common Services
• Network Address Translation
• NAT Scale
Inter-VRFs
Communication
• VRF Aware Service Infrastructure (VASI)
High Availability
• Dual Box Design
• Stateless Redundancy
• Stateful Redundancy
PE GW
6
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• VRF NAT supports MPLS/VPN for
– Communication between remote hosts in different VPNs and Internet common servers.
– Intra-VPN communication.
• VRF-Aware Service Infrastructure (VASI) for
– Traffic flows and routing exchange across different VRFs
– VASI is implemented by using virtual interface pairs (vasileftx, vasirightx), where each of the interfaces in the pair is associated with a different VRF instance.
– Apply services such as NAT, ACL, Policing, ZBFW, IPsec, PBR.
VRF Aware NAT & VASI
7
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Connectivity Model Summary
Model 1 Model 2 Model 3
Cloud Gateway
Autonomous System
GW and PE are in different
BGP AS – Cloud Services is
managed outside the business
VPN network
GW and PE are in different
BGP AS - Cloud Services is
managed outside the
business VPN network
GW and PE are in the same
BGP AS – Cloud Services is
managed as part of
business VPN network
Connectivity to the
VPN network
Inter-AS Option A (eBGP + back
to back VRF)
NAT Inside interface
Inter-AS Option B (eBGP +
label)
NAT inside interface
MP-iBGP
NAT inside interface
(a) Connectivity to
the Cloud in Global
NAT outside interface NAT outside interface N/A
(b) Connectivity to
the Cloud in VRF
Requires VASI
NAT outside in VASIleft
Requires VASI
NAT outside in VASIleft
Requires VASI
NAT outside in VASIleft
Routing over VASI iBGP iBGP eBGP
the most
common
AS: Autonomous System
8
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• HCS service in global routing table
• Inter-AS Option A
• VRF/VLAN sub-interface as VRF aware NAT inside Interface
• Global interface as NAT outside interface
Connectivity Model 1a
N x eBGP
VRFR VRFR
C_NetworkR
PE GW
AS577
Global
AS65004
C_NetworkB
C_NetworkG
VRFB VRFB
VRFG VRFG
HCS
ip nat inside ip nat outside
PE – Provider Edge Router; GW – Cloud Gateway Router; SR – Service Router
S_Network SR
9
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• HCS service in VRF
• Inter-AS Option A
• VRF/VLAN sub-interface as VRF aware NAT inside Interface
Connectivity Model 1b
N x eBGP
VRFR VRFR
C_NetworkR
PE GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
VRFB VRFB
VRFG VRFG
HCS
ip nat inside ip nat outside VASILeftx VASIRightx
• VASI to facilitate Inter-VRF communication
• VASIleft VRF interface as NAT outside interface
S_Network SR
10
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• HCS service in global route table
• Inter-AS Option B
• MPLS as VRF Aware of NAT inside interface
• Global Interface as NAT outside interface
Connectivity Model 2a
1 x eBGP
C_NetworkR
PE GW
AS577
Global
AS65004
C_NetworkB
C_NetworkG
MPLS MPLS HCS
ip nat inside ip nat outside
S_Network SR
11
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• HCS service in VRF
• Inter-AS Option B
• MPLS as VRF Aware of NAT inside interface
Connectivity Model 2b
• VASI to facilitate Inter-VRF communication
• VASILeft VRF interface as NAT outside interface
1 x eBGP
C_NetworkR
PE GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
MPLS MPLS HCS
ip nat inside ip nat outside VASILeftx VASIRightx
S_Network SR
12
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• HCS service in VRF
• MP-iBGP
• MPLS as VRF Aware of NAT inside interface
Connectivity Model 3
• VASI to facilitate Inter-VRF communication
• VASILeft VRF interface as NAT outside interface
MP-iBGP
C_NetworkR
PE GW
AS65004
MPLS
C_NetworkB
C_NetworkG
MPLS/VPN
ip nat outside VASILeftx VASIRightx
S_Network SR
MPLS
ip nat inside
MP-iBGP
13
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Inter-AS Option A is the most secure and easiest to provision
• Inter-AS Option A may face manageability challenge as #s of VRFs grow
Connectivity Model 1 – Control Plane
N x eBGP
VRFR VRFR
C_NetworkR
PE GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
VRFB VRFB
VRFG VRFG
HCS
• GW and SR can run Static/IGP/BGP to exchange routes, though BGP scales and seamless
• iBGP can run in the VASI pairs to exchange routes between VRFs
S_Network SR
eBGP
AS223
iBGP
202.255.254.1 202.255.254.1 202.255.254.1
202.255.254.1
10.254.254.4 10.254.254.4
201.255.254.1 201.255.254.1
Cloud Service Network Advertised to the Customers
NAT Pool Advertised to the Cloud
14
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Customer initiate the connection to the cloud
• Routing lookup performed before VRF-Aware NAT translation
Connectivity Model 1 – Data Plane
N x eBGP
VRFR VRFR
C_NetworkR
PE GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
VRFB VRFB
VRFG VRFG
HCS
• VASI allows customer VRF traffic flow to the Cloud Service VRF and Vice Versa
• For the return traffic, NAT performed in customer VRF first, then make the routing lookup.
S_Network SR
eBGP
AS223
iBGP 10.254.254.4(S)|202.255.254.1(D) 201.254.254.1(S)|202.255.254.1(D)
(D)201.254.254.1|(S)202.255.254.1 (D)10.254.254.4|(S)202.255.254.1
15
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Connectivity Model 1 – Configuration interface GigabitEthernet0/0/0.2 description PE facing interface for VRFR encapsulation dot1Q 2 vrf forwarding VRFR ip address 192.255.254.2 255.255.255.252 ip nat inside ! interface vasileft1 vrf forwarding VRFR ip address 100.4.4.1 255.255.255.252 ip nat outside ! interface vasiright1 vrf forwarding VRFS ip address 100.4.4.2 255.255.255.252 ! interface GigabitEthernet0/0/1 description Cloud facing interface for Service VRF vrf forwarding VRFS ip address 99.99.70.2 255.255.255.0 ! access-list 4 permit 10.254.254.0 0.0.0.255 ! ip nat pool pool-PC 201.255.254.1 201.255.254.1 prefix-length 30 ip nat inside source list 4 pool pool-PC vrf VRFR overload
router bgp 577 address-family ipv4 vrf VRFS bgp router-id 1.1.1.1 bgp log-neighbor-changes neighbor 99.99.70.1 remote-as 223 neighbor 99.99.70.1 description PEERING to SR neighbor 99.99.70.1 active neighbor 100.4.4.1 remote-as 577 neighbor 100.4.4.1 next-hop-self neighbor 100.4.4.1 description PEERING to VASI VRFR interface neighbor 100.4.4.1 active ! address-family ipv4 vrf VRFR bgp router-id 4.4.4.4 redistribute static neighbor 100.4.4.2 remote-as 577 neighbor 100.4.4.2 description PEERING to VASI VRFS interface neighbor 100.4.4.2 activate neighbor 100.4.4.2 prefix-list VRF_Pool out neighbor 192.255.254.1 remote-as 65004 neighbor 192.255.254.1 description PEERING to PE neighbor 192.255.254.1 activate ! ip prefix-list VRF_Pool seq 5 permit 201.255.254.1/32 ! Ip route vrf VRFR 201.255.254.1 255.255.255.255 null0
16
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Inter-AS Option B - single eBGP session to exchange VPN routes and labels
• Label spoofing could be concern
Connectivity Model 2 – Control Plane
1 x eBGP
C_NetworkR
PE GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
MPLS MPLS HCS
• GW and SR can run Static/IGP/BGP to exchange routes, though BGP scales and seamless
• iBGP can run in the VASI pairs to exchange routes between VRFs
S_Network SR
eBGP
AS223
iBGP
202.255.254.1 202.255.254.1 L2|202.255.254.1
202.255.254.1
10.254.254.4 10.254.254.4|L6
201.255.254.1 201.255.254.1
Cloud Service Network Advertised to the Customers
NAT Pool Advertised to the Cloud
17
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Customer initiate the connection to the cloud
• Label disposition, followed by routing lookup, then performed VRF-Aware NAT translation
Connectivity Model 2 – Data Plane
1 x eBGP
C_NetworkR
PE GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
MPLS MPLS HCS
• VASI allows customer VRF traffic flow to the Cloud Service VRF and Vice Versa
• For the return traffic, NAT performed in customer VRF first, then make the routing lookup, then impose the label
S_Network SR
eBGP
AS223
iBGP 10.254.254.4(S)|202.255.254.1(D)|L2 201.254.254.1(S)|202.255.254.1(D)
(D)201.254.254.1|(S)202.255.254.1 L6|(D)10.254.254.4|(S)202.255.254.1
18
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Connectivity Model 2 – Configuration interface GigabitEthernet0/0/0 description PE facing interface ip address 192.255.254.2 255.255.255.252 ip nat inside mpls bgp forwarding ! interface vasileft1 vrf forwarding VRFR ip address 100.4.4.1 255.255.255.252 ip nat outside ! interface vasiright1 vrf forwarding VRFS ip address 100.4.4.2 255.255.255.252 ! interface GigabitEthernet0/0/1 description Cloud facing interface for Service VRF vrf forwarding VRFS ip address 99.99.70.2 255.255.255.0 ! access-list 4 permit 10.254.254.0 0.0.0.255 ! ip nat pool pool-PC 201.255.254.1 201.255.254.1 prefix-length 30 ip nat inside source list 4 pool pool-PC vrf VRFR overload
router bgp 577 neighbor 192.255.254.1 remote-as 65004 neighbor 192.255.254.1 description PEERING to PE ! address-family ipv4 vrf VRFS bgp router-id 1.1.1.1 bgp log-neighbor-changes neighbor 99.99.70.1 remote-as 223 neighbor 99.99.70.1 description PEERING to SR neighbor 99.99.70.1 active neighbor 100.4.4.1 remote-as 577 neighbor 100.4.4.1 next-hop-self neighbor 100.4.4.1 active neighbor 100.4.4.1 description PEERING to VASI VRFR interface ! address-family ipv4 vrf VRFR bgp router-id 4.4.4.4 redistribute static neighbor 100.4.4.2 remote-as 577 neighbor 100.4.4.2 description PEERING to VASI VRFS interface neighbor 100.4.4.2 activate neighbor 100.4.4.2 prefix-list VRF_Pool out ! address-family vpnv4 neighbor 192.255.254.1 active neighbor 192.255.254.1 send-community both ! ip prefix-list VRF_Pool seq 5 permit 201.255.254.1/32 ! Ip route vrf VRFR 201.255.254.1 255.255.255.255 null0
19
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• VASI becomes VRF termination point in the GW, an ideal place to apply per VRF Security and QoS policy
Connectivity Model 2 – Configuration (cont’d)
interface vasileft1
vrf forwarding VRFR
ip address 100.4.4.1 255.255.255.252
ip access-group VASI-1-LEFT-IN in
ip access-group VASI-1-LEFT-OUT out
ip nat outside
service-policy output Police_Cloud_ACCESS_VRFR_10meg*
interface vasiright1
vrf forwarding VRFS
ip address 100.4.4.2 255.255.255.252
ip access-group VASI-1-RIGHT-IN in
ip access-group VASI-1-RIGHT-OUT out
*Queuing Policy is not supported, only policing and marking
20
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Cloud service is part of business VPN network
• MP-iBGP – full mesh with all other PEs/RR/SR to exchange VPN routes and labels
Connectivity Model 3 – Control Plane
MP-iBGP
C_NetworkR
PE GW
AS65004
MPLS
C_NetworkB
C_NetworkG
MPLS MPLS/VPN
• eBGP can run in the VASI pairs to exchange routes between VRFs
S_Network SR
MP-iBGP
eBGP
202.255.254.1|L1 202.255.254.1 L2|202.255.254.1
202.255.254.1
10.254.254.4 10.254.254.4|L6
201.255.254.1 201.255.254.1|L3
Cloud Service Network Advertised to the Customers
NAT Pool Advertised to the Cloud
21
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Customer initiate the connection to the cloud
• Label disposition, followed by routing lookup, then performed VRF-Aware NAT translation
Connectivity Model 3 – Data Plane
MP-iBGP
C_NetworkR
PE GW
AS65004
MPLS
C_NetworkB
C_NetworkG
MPLS MPLS/VPN
• VASI allows customer VRF traffic flow to the Cloud Service VRF and Vice Versa
• For the return traffic, NAT performed in customer VRF first, then make the routing lookup, then impose the label
S_Network SR
MP-iBGP
eBGP 10.254.254.4(S)|202.255.254.1(D)|L2 201.254.254.1(S)|202.255.254.1(D)|L1
L3|(D)201.254.254.1|(S)202.255.254.1 L6|(D)10.254.254.4|(S)202.255.254.1
22
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Connectivity Model 3 – Configuration interface GigabitEthernet0/0/0 description MPLS VPN facing interface ip address 192.255.254.2 255.255.255.252 ip nat inside mpls ip ! interface vasileft1 vrf forwarding VRFR ip address 100.4.4.1 255.255.255.252 ip nat outside ! interface vasiright1 vrf forwarding VRFS ip address 100.4.4.2 255.255.255.252 ip policy route-map PBR_FW ! interface GigabitEthernet0/0/1 description Service facing interface ip address 99.99.70.2 255.255.255.0 mpls ip ! route-map PBR_FW permit 10 match ip address PBR_FW set ip nexthop recursive vrf FW_VRF 202.255.254.1 ! access-list 4 permit 10.254.254.0 0.0.0.255 ! ip nat pool pool-PC 201.255.254.1 201.255.254.1 prefix-length 30 ip nat inside source list 4 pool pool-PC vrf VRFR overload
router bgp 65004 neighbor 192.255.254.100 remote-as 65004 neighbor 192.255.254.100 description PEERING to RR neighbor 192.255.254.100 update-source loopback0 ! address-family ipv4 vrf VRFS redistribute connected neighbor 100.4.4.1 remote-as 65534 neighbor 100.4.4.1 local-as 65535 neighbor 100.4.4.1 update-source vasiright1 neighbor 100.4.4.1 active ! address-family ipv4 vrf VRFR redistribute static neighbor 100.4.4.2 remote-as 65535 neighbor 100.4.4.2 local-as 65534 neighbor 100.4.4.2 update-source vasileft1 neighbor 100.4.4.2 activate neighbor 100.4.4.2 prefix-list VRF_Pool out default-information originate ! address-family vpnv4 neighbor 192.255.254.100 active neighbor 192.255.254.100 send-community both ! ip prefix-list VRF_Pool seq 5 permit 201.255.254.1/32 ! Ip route vrf VRFR 201.255.254.1 255.255.255.255 null0
23
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Design of NAT Pool
Pool per VRF Shared Pool by all VRFs
1. Easy of maintenance
2. Easy of debugging
3. Add/Remove customers without service
disruption
1. Efficient use of addresses
2. Less configuration
3. Removing of one customer cause interruption
of all other customers NAT
ip nat pool customer1-nat-pool 15.1.1.1 15.1.1.255 prefix-length 24
!
ip access-list extended customer1-acl
deny ip <router-generated-ip>
permit ip 10.0.0.0 0.255.255.255
!
ip nat inside source list customer1-acl pool customer1-nat-pool overload
vrf customer1-vrf
!
ip nat pool customer2-nat-pool 16.1.1.1 16.1.1.255 prefix-length 24
!
ip access-list extended customer2-acl
deny ip <router-generated-ip>
permit ip 10.0.0.0 0.255.255.255
!
ip nat inside source list customer2-acl pool customer2-nat-pool overload
vrf customer2-vrf
ip nat pool shared-nat-pool 15.1.1.1 15.1.255.255 prefix-length 16
!
ip access-list extended shared-cust-acl
deny ip <router-generated-ip>
permit ip 10.0.0.0 0.255.255.255
!
ip nat inside source list shared-cust-acl pool shared-nat-pool overload
vrf customer1-vrf
ip nat inside source list shared-cust-acl pool shared-nat-pool overload
vrf customer2-vrf
24
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
What Mode of NAT to Run - NAT vs. CGN
Traditional NAT Carrier Grade NAT (CGN)
Session Entry full 5 tuples – {protocol, source address, source
port, destination address, destination port}
3 tuples - {protocol, source address,
source port}
Default timeout 24 hrs for TCP 15 mins for TCP
Outside mapping rule
(ip nat outside source)
Supported Not supported
EIM/EIF Not Supported Supported
High Speed Logging
(HSL)
Log full tuples No destination info in the logging
record
Bulk logging and Port
Block Allocation
Not Supported Supported
Salability - Double than traditional NAT
License No license required Require license
25
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Traditional NAT
Pro Inside global Inside local Outside local Outside global
tcp 26.1.1.6:1024 27.1.1.10:29439 26.1.1.2:23 26.1.1.2:23
• CGN
Pro Inside global Inside local Outside local Outside global
tcp 26.1.1.6:1024 27.1.1.10:11806 --- ---
NAT vs. CGN – Session Entry
26
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Endpoint-Independent Mapping (EIM) provides a stable, long-term binding where internal hosts may connect by utilizing the same NAT binding for multiple external hosts (as long as the internal port does not change)
• Endpoint-Independent Filtering (EIF) is closely related to EIM, and controls which external servers may access a host using an established binding
• This is typical for peer-to-peer applications and some Internet messenger protocols.
NAT vs. CGN – EIM/EIF
inside outside
SrcIP:Port DstIP:Port
X:x Y1:y1
SrcIP:Port DstIP:Port
X1:x1 Y1:y1
SrcIP:Port DstIP:Port
X:x Y2:y2
SrcIP:Port DstIP:Port
X1:x1 Y2:y2
EIM implies X1:x1 = X2:x2 for all Y:y (Y1:y1 and Y2:y2)
Pro Inside global Inside local Outside local Outside global
tcp 26.1.1.6:1024 27.1.1.10:11806 --- ---
CGN
27
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• High speed NAT device generate NAT transaction events (creation/deletion) in the rate of >100k events/sec, syslog is not able to support it.
• HSL enables NAT datapath directly export the transaction records (NetFlowv9-like) to an external collector.
NAT vs. CGN – High Speed Logging (HSL)
Field Format
Source IP address IPv4 address
Translated source IP address IPv4 address
Destination IP address IPv4 address
Translated destination IP address IPv4 address
Original source port 16-bit port
Translated source port 16-bit port
Original destination port 16-bit port
Translated destination port 16-bit port
VRF ID 32-bit ID
Protocol 8-bit value
Event 0-Invalid
1-Adds event
2-Deletes event
Unix timestamp in milliseconds 64-bit value
Destination Info not available in CGN
Mode
Destination Info not available in CGN
Mode
28
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Problem: High setup/teardown rates on NAT devices cause customers to have to store Terabits of data a day for NAT HSL. Customer want to see this volume of logging significantly reduced.
• Solution: Provide each end user with a block of ports. Only log when the block get (dis)associated with a user.
NAT vs. CGN – Bulk Logging and Port Block Allocation (BPA)
Field Format
Source IP address IPv4 address
Translated source IP address IPv4 address
VRF ID 32-bit ID
Protocol 8-bit value
Event 0-Invalid
1-Adds event
2-Deletes event
Unix timestamp in milliseconds 64-bit value
Port block start 16-bit port
Port block step size 16-bit step size
Number of ports in the block 16-bit number
For example: a BPA configuration with set size 8 and step size of 4.
Set 0 = {1024, 1028, 1032, 1036, 1040, 1044, 1048, 1052}
Set 1 = {1025, 1029, 1033, 1037, 1041, 1045, 1049, 1053}
Set 2 = {1026, 1030, 1034, 1038, 1042, 1046, 1050, 1054}
Set 3 = {1027, 1031, 1035, 1039, 1043, 1045, 1051, 1055}
…
29
Cloud Gateway HA Design
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Dual-GWs; Dual-PEs; Dual-SRs
• Fast Failure Detection: BFD (sub-second) – may not all platforms support BFD
• Common Failure Detection: BGP (~tens of sec)
High Availability Design
• BGP determines the active path, symmetric routing and convergence time
• GWs are in (stateless) active/standby from NAT perspective
N x eBGP/BFD
VRFR VRFR
C_NetworkR
PE1 GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
HCS S_Network SR1
eBGP/BFD
AS223
GW2
VRFR VRFR
GW1
SR2 Service VRF
iBGP
PE2
31
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• PE1-GW1-SR1 active path; PE2-GW2-SR2 standby path.
• GW1-SR1 BGP session down
• GW1 withdraw S_Network from PE1
• PE2-GW2-SR2 become the best, GW2 will begin to set up NAT translations
Failover Scenario – GW1-SR1 BGP session down
N x eBGP/BFD
VRFR VRFR
C_NetworkR
PE1 GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
HCS S_Network SR1
eBGP/BFD
AS223
GW2
VRFR VRFR
GW1
SR2 Service VRF
iBGP
PE2
202.255.254.1
202.255.254.1
202.255.254.1
32
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• PE1-GW1-SR1 active path; PE2-GW2-SR2 standby path.
• PE1-GW1 BGP session down
• GW1 is still advertising the NAT_Pool to SR1, which cause SR1 to blackhole customer traffic to GW1!
Failover Scenario – PE1-GW1 BGP session down
N x eBGP/BFD
VRFR VRFR
C_NetworkR
PE1 GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
HCS S_Network SR1
eBGP/BFD
AS223
GW2
VRFR VRFR
GW1
SR2 Service VRF
iBGP
PE2
201.255.254.1
201.255.254.1
33
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Solution: BGP VRF Aware Conditional Advertisement
• The condition is that C_networkR exist in BGP VRFR table in GW1, then GW1 can advertise NAT_Pool to VASIRight, otherwise withdraw NAT_Pool back from VASIRight
Failover Scenario – PE1-GW1 BGP session down (cont’d)
N x eBGP/BFD
VRFR VRFR
C_NetworkR
PE1 GW
AS577
Service VRF
AS65004
C_NetworkB
C_NetworkG
HCS S_Network SR1
eBGP/BFD
AS223
GW2
VRFR VRFR
GW1
SR2 Service VRF
iBGP
PE2
201.255.254.1
201.255.254.1
10.254.254.4
10.254.254.4
34
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Failover Scenario – PE1-GW1 BGP session down (cont’d)
interface GigabitEthernet0/0/0.2 description PE facing interface for VRFR encapsulation dot1Q 2 vrf forwarding VRFR ip address 192.255.254.2 255.255.255.252 ip nat inside bfd interval 50 min_rx 50 multiplier 3 ! interface vasileft1 vrf forwarding VRFR ip address 100.4.4.1 255.255.255.252 ip nat outside ! interface vasiright1 vrf forwarding VRFS ip address 100.4.4.2 255.255.255.252 ! interface GigabitEthernet0/0/1 description Cloud facing interface for Service VRF vrf forwarding VRFS ip address 99.99.70.2 255.255.255.0 ! access-list 4 permit 10.254.254.0 0.0.0.255 ! ip nat pool pool-PC 201.255.254.1 201.255.254.1 prefix-length 30 ip nat inside source list 4 pool pool-PC vrf VRFR overload ! ip prefix-list VRF_Pool seq 5 permit 201.255.254.1/32 ip prefix-list p1-adv-1 seq 5 permit 201.255.254.1/32 ip prefix-list p1-exist-1 seq 5 permit 10.254.254.4/32
router bgp 577 address-family ipv4 vrf VRFS bgp router-id 1.1.1.1 bgp log-neighbor-changes neighbor 99.99.70.1 remote-as 223 neighbor 99.99.70.1 description PEERING to SR neighbor 99.99.70.1 active neighbor 100.4.4.1 remote-as 577 neighbor 100.4.4.1 next-hop-self neighbor 100.4.4.1 description PEERING to VASI VRFR interface neighbor 100.4.4.1 active ! address-family ipv4 vrf VRFR bgp router-id 4.4.4.4 redistribute static neighbor 100.4.4.2 remote-as 577 neighbor 100.4.4.2 description PEERING to VASI VRFS interface neighbor 100.4.4.2 activate neighbor 100.4.4.2 advertise-map ADV-1 exist-map EXIST-1 neighbor 100.4.4.2 prefix-list VRF_Pool out neighbor 192.255.254.1 remote-as 65004 neighbor 192.255.254.1 description PEERING to PE neighbor 192.255.254.1 activate ! route-map ADV-1 permit 10 match ip address prefix-list p1-adv-1 ! route-map EXIST-1 permit 10 match ip address prefix-list p1-exist-1 ! Ip route vrf VRFR 201.255.254.1 255.255.255.255 null0
35
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Redundant ESP / RP on ASR 1006 and ASR 1013
• Zero packet loss on RP fail-over
• < 50ms loss for ESP fail-over
• Intra-chassis Stateful Switchover (SSO) Support for NAT
• IOS XE also provides full support for Network Resiliency
– NSR/GR for BGP
– BFD SSO
• Support for ISSU
GW Intra-Chassis Redundancy - ASR 1000 built for Carrier-Grade HA
RP
CPU RP
CPU
ES
P
QFP Crypto
Assist. PPE BQS
FECP
Crypto ES
P
QFP Crypto
Assist. PPE BQS
FECP
Crypto
RP
CPU
Crypto
Assist.
RP
CPU
ES
P
QFP PPE BQS
FECP
Crypto ES
P
QFP Crypto
Assist. PPE BQS
FECP
Crypto
SIP
SPA SPA
IOCP SPA
Aggreg.
SIP
SPA SPA
IOCP SPA
Aggreg.
SIP
SPA SPA
IOCP SPA
Aggreg.
36
Performance, Scalability & Operation Best Practice
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Midplane
ASR1000 Building Blocks
ES
P FECP
QFP Crypto
Assist.
interconnect
PPE BQS
ES
P
FECP
QFP Crypto
Assist.
interconn.
PPE BQS
FECP
Crypto
Assist.
interconnect
RP
CPU
interconn. GE switch S
IP
SPA SPA
IOCP SPA
Aggreg.
interconnect
RP
CPU
interconn. GE switch
SIP
SPA SPA
IOCP SPA
Aggreg.
interconnect
SIP
SPA SPA
IOCP SPA
Aggreg.
interconnect
Route Processor
Handles control plane
Manages system Embedded Service Processor
Handles forwarding plane traffic
SPA Interface Processor
Houses SPA’s
Buffer packets in & out
• Route Processor (RP) • Handles control plane traffic • Manages system
• Embedded Service Processor (ESP) • Handles forwarding plane traffic
• SPA Interface Processor (SIP) • Shared Port Adapters provide interface
connectivity
• Centralized Forwarding Architecture • All traffic flows through the active ESP,
standby is synchronized with all flow state with a dedicated 10-Gbps link
• Distributed Control Architecture • All major system components have a
powerful control processor dedicated for control and management planes
38
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
NAT <> ESP Resources Dependency
QFP complex
Crypto
(Nitrox-II
CN2430)
FECP GE, 1Gbps
I2C
SPA Control
SPA Bus
ESI, 11.2Gbps
SPA-SPI, 11.2Gbps
Hypertransport, 10Gbps
Other
RPs RPs RPs ESP SIPs
E-RP* PCI*
E-CSR
TCAM Resource
DRAM
Packet Buffer
DRAM
SA table
DRAM
Dispatcher Packet Buffer
DDRAM
Boot Flash
(OBFL,…)
JTAG Ctrl
Reset / Pwr Ctrl
…
Packet Processor Engines
PPE1 PPE2 PPE3 PPE4 PPE5
PPE6 PPE7 PPE8 PPE40
BQS
Reset / Pwr Ctrl
Interconnect
SPI Mux
Interconnect
EEPROM
Temp Sensor
• NAT sessions
• Memory for FECP
• QFP client / driver
• Statistics
• ACL ACEs copy
• NAT config objects
• NAT VFR re-assembly • ACL/ACE, Route-map
39
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
ASR1000 NAT Scalability (uni-dimensional)
ASR 1001 ASR 1002-X ESP5
ESP10 ESP20 ESP40 ESP100 ESP200
NAT
Sessions
(classic)
250k 2M 250k 1M 2M 2M 4M 4M
NAT
Sessions
(CGN)
500k 4M 500k 1.75M 4M 4M 12M 12M
NAT Pools - 1200 - - 1200 1200 1200 1200
VRFs for
VRF-Aware
NAT
4k 4k 1k 1k 4k 4k 4k 4k
Route-maps
w/ NAT 1024 1024 1024 1024 1024 1024 1024 1024
40
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
ASR1000 NAT Performance (uni-dimensional)
g ASR 1001 ASR 1002-X ESP5
ESP10 ESP20 ESP40 ESP100 ESP200
NAT Session
Setup Rate 50cps 230cps(pat) 50kcps 100kcps
200kcps(dyn)
139kcps(pat)
200kcps(dyn)
95kcps(pat) 250kcps(pat) 300kcps(pat)
NAT (classic)
Performance 3Mpps 10Mpps 3Mpps 6Mpps 8Mpps 9Mpps 23Mpps 45Mpps
NAT (CGN)
Performance 2.2Mpps - 2.2Mpps 5Mpps 7Mpps 7Mpps 18Mpps 34Mpps
NAT (classic)
Throughput 5Gbps 36Gbps 5Gbps 10Gbps 20Gbps 40Gbps 100Gbps 200Gbps
41
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Application Layer Gateway (ALG)
ALG VFR vTCP L4 VRF HA
FTP Yes No tco Yes Yes
H323 No Yes tcp,udp Yes Yes
RTSP Yes Yes tcp Yes Yes
SCCP No No tcp Yes Yes
SIP Yes Yes tcp,udp Yes Yes
TFTP No N/A udp Yes Yes
NETBIOS No No tcp,udp Yes Yes
RCMD No No tcp Yes Yes
LDAP No No tcp Yes Yes
DNS Yes Yes tcp,udp Yes Yes
SUNPRC Yes No tcp Yes Yes
MSRPC Yes No tcp Yes Yes
PPTP No tcp Yes Yes
• ASR 1000 support comprehensive ALGs
• With ALG traffic, "any any" ACL is not supported. This could lead to undesired payload translations, causing unexpected application behavior
42
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
• Isarflow
• Lancope
• ActionPacked
ASR 1000 HSL Supported Collector
43
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Key System Resources to Monitor
IOS
Forwarding
Manager
Forwarding
Manager
QFP Client
Driver
Datapath
SIP
show proc cpu sort show mem stat
RP memory RP CPU
TCAM
resource DRAM
pkt memory
crypto assist QFP
ESP memory
show plat
software status
control-processor
brief
show plat
software status
control-processor
brief
show plat
software status
control-processor
brief
FECP CPU
show plat software
status control-
processor brief
show plat hardware
qfp active infra
exmem statistics
show plat hardware
qfp active datapath
util summary
show plat hardware
qfp active tcam
resource-manager
-usage
75%
85%
44
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
ASR 1000 Cloud Gateway Monitoring Guide (1)
• It is general best practice that ASR 1000 in live deployment RP/IOS/ESP CPU and Memory utilization do not exceed 75% in steady state
• It is general best practice that ASR 1000 in live deployment QFP DRAM utilization do not exceed 85% in steady state
45
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
ASR 1000 Cloud Gateway Monitoring Guide (2)
• For TCAM monitoring, keep an eye on syslog:
%QFPTCAMRM-6-TCAM_RSRC_ERR: F0: QFP_sp: Allocation failed because of insufficient TCAM resources in the system
• Recommendations
1. Test out TCAM utilization before making changes
2. Always there should be unused TCAM entries which are = or > the size of biggest ACL on the router.
• Be aware of the TCAM deny jump issue
46
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
SET the Limit
• Set NAT max-entries per system to no more than platform scale: ip nat translation max-entries <number of entries>
Be aware of that
1. NAT sessions scaling numbers are based on a few pools
2. PAT session scaling numbers are expected to be reduced while the number of overload pools are rising
3. One data point we have is ESP20 support 500k sessions w/ 1200 overload pools vs. 2M session w/ a few pools
• Set NAT max-entries per VRF to prevent single customer starving entire system translation limit:
ip nat translation max-entries vrf <vrf_name> <number of entries>
47
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Features Interaction
• This architecture is proven with following features on Cloud Gateway, do not enable more features unless been tested prior to deployment.
– VRF Aware NAT + VASI + MP-BGP
– On VASI: ACL, Policing/Marking MQC, PBR, eBGP or iBGP
48
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Common Issues - TCAM Deny-Jump (1)
• Problem Description:
In ASR 1000 IPsec/FW/NAT deployment, user may see following message:
“%CPP_FM-3-CPP_FM_TCAM_ERROR: F0: cpp_sp: TCAM limit exceeded…”
• Error Message Explanation:
This is an protection mechanism prevents system from crashing with WATCH-DOG timeout error or malloc failure.
• Root Cause Analysis:
1. Classification engine in the TCAM can only represent permit.
2. System convertes the DENY entries into PERMIT ones using cross product
3. This recursive nature cause the required number of entries to “explode”.
49
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Common Issues - TCAM Deny-Jump (2) • Workaround:
1. Before deploying the platform in production, apply the configuration in lab
2. Modify the ACLs to use multiple specific permit statement, and try to reduce or eliminate the explicit use of deny statement
3. Use PBR to bypass NAT
4. Static NAT
• Solutions:
1. IOS XE3.10 introduced the SW classification engine to handle deny-jump like classification
2. System still use TCAM as long as it has room, in case TCAM does not fit, it will switch to SW classification engine.
Original NAT Config VASI & PBR to bypass NAT
ip nat inside source list NAT-ACL pool NAT-POOL overload
!
ip access-list extended NAT-ACL
deny ip any 129.25.0.0 0.0.255.255
permit ip 172.19.0.0 0.0.0.255 any
ip nat inside source list NAT-ACL pool NAT-POOL overload
!
interface GigabitEthernet0/0/1
description nat inside interface
ip address 6.1.1.1 255.255.255.0
ip nat inside
ip policy route-map no-NAT-rmap
interface vasileft1
ip address 13.1.1.1
!
interface vasiright1
ip address 13.1.2.1 255.255.255.0
!
ip access-list extended NAT-ACL
permit ip 172.19.0.0 0.0.0.255 any
ip access-list extended bypass-NAT
permit ip any 129.25.0.0 0.0.255.255
!
route-map no-NAT-rmap permit 10
match ip address bypass-nat
set interface vasileft1
Original NAT Config Identity NAT
ip nat inside source list NAT-ACL pool NAT-POOL overload
!
ip access-list extended NAT-ACL
deny ip host 172.19.1.1 any
permit ip 172.19.0.0 0.0.0.255 any
ip nat inside source static 172.19.1.1 172.19.1.1 no-alias
ip nat inside source list NAT-ACL pool NAT-POOL overload !
ip access-list extended NAT-ACL
permit ip 172.19.0.0 0.0.0.255 any
50
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Common Issues - NAT ADDR ALLOC FAILURE (1) • Problem Description:
In ASR 1000 PAT/Overload configuration, system get error message:
"%NAT-6-ADDR_ALLOC_FAILURE: Address allocation failed; pool 1 may be exhausted”
• Debug Information that should be gathered: show platform hardware qfp active feature nat data pool
show platform hardware qfp active feature nat data port
show platform hardware qfp active feature nat data stat
show platform hardware qfp active feature nat data base
show ip nat translation | inc <global address of interest>
• Common Reason for Failure:
1. Customer has a small pool which is being consumed by non-PATTAble binds.
2. A non-PATtable bind will show in 'sh ip nat trans' as a single local associated with a single global IP address.
3. It consumes an entire address in the pool.
--- 213.252.7.132 172.16.254.242 ---
51
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Common Issues - NAT ADDR ALLOC FAILURE (2) • Solution 1
1. A non-PAttable bind could be created by packet with a non-PATTable protocol.
2. The best way to prevent this is to tighten the ACL to exclude non-PAttable protocols.
• Solution 2
1. A non-PAttable bind could be created by ALG like DNS which does not have ports in its L7 header has requested a global NAT address.
2. Often customers do not need the DNS ALG so the solution is to turn it off.
3. Below shows the most common ALGs which produce non-PAttable binds being turned off.
access-list 100 permit udp 13.1.0.0 0.0.255.255 any
access-list 100 permit tcp 13.1.0.0 0.0.255.255 any
access-list 100 permit icmp 13.1.0.0 0.0.255.255 any
no ip nat service dns udp
no ip nat service dns tcp
no ip nat service netbios-ns tcp
no ip nat service netbios-ns udp
no ip nat service netbios-ssn
no ip nat service netbios-dgm
no ip nat service ldap
52
Summary and Take Away
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
NAT Deployment in Cloud Networks Summary and Take Away …
• Follow proven connectivity models
• Stateless failover with BGP/BFD
• High scale, high performance NAT on ASR 1000
• Monitor key system resources proactively
200Gbps
Cloud Gateway
HA BGP VASI 12M Sess ALG HSL Connectivity NAT/CGN
54
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Relevant Sessions at Cisco Live 2014
Breakout Sessions
• BRKSPG-2602 - IPv4 Exhaustion: NAT and Transition to IPv6 for Service Providers
• BRKARC-2019 – Operating an ASR 1000
• BRKARC-2021 - IOS XE Advanced Troubleshooting (NAT, VPN, FW packet forwarding)
55
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Complete Your Online Session Evaluation
• Give us your feedback and you could win fabulous prizes. Winners announced daily.
• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
56
© 2014 Cisco and/or its affiliates. All rights reserved. BRKDCT-2448 Cisco Public
Continue Your Education
• Demos in the Cisco Campus (ASR1001-X Live Demo)
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
57
Thank you.