OpenStack DVR_What is DVR?

What is OpenStack DVR (Distribute Virtual Router)

내것인듯 내것아닌 내것같은 Neutron/DVR

Yongyoon SHIN. SDN section, ETRI

Let’s Talk

• OpenStack and Distribute Virtual Router

– OpenStack

– What is DVR

• DVR Case

– East/West Traffic

– SNAT

– Floating IP

OpenStack Naming

• OpenStack Version Naming – Austin: Austin, Texas – Bexar: San Antonio, Texas – Cactus: Cactus, Texas – Diablo: Santa Clara, CA – Essex: Boston, Massachusetts – Folsom: San Francisco, CA – Grizzly: San Diego, CA – Havana: Portland, Oregon – Icehouse: Hong Kong – Juno: Atlanta, Georgia – Kilo: Paris, France – Liberty: Vancouver, British Columbia – M*: Tokyo, Japan (https://wiki.openstack.org/wiki/Release_Naming/M_Proposals) – N*: Austin, Texas – O*: Barcelona, Spain

Quantum

Neutron

Nova Network

* 한국은 Vancouver summit에서 2019년 summit 신청 (Naming 순서는 ‘W’) .. 2019년에도 OpenStack이 지금처럼 번성할까? ..

OpenStack Release

• Main Schedule (Neutron)

OpenStack Source

• github 사용

• Neutron

– 현재까지 업데이트 중

• 2015.06.11. 현재 8일전 update

3 Main Services in Neutron

• East-West Communication

– Data Center 내부, VM 사이의 IP Traffic 처리

• Floating IP (aka DNAT: Destination NAT)

– Public Network을 통하여 직접 VM에 접근 가능할 수 있도록 VM에 Public IP를 제공

• Shard IP (aka SNAT: Source NAT)

– Data Center 내부 VM들이 외부 IP를 공용으로 사용하여 public Network에 접속할 수 있도록 함

Main Idea of DVR

• Network Node의 병목현상 해소

– L3 Agent 기능을 모든 Compute Node에 분산

• 네트워크 노드의 단일 장애점 및 트래픽 집중문제 해결

Compute Node A

VM VM VM

VM VM VM

10.0.10.0/24 129.254.172.0/24

129.254.172.0/24 20.0.20.0/24

DVR Public Network

(129.254.172.0/24)

Compute Node B

VM VM VM

VM VM VM

10.0.10.0/24 129.254.172.0/24

129.254.172.0/24 20.0.20.0/24

DVR

DVR vs without DVR

• Without DVR

• With DVR

Compute Node A

VM

Network Node

Compute Node B

br-int

br-tun

VM

br-int

br-tun

br-int

br-tun

eth eth eth

VR

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

OpenStack DVR

• OpenStack major work

– Juno version release (~ing)

• 리눅스의 Name Space 사용

OpenStack DVR

• Network/Compute Node NameSpace

Network Node

Compute Node

VM01 (Network A)

VM02 (Network B)

br-int

br-ex br-tun

dnsmasq dnsmasq

br-int

br-ex br-tun

dhcp NameSpace dhcp NameSpace

Router NameSpace Router NameSpace

SNAT NameSpace

patch-tun

patch-int

patch-tun

patch-int

eth0 eth0 vxlan-xxxx vxlan-yyyy vxlan-xxxx vxlan-yyyy

FIP NameSpace

qvoVM1 qvoVM2

qrouter-xxx

fip-xxx

qr-aaa qr-bbb

rfp-xxx

fpr-xxx

fg-yyy

qdhcp-kkk qdhcp-qqq

tap123abc tap456def

int-br-ex

phy-br-ex

qrouter-xxx

snat-xxx

qr-aaa qr-bbb sg-123aaa sg-456bbb

qg-789abc

External

Inter subnet routing East-West

• ML2 plugin

• l2pop mechanism driver enabled

• Openvswitch mechanism driver enavled

• ovs agent installed on all of compute nodes

• Tunneling (only use VXLAN / GRE)

– Kilo version support VLAN


• Compute Node Router

– Compute Node A

– Compute Node B

• Controller Node - IP: 192.168.56.20 - DevStack All_in_one • Compute Node - IP: 192.168.56.25 - DevStack Compute • Network Node - IP: 192.168.56.30 - DevStack Network


• Router namespace created on both nodes

– Same interfaces, MAC, IP


• ARP Table

– Compute Node A

• vm01(100.0.0.2) / vm03(100.0.20.2)

• ARP Table: vm01/vm02/vm03

– Compute Node B

• vm02(100.0.10.2)

• ARP Table: vm01/vm02/vm03


• Flows

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤

⑥

⑦

⑧

src MAC dst MAC src IP dest IP

Blue VM Blue Router Interface Blue VM Red VM


• Flows

– ARP Tables

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ② ③

④

⑤

⑥

⑦

⑧


Red Router Interface Red VM Blue VM Red VM


• Flows

– Matched Red VM MAC

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤

⑥

⑦

⑧




• Flows

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤

⑥

⑦

⑧


Compute Node A MAC Red VM Blue VM Red VM


• Flows

– Encapsulation via VXLAN/GRE tunnel

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤ VXLAN / GRE

⑥

⑦

⑧




• Flows

– Tunnel encapsulation is stripped, tagged with a local VLAN

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤

⑥

⑦

⑧




• Flows

– Traffic matched remote host MAC

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤

⑥

⑦

⑧




• Flows

– Traffic matched remote host MAC, Red Network VLAN tag

– VLAN tag stripped

Compute Node A

VM

Compute Node B

br-int

br-tun

VM

br-int

br-tun

eth eth

VR VR

① ②

③

④

⑤

⑥

⑦

⑧




• East/West Traffic Flow summary Compute Node B

VM03 (Network A)

br-int

br-ex br-tun

Router NameSpace

patch-tun

patch-int

eth0 vxlan-xxxx vxlan-yyyy

qvoVM3

qrouter-xxx

qr-aaa qr-bbb

Compute Node A

VM01 (Network A)

VM02 (Network B)

br-int

br-ex br-tun

Router NameSpace

patch-tun

patch-int


FIP NameSpace

qvoVM1 qvoVM2

qrouter-xxx

fip-xxx

qr-aaa qr-bbb

rfp-xxx

fpr-xxx

fg-yyy

Distribute Virtual Routing - SNAT

• SNAT : Source NAT

– Floating IP를 설정하지 않은 VM이 외부 통신

– 라우터는 외부 네트워크에서 단일 IP를 할당 받음 (VM은 IP 공유)

• Still Centralized

– 현재 완전 분산 형태로 진행 중 (neutron blueprint)

• SNAT를 사용하지 않아도 되는 경우 발생 (VM이 외부망을 사용할 필요 없음) Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

eth1

br-ex

qr1 qr2 sg1 sg2 qg


• Network node

– Neutron l3_agent.ini

– Network node Interface


• Network Node IP Rule


• Compute Node A – IP Rule

• Compute Node B – IP Rule

모든 컴퓨트 노드의 라우팅 룰은 라우터와 연결된 모든 서브넷에 대한 내용

dvr_snat 노드의 SNAT 네임스페이스에 대한 내용


• Flows

Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

① ②

③

④

⑤

⑥

⑦


Blue VM Blue qr Interface Blue VM ex) Google IP

eth1

br-ex

qr1 qr2 sg1 sg2 qg ⑧


• Flows

Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

① ② ③

④

⑤

⑥

⑦


Blue VM Blue qr Interface Blue VM ex) Google IP

eth1

br-ex



• Flows

Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

① ②

③

④

⑤

⑥

⑦


Blue qr Interface Blue sg Interface Blue VM ex) Google IP

eth1

br-ex



• Flows

Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

① ②

③

④

⑤

⑥

⑦


Compute Node A MAC Blue sg Interface Blue VM ex) Google IP

eth1

br-ex



• Flows

Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

① ②

③

④

⑤

⑥

⑦


Blue qr Interface Blue sg Interface Blue VM ex) Google IP

eth1

br-ex



• Flows

Compute Node A

VM

Network Node

br-int

br-tun

br-int

br-tun

eth1 eth0

VR VR(SNAT)

① ②

③

④

⑤

⑥

⑦


qg Interface Default GW MAC qg Interface ex) Google IP

eth1

br-ex

qr1 qr2 sg1 sg2 qg

⑧


• SNAT Traffic Flow summary Network Node

Compute Node

VM01 (Network A)

VM02 (Network B)

br-int

br-ex br-tun

dnsmasq dnsmasq

br-int

br-ex br-tun



SNAT NameSpace

patch-tun

patch-int

patch-tun

patch-int


FIP NameSpace

qvoVM1 qvoVM2

qrouter-xxx

fip-xxx

qr-aaa qr-bbb

rfp-xxx

fpr-xxx

fg-yyy

qdhcp-kkk qdhcp-qqq

tap123abc tap456def

int-br-ex

phy-br-ex

qrouter-xxx

snat-xxx


qg-789abc

Distribute Virtual Routing – Floating IP

• Compute Node의 FIP(Floating IP) namespace

– rfp: Router NameSpace에 생성

– fpr: FIP NameSpace에 생성 Compute Node

VM01 (Network A)

VM02 (Network B)

br-int

br-ex br-tun

Router NameSpace

patch-tun

patch-int


FIP NameSpace

qvoVM1 qvoVM2

qrouter-xxx

fip-xxx

qr-aaa qr-bbb

rfp-xxx

fpr-xxx

fg-yyy

Veth pair

Compute Node A

VM

br-int

VR

qr

FIP

br-ex

eth0

rfp - 129.254.171.0/24 fpr - 129.254.171.0/24

fg


• FIP NameSpace 생성 및 External Network 확인

– Compute Node A

– Compute Node B


• veth pare

– Compute Node A

• Router NameSpace

– Compute Node B

• Router NameSpace


• Veth pair

– Compute Node A

• FIP NameSpace

• External Device: fg

– Compute Node B

• FIP NameSpace

• External Device: fg


• Routing Rule

– Compute Node A

– Compute Node B


• Floating IP Packet Tracking

– Compute Node A

– Compute Node B


Compute Node A

VM

br-int

VR

① qr

FIP

br-ex

eth0

rfp - 129.254.171.0/24 fpr - 129.254.171.0/24

fg


Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP

④

⑤

②

③


Compute Node A

VM

br-int

VR

①

②

qr

FIP

br-ex

eth0

rfp - 129.254.171.0/24 fpr - 129.254.171.0/24

fg


Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP

③ ④

⑤


Compute Node A

VM

br-int

VR

① qr

FIP

br-ex

eth0

rfp - 129.254.171.0/24 fpr - 129.254.171.0/24

fg


rfp fpr Blue VM Floating IP ex) Google IP

④ ⑤

②

③


Compute Node A

VM

br-int

VR

① qr

FIP

br-ex

eth0

rfp - 129.254.171.0/24 fpr - 129.254.171.0/24

fg


fg default GW Blue VM Floating IP ex) Google IP

⑤ ④

②

③

Distribute Virtual Routing - Floating IP

• Floating-IP Traffic Flow summary Network Node

Compute Node

VM01 (Network A)

VM02 (Network B)

br-int

br-ex br-tun

dnsmasq dnsmasq

br-int

br-ex br-tun



SNAT NameSpace

patch-tun

patch-int

patch-tun

patch-int


FIP NameSpace

qvoVM1 qvoVM2

qrouter-xxx

fip-xxx

qr-aaa qr-bbb

rfp-xxx

fpr-xxx

fg-yyy

qdhcp-kkk qdhcp-qqq

tap123abc tap456def

int-br-ex

phy-br-ex

qrouter-xxx

snat-xxx


qg-789abc

OpenStack DVR Summary

특성 Forwarding Mechanism

Local • Source/Destination IP는 동일한 Subnet에 속함 • VM들은 동일한 노드에 배열

br-int는 관련된 MAC Address를 학습하고 로컬 트래픽을 전달

Remote • Source/Destination IP는 동일한 Subnet에 속함 • VM들은 서로 다른 노드에 배열

Segmentation type(VXLAN/GRE)에 따라 전달

East/West • Source/Destination IP는 다른 Subnet에 속함 컴퓨트 노드의 라우터 인터페이스에 의해 전달

SNAT (Source NAT)

• Destination IP는 컴퓨트 노드가 알고 있는 Subnet에 속하지 않음

• VM에 Floating IP 설정이 되어 있지 않음

dvr_snat 노드 상의 SNAT 네임스페이스에 존재하는 IP 매칭에 따라 전달

FIP (Floating IP)

• Destination IP는 컴퓨트 노드가 알고 있는 Subnet에 속하지 않음

• VM에 Floating IP 설정

컴퓨트 노드의 FIP 네임스페이스와 연결된 컴퓨트 노드 라우터에 전달

OpenStack에서는 앞으로…

• Juno 배포 이후 업데이트 예정 – L3 고가용성 지원

• HA 지원 / Cascading 관련 내용 Candidate

– Security Groups 개선 • VM의 증가로 인하여 iptables의 필터링 규칙 폭증 -> 패킷 필터링 성능 저하

• Kilo 배포 이후 추가 사항 – VPNaaS (Virtual Private Network as a Service) 지원 – VLAN 지원 – IPv6 지원 – 분산 DHCP 서비스 – 분산 SNAT 서비스

Yongyoon. SHIN (SDN Research Section, ETRI) Email: [email protected]

http://uni2u.meximas.com

Thank you

OpenStack DVR_What is DVR?

Internet

Transcript of OpenStack DVR_What is DVR?