Post on 20-Sep-2020
NFV multi-domaines : de la théorie à la pratique
Mathieu GOESSENS : IMT AtlantiqueAhmed BOUABDALLAH : IMT Atlantique
Karine GUILLOUARD : Orange Labs
Journées Cloud12/13 septembre 2018Troyes (France)
Motivations
Network services specificities
Implementation of an NFV multidomain platform
CONTENT
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
Motivations
vnf
vnfvnf
vnf
Domain 1
vnfvnf
vnf
Domain 1
vnf
Domain 2
3
Domain 2
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
Network services specificities
We define network services as low level infrastructures services : Routing, Firewalling, NAT, QoS, DHCP, VPNs, DNS …
Most of those services are tied with lower levels of TCP/IP stack :− Configuration relies on IP addresses, ports, interfaces…→ Changing those IPs means to change the configuration ...− Some of those services (routing, firewalling, NAT...) are handled by software in
kernel not in userland.→ Not as easy to manipulate than high level services (example : copying states of a
statefull firewall versus playing with HTTP)
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
4
Network services specificities
Example : a routing table
Example : a firewall table
Destination Nexthop
192.168.1.0/24 eth0Default (0.0.0.0/0) 192.168.1.254
Source IP Dest IP Src Port Dst Port Other Action192.168.0.0/24 192.168.0.0/24 * * N/A ACCEPT
192.168.0.0/24 Facebook.com 80, 443 N/A DROP
192.168.0.0/24 0.0.0.0/0 * * N/A ACCEPT,
SNAT FROM$PUBLIC_IP
0.0.0.0/0 192.168.0.0/24 * * TCP_state = Established
ACCEPT
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
5
Network services specificities
Example : a DNS zonezone exemple.fr on ns.exemple.fr
exemple.fr NS 192.0.2.53ns.exemple.fr A 192.0.2.53exemple.fr A 192.0.2.80www.exemple.fr A 192.0.2.80
zone .fr :exemple.fr NS 192.0.2.53ns.exemple.fr A 192.0.2.53
Changing the Nameserver IP means changing your own DNS zone.But also the .fr zone… Then you have to talk to your provider, think (even more) about TTL (may be different in each zone), DNS propagation, handling inconsistency meanwhile …When you add secondaries servers :
→ Only IPs are allowed. No DNS. If you renumber you have to change everything, everywhere.
zone "exemple.fr" {type master;file "exemple.fr.db";Allow-transfer {192.0.2.153;};
};
zone "exemple.fr" {type slave;file "exemple.fr.db";Master {192.0.2.53;};
};
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
6
Network services specificities
Some network services are too tied with IP addresses etc to allow to easily manipulate them.
Moving those services implies either to keep those addresses consistent accros domains or to reconfigure everything…(Try to ask your local net admin if he would like to change the router IPs ...)
"Renumbering, especially for medium to large sites and networks, is widelyviewed as an expensive, painful, and error-prone process, and is therefore avoided by network managers as much as possible."
Renumbering Still Needs WorkB. Carpenter, R. Atkinson, H. Flinck, IETF RFC 5887 - May 2010https://tools.ietf.org/html/rfc5887
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
7
Requirements for an implementation of an NFV multidomains platform
We would like to be as close as possible from ETSI Mano framework, to support standards such TOSCA :
Thus we will rely on : − An orchestrator to manage our VNF, VFNFG etc
− A VIM to have the underlying infrastructure.
− A global Network manager, to keep consistent networking accross domains allowing to keep consistent address plans and to be able to have SFC and VFNFG accross domains.
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
8
Components for an implementation of an NFV multidomains platform
For the VIM, our survey showed that we have two serious candidates : Openstack and Kubernetes.→ As Openstack is already a mature, and stable solution, we choose to rely on it.
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
9
Source docs.openstack.org
Components for an implementation of an NFV multidomains platform
As we choose Openstack as a VIM and Openstack provides its own NFV Orchestrator : Tacker. We choose to use it to leverage its integration with the Openstack community & plateform.
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
10
Source wiki.openstack.org/Tacker
Components for an implementation of an NFV multidomains platform
We also need to have a coherent network view to keep our addressplan consistent and to be able do SFC/VNFFG across domains :→ Openstack Tricircle trough its Centralized Neutron which acts as a « Master »
across Regions and their local Neutron.
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
11
Source wiki.openstack.org/Tricircle
Multidomain NFV Platform
Allows us to have a global and consistent view of Neutron networks accros Regions ( datacenters) to manage NFV services
(running in a consumer grade PC with 32GB of Ram)NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
12
Multidomain NFV Platform
But networking consistency is still partialWe have consistent Neutron Networks accros our 3 regions :- private- net0- net1- net_mgmt
But having consistent Neutron Network for public addresses in another issue.
Locally it works but having that in a production network would imply to have your IP range announcement consistent across regions.
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
13
Conclusion
While investigating around our usecase of managing Virtual Network Functions across domains, we identified essential features required for such services.
We proposed a first version of a running platform to test those features.– This platform will be used to experimentally investigate the dynamic delegation of
control in multidomain NFV
We also identify that even if there is a broad literature the implementations are not fully mature for such advanced usecases.– There are still many rooms for improvements for such a platform, for example to
properly isolate administrative domains …– Implementations are still in heavy developpement (for example dynamic
forwarding graphs)
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
14
Multidomain NFV Platform
Kolla-ansible− « Kolla’s mission is to provide production-ready containers and deployment tools
for operating OpenStack clouds. »− Allow us to easily test Tacker− Do not allow to easily use Tricircle− Nice funny feature : make you part of a DDOS Botnet by default (memcached
CVE-2018-1000115)
− http://lists.openstack.org/pipermail/openstack-dev/2018-March/128733.html− or https://frama.link/kollaDDOS − And did not plan to ship corrected packages : «The patch is merged on all
branches but not released»
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
15
Multidomain NFV Plateform
Tacker«Tacker - OpenStack NFV Orchestration»− Hardcode RegionName=RegionOnehttps://github.com/openstack/tacker/blob/fd0600a80f7111e5e231146c97e5cce42
f66d348/devstack/lib/tacker#L433https://bugs.launchpad.net/tacker/+bug/177770
→ Corrected
− Override Neutron configurationhttps://github.com/openstack/tacker/blob/master/devstack/lib/tacker#L218https://bugs.launchpad.net/tacker/+bug/1777170→ Still Waiting
→ We corrected it locally
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
16
Multidomain NFV Plateform
Tricircle− « The Tricircle is to provide networking automation across Neutron in multi-region
OpenStack deployments »− Allow you to deploy cross region neutron networks.− Pre definied and tested topologies− Other Topologies are possible but not documented, not tested, and can trigger
interesting bugs..− Documentation is sometime confusing.
« Non-local router can work only for cross Neutron east-west networking purpose if no external network is set to the router. »
« Non-local router can serve as the centralized north-south traffic gateway if external network is attached to the router, and support east-west traffic at the same time. »
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
17
Multidomain NFV Platform
As said earlier, status of cross-neutron router for North/South + East West networking is a bit confusing.− Documentation is not clear− No proposed topology use it.
But its available and a really interesting topology.In that case Neutron create two routers. One for north/south, one for east/west. − The east/west have default route to north south− The north/south have a default route to east/west− Once you attach an external network. The N/S router default route change to it.− But no specific routes are added for the N/S router.→ It cannot reach other regions VMs. The cannot reach internet
Seems to be a know issue, but no real bug report, no proposed solution.
We have our own workaround (manually add routes). We will propose it to the community : http://paste.openstack.org/show/729915/
https://frama.link/Tricircle2routers
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
18
Multidomain NFV Plateform
Tricircle VM may receive wrong routing information :− Tricircle allocate some shadow openstack ports for the cross neutron networking− In some circonstencies, the Neutron DHCP server(s) can give VMs incorrect
routing informations, giving them a non reachable (non existing) gateway.− Known issue, work in progress and no usuable workaround proposed. − We proposed one : Ancipate the probable wrong routing informations to be send
and simply assign those IPs to the router.https://bugs.launchpad.net/tricircle/+bug/1683261or https://frama.link/TricircleRouting
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
19
Multidomain NFV Plateform
Tricircle floating IP across global Neutron− Floating IP are Openstack way to define public and stable IPs for VM/VNF− Tested with Tricircle
Works locallyDoes not works across Neutron
Lets just check how it works and re-do it− Make the router listen on an IP− SNAT and DNAT from this IP− Script it !http://paste.openstack.org/show/729741/ https://frama.link/TricircleFloating
NFV multi-domaines : de la théorie à la pratiqueM. Goesssens / A. Bouabdallah / K.GuillouardJournées Cloud - 12/13 septembre 2018 - Troyes (France)
20