1.2012.12.21CloudStack + MidoNet Virtual Router CloudStack MidoNet Xmas Present MidoNet DHCP, Source NAT Router VM MidoNet Agent VM CloudStack MidoNet (Ingress )IP 1 Intelligence at the edge Ingress Point CloudStack MidoNet MidoNet FacebookPage Email : info@midokura.com Facebook Page : http://www.facebook.com/MidokuraJapan/

2. How to replace CloudStacks networking functionality This article explains why one might want to replace or extend CloudStacks existing networking technology. We also go into moderate technical detail on how to do so, using the CloudStack integration of Midokuras MidoNet [1] network virtualization technology as a case study.Why replace CloudStacks networking functionality? Out of the box, CloudStack provides most of its services (such as DHCP, Source NAT) via the CloudStack Virtual Router. The CloudStack Virtual Router is an actual VM; at the time of writing (Dec.21, 2012), it consists of a Debian image with haproxy, dnsmasq etc installed. The number of these Virtual Router VMs launched depends on the type of Guest Network used. For example, when using Advanced networking and Isolated Guest Networks, one Virtual Router VM will be launched per Guest Network. In the VPC mode, the Virtual Router is also responsible for routing traffic among the isolated networks within the VPC. The CloudStack Virtual Router has several disadvantages; one example is inefficient routing of traffic in the VPC mode. Although there may be a large number of VMs running on multiple hosts, there is only one Virtual Router VM responsible for the entire VPC traffic, running on one host. Thus, to route among isolated networks, all traffic must pass through this host and through the Virtual Router VM. Lets take an example (Figure 1): Figure 1: Traffic flow when using CloudStack Virtual Router 3. In the diagram above, if Guest VM 1 on Network 1 wants to send to Guest VM 2 ofNetwork 2, the traffic will first go from Host A to Host B and into the Virtual Router VM,before being directed to Host C. This makes Host B and the Virtual Router VM abottleneck, and for a heavy traffic environment, this is not a scalable solution.Ideally, the traffic in the example above should go directly from VM 1 on Host A to VM2 on Host C, with no reliance on the Virtual Router VM. This is more clearly illustratedwhen you consider a logical topology of the network. By removing the hosts from theequation, we can see that the VMs are connected in the logical network as shown inFigure 2. Note that we have removed the notion of Virtual Router VM and replaced itwith a logical construct called Logical Router, which is a virtual entity that you canapply configurations such as routing and ACL to, and it is not a separate VM. Figure 2: Example logical topologyIf the hypervisor host sending traffic is aware of the logical topology shown above,before a packet is sent out into the physical wire, it can determine the host IP of thedestination VM, and create an IP tunnel to send the packet directly to that host. Thiseffectively removes any reliance on the Virtual Router VM and avoids potential trafficbottleneck.At an extremely high level, this is exactly what MidoNet, a virtual networking platform,does. Through the MidoNet REST API, users can construct logical topologies which arestored in a distributed, fault-tolerant database, and the MidoNet agent running on eachhost can manage tunnels and forward packets to other MidoNet hosts. To plug theVMs into MidoNet networks, however, it must be integrated with CloudStack. Once fullyintegrated, the traffic from VM 1 to VM 2 would flow as shown in Figure 3. 4. Figure 3: Traffic flow using MidoNet instead of CloudStack Virtual Router Enabling this kind of virtualized networking to the cloud platform was one of the main motivations for extending CloudStacks networking functionality. In the rest of this article, we will focus on the details of the actual CloudStack-MidoNet integration work currently in progress.Creating a NetworkElement Network Elements represent CloudStack network components which provide network services (e.g. Source NAT, DHCP). In our case, the MidoNetElement represents MidoNet, and takes care of communicating with the MidoNet API in order to set up the virtual topology and provide various network services like Source NAT and DHCP. Most of the logic of our integration is implemented here. To create your own Element, you need to extend com.cloud.network.element. NetworkElement. Below, we give an overview of the methods the Element must implement, an idea of what each does, and we discuss what the MidoNetElement does as a concrete example. 5. getCapabilities()When you create a network offering, you choose the capabilities you want yournetwork to have (e.g. Source NAT, DHCP), and you then choose the Provider foreach of these capabilities. If and only if a capability is listed by your ElementsgetCapabilities() call, your Element will show up in the dropdown list for Providers ofthat Capability. See screenshot below where we choose MidoNet as the provider forDHCP services in a new NetworkOffering (Figure 4).Figure 4: Screenshot of choosing Capabilities and Providers in a new NetworkOfferingimplement()Called when the first VM is launched on a guest network which uses this Element. Ifusing CloudStacks Virtual Router, the Virtual Router VM would be launched at thistime. In our integration, we add a MidoNet virtual router to the MidoNet virtualtopology at this point, ready for VMs to use when they launch.prepare()Called when a new NIC is added to the guest network, usually meaning just before aVM is launched. In our integration, we add the NIC to the MidoNet virtual topology,for example setting the NIC to be connected to the guest networks MidoNet virtual 6. router. This means when the VM comes up, its NIC will already be correctly connected into the MidoNet topology. release() Called when a NIC is removed from the guest network, usually meaning just after a VM is shut down. In our integration, we remove the NIC from the MidoNet virtual topology, the virtual topology equivalent of plugging the network cable out of the router.Creating a GuestNetworkGuru When a new Guest Network is created in CloudStack (e.g. via the UI or API), CloudStack needs to decide which Guru to use for that network. It looks at all of the Gurus defined in components.xml [2], and tries to load each one. Each Guru has a canHandle() method which inspects the attributes of the Guest Network being created, and returns true if it can handle that network, false if it cant. When creating your own Guru, you need to keep in mind two things: Ensure your canHandle() method returns true only for Guest Networks you can handle and not those you cant For example, the ExternalGuestNetworkGuru returns true if the network isAdvanced and Isolated, false otherwise Ensure your Guru doesnt clash with another If there is another Guru which can handle the same types of networks asyour Guru, two guest networks will be created in CloudStack when you tryto create one Extending CloudStack Networks[2] recommends that you remove theconflicting Guru from components.xml when deploying to avoid this issue To create your own Guru, you need to extend com.cloud.network.guru.GuestNetworkGuru. Below, we give an overview of the callbacks the Guru must implement, and an idea of what each does. In our case, there was very little we needed to do in the Guru, as the Element class takes care of most of the logic. 7. design() Called when the guest network is created, but before any VMs exist on the network. The parent class (GuestNetworkGuru) does preparation such as setting the gateway IP based on the CIDR specified by the user. implement() Called when the first virtual machine is about to be started on the guest network. reserve() Called when a new virtual machine is starting in the network. release() Called when a virtual machine is stopped in the network (release its allocated IP etc). shutdown() Called when all virtual machines within the network have been stopped, and the guest network is garbage collected. trash() Called when a guest network is being deleted.Getting CloudStack to load your classes First, make sure the network classes you have created are registered in components.xml. For example, under We added: and under we added our Provider, MidoNet: 8. Restarting the management server will cause CloudStack to load these classes, and you can now create a NetworkOffering from the UI which uses the Capabilities (e.g. Source NAT, DHCP) provided by your Provider.Creating a NetworkOffering from the UI In the UI, go to Service Offerings and choose Network Offerings from the dropdown. Click Add New Network Offering, and under Supported Services, choose your Provider for any of the Capabilities you wish to use in the Offering, then save the Offering.Create a Guest Network using your NetworkOffering When creating a Guest Network in the UI, the NetworkOffering can be chosen from a dropdown menu. However, there are a few hurdles to watch out for. First, your Provider (e.g. MidoNet) must be enabled on the underlying Physical Network, or else your NetworkOffering wont appear in the dropdown list. You can enable your provider on the physical network using the CloudStack API. For example, to enable the MidoNet Provider using the CloudStack command line tool CloudMonkey [3], you would use a sequence of commands like this: cloudmonkey add networkserviceprovider name=MidoNet physicalnetworkid=$PHYSICALNETWORKID cloudmonkey update networkserviceprovider state=Enabled id=$MIDONETSERVICEPROVIDERID It is also possible to add your Provider to the CloudStack UI - this then allows you to toggle your Provider on and off in Infrastructure > Zones > $ZONENAME > Physical Networks > $PHYSICALNETWORKNAME > NetworkProviders instead of enabling it via the API. We wont go into this part in detail here, but if you search for the Providers which are currently shown in the UI in ui/scripts/system.js, youll quickly see which parts you need to add in that file to display your Provider. 9. Another point to check at this stage is to ensure that the IsolationMethod of the underlying physical network matches what you specified in the canHandle() method of your GuestNetworkGuru. Once all of the above is in place, your NetworkOffering should show up in the dropdown list for Create Guest Network, and you should be able to successfully create a network which uses your new Provider.Plug in the NICThe final step is to plug the VMs into the MidoNet network for connectivity.The method for implementing this in CloudStack differs depending on the hypervisorbeing used. For the first step of our integration, we are using the KVM hypervisor. Forthe KVM case, we override the existing CloudStack VIFDriver so that in the plug method,we: Create a TAP interface (NIC) for the new VM to use Plug the TAP interface (NIC) into the MidoNet datapath, so that traffic to and fromthe VM flows through the MidoNet datapathConclusionWe hope that this overview gave you a concrete example of how to extend CloudStacknetworking, and an idea of what we are aiming to achieve with our MidoNet integrationin CloudStack. We think that the scalability and flexibility of MidoNets virtual networkingcan be a great addition to CloudStack networking.For further information on extending CloudStack networking, see [4].References[1] Midokura MidoNet:http://www.midokura.com/midonet/[2] Extending CloudStack Networking (components.xml section): 10. http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/Extendin g_CloudStack_Networking#Loading_the_components_into_CloudStack[3] CloudStack CloudMonkey CLI: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+CloudMonke y+CLI[4] Extending CloudStack Networking: http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/Extendin g_CloudStack_Networking