Demo: OpenStack + Cumulus VX "Rack-on-a-Laptop" Part II (L3 to the Host)

Follow

Important: This demo is not approved for use in a production environment and is for demonstration purposes only.

One of the greatest strengths of Cumulus Linux is its powerful, yet easy to configure routing technology. This demonstration shows how to leverage BGP unnumbered interfaces all the way down to the OpenStack hosts. This implementation has many potential advantages:

  1. Significant reduction in IP addressing in both the server and switch environment.
  2. Easier VTEP path troubleshooting because each entity is configured with BGP and has its own autonomous system number (ASN).  VTEPs are placed on the hosts in this OpenStack environment.
  3. No spanning tree protocol (STP) at all in the entire infrastructure, resulting in an infrastructure that can scale without the worries of traditional limitations.
  4. Load balancing achieved using layer 3 ECMP, instead of layer 2 load balancing mechanisms.
  5. Unparalleled server/rack mobility. It's easy to take a physical server from one rack and move it to another rack without any IP addressing changes.
  6. It uses Quagga, which is available for any Linux distribution, but tested and optimized for use with Cumulus Linux switches.

{{table_of_contents}}

Overview

This demo follows the Demo: OpenStack + Cumulus VX "Rack-on-a-Laptop" Part I (L2+MLAG, ML2) article. Unlike the previous demo, which utilizes the Cumulus Linux Modular Layer 2 (ML2) Mechanism Driver for OpenStack with the VLAN type driver, this demo utilizes the VXLAN type driver, with layer 3 networking used throughout from the spine switches all the way to the actual hosts. This is a different methodology for networking that adds simplicity and mobility for system administrators and networking engineers alike.

Prerequisites

This demo requires a basic understanding of OpenStack, layer 3 networking technologies and the BGP unnumbered routing protocol methodology.

System Requirements

This demo runs in an Oracle VirtualBox VM, which has the following minimum requirements:

Feature Minimum Requirements
Operating system Windows/Mac OS/Linux
Oracle VirtualBox Version 5.0
RAM

8GB

Tip: Shut down all other memory-intensive apps before running the demo.

Hard disk 60GB
CPU Intel Core i5
Web Browser Chrome, Firefox, Safari

Preparing the Environment

This demo occurs within a series of VirtualBox VMs. The VMs are described as follows:

VM Name OS Purpose
RDO Server1 CentOS7/RDO (Liberty Release)

RDO project network/controller/compute node. Uses the Linux bridge and not OVS bridge for simplicity.

Note: For simplicity, the network node, controller and a compute node have been combined on a single server. These are typically separate in a more realistic environment.

RDO Server2 CentOS7/RDO (Liberty Release) RDO project compute node.
CumulusLeaf1-2.5.5 Cumulus VX 2.5.5 Leaf (top of rack) switch 1.
CumulusLeaf2-2.5.5 Cumulus VX 2.5.5 Leaf (top of rack) switch 2.
CumulusSpine1-2.5.5 Cumulus VX 2.5.5 Spine switch 1.
Ext_Rtr Debian Jessie External router.

To prepare the demo environment, do the following:

  1. Download the OpenStack demo OVA file from the Cumulus Networks Box.com account. The image file is approximately 4.2GB in size.

  2. Launch VirtualBox.

    1. Import the OVA by selecting File -> Import Appliance….

      Note: If you are reinstalling the demo, make sure you previously deleted the VMs and all associated files.

    2. Accept the default values when prompted during the configuration of the appliance.

      Important: Ensure the Reinitialize the MAC address of all network cards option is unchecked when prompted.

    Note: The import process can take 2 to 3 minutes, depending on your hardware.

  3. Start all six virtual machines imported into VirtualBox.

    Note: The start order does not matter, however you should wait a couple of minutes for the six VMs to complete the boot process.

  4. Once all six VMs are up and running, open the following browser tabs and log in to each server and switch using the username and passwords below:

    Tab URL Application Authentication (Username/Password)
    http://localhost:8080 Horizon Dashboard demo/cumulus
    http://localhost:8800 Server1 - Controller / Network Node / Compute Node cumulus/cumulus
    http://localhost:8801 Server2 - Compute Node cumulus/cumulus
    http://localhost:8802 Cumulus Leaf 1 cumulus/cumulus
    http://localhost:8803 Cumulus Leaf 2 cumulus/cumulus
    http://localhost:8804 Cumulus Spine 1 cumulus/cumulus
    http://localhost:8805 External Router (to Internet) cumulus/cumulus

    Note: These browser tabs are provided for your convenience so that you do not need to use the VirtualBox console.

    Important: If you do not log in through the browser, it may time out after 60 seconds.

    Note: The browser view of the switches and Linux compute nodes may not be typical and secure at a customer site. This is for demo simplicity and convenience purposes. However, the Horizon dashboard is a browser-based UI that OpenStack customers use.

  5. Run the following commands on the Cumulus VX leaf switches to confirm the baseline configuration:

    cumulus@leaf1:~$ ifquery -a
    auto lo 
    iface lo inet loopback
      address 192.168.200.X/32
    
    auto eth0
    iface eth0
      address 192.168.100.X/24
      post-up ip route add 0.0.0.0/0 via 192.168.100.1
    
    auto swp32s0
    iface swp32s0 
       alias to server1
    
    auto swp49
    iface swp49
       alias to spine1
    cumulus@leaf1$ netshow int
    --------------------------------------------------------------------                                     
    To view the legend,  rerun "netshow" cmd with the  "--legend" option                                     
    --------------------------------------------------------------------                                     
        Name                  Speed         MTU  Mode        Summary                                         
    --  --------------------  ----------  -----  ----------  --------------------------------------------    
    UP  eth0                  10G          1500  Mgmt        IP: 192.168.100.3/24                            
    UP  lo                    N/A         16436  Mgmt        IP: 127.0.0.1/8, 192.168.200.3/32, ::1/128      
    UP  swp32s0 (to server1)  10G(4x10G)   1500  Unnumbered  remote port: enp0s9 , remote device: server1    
    UP  swp49 (to spine1)     10G          1500  Unnumbered  remote port: swp1 , remote device: spine1 
    
    
    cumulus@leaf1$ netshow lldp
    --------------------------------------------------------------------                                              
    To view the legend,  rerun "netshow" cmd with the  "--legend" option                                              
    --------------------------------------------------------------------                                              
    Local Port            Speed       Mode              Remote Port        Remote  Host                               
    --------------------  ----------  ----------  ----  -----------------  --------------                             
    eth0                  10G         Mgmt        ====  eth0               spine1
                                                  ====  08:00:27:d0:d5:38  server1
                                                  ====  eth0               leaf2
    swp32s0 (to server1)  10G(4x10G)  Unnumbered  ====  enp0s9             server1
    swp49 (to spine1)     10G         Unnumbered  ====  swp1               spine1
    cumulus@leaf1$ sudo vtysh -c "show run"
    Building configuration...                                                                                
                                                                                                             
    Current configuration:                                                                                   
    !                                                                                                        
    log file /var/log/quagga/bgpd.log                                                                        
    log timestamp precision 6                                                                                
    hostname leaf1-quagga                                                                                    
    username cumulus nopassword                                                                              
    !                                                                                                        
    service integrated-vtysh-config                                                                          
    !                                                                                                        
    password cn321                                                                                           
    enable password cn321                                                                                    
    !                                                                                                        
    interface eth0                                                                                           
     link-detect                                                                                             
    !                                                                                                        
    interface lo                                                                                             
     link-detect                                                                                             
    !                                                                                                        
    interface swp32s0                                                                                        
     ipv6 nd ra-interval 5                                                                                   
     link-detect                                                                                             
     no ipv6 nd suppress-ra                                                                                  
    !                                                                                                        
    interface swp49                                                                                          
     ipv6 nd ra-interval 5                                                                                   
     link-detect                                                                                             
     no ipv6 nd suppress-ra                                                                                  
    !                                                                                                        
    router bgp 65103                                                                                         
     bgp router-id 192.168.200.3                                                                             
     neighbor swp49 interface                                                                                
     neighbor swp49 remote-as external                                                                       
     neighbor swp49 capability extended-nexthop                                                              
     neighbor swp32s0 interface                                                                              
     neighbor swp32s0 remote-as external                                                                     
     neighbor swp32s0 capability extended-nexthop                                                            
    !                                                                                                        
    ip forwarding                                                                                            
    ipv6 forwarding                                                                                          
    !                                                                                                        
    line vty                                                                                                 
    !                                                                                                        
    end 
  6. Run the following commands on the Cumulus spine switch to confirm the baseline configuration:
    cumulus@spine1$ ifquery -a
    auto lo
    iface lo inet loopback 
      address 192.168.200.5/32
    
    auto eth0 
    iface eth0 
      address 192.168.100.5/24
      post-up ip route add 0.0.0.0/0 via 192.168.100.1
    
    auto swp1
    iface swp1
      alias to leaf1
    
    auto swp2
    iface swp2
      alias to leaf2
    
    cumulus@spine1$ netshow int
    --------------------------------------------------------------------                                     
    To view the legend,  rerun "netshow" cmd with the  "--legend" option                                     
    --------------------------------------------------------------------                                     
        Name             Speed      MTU  Mode        Summary                                                 
    --  ---------------  -------  -----  ----------  ------------------------------------------              
    UP  eth0             10G       1500  Mgmt        IP: 192.168.100.5/24                                    
    UP  lo               N/A      16436  Mgmt        IP: 127.0.0.1/8, 192.168.200.5/32, ::1/128              
    UP  swp1 (to leaf1)  10G       1500  Unnumbered                                                          
    UP  swp2 (to leaf2)  10G       1500  Unnumbered
    
    cumulus@spine1$ netshow lldp
    --------------------------------------------------------------------                                     
    To view the legend,  rerun "netshow" cmd with the  "--legend" option                                     
    --------------------------------------------------------------------                                     
    Local Port       Speed    Mode              Remote Port        Remote  Host                              
    ---------------  -------  ----------  ----  -----------------  --------------                            
    eth0             10G      Mgmt        ====  08:00:27:d0:d5:38  server1                                   
                                          ====  eth0               leaf2                                     
                                          ====  eth0               leaf1                                     
    swp1 (to leaf1)  10G      Unnumbered  ====  swp49              leaf1                                     
    swp2 (to leaf2)  10G      Unnumbered  ====  swp49              leaf2 
    
    cumulus@spine1$ sudo vtysh -c "show run"
    Building configuration...                                                                                
                                                                                                             
    Current configuration:                                                                                   
    !                                                                                                        
    log file /var/log/quagga/bgpd.log                                                                        
    log timestamp precision 6                                                                                
    hostname spine1-quagga                                                                                   
    username cumulus nopassword                                                                              
    !                                                                                                        
    service integrated-vtysh-config                                                                          
    !                                                                                                        
    password cn321                                                                                           
    enable password cn321                                                                                    
    !                                                                                                        
    interface eth0                                                                                           
     link-detect                                                                                             
    !                                                                                                        
    interface lo                                                                                             
     link-detect                                                                                             
    !                                                                                                        
    interface swp1                                                                                           
     ipv6 nd ra-interval 5                                                                                   
     link-detect                                                                                             
     no ipv6 nd suppress-ra                                                                                  
    !                                                                                                        
    interface swp2                                                                                           
     ipv6 nd ra-interval 5                                                                                   
     link-detect                                                                                             
     no ipv6 nd suppress-ra                                                                                  
    !                                                                                                        
    router bgp 65105                                                                                         
     bgp router-id 192.168.200.5                                                                             
     neighbor swp1 interface                                                                                 
     neighbor swp1 remote-as external                                                                        
     neighbor swp1 capability extended-nexthop                                                               
     neighbor swp2 interface                                                                                 
     neighbor swp2 remote-as external                                                                        
     neighbor swp2 capability extended-nexthop                                                               
    !                                                                                                        
    ip forwarding                                                                                            
    ipv6 forwarding                                                                                          
    !                                                                                                        
    line vty                                                                                                 
    !                                                                                                        
    end 
    
  7. Configure the OpenStack host interfaces. A virtual IP address is required to use as the VXLAN tunnel end point (VTEP). In this demo, the VTEP is configured on the loopback interface. This is because the server sourced a packet to another VTEP, and it uses the IP configured on the loopback. Attempts were made to use a dummy interface and use that as the source IP but the source IP address selection in Linux would select the wrong IP and results in VXLAN forwarding failures. Fortunately, this problem is easily solved by setting a route map with ip protocol bgp, which sets the source IP address setting to whatever IP address you want for all locally-sourced packets. This configuration is now used in the setup.

    The interfaces facing the Cumulus VX leaf switches must be up and have IPv6 enabled.

    The interfaces facing the external router only need to be enabled. In this demo external networks are provided using the OpenStack flat network design, as configured in the following files:

    /etc/sysconfig/network-scripts/ifcfg-lo:1

    NAME="lo:1"
    DEVICE="lo:1"
    ONBOOT=yes
    NETBOOT=no
    IPV6INIT=yes
    BOOTPROTO=static
    NETMASK=255.255.255.255
    IPADDR=192.168.200.1
    TYPE=Ethernet

    /etc/sysconfig/network-scripts/ifcfg-enp0s9

    NAME="enp0s9"
    DEVICE="enp0s9"
    ONBOOT=yes
    NETBOOT=no
    IPV6INIT=yes
    BOOTPROTO=none
    TYPE=Ethernet

    /etc/sysconfig/network-scripts/ifcfg-enp0s10 (on the network node)

    NAME="enp0s10"
    DEVICE="enp0s10"
    ONBOOT=yes
    NETBOOT=no
    IPV6INIT=yes
    BOOTPROTO=none
    TYPE=Ethernet
  8. Configure the Neutron layer 3 agent. This demo uses the Linux bridge agent. It is easier to comprehend how OpenStack forwards frames using this agent. The other agent, OVS, is much more complex but contains many more features than the Linux bridge, such as Distributed Virtual Routing for better load distribution of east-west traffic.

    /etc/neutron/l3_agent.ini

    [DEFAULT]
    # Example of interface_driver option for LinuxBridge
    interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
    
    # The working mode for the agent. Allowed values are:
    # - legacy: this preserves the existing behavior where the L3 agent is
    #   deployed on a centralized networking node to provide L3 services
    #   like DNAT, and SNAT. Use this mode if you do not want to adopt DVR.
    # - dvr: this mode enables DVR functionality, and must be used for an L3
    #   agent that runs on a compute host.
    # - dvr_snat: this enables centralized SNAT support in conjunction with
    #   DVR. This mode must be used for an L3 agent running on a centralized
    #   node (or in single-host deployments, e.g. devstack).
    # agent_mode = legacy
    agent_mode = legacy
  9. Configure the Neutron layer 2 agent.

    /etc/neutron/plugins/ml2/ml2_conf.ini

    [ml2]
    # (ListOpt) List of network type driver entrypoints to be loaded from
    # the neutron.ml2.type_drivers namespace.
    #
    # type_drivers = local,flat,vlan,gre,vxlan,geneve
    type_drivers = vlan,vxlan,flat
    
    
    # (ListOpt) Ordered list of network_types to allocate as tenant
    # networks. The default value 'local' is useful for single-box testing
    # but provides no connectivity between hosts.
    #
    # tenant_network_types = local
    tenant_network_types = vxlan
    
    
    # (ListOpt) Ordered list of networking mechanism driver entrypoints
    # to be loaded from the neutron.ml2.mechanism_drivers namespace.
    # mechanism_drivers =
    mechanism_drivers = linuxbridge,l2population
    
    [ml2_type_flat]
    # (ListOpt) List of physical_network names with which flat networks
    # can be created. Use * to allow flat networks with arbitrary
    # physical_network names.
    #
    flat_networks = physnet1
    
    
    [ml2_type_vxlan]
    # (ListOpt) Comma-separated list of <vni_min>:<vni_max> tuples enumerating
    # ranges of VXLAN VNI IDs that are available for tenant network allocation.
    #
    # vni_ranges =
    vni_ranges =10:100
    

    /etc/neutron/plugins/ml2/linuxbridge_agent.ini

     [linux_bridge]
    # for the flat network external net
    physical_interface_mappings = physnet1:enp0s10
    
    [vxlan]
    # (BoolOpt) enable VXLAN on the agent
    # VXLAN support can be enabled when agent is managed by ml2 plugin using
    # linuxbridge mechanism driver.
    enable_vxlan = True
    #
    # (IntOpt) use specific TTL for vxlan interface protocol packets
    # Default is set to ttl of 1
    ttl = 10
    
    # (StrOpt) Local IP address to use for VXLAN endpoints (required)
    local_ip = 192.168.200.1
    #
    # (BoolOpt) Flag to enable l2population extension. This option should be used
    # in conjunction with ml2 plugin l2population mechanism driver (in that case,
    # both linuxbridge and l2population mechanism drivers should be loaded).
    # It enables plugin to populate VXLAN forwarding table, in order to limit
    # the use of broadcast emulation (multicast will be turned off if kernel and
    # iproute2 supports unicast flooding - requires 3.11 kernel and iproute2 3.10)
    l2_population = True
    
    [securitygroup]
    # Firewall driver for realizing neutron security group function
    # firewall_driver = neutron.agent.firewall.NoopFirewallDriver
    # Example: firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
    
  10. Configure Quagga on the compute nodes for layer 3 connectivity to the hosts. The demo uses the Cumulus Linux Quagga package from the Cumulus Networks GitHub repository, compiled to work with CentOS7, and generated RPMs and SRPMs. RPM uses systemd to manage bringing up of the zebra and bgpd services. In this example the bgpd and zebra configurations are in separate files. A future version of this demo will merge both into a single Quagga.conf file.

    /etc/quagga/bgpd.conf

    router bgp 65101
     network 192.168.200.1/32
     neighbor enp0s9 interface
     neighbor enp0s9 remote-as external
     neighbor enp0s9 capability extended-nexthop
    !
    route-map setvtepsrc permit 10

    /etc/quagga/zebra.conf

    interface enp0s3                                                                            
     link-detect                                                                                
    !
    interface enp0s8                                                                            
     link-detect                                                                                
    !
    interface enp0s9                                                                            
     link-detect                                                                                
     no ipv6 nd suppress-ra                                                                     
     ipv6 nd ra-interval 5
    !
    interface enp0s10                                                                           
     link-detect                                                                                 
    !
    route-map setvtepsrc permit 10
     set src 192.168.200.1
    !
    ip forwarding                                                                               
    !
    ip protocol bgp route-map setvtepsrc  
    
  11. Confirm the instances on the Horizon dashboard:

    • Click the Instances tab under the System Menu section to confirm that there are instances running.
    • Click the Admin -> Networks and Admin -> Routers section to confirm that there are no networks and routers shown.
    • Click the System -> Hypervisor section to confirm that OpenStack can see two hypervisors (server1, server2).

Running the Demo

You can run the demo with a single tenant and either one or two networks.

Demo One: Single Tenant, One Network

In this scenario, you use OpenStack HEAT to create one broadcast domain that spans two compute nodes. Each compute node has one OpenStack VM in the broadcast domain. The broadcast domain runs over VLXLAN tunnels that are also dynamically created by OpenStack. To manage BUM (Broadcast, Unknown unicast and Multicast) packets, OpenStack has technology similar to LNV in Cumulus Linux called L2population. L2population depends on the VXLAN kernel module features added between 3.7 and 3.11, so is it generally better to run a kernel 3.11+. The demo server uses Kernel 4.4.

In this demo environment OpenStack creates a VXLAN domain. It picks a VXLAN number identifier (VNI) from a range provided to Neutron. On the compute nodes — server1 and server2 — a new bridge is created that contains the interface to the OpenStack VM and a VXLAN interface.

No dynamic provisioning occurs on the Cumulus VX switch.

All VTEP IPs are distributed between the hosts, using BGP unnumbered. 

Note: To access the netshow int output that shows OpenStack data, run source keystonerc_demo from the terminal to log in to OpenStack as an admin.

To run this demo, log into the RDO Server1 (http://localhost:8800) and follow the instructions on the /etc/MOTD message.

cd $HOME/cumulus_demo
./one_tenant_subnet_demo.sh

Using the demo script, you can start, verify and destroy.

The demo script walks you through the steps you need to take to view the demo in action. Specifically:

  1. Log in to the Horizon dashboard, then connect to the VMs using a console after they are created and ping the VMs.
  2. Observe OpenStack networking using netshow, ip link show and ip netns commands.
  3. Ping from the VMs to the external router.

Verifying the Demo

Here are some screen shots from the OpenStack Horizon dashboard and from a leaf switch that show what happens after provisioning a single subnet in a single tenant.

 

Demo Two: Single Tenant, Two Networks

This is similar to the one tenant/one subnet demo, but instead of creating only one broadcast domain, it creates two subnets in a single tenant, and puts two OpenStack VMs on each subnet. Again, it uses OpenStack HEAT to perform this task.

The demo is also performed from RDO Server1 (http://localhost:8800) and the MOTD has instructions on how to execute the demo.

cd $HOME/cumulus_demo
./two_tenant_subnets_demo.sh

Using the demo script, you can start, verify and destroy. 

The demo script walks you through the steps you need to take to view the demo in action. Specifically:

  1. Log in to the Horizon dashboard, then connect to the VMs using a console after they are created and ping the VMs.
  2. Observe OpenStack networking using netshow, ip link show and ip netns commands.
  3. Ping from the VMs to the external router. 

Verifying the Demo

Here are some diagrams from the OpenStack Horizon dashboard and from a leaf switch that show what happens after provisioning two networks in a single tenant:

 

That's it!

Caveats

  • This is for demo purposes only, and not for production use cases.
  •  VXLAN offload NICs are recommended for use in a physical environment.

FAQs

Q: How can I get help on running this demo?

A: Although this demo is unsupported, feel free to join in the discussion on the Open Networking Community post.

Q: After a Cumulus VX instance boots up it reports an error message stating that DBUS is not installed. Is this an error of concern?

A. No. This is a benign error, and can be ignored.

Q: In the single tenant/2 network demo, when I run netshow interface or ip addr show, the default gateways of the OpenStack instances (VM) are not present. Where are the default gateways of the subnet?

A: The default gateways of the subnets are located in an IP namespace located on the network node (server1). Run the following command to view the "router" config:

[cumulus@server1] source $HOME/keystonerc_demo
[cumulus@server1 cumulus_demo(keystone_demo)]$ netshow int
(code done..output todo)

Q: When I reboot the OpenStack server using the reboot or shutdown commands, the VM appears to hang. What do I do?

A: Force a reload via the VirtualBox Manager.

Q: May I re-distribute this demo OVA file to customers and prospects?

A: Yes. The OVA contains language that it is being provided on an as-is basis with no warranty nor support.

 

Have more questions? Submit a request

Comments

Powered by Zendesk