Open Ethernet Networking (OpEN) API Guide and Reference Manual  3.6.0.3
Useful OpenSource tools

Overview

In this section, several tools which are useful for querying the state of OpenStack VTEP are presented.

Tool Basics

OpenStack VTEP is based largely on the Open vSwitch open source project. Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. You can learn more at http://www.openvswitch.org

Open vSwitch provides a couple of tools which can be used to inspect the state of OpenStack VTEP, and the virtual switches that it manipulates. These tools are useful for collecting data when reporting bugs to Broadcom, or for diagnosing the operation and configuration of OpenStack VTEP.

Both of these tools are Linux applications which may be run natively on any compute or controller node in the cluster. In some cases, the tool must be run on a specific host, and in other cases, the tool accepts an argument that specifies the IP address of a target host, allowing you to gather data from all hosts in the cluster from a single location.

The tools discussed here are:

You can view the man pages of these tools on any host in the cluster by using the man command. For example:

$ man ovs-vsctl

Inspecting virtual switches with ovs-vsctl

The ovs-vsctl command can be used to inspect the configuration of the Open vSwitch bridges and tunnels that are located on compute nodes in the cluster.

Use of tunnels in OpenStack

OpenStack, in configurations which support OpenStack VTEP, uses Open vSwitch bridges to connect tenant virtual machines to each other. There are two cases to consider.

First, source and destination VMs that are located on the same host will be attached to ports on an Open vSwitch bridge named br-int. Traffic flowing from one of these VMs to the other will flow through br-int, without leaving the host.

Packets destined to a tenant VM that is executing on a different host than the source will leave the host and be forwarded out a tunnel which originates on the source host, and terminates on the host where the destination VM is executing. OpenStack creates an overlay network that connects each node in the cluster to the remaining nodes in the cluster. A tunnel is a path from one node to another that originates at some node x and terminates at some node y. For each path (x, y) there is a path (y, x) in the opposite direction. Packets flowing from some node x to some node y are encapsulated using GRE or VXLAN. Open vSwitch 1.10.2 and later support both GRE and VXLAN.

At the source and destination nodes, the tunnels are patched into a second Open vSwitch bridge named br-tun. br-tun and br-int are connected to each other with virtual patch cables.

Packets that flow from a VM destined to another host will arrive at br-int. br-int, as the result of mac learning, will forward the packet to br-tun. br-tun will encapsulate the packet using GRE or VXLAN, and then send the packet to the hardware interface will transmit it over the cluster local network. On the destination host, the packet will arrive at br-tun, the GRE or VXLAN encapsulation will be removed, and the packet will be forwarded to br-int which will then handle forwarding the packet to the virtual port where the destination VM is attached.

Use of tunnels in OpenStack

VXLAN tunnels are created by vtepd on compute nodes and patched into the same br-tun virtual switch used by OpenStack. These tunnels connect compute nodes to OpenStack VTEP gateways. Like the OpenStack tunnels described above, the connectivity is bi-directional - for each compute node x and switch a, if a tunnel exists in the direction of x to a, then a corresponding tunnel will exist in the direction of a to x. It is in this way that tenant networks are extended to reach physical devices plugged into the OpenStack VTEP gateways. Packets destined for the a VTEP gateway device from a tenant VM will arrive at br-int, be forwarded to br-tun, and then sent out as VXLAN encapsulated packets to the OpenStack VTEP gateway.

Using ovs-vsctl

With the above background in mind, let's take a look at the output that is generated by ovs-vsctl during the operation of the cluster. In this example, the cluster consists of a controller (which is also configured to run compute), and a compute node. An OpenStack VTEP switch is also connected to the cluster.

The compute node IP address is 192.168.3.2, the controller node is at 192.168.3.3, and the OpenStack VTEP gateway is at 192.168.3.100.

To see the configuration of the virtual switches and the ports, use the "show" command:

solmaz:~/devstack$ sudo ovs-vsctl show
5ee4d85c-f0c9-4ccc-be1a-a4ea685c1c8e
    ovs_version: "1.10.2"

The above shows the initial state before OpenStack is run - no virtual switches. It also demonstrates how ovs-vsctl can be used to display the version of Open vSwitch. Here, it reports 1.10.2, which is the minimum version supported by OpenStack VTEP.

If we start the controller (and compute) on a node (as is done by the configuration documented by this release), we see the following:

solmaz:~/devstack$ ./stack.sh
...
solmaz:~/devstack$ sudo ovs-vsctl show
5ee4d85c-f0c9-4ccc-be1a-a4ea685c1c8e
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    ovs_version: "1.10.2"

Here, we see two virtual bridges: br-tun and br-int. br-int is the integration bridge, into which VM machines are tapped. br-tun is the tunnel bridge, and it is here tunnels to other nodes in the cluster, and to OpenStack VTEP gateways, are created and attached. Notice how both bridges are wired together. Each switch has a port, the name of which is the same name as the switch (e.g., Port br-tun). Each also has a patch port (e.g., Port patch-int), which has an associated peer (options: {peer-patch-tun} in the case of patch-int. This is the way the bridges are connected to each other, and represents the path via which packets flow, as described earlier.

Now let's start the compute node in our two-node cluster, and look at the output of ovs-vsctl on the compute node:

solmaz:~/devstack$ ./stack.sh
...
solmaz:~/devstack$ sudo ovs-vsctl show
a62c923c-b8ae-4654-bf1b-399bf61bf58f
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port "vxlan-c0a80303"
            Interface "vxlan-c0a80303"
                type: vxlan
                options: {in_key=flow, local_ip="192.168.3.2", out_key=flow, remote_ip="192.168.3.3"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "1.10.2"

The compute node is the same as we saw with the controller except for one additional port that has been attached to the br-tun bridge which establishes a VXLAN tunnel in the direction of the controller node. The port which represents the tunnel is named vxlan-c0a80303, and it has the type vxlan. local_ip defines the IP address of the head of the tunnel, and remote_ip defines the IP address of the tunnel end.

If we run ovs-vsctl on the controller, we will see there is now a tunnel back to the compute node:

solmaz:~/devstack$ sudo ovs-vsctl show
5ee4d85c-f0c9-4ccc-be1a-a4ea685c1c8e
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-c0a80302"
            Interface "vxlan-c0a80302"
                type: vxlan
                options: {in_key=flow, local_ip="192.168.3.3", out_key=flow, remote_ip="192.168.3.2"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    ovs_version: "1.10.2"

A minimal amount of python knowledge is required to configure vtepc. Basically the script you write will declare three lists which describe gateways (OpenStack VTEP switches), tenants, and devices. A fourth list maps tenants to devices. vtepc will use this information to configure vtepd instances that are running in the cluster. Entries in these lists are python dictionaries which define various attributes of the respective objects being configured.

VTEP Tunnels

In our implementation, devstack launches vtepd on all compute nodes. If we manually launch vtepd on the VTEP gateway at 192.168.3.100, launch enough VMs so that both nodes have tenant VMs running, and we start vtepc, we see tunnels created on each node of the cluster in the direction of the VTEP gateway switch. The following show the controller state, where the tunnel to the switch is named vx1.

solmaz:~/devstack$ sudo ovs-vsctl show
5ee4d85c-f0c9-4ccc-be1a-a4ea685c1c8e
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-c0a80302"
            Interface "vxlan-c0a80302"
                type: vxlan
                options: {in_key=flow, local_ip="192.168.3.3", out_key=flow, remote_ip="192.168.3.2"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vx1"
            Interface "vx1"
                type: vxlan
                options: {in_key=flow, local_ip="192.168.3.3", out_key=flow, remote_ip="192.168.3.100"}
    Bridge br-int
        Port "tap099aa406-bc"
            tag: 1
            Interface "tap099aa406-bc"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap457d0cd1-e1"
            tag: 1
            Interface "tap457d0cd1-e1"
    ovs_version: "1.10.2"

On the compute node, we also have a tunnel named vx1 which terminates on the switch:

solmaz:~/devstack$ sudo ovs-vsctl show
a62c923c-b8ae-4654-bf1b-399bf61bf58f
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port "tapf94338b3-d0"
            tag: 1
            Interface "tapf94338b3-d0"
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Port "vx1"
            Interface "vx1"
                type: vxlan
                options: {in_key=flow, local_ip="192.168.3.2", out_key=flow, remote_ip="192.168.3.100"}
        Port "vxlan-c0a80303"
            Interface "vxlan-c0a80303"
                type: vxlan
                options: {in_key=flow, local_ip="192.168.3.2", out_key=flow, remote_ip="192.168.3.3"}
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
    ovs_version: "1.10.2"

Inspecting virtual switches with ovsdb-client

The ovsdb-client can be used to query the database associated with any node in the cluster, as well as the database located on the OpenStack VTEP gateway device. The main reason for dumping the database would be to supply it along with a bug report. In this section, a quick walkthrough of the database to illustrate some tables that are used when diagnosing problems will be provided.

Running ovsdb-client

ovsdb-client can be run from any host in the cluster (other than the VTEP gateway switches). In fact, it can be run anywhere in your intranet that is able to reach the cluster via its management interface (eth0 in our sample setup).

OVSDB is hosting on each node in the cluster two databases that can be useful in diagnosing problems. The most important of these is hardware_vtep. To obtain a dump of hardware_vtep from a cluster database (compute node or VTEP gateway), determine the IP address of the management port, and issue the following command (here, we are pointing at a controller located at 10.14.240.236):

$ ovsdb-client dump tcp:10.14.240.236:6632 hardware_vtep > controller_hwvtep.txt

The text file controller_hwvtep.txt can then be sent along to Broadcom when reporting a bug. It also is good to gather the data from the other nodes. From the compute node at 10.14.240.194:

$ ovsdb-client dump tcp:10.14.240.194:6632 hardware_vtep > compute_hwvtep.txt

And from the VTEP gateway switch at 10.14.240.118:

$ ovsdb-client dump tcp:10.14.240.118:6632 hardware_vtep > gateway_hwvtep.txt

Collecting hardware_vtep from all nodes will provide a complete picture of the entire distributed database of the OpenStack VTEP side cluster.

The following sections highlight a couple of the tables in the collected database than can be useful for diagnosing problems. It is out of the scope of this document to give a detailed picture of the hardware_vtep table as it is used by OpenStack VTEP. However, the tables illustrated here were obtained at the same time the ovs-vsctl command earlier in this document was used to read state of virtual switches after vtepc and VMs were launched, so this example shows how entries in the database relate to the tunnels that were created in the OpenStack VTEP cluster.

Dumping the compute node database

The following illustrates the output seen by running the command against the controller node in our cluster. The controller in our cluster is configured to run VMs, so it acts as a compute node as well. The same basic output is to be expected on all compute nodes in the cluster.

solmaz:~$ ovsdb-client dump tcp:10.14.240.236:6632 hardware_vtep
Global table
_uuid                                managers switches
------------------------------------ -------- --------------------------------------
18e94032-854a-49dc-ad71-a2d0f49348d8 []       [7d885046-d6db-4865-b645-5dfb7a42e403]
Logical_Binding_Stats table
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local
----- ---------------- -------------- ------------------ ----------------
Logical_Router table
_uuid description name static_routes switch_binding
----- ----------- ---- ------------- --------------
Logical_Switch table
_uuid                                description name                            tunnel_key
------------------------------------ ----------- ------------------------------- ----------
777f53fe-9e0f-4c7d-ac4e-4b5ea2eeabca ""          broadcom_openstack_demo_private 1001
Manager table
_uuid inactivity_probe is_connected max_backoff other_config status target
----- ---------------- ------------ ----------- ------------ ------ ------
Mcast_Macs_Local table
MAC         _uuid                                ipaddr locator_set                          logical_switch
----------- ------------------------------------ ------ ------------------------------------ ------------------------------------
unknown-dst 5694974a-1116-4177-858a-e2aeabef5fa1 ""     e43b904b-4734-49b2-b018-8c1fd135b142 777f53fe-9e0f-4c7d-ac4e-4b5ea2eeabca
Mcast_Macs_Remote table
MAC         _uuid                                ipaddr locator_set                          logical_switch
----------- ------------------------------------ ------ ------------------------------------ ------------------------------------
unknown-dst 7fc768db-579f-43c5-a73c-7b5acd586644 ""     a01c8d47-8391-4c8a-9542-806eba86e99a 777f53fe-9e0f-4c7d-ac4e-4b5ea2eeabca
Physical_Locator table
_uuid                                bfd bfd_status dst_ip          encapsulation_type
------------------------------------ --- ---------- --------------- ------------------
39bcd31d-27a2-4f6d-983a-e12e9c8e8e57 {}  {}         "192.168.3.100" "vxlan_over_ipv4"
b8de8e29-e554-4d12-9eeb-80aef4584bb4 {}  {}         "192.168.3.3"   "vxlan_over_ipv4"
Physical_Locator_Set table
_uuid                                locators
------------------------------------ --------------------------------------
a01c8d47-8391-4c8a-9542-806eba86e99a [39bcd31d-27a2-4f6d-983a-e12e9c8e8e57]
e43b904b-4734-49b2-b018-8c1fd135b142 [b8de8e29-e554-4d12-9eeb-80aef4584bb4]
Physical_Port table
_uuid description name port_fault_status vlan_bindings vlan_stats
----- ----------- ---- ----------------- ------------- ----------
Physical_Switch table
_uuid                                description        management_ips name   ports switch_fault_status tunnel_ips
------------------------------------ ------------------ -------------- ------ ----- ------------------- ---------------
7d885046-d6db-4865-b645-5dfb7a42e403 "Broadcom Gateway" []             br-int []    []                  ["192.168.3.3"]
Ucast_Macs_Local table
MAC _uuid ipaddr locator logical_switch
--- ----- ------ ------- --------------
Ucast_Macs_Remote table
MAC                 _uuid                                ipaddr locator                              logical_switch
------------------- ------------------------------------ ------ ------------------------------------ ------------------------------------
"90:e2:ba:19:b1:ed" 7fbe5283-77a9-465a-a675-39f47a965b09 ""     39bcd31d-27a2-4f6d-983a-e12e9c8e8e57 777f53fe-9e0f-4c7d-ac4e-4b5ea2eeabca

Key table entries include:

The Physical_Switch table describes the virtual switch. The name of the switch should be br-int, and the tunnel_ips field should contain of an array with a single entry, which contains the IP address on the cluster network where tunnels in the cluster should terminate in order to reach this switch.

The Physical_Locator table should have two entries in our example. The first entry (b8de8e29-e554-4d12-9eeb-80aef4584bb4) further defines the parameters of tunnels that terminate on this switch. The dest_ip for this entry will match the entry in the tunnel_ips array in the Physical_Switch table. The other entries in the Physical_Locator table represent tunnels that originate at the switch and terminate on VTEP gateways.

For each tenant configured by vtepc to access devices on this gateway, there is an entry in the Logical_Switch table. The name of the logical switch is of the form "broadcom_openstack_<tenant name>_<network> where:

tenant_name: the symbolic name for the tenant used by OpenStack and configured by vtepc; network: the symbolic name assigned by OpenStack to the network assigned to the tenant.

Tunnel_key field (in this example, 1001) is the VNID that will be used to identify tunnel traffic as belonging to the specific tenant. This id is assigned to the tenant by OpenStack.

The final table covered here, Ucast_Macs_Remote, is written to the database by vtepc to map mac addresses to entries in the Physical_Locator table. In simple terms, it defines which tunnel can be used to reach a device on the specified OpenStack VTEP gateway with the given mac address.

Dumping the VTEP gateway database

The following notable differences exist in the hardware_vtep database for the OpenStack VTEP gateway:

Useful ICOS CLI Commands

There are a couple of commands that are useful in diagnosing the status of the VXLAN tunnels on the switch.

Checking the status of VXLAN can be done with the show vxlan command. The following show that VXLAN is enabled on the switch:

(Routing) >enable
(Routing) #show vxlan
VXLAN Admin Mode............................... Enable
Destination UDP Port........................... 4789
Maximum Allowed Limits or Table Sizes
------------------------------------------------------
Tenant Table Size.............................. 1024
Access Ports Table Size........................ 2048
Tunnel/Network Reference Ports Table Size...... 8192
Current Entries Count or Table Usage
------------------------------------------------------
Tenant Table Entries........................... 0
Access Port Entries............................ 0
Tunnel/Network Reference Port entries.......... 0
VXLAN ID  Source Address   VLAN  Access Port(s)        Remote TEP(s)
--------- ---------------- ----- --------------------- -----------------

You can also use the show vxlan command to view the status of individual vteps. Assume that a VTEP for tenant 1002 exists and terminates at the IP address 192.168.3.4, Then:

show vxlan 1002 vtep 192.168.3.4

will display information about the vtep (packet transmissions, whether or not the VTEP is reachable, etc.).