Open vSwitch

Open vSwitch is an open source implementation of virtual switches. Widely used in the cloud computing industry to provide network administrators with visibility and control of traffic between and within virtual cloud hosts, Open vSwitch is designed to solve network problems with a virtualization solution that works with controller software to enable distributed virtual switching technology. This means that the switch and controller software can create clustered network configurations across multiple servers, eliminating the need for separate network configurations on each cloud host and physical host. This switch also supports VLAN trunking, visibility through NetFlow, sFlow and RSPAN, and management through the OpenFlow protocol. It also has several other features: strict flow control, which is enabled by the OpenFlow switch protocol; and remote management capabilities, which enable more control through network policies.

In terms of Flow controllers or management tools for virtual switches, OvS requires the use of third-party controllers or management tools to implement complex forwarding policies. For example, if OvS supports the OpenFlow protocol, we can use any controller that supports the OpenFlow protocol to remotely manage OvS. However, this does not mean that OvS must have a controller to work. Without an external controller attached, OvS itself can rely on MAC address learning for Layer 2 packet forwarding functions, just like Linux Bridge.

In short, Open vSwitch, the open OpenFlow switch, is capable of product-level quality, meaning it can be deployed for use in a number of production environments. It supports not only basic Layer 2 switching, but also standard management machine interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSAPN, CLI, LACP, 802.1ag), which can be well integrated with SDN systems.

List of Open vSwitch features.

• Support for enabling internal VM communications to be monitored via NetFlow, sFlow, IPFIX, SPAN, RSPAN and GRE-tunneled mirrors.
• Support for the LACP (IEEE 802.1AX-2008, multi-port binding) protocol.
• Support for the standard 802.1Q VLAN model as well as Trunk mode.
• Support for BFD and 802.1ag link status monitoring.
• Support for STP (IEEE 802.1D-1998).
• Support for fine-grained QoS.
• Support for HFSC system-level traffic control queues.
• Support traffic control policy for per-VM NIC traffic.
• Support for multi-port binding based on source MAC load balancing mode, primary and secondary mode, and L4 hash mode.
• Support for OpenFlow protocol (including many virtualization enhancements).
• Support for IPV6
• Support for multiple tunneling protocols (GRE, VXLAN, IPsec, GRE and VXLAN over IPsec)
• Support for remote configuration via C or Python interfaces.
• Support for kernel-state and user-state forwarding engine settings.
• Support for send cache engines for multi-list forwarding.
• Support for forwarding layer abstraction for easy orientation to new software or hardware platforms.

System Architecture

OvS consists of three main components.

• The main user space components are ovsdb-server and ovs-vswitchd
• The kernel space consists of the datapath kernel module
• The top Controller represents the OpenFlow controller, which is connected to OvS through the OpenFlow protocol

• ovs-vswitchd: The main module that implements kernel datapath upcall processing and ofproto table lookup, as well as the dpdk datapath handler.
• ovsdb-server: database server, using the currently accepted ovsdb protocol.
• ovs-vsctl: bridge, interface creation, deletion, setup, query, etc.
• ovs-dpctl: configuration of vswitch kernel module
• ovs-appctl: send command messages to ovs-vswithchd, check the status of different modules
• ovs-ofctl: send down flow table information. This command can configure other openflow switches (with openflow protocol)

ovsdb

ovsdb is an OvS lightweight database service that stores configuration information for the entire OvS, including interfaces, switch contents, VLANs, virtual switch creation, NIC additions, and other information and operations, all stored by ovsdb in a conf.db file (JSON format) and served via db.sock. vswitchd works based on the configuration information in the database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

ovsdb-server /etc/openvswitch/conf.db 
    -vconsole:emer -vsyslog:err 
    -vfile:info 
    --remote=punix:/var/run/openvswitch/db.sock 
    --private-key=db:Open_vSwitch,SSL,private_key 
    --certificate=db:Open_vSwitch,SSL,certificate 
    --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir 
    --log-file=/var/log/openvswitch/ovsdb-server.log 
    --pidfile=/var/run/openvswitch/ovsdb-server.pid
    --detach 
    --monitor

• /etc/openvswitch/conf.db : is the database file location, ovsdb-server needs this file to start, you can use ovsdb-tool create command to create and initialize this database file.
• --remote=punix:/var/run/openvswitch/db.sock : implements a Unix Sockets connection through which the OvS master process ovs-vswitchd or other command tool (e.g. ovsdb-client) manages ovsdb.
• /var/log/openvswitch/ovsdb-server.log : The runtime log file of ovsdb-server.

ovs-vswitchd

ovs-vswitchd is essentially a daemon, the core component of OvS. ovs-vswitchd and Datapath work together to implement Flow-based Switching for OvS.

• Can communicate with OpenFlow controllers via the OpenFlow protocol
• Communicates with the ovsdb-server database service using the ovsdb protocol
• Uses netlink and Datapath kernel modules to communicate.

ovs-vswitchd supports multiple independent Datapaths. ovs-vswitchd requires the Datapath kernel module to be loaded for proper operation. ovs-vswitchd reads the configuration information from ovsdb-server at startup and then automatically configures the Datapaths and OvS Switches’ When the configuration content in ovsdb is modified, ovs-vswitched automatically updates its configuration to keep the data synchronized. ovs-vswitchd can also obtain flow table entries from the OpenFlow controller. ovs-vswitchd can also obtain flow table entries from the OpenFlow controller.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

ovs-vswitchd unix:/var/run/openvswitch/db.sock 
    -vconsole:emer 
    -vsyslog:err 
    -vfile:info 
    --mlockall 
    --no-chdir 
    --log-file=/var/log/openvswitch/ovs-vswitchd.log 
    --pidfile=/var/run/openvswitch/ovs-vswitchd.pid 
    --detach 
    --monitor

datapatch

In the semantics of OpenFlow Switch rules, a technical term called Datapath is given to switches, or bridges. open vSwitch’s kernel module openvswitch.ko implements multiple Datapaths, each of which can have multiple Ports. Datapath listens to the NIC interface devices, matches the packets in the flow table, and returns the corresponding Actions to Datapath as a description of the data processing behavior after finding the matching flow table entry. The kernel module information of Datapath is as follows.

1
2
3
4
5
6
7
8
9

# modinfo openvswitch
filename:       /lib/modules/3.10.0-327.el7.x86_64/kernel/net/openvswitch/openvswitch.ko
license:        GPL
description:    Open vSwitch switching datapath
rhelversion:    7.2
srcversion:     F75F2B83324DCC665887FD5
depends:        libcrc32c
intree:         Y
...

More information on Datapath can be found at The Design and Implementation of Open vSwitch.

How it works

The Bridge processes data frames according to the following rules.

• A frame received on a Port does not send this frame to this Port again.
• Incoming frames learn their Source MAC address.
• If the frame is a multicast or broadcast packet (determined by the Layer 2 MAC address) it is forwarded to all ports except the receiving port, and if the upper layer protocol is interested, it is also passed to the upper layer for processing.
• If the address of the data frame cannot be found in the CAM (MAC-Port Mapping) table, it is forwarded to all ports outside the receiving port.
• If it can be found in the CAM table, it is forwarded to the corresponding Port, or not sent if both sending and receiving are the same Port.
• The bridge works in promiscuous mode, and data frames with all MAC addresses can pass through.

The user space ovs-vswitchd and the kernel module Datapath determine the forwarding of packets as follows.

• The kernel-state Datapath listens for incoming packets from the interface device.
• If Datapath does not find a matching flow table entry in the kernel-state flow table cache, it passes (upcalls) the packet to the user-state ovs-vswitchd daemon for processing.
• (Optional) User-state ovs-vswitchd has full flow table entries and communicates with the OpenFlow controller or the ovs-ofctl command-line tool via the OpenFlow protocol, mainly to receive flow table entries from the OpenFlow controller’s southbound interface. Or, depending on the flow table entry settings, ovs-vswitchd may send network packets to the OpenFlow controller for processing as Packet-In messages.
• ovs-vswitchd updates the kernel-state Flow Table upon receiving a message from the OpenFlow controller or the ovs-ofctl command-line tool. Alternatively, based on the locality principle, the user-state ovs-vswitchd injects into the Flow Table the flow table entries that were not cached by the Datapath that was just executed.
• ovs-vswitchd reinjects the packets into Datapath after matching the Flow Table entries.
• Datapath accesses the Flow Table again to get the flow table entries for matching.
• Finally, the network packet is forwarded or dropped by Datapath based on the Flow Table entry Actions.

The above, Datapath and ovs-vswitchd interworking contains two ways of handling network packets.

• Fast Path : After Datapatch is loaded into the kernel, it registers a hook function on the NIC, which is called whenever a network packet arrives at the NIC, starts unpacking the network packet layer by layer (MAC layer, IP layer, TCP layer, etc.), then matches it with a flow table entry, and if a matching flow table entry is found then the network packet is processed according to the established policy (e.g. Modify MAC, modify IP, modify TCP port, which NIC to send it from, etc.), and then send the network packet from the NIC. This process is all done in the kernel, so it is very fast and is called Fast Path.
• Slow Path: The kernel state is not allocated much memory, so the kernel state can save very few stream table entries, and often the old ones are discarded when new stream table entries arrive. If you cannot find a stream table entry in the kernel state, you need to look it up in the user state. Network packets are sent to ovs-vswitchd via netlink (a mechanism for interaction between the kernel state and the user state). ovs-vswitchd has a listening thread, and when it finds a network packet from the kernel state, it enters its own processing and reinjects the network packet into Datapath again. Obviously, processing in the user state is relatively slow, so the value is called Slow Path.

In the user state ovs-vswtichd does not need to skimp on memory, it contains all the flow table entries that may have been issued by the OpenFlow controller via the OpenFlow protocol or set by the OvS command line tool ovs-ofctl. ovs-vswtichd will match network packet information layer by layer until it finds a flow table entry for processing. If none is found, the default stream table entry is usually used, such as dropping the packet.

When a stream table entry is finally matched, it is sent down to the kernel via the netlink protocol based on the “locality principle (local data is accessed frequently over time, which is the basis of cache design)”, and when the policy is sent down to the kernel, if the kernel does not have enough memory space, it will starts eliminating some of the old policies. This ensures that the next network packet of the same type can be matched directly from the kernel, thus speeding up execution. Due to the proximate effect, the next network packet should most likely be able to match this policy. For example, if a file is transferred, a steady stream of network packets of the same type will arrive.

Command line tools

• ovs-vsctl : Used to manage the configuration information of ovs-vswitchd.
• ovs-ofctl : Used to manage the flow table information of OvS.
• ovs-pki : Used to manage the TSL communication framework between OvS and OpenFlow Controller.
• ovs-dpctl : Used to manage Datapath, such as viewing Datapath information.
• ovs-appctl : Application layer command set, e.g. simulate packets for testing OvS Switch data forwarding flow.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# 查看 OvS Log 级别
ovs-appctl vlog/list

# 设置 Log 级别
ovs-appctl vlog/set {module name}:{console, syslog, file}:{off, emer, err, warn, info, dbg}
ovs-appctl vlog/set stp:file:dbg

# 查看 OvS 版本
ovs-ofctl -V

# 查询指令历史记录
ovsdb-tool show-log [-mmm]

# 修改 OpenFlow 网络端口编号
ovs-vsctl add-port <bridge> <interface> -- set Interface <interface> ofport_request=100

# 设置 Interface 类型为 internal
ovs-vsctl set Interface <interface> type=internal

# 开启指定 Bridge 的 STP
ovs-vsctl set bridge <bridge> stp_enable=true

# 关闭指定 Bridge 的 STP
ovs-vsctl set bridge <bridge> stp_enable=false

# 查询指定 Bridge 的 STP 的配置信息
ovs-vsctl get bridge <bridge> stp_enable

# 设置指定 Bridge 的 STP Priority
ovs−vsctl set bridge <bridge> other_config:stp-priority=0x7800

# 设置指定 Bridge 的 STP Cost
ovs−vsctl set bridge <bridge> other_config:stp-path-cost=10

Installation and Deployment

OS : CentOS7

Step1. Shut down SELinux, otherwise ovsdb-server Manager will not work properly.

1
2
3
4
5

[root@localhost ~]# setenforce 0

[root@localhost ~]# cat /etc/selinux/config | grep -v ^#
SELINUX=disabled
SELINUXTYPE=targeted

Step 2. yum install

1

yum install openvswitch openvswitch-test

Step 3. Start the service

1
2
3

systemctl enable openvswitch
systemctl start openvswitch
systemctl status openvswitch

View the current version of OvS:

1
2
3

[root@localhost ~]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    ovs_version: "2.0.0"

View a list of OvS service processes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

ovsdb-server /etc/openvswitch/conf.db 
    -vconsole:emer -vsyslog:err 
    -vfile:info 
    --remote=punix:/var/run/openvswitch/db.sock 
    --private-key=db:Open_vSwitch,SSL,private_key 
    --certificate=db:Open_vSwitch,SSL,certificate 
    --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir 
    --log-file=/var/log/openvswitch/ovsdb-server.log 
    --pidfile=/var/run/openvswitch/ovsdb-server.pid
    --detach 
    --monitor

ovs-vswitchd unix:/var/run/openvswitch/db.sock 
    -vconsole:emer 
    -vsyslog:err 
    -vfile:info 
    --mlockall 
    --no-chdir 
    --log-file=/var/log/openvswitch/ovs-vswitchd.log 
    --pidfile=/var/run/openvswitch/ovs-vswitchd.pid 
    --detach 
    --monitor

To view the loaded kernel modules.

1
2
3
4
5

[root@localhost ~]# lsmod | grep openvswitch
openvswitch            70743  0
vxlan                  37584  1 openvswitch
gre                    13808  1 openvswitch
libcrc32c              12644  2 xfs,openvswitch

ovs-db

The carrier of ovs-db on the operating system is the JSON file /etc/openvswitch/conf.db, whose contents can be viewed by executing the command ovsdb-client dump, e.g.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

[root@localhost ~]# ovsdb-client dump
Bridge table
_uuid controller datapath_id datapath_type external_ids fail_mode flood_vlans flow_tables ipfix mirrors name netflow other_config ports protocols sflow status stp_enable
----- ---------- ----------- ------------- ------------ --------- ----------- ----------- ----- ------- ---- ------- ------------ ----- --------- ----- ------ ----------

Controller table
_uuid connection_mode controller_burst_limit controller_rate_limit enable_async_messages external_ids inactivity_probe is_connected local_gateway local_ip local_netmask max_backoff other_config role status target
----- --------------- ---------------------- --------------------- --------------------- ------------ ---------------- ------------ ------------- -------- ------------- ----------- ------------ ---- ------ ------

Flow_Sample_Collector_Set table
_uuid bridge external_ids id ipfix
----- ------ ------------ -- -----

Flow_Table table
_uuid flow_limit groups name overflow_policy
----- ---------- ------ ---- ---------------

IPFIX table
_uuid cache_active_timeout cache_max_flows external_ids obs_domain_id obs_point_id sampling targets
----- -------------------- --------------- ------------ ------------- ------------ -------- -------

Interface table
_uuid admin_state bfd bfd_status cfm_fault cfm_fault_status cfm_health cfm_mpid cfm_remote_mpids cfm_remote_opstate duplex external_ids ifindex ingress_policing_burst ingress_policing_rate lacp_current link_resets link_speed link_state mac mac_in_use mtu name ofport ofport_request options other_config statistics status type
----- ----------- --- ---------- --------- ---------------- ---------- -------- ---------------- ------------------ ------ ------------ ------- ---------------------- --------------------- ------------ ----------- ---------- ---------- --- ---------- --- ---- ------ -------------- ------- ------------ ---------- ------ ----

Manager table
_uuid connection_mode external_ids inactivity_probe is_connected max_backoff other_config status target
----- --------------- ------------ ---------------- ------------ ----------- ------------ ------ ------

Mirror table
_uuid external_ids name output_port output_vlan select_all select_dst_port select_src_port select_vlan statistics
----- ------------ ---- ----------- ----------- ---------- --------------- --------------- ----------- ----------

NetFlow table
_uuid active_timeout add_id_to_interface engine_id engine_type external_ids targets
----- -------------- ------------------- --------- ----------- ------------ -------

Open_vSwitch table
_uuid                                bridges cur_cfg db_version external_ids                                       manager_options next_cfg other_config ovs_version ssl statistics system_type system_version
------------------------------------ ------- ------- ---------- -------------------------------------------------- --------------- -------- ------------ ----------- --- ---------- ----------- --------------
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f []      0       "7.3.0"    {system-id="257b9b47-87e7-4404-9af8-37f98b04f2bd"} []              0        {}           "2.0.0"     []  {}         unknown     unknown       

Port table
_uuid bond_downdelay bond_fake_iface bond_mode bond_updelay external_ids fake_bridge interfaces lacp mac name other_config qos statistics status tag trunks vlan_mode
----- -------------- --------------- --------- ------------ ------------ ----------- ---------- ---- --- ---- ------------ --- ---------- ------ --- ------ ---------

QoS table
_uuid external_ids other_config queues type
----- ------------ ------------ ------ ----

Queue table
_uuid dscp external_ids other_config
----- ---- ------------ ------------

SSL table
_uuid bootstrap_ca_cert ca_cert certificate external_ids private_key
----- ----------------- ------- ----------- ------------ -----------

sFlow table
_uuid agent external_ids header polling sampling targets
----- ----- ------------ ------ ------- -------- -------

The use of graphical tools allows for a more user-friendly view of.

The relationship between the database tables is shown in the following diagram.

The Open_vSwitch table is the root of the OvS DB.

It has the following table structure, which records the configuration information of ovs-vswitchd, e.g.

• Configuration of the Bridge device
• The configuration of the OvS itself
  • other_config:stats-update-interval: the interval to write statistics to the database
  • other_config:flow-restore-wait: for hot-upgrade, if True then no packets are processed. The general procedure is: first stop ovs-vswitchd, then set this value to True, then start ovs-vswitchd, which does not process any packets, then use ovs-ofctl to restore the flow table to a correct state, and finally set this value to False to start processing packets.
  • other_config:flow-limit: Specify the number of flow entries in the Flow Table
  • other_config:n-handler-threads: the number of threads used to process the new Flow
  • other_config:n-revalidator-threads: the number of threads used to validate the Flow
  • other_config:enable-statistics: whether to count the following items
    • statistics:cpu: statistics on the number of CPUs, threads
    • statistics:load_average: system load
    • statistics:memory: total RAM, Swap
    • statistics:process_NAME: statistics memory size, cpu time, etc. (with NAME replaced by a process name)
    • statistics:file_systems: mount point, size, used
  • client request id: cur_cfg and next_cfg, when a Client finishes modifying the database, next_cfg adds 1, then wait for OvS to apply the changes, when the application is finished, cur_cfg adds 1, at this time cur_cfg equals next_cfg. This configuration is obviously there to ensure consistency for highly concurrent requests.
  • The configuration for ovsdb-server points to the Manager table, ovs-vswitchd as one of the Clients of ovsdb-server, and the Manager configures the DB Connection field.
  • Configuration of the SSL DB Connection: points to the SSL table, which mainly configures the path to the private key, certificate, etc. files needed for SSL secure communication.

Operational Objects of Open vSwitch

The database structure is a mapping of a software’s resource model design, and the following is an overview of the characteristics and roles of each resource object based on the resource model of OvS.

Manager

The Manager object is used to configure the connection of ovsdb-server so that Clients (e.g. ovs-vswitchd, ovs-vsctl, host) can perform DB operations on ovsdb-server remotely. server is the RPC interface for ovs-db to provide management. The manager_options field in the Open_vSwitch table is loaded as the listening port.

The table structure of Manager

The most important field is target, which records information about the ovsdb-server’s listening parameters.

• Active database connection methods.
  • ssl:ip[:port] : listens on the Port of the specified Remote IP, protocol is SSL
  • tcp:ip[:port] : Listens on the Port of the specified Remote IP, protocol is TCP
  • unix:FILE : Unix domain socket named FILE
• Passive database connection methods:
  • pssl:[port][:ip] : Listens on the specified Port of the local IP, protocol is SSL
  • ptcp:[port][:ip] : Listens on the specified Port of the local IP, protocol is TCP

Set by the following command.

1
2
3
4
5
6

ovs-vsctl set-manager TARGET...      # set the list of managers to TARGET...

# Active Listener
ovs-vsctl set-manager tcp:1.2.3.4:6640
# Passive Listener 
ovs-vsctl set-manager ptcp:6640

NOTE: TCP-based DB Connection that allows ovs-vsctl to control ovsdb-server even on remote machines.

Configuring Manager OvS DB Connection

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[root@localhost ~]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    ovs_version: "2.0.0"

[root@localhost ~]# ovs-vsctl set-manager ptcp:8881

[root@localhost ~]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "ptcp:8881"
    ovs_version: "2.0.0"

[root@localhost ~]# ovsdb-client dump
...
Manager table
_uuid                                connection_mode external_ids inactivity_probe is_connected max_backoff other_config status                                                               target
------------------------------------ --------------- ------------ ---------------- ------------ ----------- ------------ -------------------------------------------------------------------- -----------
84f0a33c-a798-40fc-a285-ed4d83121d3e []              {}           []               false        []          {}           {bound_port="8881", sec_since_connect="0", sec_since_disconnect="0"} "ptcp:8881"

Check that the Port is turned on properly.

1
2

[root@localhost ~]# netstat -lpntu | grep 8881
tcp        0      0 0.0.0.0:8881            0.0.0.0:*               LISTEN      3024/ovsdb-server

To perform a connection from a remote computer.

1
2
3
4

[root@localhost ~]# ovs-vsctl --db=tcp:192.168.1.109:8881 show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "ptcp:8881"
    ovs_version: "2.0.0"

NOTE : Note the firewall interference factor.

SSL

If the Open vSwitch is configured to connect to the OpenFlow controller over the network (between ovs-vswitchd and ovs-controller), then it is recommended that you use OpenSSL to build secure network communications. Bidirectional SSL ensures the integrity and and security of the OpenFlow connection to each other.

To establish an SSL connection, you first need to obtain the relevant CA certificates and record these certificates in the SSL table. When you start ovsdb-server, you can use the option -private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert =db:Open_vSwitch,SSL,ca_cert to specify that these parameters are applied.

SSL Table Structure.

SSL Table Properties :

• private_key: private key
• ca_cert: CA root certificate
• bootstrap_ca_cert (Boolean): If True, the latest CA certificate is obtained from the Controller every time ovsdb-server is started. Otherwise, the old CA certificate will continue to be used.
• certificate: the certificate issued by CA

The public key of the host (client) is included in the certificate issued by CA, and then signed by the private key of the CA center, which guarantees that the certificate is legitimate. In order to verify the CA signature, the public key of CA is also needed and put into the CA certificate pointed to by ca_cert. The CA’s own public key also needs to be guaranteed by the CA with higher signature or the CA (root CA) itself.

SSL verification process:

1. client holds CA root certificate (ca_cert) and verifies it with CA private key (CA root certificate is self-issued by CA through its own private key), if the verification is successful, it means the CA is the root CA and can guarantee itself.

2. the client holds the certificate (certificate, containing the client’s public key) issued by the CA and verifies it with the CA root certificate (the certificate issued by the CA can be unlocked with the CA’s private key), and returns the client’s public key if the verification is successful.

3. the client holds its own private key and verifies it with the client’s public key unlocked by the CA, and if the verification is successful, the client is guaranteed by the CA.

4. The client establishes an SSL connection with the server.

   

5. In short, the client needs to ensure that it has its own private key, a CA-issued certificate, and a CA root certificate in order to complete SSL authentication.

1
2
3
4
5
6

# 查询 SSL 连接
ovs-vsctl get-ssl
# 设置 SSL 证书
ovs-vsctl set-ssl sc-privkey.pem sc-cert.pem cacert.pem
# 删除 SSL 连接
ovs-vsctl del-ssl

Configuring SSL Connection

Self-issuing CA Root Certificate:

• Step 1. Generate the RSA private key for the CA root certificate

1
2
3
4

mkdir ~/OVS_CA
cd ~/OVS_CA

openssl genrsa -out caprivate.key 1024

• Step 2. generate the signing request (CSR) for the CA root certificate

1

openssl req -key caprivate.key -new -subj "/C=CN/ST=CN/L=CN/O=CN/CN=fanguiju@163.com" -out cacertificate.req

• Step 3. issue your own CA root certificate (with the CA’s public key included)

1

openssl x509 -req -in cacertificate.req -signkey caprivate.key -out cacertificate.pem

PS : Self-issued, i.e., you guarantee yourself and issue your own CSR with your own private key, so it is also called a root certificate.

Issuance of client certificate.

• Step 1. Generate the client private key

1
2
3
4

mkdir ~/ClientCerts
cd ~/ClientCerts

openssl genrsa -out cliu8private.key 1024

• Step 2. generating client-side signature requests

1

openssl req -key cliu8private.key -new -subj "/C=CN/ST=CN/L=CN/O=CN/CN=cliu8@163.com" -out cliu8certificate.req

• Step 3. CA center issues client certificate (CA center guarantees the client)

1
2
3
4

cp ~/OVS_CA/caprivate.key ~/OVS_CA/cacertificate.pem ~/ClientCerts

openssl x509 -req -in cliu8certificate.req -CA cacertificate.pem -CAkey caprivate.key -out cliu8certificate.pem -CAcreateserial
123

Configuring the SSL Connection.

1
2
3

ovs-vsctl del-manager
ovs-vsctl set-manager pssl:8881
ovs-vsctl set-ssl /root/ClientCerts/cliu8private.key /root/ClientCerts/cliu8certificate.pem /root/OVS_CA/cacertificate.pem

NOTE : set-ssl must specify the absolute path, otherwise the certificate file will not be loaded correctly.

VIEW MODIFY.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[root@localhost newcerts]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "pssl:8881"
    ovs_version: "2.0.0"

[root@localhost ~]# ovsdb-client dump
...
SSL table
_uuid                                bootstrap_ca_cert ca_cert                          certificate                              external_ids private_key
------------------------------------ ----------------- -------------------------------- ---------------------------------------- ------------ ------------------------------------
f8d8f0f5-f87b-430b-9d48-123a74a6804f false             "/root/OVS_CA/cacertificate.pem" "/root/ClientCerts/cliu8certificate.pem" {}           "/root/ClientCerts/cliu8private.key"

Authentication.

• At this point, if you do not enter the PKI configuration, you will not be able to pass SSL authentication

1
2
3
4
5
6

[root@localhost ~]# ovs-vsctl --db=ssl:192.168.1.109:8881 show
2019-02-23T06:46:26Z|00001|stream_ssl|ERR|Private key must be configured to use SSL
2019-02-23T06:46:26Z|00002|stream_ssl|ERR|Certificate must be configured to use SSL
2019-02-23T06:46:26Z|00003|stream_ssl|ERR|CA certificate must be configured to use SSL
2019-02-23T06:46:26Z|00004|reconnect|WARN|ssl:192.168.1.109:8881: connection attempt failed (Protocol not available)
ovs-vsctl: ssl:192.168.1.109:8881: database connection failed (Protocol not available)

• Copy copies of cliu8private.key, cliu8certificate.pem, and cacertificate.pem to the client

1

scp cliu8private.key cliu8certificate.pem cacertificate.pem root@192.168.1.110:~/certs

• Request an SSL connection on the client side

1
2
3
4

[root@localhost certs]#  ovs-vsctl --db=ssl:192.168.1.109:8881 --private-key=cliu8private.key --certificate=cliu8certificate.pem --ca-cert=cacertificate.pem show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "pssl:8881"
    ovs_version: "2.0.0"

NOTE :

1
2
3
4

PKI configuration (required to use SSL):
  -p, --private-key=FILE  file with private key
  -c, --certificate=FILE  file with certificate for private key
  -C, --ca-cert=FILE      file with peer CA certificate

Bridge

Bridge means bridge, but in Linux semantics Bridge has the same meaning as Ethernet switch and is the core object of Open vSwitch. The following is a list of common OvS Bridge operations.

Bridge Common Operation Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

Bridge commands:
  add-br BRIDGE               create a new bridge named BRIDGE
  add-br BRIDGE PARENT VLAN   create new fake BRIDGE in PARENT on VLAN
  del-br BRIDGE               delete BRIDGE and all of its ports
  list-br                     print the names of all the bridges
  br-exists BRIDGE            exit 2 if BRIDGE does not exist
  br-to-vlan BRIDGE           print the VLAN which BRIDGE is on
  br-to-parent BRIDGE         print the parent of BRIDGE
  br-set-external-id BRIDGE KEY VALUE  set KEY on BRIDGE to VALUE
  br-set-external-id BRIDGE KEY  unset KEY on BRIDGE
  br-get-external-id BRIDGE KEY  print value of KEY on BRIDGE
  br-get-external-id BRIDGE  list key-value pairs on BRIDGE

# Bridge 管理操作
ovs-vsctl show
ovs-vsctl add-br <bridge>
ovs-vsctl del-br <bridge>
ovs-vsctl --if-exists del-br <bridge>
ovs-vsctl add-port <bridge> <port>|<interface>
ovs-vsctl del-port <bridge> <port>|<interface>
ovs-vsctl list <bridge>|<port>|<interface>
ovs-vsctl list bridge <bridge>
# 创建 Bridge 的同时为 Bridge 添加 Port/Interface
ovs−vsctl add−br <bridge> -- add−port <bridge> <port>|<interface>

• Creating a virtual switch.

1
2
3
4
5
6
7
8
9

[root@localhost ~]# ovs-vsctl add-br ubuntu_br
[root@localhost ~]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "ptcp:8881"
    Bridge ubuntu_br
        Port ubuntu_br
            Interface ubuntu_br
                type: internal
    ovs_version: "2.0.0"

• Creating a virtual “netline” (VETH pair).

1
2
3

[root@localhost ~]# ip link add first_br type veth peer name first_if
[root@localhost ~]# ip link add second_br type veth peer name second_if
[root@localhost ~]# ip link add third_br type veth peer name third_if

• View newly created devices.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

[root@localhost ~]# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 00:0c:29:a2:2e:a4 brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether f6:66:5f:f9:ba:17 brd ff:ff:ff:ff:ff:ff
4: ubuntu_br: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1000
    link/ether 46:ca:9f:ed:0b:46 brd ff:ff:ff:ff:ff:ff
5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
    link/ether 52:54:00:69:fc:47 brd ff:ff:ff:ff:ff:ff
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:69:fc:47 brd ff:ff:ff:ff:ff:ff
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN mode DEFAULT qlen 1000
    link/ether fe:54:00:b8:9a:cf brd ff:ff:ff:ff:ff:ff
8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN mode DEFAULT qlen 1000
    link/ether fe:54:00:43:07:06 brd ff:ff:ff:ff:ff:ff
9: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN mode DEFAULT qlen 1000
    link/ether fe:54:00:ce:e6:40 brd ff:ff:ff:ff:ff:ff
10: first_if@first_br: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 46:df:83:a6:66:bc brd ff:ff:ff:ff:ff:ff
11: first_br@file_if: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 72:90:6d:0d:4b:e0 brd ff:ff:ff:ff:ff:ff
12: second_if@second_br: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 6a:68:8f:38:86:a2 brd ff:ff:ff:ff:ff:ff
13: second_br@second_if: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 86:8d:e3:32:03:b4 brd ff:ff:ff:ff:ff:ff
14: third_if@third_br: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 5e:72:4c:9b:4d:18 brd ff:ff:ff:ff:ff:ff
15: third_br@third_if: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether a2:cf:5f:72:da:23 brd ff:ff:ff:ff:ff:ff

• Connecting one end of the virtual “network cable” to the virtual switch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

[root@localhost ~]# ovs-vsctl add-port ubuntu_br first_br
[root@localhost ~]# ovs-vsctl add-port ubuntu_br second_br
[root@localhost ~]# ovs-vsctl add-port ubuntu_br third_br

[root@localhost ~]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "ptcp:8881"
    Bridge ubuntu_br
        Port ubuntu_br
            Interface ubuntu_br
                type: internal
        Port first_br
            Interface first_br
        Port second_br
            Interface second_br
        Port third_br
            Interface third_br
    ovs_version: "2.0.0"

NOTE: The eponymous Port of Bridge ubuntu_br is generally the management port of the Bridge, analogous to the management port of a physical switch, for which an IP address is generally configured.

Controller

OvS is an OpenFlow Switch implementation, which can unify the management of all distributed Bridges through OpenFlow Controlle. The so-called “management” is actually the management of the Bridge’s Flow Table. The core of SDN Controller is to control the issuance of policies and use them to make decisions about the flow of data .

Controller has two types.

• Primary Controller: Really controls the Bridge’s Flow Table. The Bridge maintains a connection to the Controller, and if the connection fails or is disconnected, it depends on the Bridge’s Fail Mode to handle it. A Bridge can connect to multiple Controllers, but the collaboration between Controllers needs to be done by the Controllers themselves.
• Service Controller: Used only for Support, occasional operation, Maintain use, Bridge’s Fail Mode will not work if Connection is disconnected from Bridge.

OvS provides the following two Fail Modes for the Bridge.

• Secure: After disconnecting, the Bridge tries to reconnect to the Controller until it succeeds, and does not maintain its own Flow Table.
• Standalone: If the Bridge fails to connect to the Controller after three attempts, it will create and maintain its own independent Flow Table.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# 查询 Bridge 的 Fail Mode
ovs-vsctl get-fail-mode ovs-br
# 设置 Bridge 的 Fail Mode
ovs-vsctl set-fail-mode ovs-br standalone
ovs-vsctl set-fail-mode ovs-br secure
# 移除 Bridge 的 Fail Mode
ovs-vsctl del-fail-mode ovs-br

# 查看 Bridge 和 Controller 的连接模式
ovs-vsctl get controller ovs-br connection-mode
# 设置 Out-of-band 连接模式
ovs-vsctl set controller ovs-br connection-mode=out-of-band
# 设置 In-band (default) 连接模式
ovs-vsctl set controller ovs-br connection-mode=in-band
# 移除 hidden flow
ovs-vsctl set bridge br0 other-config:disable-in-band=true

Controller table structure:

Controller Common Commands

1
2
3
4
5
6
7
8

# 设置 Controller
ovs-vsctl set-controller <bridge> tcp:<controller_ip>:6633
# 设置 Multi Controller
ovs-vsctl set-controller <bridge> tcp:<controller_ip1>:6633 tcp:<controller_ip2>:6633
# 获取 Bridge 的 Controller
ovs-vsctl get-controller <bridge>
# 移除 Controller
ovs-vsctl del-controller <bridge>

Install Floodlight

There are various types of OpenFlow Controllers, such as OpenDaylight, etc. A detailed list can be found at OpenFlow Controllers in GENI". This article takes Floodlight as an example to feel the controller’s flow control function for Bridge.

Official website: http://www.projectfloodlight.org/getting-started/ Official deployment manual: https://floodlight.atlassian.net/wiki/spaces/floodlightcontroller/pages/1343544/Installation+Guide

Installing Floodlight ：

1
2
3
4
5
6
7
8
9

git clone git://github.com/floodlight/floodlight.git
cd floodlight/
git submodule init
git submodule update
ant

sudo mkdir /var/lib/floodlight
sudo mkdir /var/log/floodlight
sudo chmod 777 /var/lib/floodlight

NOTE : Tag version 1.2 is used here

Running Floodlight in the Terminal：

1

nohup java -jar ~/floodlight/target/floodlight.jar > /var/log/floodlight/floodlight.log 2>&1 &

Check the Controller’s listening port.

1
2
3
4
5

[root@localhost ~]# cat /var/log/floodlight/floodlight.log | grep "Listening for switch connections"
09:28:16.102 INFO [n.f.c.i.OFSwitchManager:main] Listening for switch connections on /0.0.0.0:6653

[root@localhost ~]# netstat -lpntu | grep 6653
tcp6       0      0 :::6653                 :::*                    LISTEN      20159/java

NOTE: Floodlight will get the Bridge connection by listening to /0.0.0.0:6653 socket.

Connecting Bridge to Controller.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

[root@localhost floodlight-1.2]# ovs-vsctl set-controller ubuntu_br tcp:192.168.1.109:6653
[root@localhost floodlight-1.2]# ovs-vsctl show
2028eafc-e1db-4ea8-b0fc-30a21fdaca0f
    Manager "ptcp:8881"
    Bridge ubuntu_br
        Controller "tcp:192.168.1.109:6633"
        Port ubuntu_br
            Interface ubuntu_br
                type: internal
        Port third_br
            Interface third_br
        Port first_br
            Interface first_br
        Port second_br
            Interface second_br
    ovs_version: "2.0.0"

NOTE: Once the Bridge is connected to the Controller, the Controller can collect, send and manage information about the Bridge (e.g. Flow Table).

Access to Web GUI.

1

http://192.168.1.109:8080/ui/index.html

Connecting KVM Virtual Machines to the OvS Bridge

Plugging three KVM virtual machines into the Bridge.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[root@localhost ~]# virsh domiflist VM1
Interface  Type       Source     Model       MAC
-------------------------------------------------------
macvtap1   direct     ubuntu_br  rtl8139     52:54:00:b8:9a:cf

[root@localhost ~]# virsh domiflist VM2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
macvtap0   direct     ubuntu_br  rtl8139     52:54:00:43:07:06

[root@localhost ~]# virsh domiflist VM3
Interface  Type       Source     Model       MAC
-------------------------------------------------------
macvtap2   direct     ubuntu_br  rtl8139     52:54:00:ce:e6:40
1234567891011121314

NOTE : VM1, 2 and 3 are all connected to the same Bridge, so as long as all three have the IP address of the network segment, they can communicate.

Check the Floodlight Dashboard.

Wait for a period of time and Floodlight will collect the Bridges on the host, Hosts on the Bridge, IP/MAC addresses of the Hosts, etc.

Floodlight Controller Common Commands

The Floodlight Dashboard is basically a display only, so most of the operations still need to be done through the command line.

• Request the DPID of all Switches on this Controller

1

curl http://localhost:8080/wm/core/controller/switches/json

• View stream table entries

1

curl http://localhost:8080/wm/staticflowentrypusher/list/all/json

• View the flow table entries for the specified Switch

1

curl http://192.168.1.109:8080/wm/core/switch/00:00:46:ca:9f:ed:0b:46/flow/json

• Add static flow table entries

1

curl -d '{"switch": "00:00:00:00:00:00:00:01", "name":"flow-mod-1", "cookie":"0", "priority":"32768", "ingress-port":"1", "active":"true", "actions":"output=2"}' http://localhost:8080/wm/staticflowentrypusher/json

• Static flow tables that allow only 10.0.0.101 and 10.0.0.103 to ping each other

1
2
3
4
5

# VM1
curl -d '{"switch": "00:00:46:ca:9f:ed:0b:46", "name":"static-flow2", "cookie":"0", "priority":"32768", "src-mac":"52:54:00:b8:9a:cf","active":"true", "actions":"output=10"}'  http://localhost:8080/wm/staticflowentrypusher/json

# VM3
curl -d '{"switch": "00:00:46:ca:9f:ed:0b:46", "name":"static-flow1", "cookie":"0", "priority":"32768", "src-mac":"52:54:00:ce:e6:40","active":"true", "actions":"output=12"}'  http://localhost:8080/wm/staticflowentrypusher/json

• Delete the specified flow table

1

curl -X DELETE -d '{"name":"flow-mod-1"}' http://localhost:8080/wm/staticflowentrypusher/json

• Delete all flow tables for a given Switch

1

curl http://localhost:8080/wm/staticflowentrypusher /clear/<dpid>/json

Mirror

The function of Mirror is to configure a Bridge to send certain packets with certain conditions to the specified Mirrored Ports.

The packet conditions are the following.

• select_all
• select_dst_port
• select_src_port
• select_vlan

The designated purpose Ports are either.

• output_port (SPAN Switched Port ANalyzer)
• output_vlan (RSPAN Remote Switched Port ANalyzer)

SPAN

Source (SPAN) port - A port that is monitored with use of the SPAN feature. Destination (SPAN) port - A port that monitors source ports, usually where a network analyzer is connected.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

ovs-vsctl add-br ovs-br
ovs-vsctl add-port ovs-br eth0
ovs-vsctl add-port ovs-br eth1
# 设置 SPAN Mirrors，将 ovs-br 上 add-port {eth0,eth1} Mirror 至 tap0
ovs-vsctl add-port ovs-br tap0 \
     -- --id=@p get port tap0 \
     -- --id=@m create mirror name=m0 select-all=true output-port=@p \
     -- set bridge ovs-br mirrors=@m
# 删除 SPAN Mirrors
ovs-vsctl clear bridge ovs-br mirrors

RSPAN

• The monitored traffic is not sent to a specified port, but Flooded to a specified VLAN
• The listening port does not have to be on the local Switch, it can be on any Switch in the specified VLAN
• S1 is a source switch
• S2 and S3 are intermediate switches
• S4 and S5 are destination switches.
• learning is disabled to enable flooding

Port and Interface

A Port is a port of a Bridge, and each Port belongs to a Bridge, while an Interface is a network interface device connected to a Port and is actually responsible for receiving and sending packets. Normally, Port and Interface have a one-to-one relationship, but when Port is configured in bond mode, Port:Interface is a 1:N relationship.

Port has the following 3 types.

• Normal: You can mount an existing NIC (physical or virtual) from HostOS to the Bridge, and OvS will generate a Port with the same name on the Bridge to handle the packets to and from this NIC. Note that networks mounted to the Bridge do not support the assignment of IP addresses.

1
2
3

# 将物理网卡接口 eth1 加入 Bridge br-ext
# br-ex 会自动创建同名的 Port eth1
ovs-vsctl add-port br-ext eth1

• Internal : is a virtual NIC interface created internally by OvS. Whenever an Internal type Port is created, OvS automatically creates an internal Interface with the same name on HostOS and mounts it to the Port. Only the Internal type Interface device supports configuring IP address information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# 创建一个 Port p0
[root@ovs ~]# ovs-vsctl add-port ovs-switch p0 -- set Interface p0 ofport_request=100

# 设定 Port p0 的 Interface p0 为 Internal 类型
[root@ovs ~]# ovs-vsctl set Interface p0 type=internal

# OvS 自动创建 Interface p0 设备
[root@ovs ~]# ip a
...
5: p0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 5a:d7:49:85:d9:da brd ff:ff:ff:ff:ff:ff
    inet6 fe80::58d7:49ff:fe85:d9da/64 scope link
       valid_lft forever preferred_lft forever

• Patch: When there are multiple OvS Bridges on HostOS, you can connect two Bridges together using Patch Port (Virtual Wire Port).Patch Port always comes in pairs, connected to two bridges, and packets received from one Patch Port will be forwarded to the other Patch Port. Similar to the VETH Pair in Linux, two bridges connected with a Patch Port are no different from one.

1
2
3
4
5
6
7

ovs-vsctl add-br br0
ovs-vsctl add-br br1

# 使用 Patch Port 连接两个 Bridge
ovs-vsctl \
-- add-port br0 patch0 -- set interface patch0 type=patch options:peer=patch1 \
-- add-port br1 patch1 -- set interface patch1 type=patch options:peer=patch0

• Tunnel: Common tunneling technologies are GRE or VxLAN. tunneling is the construction of a virtual network on top of an existing physical network, with the upper layer applications related only to the virtual network, thus enabling a more flexible virtual network configuration than the physical network and enabling L2 communication across hosts. Tunneling allows Bridges on different hosts to interoperate as if they were connected to a single large Bridge.

1
2
3
4
5
6
7
8
9

# Host A，192.168.7.21
ovs-vsctl add-br br-vxlan
# Host B，192.168.7.23
ovs-vsctl add-br br-vxlan

# Host A 上添加连接到 Host B 的 Tunnel Port
ovs-vsctl add-port br-vxlan tun0 -- set Interface tun0 type=vxlan options:remote_ip=192.168.7.23
# Host B 上添加连接到 Host A 的 Tunnel Port
ovs-vsctl add-port br-vxlan tun0 -- set Interface tun0 type=vxlan options:remote_ip=192.168.7.21

• netdev: generic NIC device (e.g. eth0, veth)
  • Receive: A netdev receives a message at L2 and processes it directly through the OvS receive function, without going through the traditional kernel TCP/IP stack.
  • send: A stream in OvS specifying that it is sent from that netdev is sent through that NIC device.
• internal: Virtual NIC device
  • Receive: The OvS receive processing function is entered when a message from the system is routed to be sent through this device.
  • send: When a flow in OvS is formulated to be sent from this internal device, the message is reinjected into the kernel stack.
• gre device: tunnel device (no matter how many GRE tunnels are created in user state, there is one and only one GRE device in kernel state)
  • Receive: When the system receives a GRE message, it is passed to the L4 layer to parse the gre header and then passed to the OvS receive processing function.
  • Send: A flow in OvS is formulated to be sent from this GRE device, and the message will be sent according to the flow table rules plus the GRE header and the outer wrapper IP, and find the route.

VLAN for Port

An important feature of the Port is VLAN Configuration, which has two types.

• Trunk Port
• Access Port

Trunk Port.

• This port is not configured with Tag, configured with trunks.
• If trunks is empty, then all VLANs are trunked and the default VLAN ID is 0, all are allowed to pass.
• If trunks is not empty, only VLANs that match the condition (ID) can pass.

In short, it is the Trunk port of the physical switch that matches the VLAN ID of the Trunks list to pass through.

Access Port.

• This Port is configured with a Tag, and packets coming in from this Port are tagged with a Tag (ID).
• If a packet coming in from another Trunk Port has a VLAN ID of its own, it will be sent out from this Port if the VLAN ID is equal to the Tag of this Port.
• Packets coming in from other Access Ports with the same Tag will also be Forwarded to this Port.
• Packets out of an Access Port are stripped of their Tag and do not carry a VLAN ID.
• If a packet arrives at the Access Port with a VLAN ID of its own, even if the VLAN ID is equal to the Tag, it is discarded and the Access Port only accepts Untag packets.

In short, packets received by the Access Port are tagged and the Access Port only accepts Untag packets, otherwise they are discarded. Packets with the same VLAN ID are forwarded to the corresponding Access Port and then untagged for retransmission.

VLAN common operation commands for Port

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# 添加 Port 并设置 VLAN tag，ID 为 3
ovs-vsctl add-port <bridge> <vlan_access_interface> tag=3 -- set interface <vlan_access_interface> type=internal
# 为已存在的 Port 设置 VLAN tag，ID 为 9
ovs-vsctl set port <port> tag=9
# 移除 VLAN tag
ovs-vsctl del-port <bridge> <vlan_access_interface>
# 查询 VLAN
ifconfig <vlan_access_interface>
# 设置 Vlan Trunk
ovs-vsctl add-port <bridge> <vlan_trunk_interface> trunk=3,4,5,6
# 添加设置 VLAN 的 Flow，ID 为 100
ovs-ofctl add-flow <bridge> in_port=1,dl_vlan=0xffff,actions=mod_vlan_vid:100,output:3
ovs-ofctl add-flow <bridge> in_port=1,dl_vlan=0xffff,actions=push_vlan:0x8100,set_field:100-\>vlan_vid,output:3
# 添加摘除 VLAN tag 的 Flow
ovs-ofctl add-flow <bridge> in_port=3,dl_vlan=100,actions=strip_vlan,output:1
# ovs-ofctl add-flow pop-vlan
ovs-ofctl add-flow ovs-br in_port=3,dl_vlan=0xffff,actions=pop_vlan,output:1

Bonding of Ports

The relationship between Port and Interface is a one-to-many relationship because OvS supports the Bond feature. Bonding is the process of “bundling” multiple Interfaces together to form a virtual connection, thus achieving high availability and high throughput.

Common bond_modes are as follows.

• active-backup: Failover, where one connection is active and the others are backup.
• balance-slb: Load balancing, load balancing based on source MAC and output VLAN.
• balance-tcp: Load balancing, must support LACP protocol to do so, can load balance based on L2, L3, L4.
• stable (LACP): Attempts to always assign a given flow to the same slave consistently.

PS: LACP (Link Aggregation Control Protocol).

Bond model for OvS:

QoS for Ports

Qos in OvS is often used in conjunction with Flow Policy. It is well known that QoS has two directions, one for Ingress and one for Egress.

Network Process QoS Implementation Principles and Methods

The most commonly used network QoS on Linux is the TC tool, which is mainly implemented by means of Queuing.

1. Classless Queuing Disciplines: default is pfifo_fast, a technique that does not classify network packets. pfifo_fast checks the corresponding Band in the priomap of TOS according to the number corresponding to the TOS in the network packet, and different Bands correspond to different queues.

   

   

2. SFQ, Stochastic Fair Queuing: There are many queues of FIFOs, and TCP sessions or UDP streams are assigned to a queue. Packets are RoundRobin’d from each queue and sent. This way one Session does not take up all the traffic. But instead of having a queue for each Session, a large number of Sessions are assigned to a limited number of queues through a Hash algorithm. This way two or more Sessions will share a queue and may affect each other. Therefore, the Hash function changes frequently so that Sessions do not always affect each other.

   

3. TBF, Token Bucket Filter: All network packets are queued for sending, but not at the head of the queue, but only when the packet has a Token. tokens are generated according to a set rate (Rate), so even if the queue is long, they are sent according to the Rate. When there are no packets in the queue, Tokens are still generated at a set rate, but they do not accumulate indefinitely until the Buckets are full, and the size of the Buckets is often set by the burst/buffer/maxburst parameter. Token, then suddenly a large number of packets come in and each one gets a Token, resulting in an instant traffic spike.

4. Classful Queuing Disciplines: typical of these is HTB, Hierarchical Token Bucket.

• Shaping: Occurs only at leaf nodes and depends on other Queues.

• Borrowing: When network resources are free, borrow some for my use.

• Rate: The set rate.

• Ceil: The maximum rate. The difference between Ceil and Rate indicates the maximum amount that can be borrowed from others.

  

Create an HTB (Hierarchical Token Bucket) tree using TC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# 创建一个 HTB 的 qdisc 在 eth0 上，句柄为 1:，default 12 表示默认发送给 1:12。
tc qdisc add dev eth0 root handle 1: htb default 12

# 创建一个 root class，然后再创建几个 sub class。
# 同一个 root class 下的 sub class 可以相互借流量，如果直接不在 qdisc下面创建一个 root class，而是直接创建三个 class，他们之间是不能相互借流量的。
tc class add dev eth0 parent 1: classid 1:1 htb rate 100kbps ceil 100kbps
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 30kbps ceil 100kbps
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 10kbps ceil 100kbps
tc class add dev eth0 parent 1:1 classid 1:12 htb rate 60kbps ceil 100kbps

# 创建叶子 qdisc，分别为 fifo 和 sfq。
tc qdisc add dev eth0 parent 1:10 handle 20: pfifo limit 5
tc qdisc add dev eth0 parent 1:11 handle 30: pfifo limit 5
tc qdisc add dev eth0 parent 1:12 handle 40: sfq perturb 10

# 设定规则：从 IP 1.2.3.4 来的并且发送给 port 80 的包，从 1:10 走；其他从 1.2.3.4 发送来的包从 1:11 走；其他的走默认。
# 实现了限速与分流。
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip src 1.2.3.4 match ip dport 80 0xffff flowid 1:10
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip src 1.2.3.4 flowid 1:11

In fact, the only thing OvS can control is the Egress QoS, which is implemented through Shaping. The Ingress QoS cannot be controlled, but only the specified packets can be dropped by Policy.

Ingress policy.

1
2

ovs-vsctl set Interface tap0 ingress_policing_rate=100000
ovs-vsctl set Interface tap0 ingress_policing_burst=10000

Egress shaping : Port QoS policy is only supported for HTB.

• QoS can be created on the Port
• A QoS can have multiple Queues
• Rules are set via Flow

Tunnel

OvS supports three types of tunnels, all of which are described in the Networking Basics Terminology/Concepts, and will not be repeated.

• GRE
• VxLAN
• IPSec_gre

Example of OvS Tunnel operation command

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# Instance1
ovs-vsctl add-br testbr
ifconfig testbr 10.0.0.1/24
ovs-vsctl add-port testbr gre0 -- set Interface gre0 type=gre options:local_ip=192.168.100.100 options:remote_ip=192.168.100.101
ovs-vsctl add-port testbr vxlan0 -- set Interface vxlan0 type=vxlan options:local_ip=192.168.100.100 options:remote_ip=192.168.100.102

# Instance2
ovs-vsctl add-br testbr
ifconfig testbr 10.0.0.2/24
ovs-vsctl add-port testbr gre0 -- set Interface gre0 type=gre options:local_ip=192.168.100.101 options:remote_ip=192.168.100.100
ovs-vsctl add-port testbr ipsec0 -- set Interface ipsec0 type=ipsec_gre options:local_ip=192.168.100.101 options:remote_ip=192.168.100.102 options:psk=password

# Instance3
ovs-vsctl add-br testbr
ifconfig testbr 10.0.0.3/24
ovs-vsctl add-port testbr vxlan0 -- set Interface vxlan0 type=vxlan options:local_ip=192.168.100.102 options:remote_ip=192.168.100.100
ovs-vsctl add-port testbr ipsec0 -- set Interface ipsec0 type=ipsec_gre options:local_ip=192.168.100.102 options:remote_ip=192.168.100.101 options:psk=password

# enable STP，避免环导致的洪泛（Flood）
# Spanning Tree Protocol，即通过协议，将一个有环的二层网络变成一颗树。
ovs-vsctl set Bridge testbr stp_enable=true

Flow Tables

Open vSwitch defines a series of Flow Tables that control the flow and structure of network packets.

According to the OpenFlow protocol, a Flow Entry should consist of two parts.

• Match Field
• Action

If the packet matches the Match then the Action is executed.

Match Field parses network packets covering all layers of the TCP/IP protocol family, with the following fields to see if they can match a value.

• Layer 1 - Tunnel ID, In Port, QoS priority, skb mark
• Layer 2 - MAC address, VLAN ID, Ethernet type
• Layer 3 - IPv4/IPv6 fields, ARP
• Layer 4 - TCP/UDP, ICMP, ND

Action mainly contains the following actions.

• Output to port (port range, flood, mirror)
• Discard, Resubmit to table x
• Packet Mangling (Push/Pop VLAN header, TOS, …)
• Send to controller, Learn

Common Flow Table Operations Commands

The management of Flow Table by OvS is mainly done through the ovs-ofctl utility.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# 查看 Bridge 的流表
ovs-ofctl show <bridge>
# 查询指定 Bridge 的流表
ovs-ofctl dump-flows <bridge>
# 添加 Bridge 的流表
ovs-ofctl add−flow <bridge> <flow>
# e.g.
ovs-ofctl add-flow ovs_br dl_src=02:a2:a2:a2:a2:a2,dl_dst=02:b2:b2:b2:b2:b2,in_port=2,dl_type=0x0800,nw_src=10.0.0.1,nw_dst=10.0.0.2,actions=output:6
# 修改 Bridge 的流表
ovs-ofctl mod−flows <bridge> <flow>
# 删除 Bridge 的指定流表
ovs-ofctl del−flows <bridge> <flow>
# e.g.
ovs-ofctl del-flows <bridge> dl_src=02:a2:a2:a2:a2:a2,dl_dst=02:b2:b2:b2:b2:b2,in_port=2,dl_type=0x0800,nw_src=10.0.0.1,nw_dst=10.0.0.2
# 删除 Bridge 的所有流表
ovs-ofctl del-flows <bridge>   # This will delete all the flow entries in the flow table