QEMU/KVM Virtualization

QEMU/KVM is currently the most popular virtualization technology. It is based on the kvm module provided by the Linux kernel, with a streamlined structure, low performance loss, and is open source and free (compared to the paid vmware), so it has become the preferred virtualization solution for most enterprises.

The current virtualization solutions of major cloud vendors are basically using KVM technology for new server instances. Even AWS, which started first and has been using Xen heavily, has switched to Nitro virtualization technology based on KVM customization since EC2 C5.

However, KVM, as an enterprise-class underlying virtualization technology, is not deeply optimized for desktop use, so if you want to use it as desktop virtualization software and replace VirtualBox/VMware, it is somewhat difficult.

This article is a summary document of my personal learning about KVM, with the goal of using KVM as desktop virtualization software.

I. Installing QUEU/KVM

The QEMU/KVM environment requires the installation of a number of components, each of which has its own role to play.

  • qemu: emulates various types of input and output devices (network cards, disks, USB ports, etc.)
    • qemu uses kvm at the bottom to emulate CPU and RAM, which is much faster than software emulation.
  • libvirt: provides simple and unified tools and APIs for managing virtual machines, shielding the underlying complex structure. (supports qemu-kvm/virtualbox/vmware)
  • ovmf: Enables UEFI support for virtual machines
  • virt-manager: GUI interface for managing virtual machines (can manage remote kvm hosts).
  • virt-viewer: Interact directly with the virtual machine through the GUI interface (can manage remote kvm hosts).
  • dnsmasq vde2 bridge-utils openbsd-netcat: Network-related component that provides virtual networking features such as Ethernet virtualization, network bridging, and NAT networking.
    • dnsmasq provides DHCP and DNS resolution for NAT virtual networks.
    • vde2: Ethernet virtualization
    • bridge-utils: As the name implies, provides network bridging related tools.
    • openbsd-netcat: The Swiss Army knife of TCP/IP.

Installation commands.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# archlinux/manjaro
sudo pacman -S qemu virt-manager virt-viewer dnsmasq vde2 bridge-utils openbsd-netcat

# ubuntu,参考了官方文档，但未测试
sudo apt install qemu-kvm libvirt-daemon-system virt-manager virt-viewer virtinst bridge-utils

# centos,参考了官方文档，但未测试
sudo yum groupinstall "Virtualization Host"
sudo yum install virt-manager virt-viewer virt-install

# opensuse
# see: https://doc.opensuse.org/documentation/leap/virtualization/html/book-virt/cha-vt-installation.html
sudo yast2 virtualization
# enter to terminal ui, select kvm + kvm tools, and then install it.

Once the installation is complete, you can’t use it directly yet, you need to do some extra work. Please continue to the next step.

1. libguestfs - Virtual Machine Disk Image Processing Tool

libguestfs is a virtual machine disk image processing tool that can be used to directly modify/view/virtual machine images, convert image formats, etc.

It provides the following list of commands.

  • virt-df centos.img : View hard disk usage
  • virt-ls centos.img / : List directory files
  • virt-copy-out -d domain /etc/passwd /tmp : Perform file copy in virtual image
  • virt-list-filesystems /file/xx.img : View file system information
  • virt-list-partitions /file/xx.img : View partition information
  • guestmount -a /file/xx.qcow2(raw/qcow2 are supported) -m /dev/VolGroup/lv_root --rw /mnt : Mount the partition directly to the host
  • guestfish : Interactive shell that runs all the above commands.
  • virt-v2v : Convert virtual machines of other formats (e.g. ova) into kvm virtual machines.
  • virt-p2v : Converts a physical machine into a virtual machine.

The above commands may be used during the learning process, it never hurts to install them in advance. The installation commands are as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# opensuse
sudo zypper install libguestfs

# archlinux/manjaro，目前缺少 virt-v2v/virt-p2v 组件
sudo pacman -S libguestfs

# ubuntu
sudo apt install libguestfs-tools

# centos
sudo yum install libguestfs-tools

2. Start QEMU/KVM

Start the libvirtd backend service via systemd.

1
2

sudo systemctl enable libvirtd.service
sudo systemctl start libvirtd.service

3. Enable non-root users to use kvm properly

Once qumu/kvm is installed, by default you need root privileges to use it properly. To make it easier to use, first edit the file /etc/libvirt/libvirtd.conf :

  • unix_sock_group = "libvirt" to uncomment this line so that the libvirt user group can use unix sockets.
  • unix_sock_rw_perms = "0770" , uncomment this line so that users can read and write unix sockets.

Then create a new libvirt user group and add the current user to it:

1
2

newgrp libvirt
sudo usermod -aG libvirt $USER

Finally, restart the libvirtd service and it should work fine:

1

sudo systemctl restart libvirtd.service

3. Enabling nested virtualization

If you need to run a virtual machine within a virtual machine (e.g. to test secure container technologies such as katacontainers within a virtual machine), then you need to enable the kernel module kvm_intel to implement nested virtualization.

1
2
3
4
5

# 临时启用 kvm_intel 嵌套虚拟化
sudo modprobe -r kvm_intel
sudo modprobe kvm_intel nested=1
# 修改配置，永久启用嵌套虚拟化
echo "options kvm-intel nested=1" | sudo tee /etc/modprobe.d/kvm-intel.conf

Verify that nested virtualization is enabled.

1
2

$ cat /sys/module/kvm_intel/parameters/nested 
Y

Now you should be able to find the virt-manager icon on your system and go inside to use it. The usage of virt-manager is similar to that of virtualbox/vmware workstation, so I won’t go into details here, you should be able to figure it out yourself.

The following is an advanced section that introduces how to manage virtual machine disks and KVM through the command line. If you are still new to kvm, we recommend that you first familiarize yourself with virt-manager through the graphical interface before reading on.

II. Virtual machine disk image management

This requires the use of two tools.

  • libguestfs: virtual machine disk image management tool, which has been introduced earlier
  • qemu-img: qemu’s disk image management tool, used to create disks, expand and shrink disks, generate disk snapshots, view disk information, convert disk formats, etc.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# 创建磁盘
qemu-img create -f qcow2 -o cluster_size=128K virt_disk.qcow2 20G

# 扩容磁盘
qemu-img resize ubuntu-server-cloudimg-amd64.img 30G

# 查看磁盘信息
qemu-img info ubuntu-server-cloudimg-amd64.img

# 转换磁盘格式
qemu-img convert -f raw -O qcow2 vm01.img vm01.qcow2  # raw => qcow2
qemu-img convert -f qcow2 -O raw vm01.qcow2 vm01.img  # qcow2 => raw

1. Importing vmware images

Import the kvm directly from the vmware ova file, this way the image should work directly (the NIC needs to be reconfigured).

1

virt-v2v -i ova centos7-test01.ova -o local -os /vmhost/centos7-01  -of qcow2

You can also extract the vmdk disk image from ova, convert the vmware vmdk file to qcow2 format, and then import it into kvm (the NIC needs to be reconfigured):

1
2
3
4

# 转换映像格式
qemu-img convert -p -f vmdk -O qcow2 centos7-test01-disk1.vmdk centos7-test01.qcow2
# 查看转换后的映像信息
qemu-img info centos7-test01.qcow2

The qcow2 image obtained by directly converting the vmdk file will report errors such as “disk cannot be mounted”. According to the Importing Virtual Machines and disk images - ProxmoxVE Docs, you need to download and install the MergeIDE.zip component from the Internet, and you need to change the hard disk type to IDE before starting the virtual machine in order to solve this problem.

2. import img image

The img image file, the so-called raw format image, also known as bare image, has faster IO speed than qcow2, but is large and does not support advanced features such as snapshots. If you do not pursue IO performance, it is recommended to convert it to qcow2 and then use it.

1

qemu-img convert -f raw -O qcow2 vm01.img vm01.qcow2

III. Virtual Machine Management

Virtual machine management can be done using the command line tool virsh / virt-install or the GUI tool virt-manager .

The GUI is very foolproof, so I won’t introduce it here, but mainly the command line tool virsh / virt-install.

First, let’s introduce a few concepts in libvirt.

  • Domain: Refers to an instance of the operating system running on the virtual machine - a virtual machine, or the configuration used to start the virtual machine.
  • Guest OS: The virtual operating system running in the domain.

In most cases, you can interpret the domain involved in the following commands as a virtual machine.

0. Setting the default URI

virsh / virt-install / virt-viewer and a series of libvirt commands, sudo virsh net-list -all will use qemu:///session as URI to connect to QEMU/KVM by default, only only the root account will use qemu:///system by default .

On the other hand, the GUI tool virt-manager will also use qemu:///system to connect to QEMU/KVM by default (same as the root account)

qemu:///system is the global qemu environment for the system, while the qemu:///session environment is isolated by user. Also qemu:///session does not have a default network, which can cause problems when creating virtual machines.

Therefore, you need to change the default URI to qemu:///system or you will definitely get screwed:

1

echo 'export LIBVIRT_DEFAULT_URI="qemu:///system"' >> ~/.bashrc

1. Virtual Machine Network

After the installation of qemu-kvm is complete, a default network is created by default in the qemu:///system environment, while qemu:///session does not provide a default network and needs to be created manually.

We usually use the qemu:///system environment just fine, you can use the following method to view and start the default network so that it will be available when you create the virtual machine later.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# 列出所有虚拟机网络
$ sudo virsh net-list --all
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   no          yes

# 启动默认网络
$ virsh net-start default
Network default started

# 将 default 网络设为自启动
$ virsh net-autostart --network default
Network default marked as autostarted

# 再次检查网络状况，已经是 active 了
$ sudo virsh net-list --all
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

You can also create a new VM network, which requires writing the xml configuration for the network manually, and then creating it with virsh net-define --file my-network.xml, which I won’t go into detail about here because I won’t use it for a while…

2. Create the virtual machine - virt-intall

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# 使用 iso 镜像创建全新的 proxmox 虚拟机，自动创建一个 60G 的磁盘。
virt-install --virt-type kvm \
--name pve-1 \
--vcpus 4 --memory 8096 \
--disk size=60 \
--network network=default,model=virtio \
--os-type linux \
--os-variant generic \
--graphics vnc \
--cdrom proxmox-ve_6.3-1.iso

# 使用已存在的 opensuse cloud 磁盘创建虚拟机
virt-install --virt-type kvm \
  --name opensuse15-2 \
  --vcpus 2 --memory 2048 \
  --disk opensuse15.2-openstack.qcow2,device=disk,bus=virtio \
  --disk seed.iso,device=cdrom \
  --os-type linux \
  --os-variant opensuse15.2 \
  --network network=default,model=virtio \
  --graphics vnc \
  --import

The -os-variant is used to set the OS-related optimization configuration, which is set by the official documentation highly recommended, and its optional parameters can be viewed via osinfo-query os.

3. virtual machine management - virsh

Once the virtual machine has been created, you can use virsh to manage the virtual machine.

To view the list of virtual machines.

1
2
3
4
5

# 查看正在运行的虚拟机
virsh list

# 查看所有虚拟机，包括 inactive 的虚拟机
virsh list --all

Use virt-viewer to log in to the virtual machine terminal using the vnc protocol.

1
2
3
4

# 使用虚拟机 ID 连接
virt-viewer 8
# 使用虚拟机名称连接，并且等待虚拟机启动
virt-viewer --wait opensuse15

Starting, shutting down, suspending (hibernating), restarting the virtual machine.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

virsh start opensuse15
virsh suuspend opensuse15
virsh resume opensuse15
virsh reboot opensuse15
# 优雅关机
virsh shutdown opensuse15
# 强制关机
virsh destroy opensuse15

# 启用自动开机
virsh autostart opensuse15
# 禁用自动开机
virsh autostart --disable opensuse15

Virtual machine snapshot management.

1
2
3
4
5
6
7
8

# 列出一个虚拟机的所有快照
virsh snapshot-list --domain opensuse15
# 给某个虚拟机生成一个新快照
virsh snapshot-create <domain>
# 使用快照将虚拟机还原
virsh snapshot-restore <domain> <snapshotname>
# 删除快照
virsh snapshot-delete <domain> <snapshotname>

Deleting a virtual machine.

1

virsh undefine opensuse15

Migration of virtual machines.

1
2
3

# 使用默认参数进行离线迁移，将已关机的服务器迁移到另一个 qemu 实例
virsh migrate 37 qemu+ssh://tux@jupiter.example.com/system
# 还支持在线实时迁移，待续

cpu/memory modifications.

1
2
3
4

# 改成 4 核
virsh setvcpus opensuse15 4
# 改成 4G
virsh setmem opensuse15 4096

Virtual machine monitoring.

1

# 待续

Modify disks, networks and other devices.

1
2
3
4
5
6
7
8

# 添加新设备
virsh attach-device
virsh attach-disk
virsh attach-interface
# 删除设备
virsh detach-disk
virsh detach-device
virsh detach-interface