kubernetes supports local volume (local volume) since version 1.10. workloads (not only statefulsets types) can take advantage of local fast SSDs to get better performance than remote volumes (e.g. cephfs, RBD).
Before the advent of local volume, statefulsets could also take advantage of local SSDs by configuring hostPath and binding to specific nodes via nodeSelector or nodeAffinity. However, the problem with hostPath is that administrators need to manually manage the directory of each node of the cluster, which is less convenient.
The following two types of applications are suitable for using local volume.
- Data caching, where applications can access data nearby for fast processing.
- Distributed storage systems, such as distributed database Cassandra , distributed file system ceph/gluster
The following will introduce how to use local volume by creating PV, PVC, Pod in a manual way, then introduce the semi-automatic way provided by external storage, and finally introduce some developments in the community.
Create a storage class
First you need to have a sc named
sc’s provisioner is
WaitForFirstConsumer means that the PV should not bind the PVC immediately, but until there is a Pod that needs to use the PVC. The scheduler will take into account the appropriate local PV when scheduling, so that it does not cause conflicts with Pod resource settings, selectors, affinity and anti-affinity policies, etc. Obviously: if the PVC is bound to the local PV first, since the local PV is bound to the node, the selectors, affinity, etc. are basically useless, so it is better to select the node first according to the scheduling policy, and then bind the local PV.
Create a PV statically
Create a 5GiB PV statically via the kubectl command; the PV uses the
/data/local/vol1 directory of node ubuntu-1; the sc of the PV is local-volume.
Retain means that after the PV is released with the PVC, the administrator needs to clean it up manually and reset the volume.
You need to specify the sc corresponding to the PV; the directory
/data/local/vol1 also needs to be created.
Using a local volume PV
Next, create a PVC with sc:local-volume associated with it, and then hook that PVC into the nginx container.
When you enter the container, you will see the mounted directory, and the size is actually the size of the disk where the PV created above is located.
index.html file in the
/data/local/vol1 directory of the host.
Then go to the IP address of the curl container and you can get the string you just wrote.
Delete the Pod/PVC, after that the PV status is changed to Released and that PV will not be bound to the PVC anymore.
Dynamic PV creation
Managing local PVs manually is obviously a lot of work, and the community provides external storage to create PVs dynamically (which is still not actually automated enough).
The official arrangement of the local volume provisioner is in the
local-volume/provisioner/deployment/kubernetes/example/default_example_provisioner_generated.yaml directory However, the official documentation is a bit confusing with fast-disk and local-storage. I use
After kubectl is created, a provisioner will be started on each node due to the daemonset type. this provisioner will monitor the
discovery directory, which is
/data/local configured above.
mypod/myclaim has been deleted and we recreate one, at this point pvc myclaim is in Pending state and the provisoner does not automatically supply the storage. Why?
It turns out that the logic of
external-storage is like this: the provisioner itself does not provide local volume, but its provisioner on each node will dynamically “discover” the mount point (discovery directory), when the provisioner of a node finds a mount point in the
/data/local/ directory, it will create a PV, the
local.path of the PV is the mount point, and set the nodeAffinity to that node.
So how do you get the mount point?
Going straight to creating a directory won’t work because provsioner expects PVs to be isolated, e.g. capacity, io, etc. Try creating a
xxx directory under
/data/local/ on ubuntu-2 and you will get this alert.
The directory is not a mount point and cannot be used.
The directory must be a real mount to work. One way is to add the drive, format it, and mount it, which is a lot of work. You can actually “trick” the provisioner into thinking it is a mounted disk by formatting the local files (loopfs) and then mounting it, thus automatically creating the PV and binding it to the PVC.
This is done as follows.
Save the following code as a file
loopmount, add execute permissions and copy it to the
/bin directory, then you can use the command to create a mount point.
Use the script to create a 6G file and mount it under
/data/local. The reason for the 6G is that the front PVC needs 5GB, and the remaining space after formatting will be a little smaller, so set the file a little bigger to bind the PVC later.
Looking at the PV, you can see that Provisioner automatically creates the PV, and kubernetes will provision that PV to the PVC myclam in front of it, and mypod is run up.
As you can see, the current version of local volume is not yet fully automated like cephfs/RBD, and still requires administrator intervention, which is obviously not a good implementation.
Someone in the community has submitted a Proposal to do dynamic provisioning of local volume based on LVM, but progress is slow. The author is a huawei employee, so I think huawei has already implemented it.
In addition to LVM-based, you can also implement dynamic provisioning of LV based on
ext4 project quota.
In addition to using disk, you can also consider using an in-memory file system to get higher io performance, just the capacity is not so ideal. Some special applications can be considered.
In general, local volume local volumes do not currently support dynamic provisioning and cannot really be promoted for use yet, but they can be used to solve some specific problems.