Kubernetes Component Unit Testing Guide

Preface

The concepts and basics of unit testing are not covered here, but you can refer to some official go guides and other materials on the web:

• LearnTesting
• TableDrivenTests

For components that need to operate k8s, the key to single testing is how to construct a k8s cluster in the single testing function for the business function to CRUD the corresponding resources. The general idea of constructing a k8s cluster is divided into two categories: using a fake client to construct a fake and constructing a real, lightweight k8s cluster in the process of single testing; the following will introduce these two methods one by one The two methods are described below.

Note: Depending on the test object, it is sufficient to choose the appropriate method that can achieve the test purpose, and it is not necessary to force the use of a particular method.

Using fake client

The fake client can only be used to CRUD various resources (but in fact this can cover most of the scenarios), some other operations such as triggering the callback event of the informer is not possible, so if the test code also wants to cover such scenarios, you need to use the following method of constructing a real cluster; using the fake client test steps The test steps are roughly as follows.

1. Construct test data
   • construct the test data, i.e., the (native and custom) resource objects needed in the various test cases
2. use the above test data to generate the fake client
   • Append these test objects to the fake client
3. replace the client used by the business function with the fake client
   • This depends on how the business function is implemented and how the k8s client is obtained

The fake client can be subdivided into the following two categories.

native client

refers to the native client-go and the typed client of each CR generated using code-generator, which provide the corresponding fake client method. is easy to construct, just use one function and add all the objects needed for testing.

1

client := fake.NewSimpleClientset(objects...)

A simple example is as follows, starting with a business function defined as follows.

1
2
3
4

// Add adds or updates the given Event object.
func Add(kubeClient kubernetes.Interface, eventObj *corev1.Event) error {
    ...
}

I need to test just two scenarios: adding an event and updating an existing event, and constructing test data for these two scenarios.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

tests := []struct {
    name             string
    objects          []runtime.Object
    event            *corev1.Event
    isErr            bool
    wantedEventCount int32
}{
    {
        name:             "exist test",
        objects:          []runtime.Object{test.GetEvent()},
        event:            test.GetEvent(),
        isErr:            false,
        wantedEventCount: 1,
    },
    {
        name:             "not exist test",
        event:            test.GetEvent(),
        isErr:            false,
        wantedEventCount: 2,
    },
}

Generate a fake client based on the test data of the case and execute the test.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
        // 生成 fake client，把需要 CRUD 的资源加入进去
        client := fake.NewSimpleClientset(tt.objects...)

        // 执行函数
        err := Add(client, tt.event)
        if tt.isErr != (err != nil) {
            t.Errorf("%s Add() unexpected error: %v", tt.name, err)
        }

        // 校验结果是否符合预期
        eventObj, err := client.CoreV1().Events(tt.event.Namespace).Get(context.TODO(), tt.event.Name, metav1.GetOptions{})
        if err != nil {
            t.Errorf("%s unexpected error: %v", tt.name, err)
        }
        if eventObj.Count != tt.wantedEventCount {
            t.Errorf("%s event Count = %d, want %d", tt.name, eventObj.Count, tt.wantedEventCount)
        }
    })
}

Note: The test function does not have to be written like the example, focus on understanding the process and the method of constructing the fake client can be.

For the controller test, if you use the lister, you need to add the required resources to the informer of the corresponding resources, so that the lister can read the corresponding resources in the business code, a simple example is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// 创建 fake client
f.client = fake.NewSimpleClientset(f.objects...)

// 创建基于 fake client 的 informer
informer := informers.NewSharedInformerFactory(f.client, 0)

// 往 informer indexer 中添加对应的资源对象
for _, s := range f.storageClasses {
    informer.Native().Storage().V1().StorageClasses().Informer().GetIndexer().Add(s)
}

After that, just replace the client and informer and run the test again.

generic client

refers to the generic client provided by controller-runtime, unlike the typed Unlike the typed client above, this client is a generic client that can be used to CRUD any resource. The test method is basically the same as above, only the method of constructing the fake client is slightly different, but the rest of the process is the same. The following will only introduce the method of constructing the fake client, and the rest of the content will not be repeated.

The biggest difference in the method of constructing the fake client of the generic client is that it needs to contain the scheme of all the resources to be CRUDed.

1
2

// 配置 scheme 和需要添加的 objects
client := fake.NewClientBuilder().WithScheme(scheme.Scheme).WithObjects(objs...).Build()

The general construction method of scheme is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

var (
    // Scheme contains all types of custom clientset and kubernetes client-go clientset
    Scheme = runtime.NewScheme()
)

func init() {
    // 添加你需要的资源的 scheme
    _ = clientgoscheme.AddToScheme(Scheme)
    _ = cosscheme.AddToScheme(Scheme)
    _ = apiextensionsv1.AddToScheme(Scheme)
}

Constructing lightweight clusters

This method can be used to test scenarios that cannot be covered by using a fake client. It is very convenient to start a real cluster in a unit test using the envtest library provided by controller-runtime, and since a real cluster is started, this method is applicable to any client.

The test steps are roughly as follows.

1. prepare the cluster-related configuration and start the cluster
   • Generally, no additional configuration is needed, a function can be started, if you want to pre-register CRD and so on, you need to configure more configuration items, you can refer to https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/envtest/server.go#L105
2. use the kube config of the cluster created above to generate various clients
3. create test data (if needed)
4. Run the tests and destroy the cluster when finished

Start the cluster as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// 生成 env，这里面有很多配置项，可以根据需要配置
testEnv := &envtest.Environment{}

// 启动环境，返回值是环境的 rest.Config，可用于生成各 kube client
config, err := testEnv.Start()
if err != nil {
    t.Fatal(err)
}

// 销毁集群
testEnv.Stop()

Note that the method needs to install kubebuilder in advance, it relies on the apiserver that the kubebuilder package provides several binary files, the local words themselves download to install it, if the CI environment requires it, you can add these files in the base image, an example.

1
2
3
4
5

RUN mkdir -p /usr/local && \
    wget https://go.kubebuilder.io/dl/2.3.1/linux/amd64 && \
    tar xvf amd64 && \
    mv kubebuilder_2.3.1_linux_amd64 /usr/local/kubebuilder && \
    rm amd64

The following is a simple example for a normal client with business functions defined as follows.

1
2
3
4

// Create creates the given CRD objects or updates them if these objects already exist in the cluster.
func Create(client clientset.Interface, crds ...*apiextensionsv1.CustomResourceDefinition) error {
    ...
}

Test preparation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// 启动测试集群
testEnv := &envtest.Environment{}
config, err := testEnv.Start()
if err != nil {
    t.Fatal(err)
}
defer testEnv.Stop()

// 生成需要的 apiextension client
apiextensionClient, _ := apiextensionsclient.NewForConfig(config)

Then you can use the above apiextensionClient to execute the test.

A simple example for the generic client is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// 启动测试集群
testEnv := &envtest.Environment{}
config, err := testEnv.Start()
if err != nil {
    t.Fatal(err)
}
defer testEnv.Stop()

// 生成 generic client
cli, err := client.New(config, client.Options{
    Scheme: scheme,
})
if err != nil {
    t.Fatal(err)
}

// 准备测试数据
sc1 := test.GetStorageClass()
sc1.Name = "sc1"
sc1.Provisioner = "example.com/test"
// 创建测试数据
if err := cli.Create(context.TODO(), sc1); err != nil {
    t.Fatal(err)
}

// 开始执行测试...

Comparison of common scenarios

fake client is simple to use, very lightweight, fast execution, but it basically can only cover resource CRUD scenarios, other operations of the business code can not be covered, there are some scenarios can be covered but need some additional operations, a little bit of trouble; construct real cluster can completely simulate the components running in the online cluster, almost all the business code can be covered as long as you want to test However, the execution speed is slow and there are special requirements for the environment; a simple way to judge which solution to choose is to use the fake client to test the fake client that can be covered, and then use the method of constructing the real cluster if the fake client cannot be covered.