Building a Kubernetes CRD Controller with Kubebuilder

The previous article, How to Extend a Kubernetes Cluster with CRD, explained what CRD is and what capabilities it can provide through a demo. This article continues to build on that demo (https://github.com/Coderhypo/KubeService) and explains how to build a CRD Controller.

CRD Controller

For the CRD (CustomResourceDefinition) itself, it is not too much to understand it as just a Schema of OpenApi, because that is its only ability and role, but for the broader statement: “CRD is used to implement xx functions”, it is actually the CRD Controller that is responsible for implementing the functions.

Kubernetes itself comes with a bunch of Controllers, and one of the three core components on the Master node: Controller Manager, is actually a collection of Controllers.

There are many Controllers in the Controller Manger that do essentially the same thing as the CRD Controller we are implementing, which is to manage specific resources.

It is also interesting to see how different Controllers communicate with each other in Kubernetes, for example, by creating Pods through Deployment.

The user creates a Deployment through Kubectl, APIServer authenticates the permission and access to the request, and then stores the resources of the Deployment in ETCD, because Kubernetes implements the List-Watch mechanism through ETCD, so the Deployment-related events of interest to The Deployment Controller receives the ADD event for the resource and processes it, i.e., creates an RS for the Deployment.

After the RS creation request is received by the APIServer, the ADD event of the RS will be published, so the ReplicaSet Controller will receive the event and process it subsequently: i.e., create the Pod.

So you can see that thanks to the event-based way Kubernetes works, the action of creating a Deployment-managed Pod is performed by both the Deployment Controller and the ReplicaSet Controller, but there is no direct communication between the two Controllers.

Because of its event-based approach, we can customize the Controller to handle events of interest, including but not limited to CR creation, modification, etc.

Kubebuilder and Operator-SDK

For building CRD Controllers, there are several mainstream tools, one is the coreOS open source Operator-SDK and the other is the Kubebuilder maintained by the K8s interest group (https://github.com/kubernetes-sigs/kubebuilder).

The Operator-SDK is part of the Operator framework, and the Operator community is relatively mature and active, and even has its own Hub to let people explore and share interesting Operators.

Kubebuilder is not so much an SDK as a code generator, which generates a working Controller by conforming to the format of comments and data structures.

Kubebuilder quick start

Probably thanks to the Kubebuilder biased code generator, it is very easy to create a Controller from scratch using Kubebuilder, and there will be many articles or topics with the title “Create a CRD Controller in x minutes”.

The official Getting Started documentation can be found at: https://book.kubebuilder.io/quick-start.html

Creating a project

First initialize the project with the Kubebuilder init command, --domain flag arg to specify the api group.

1

kubebuilder init --domain o0w0o.cn --owner "Hypo"

When the project is created, you will be reminded whether to download the dependencies or not, and then you will find that most of the Kubernetes code is already in your GOPATH.

Creating Api

Once the project is created, you can create the api.

1
2

kubebuilder create api --group app --version v1 --kind App
kubebuilder create api --group app --version v1 --kind MicroService

While creating the api, you will find that Kubebuilder will create some directories and source files for you.

1. pkg.apis contains the default data structures for the resources App and MicroService.
2. pkg.controller contains the two default controllers for App and MicroService.

resource type

Kubebuilder has already created and default structure for you.

1
2
3
4
5
6
7
8
9

// MicroService is the Schema for the microservices API
// +k8s:openapi-gen=true
type MicroService struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MicroServiceSpec   `json:"spec,omitempty"`
    Status MicroServiceStatus `json:"status,omitempty"`
}

All you have to do is expand, using MicroService as an example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

type Canary struct {
	// +kubebuilder:validation:Maximum=100
	// +kubebuilder:validation:Minimum=1
	Weight int `json:"weight"`

	// +optional
	CanaryIngressName string `json:"canaryIngressName,omitempty"`

	// +optional
	Header string `json:"header,omitempty"`

	// +optional
	HeaderValue string `json:"headerValue,omitempty"`

	// +optional
	Cookie string `json:"cookie,omitempty"`
}

type DeployVersion struct {
	Name     string                `json:"name"`
	Template appsv1.DeploymentSpec `json:"template"`

	// +optional
	ServiceName string `json:"serviceName,omitempty"`

	// +optional
	Canary *Canary `json:"canary,omitempty"`
}

type ServiceLoadBalance struct {
	Name string             `json:"name"`
	Spec corev1.ServiceSpec `json:"spec"`
}

type IngressLoadBalance struct {
	Name string                        `json:"name"`
	Spec extensionsv1beta1.IngressSpec `json:"spec"`
}

type LoadBalance struct {
	// +optional
	Service *ServiceLoadBalance `json:"service,omitempty"`
	// +optional
	Ingress *IngressLoadBalance `json:"ingress,omitempty"`
}

// MicroServiceSpec defines the desired state of MicroService
type MicroServiceSpec struct {
	// +optional
	LoadBalance        *LoadBalance    `json:"loadBalance,omitempty"`
	Versions           []DeployVersion `json:"versions"`
	CurrentVersionName string          `json:"currentVersionName"`
}

The complete code is available at: https://github.com/Coderhypo/KubeService/blob/master/pkg/apis/app/v1/microservice_types.go

If you haven’t understood the Kubernetes controller code before, you may be wondering about the default generated Controller. The default MicroService Controller is named ReconcileMicroService, which has only one main method that is.

1

func (r *ReconcileMicroService) Reconcile(request reconcile.Request) (reconcile.Result, error)

Before making this controller work, you need to register the events to follow in the add method in pkg.controller.microservice.micriservice_controller.go, and when any event of interest occurs, Reconcile will be called. When an event occurs, it compares the current state of the resource with the expected state, and corrects it if it is inconsistent.

For example, if MicroService manages versions through Deployment, Reconcile has to determine whether each version of Deployment exists and whether it meets expectations.

The specific ReconcileMicroService code can be found at: https://github.com/Coderhypo/KubeService/blob/master/pkg/controller/microservice/microservice_controller.go

Running

Once the structure of CR has been defined and the Controller code has been completed, you can try to run it. kubebuilder can be run with a cluster configured locally in kubeconfig (minikube is recommended for creating a development cluster quickly).

First remember to add the schema to the init method in main.go.

1
2
3
4
5
6
7

func init() {
    _ = corev1.AddToScheme(scheme)
    _ = appsv1.AddToScheme(scheme)
    _ = extensionsv1beta1.AddToScheme(scheme)
    _ = apis.AddToScheme(scheme)
    // +kubebuilder:scaffold:scheme
}

Then make Kubebuilder regenerate the code:

1

make

Then apply the CRD yaml under config/crd to the current cluster:

1

make install

Run CRD Controller locally (you can execute the main function directly):

1

make run