1. Theoretical Overview
We know that Kubernetes provides a CRD mechanism to scale resources, and in some scenarios we can use CRD to scale resources stored through Kubernetes, i.e., use Kubernetes as storage. According to the official statement CRD + Controller = Operator, kubebuilder can generate such a code framework, and can be customized to generate Controller or not. The client-related code can be generated by code-generator, which generates informer, listster, clientset, and other series of code. In this way, we only need to fill the generated code with business code, and do not need to care about the code outside the framework, which can greatly improve the development efficiency and reduce the threshold of CRD. In this article, we introduce ConfigMapHistory, a CRD, as an example, and take you through building and deploying a CRD from zero to one.
2. kubebuilder installation
The kubebuilder installation documentation can be found in the official file Install kubebuilder.
1
2
3
4
5
6
7
8
9
10
|
os=$(go env GOOS)
arch=$(go env GOARCH)
# download kubebuilder and extract it to tmp
curl -L https://go.kubebuilder.io/dl/2.3.1/${os}/${arch} | tar -xz -C /tmp/
# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.3.1_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin
|
3. kubebuilder initialization
kubebuilder code initialization.
1
2
3
4
5
6
|
# github.com/opskumu 以实际仓库为主，本实例为 crd-example
mkdir -p $GOPATH/src/github.com/opskumu/crd-example
cd $GOPATH/src/github.com/opskumu/crd-example
# 域名以实际域名为主，本实例为 opskumu.com
kubebuilder init --domain opskumu.com
kubebuilder edit --multigroup=true
|
To view the current directory structure of the generated code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# tree -L 2 ./
.
├── Dockerfile
├── Makefile
├── PROJECT
├── bin
│ └── manager
├── config
│ ├── certmanager
│ ├── default
│ ├── manager
│ ├── prometheus
│ ├── rbac
│ └── webhook
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go
|
4. kubebuilder Create API
1
2
3
4
5
6
7
8
9
10
|
# kubebuilder create api --group test --version v1 --kind ConfigMapHistory
Create Resource [y/n] # 创建 ConfigMapHistory 资源，此处选择 y
y
Create Controller [y/n] # 创建控制器，此处不需要，选择 n
n
# tree apis/test/v1 # apis 目录存放 group test v1 相关的资源 API
apis/test/v1
├── configmaphistory_types.go
├── groupversion_info.go
└── zz_generated.deepcopy.go
|
5. Customize the code and generate the client code
Generate apis/test/v1/configmaphistory_types.go code by default as follows.
1
2
3
4
5
6
7
8
9
10
11
|
......
// +kubebuilder:object:root=true
// ConfigMapHistory is the Schema for the configmaphistories API
type ConfigMapHistory struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ConfigMapHistorySpec `json:"spec,omitempty"`
Status ConfigMapHistoryStatus `json:"status,omitempty"`
}
......
|
The purpose of creating a resource of type ConfigMapHistory is to store the history of the ConfigMap, so some changes need to be made to the above code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
......
// +genclient
// +kubebuilder:object:root=true
// ConfigMapHistory is the Schema for the configmaphistories API
type ConfigMapHistory struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
ConfigMap string `json:"configMap"` // 关联 ConfigMap 名
Data map[string]string `json:"data,omitempty"` // 对应 ConfigMap 版本数据
BinaryData map[string]byte `json:"binaryData,omitempty"` // 对应 ConfigMap 版本二进制数据
}
......
|
Note that // +genclient is added additionally to tag the subsequent code-generator to generate the client code. after the changes to the types file, the make manifests command can be executed to generate the CRD file.
1
2
3
|
# make manifests
# ls config/crd/bases
test.opskumu.com_configmaphistories.yaml
|
If the code structure changes, you need to run make manifest again to generate a new CRD file.
Create apis/test/v1/doc.go, apis/test/v1/register.go, hack/tools.go and hack/update-codegen.sh files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// +k8s:deepcopy-gen=package
// 注意 groupName 和实际相同
// +groupName=test.opskumu.com
// Package v1 is the v1 version of the API.
package v1
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package v1
import (
"k8s.io/apimachinery/pkg/runtime/schema"
)
// SchemeGroupVersion is group version used to register these objects.
var SchemeGroupVersion = GroupVersion
func Resource(resource string) schema.GroupResource {
return SchemeGroupVersion.WithResource(resource).GroupResource()
}
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
// +build tools
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// This package imports things required by build scripts, to force `go mod` to see them as dependencies
package tools
import _ "k8s.io/code-generator"
|
tools.go is mainly used to introduce the k8s.io/code-generator dependency.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
#!/usr/bin/env bash
# Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}
# generate the code with:
# --output-base because this script should also be able to run inside the vendor dir of
# k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
# instead of the $GOPATH directly. For normal projects this can be dropped.
bash "${CODEGEN_PKG}"/generate-groups.sh "client,informer,lister" \
github.com/opskumu/crd-example/generated github.com/opskumu/crd-example/apis \
test:v1 \
--output-base "$(dirname "${BASH_SOURCE[0]}")/../../../.." \
--go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt
# To use your own boilerplate text append:
# --go-header-file "${SCRIPT_ROOT}"/hack/custom-boilerplate.go.txt
|
The above code can be found in sample-controller doc.go/register.go/ tools.go/update-codegen.sh. The CRD in this article is generated by kubebuilder, so the content is slightly different.
After creating the above files, execute the following command to generate the code.
1
2
3
4
5
6
|
# 注意此处版本以下几个包要一致，否则可能出现不兼容的情况
# K8SVERSION=v0.19.6
# go get -v k8s.io/code-generator@${K8SVERSION}
# go get -v k8s.io/client-go@${K8SVERSION}
# go get -v k8s.io/apimachinery@${K8SVERSION}
# go mod vendor
|
Put the relevant dependency packages in the vendor so that hack/update-codegen.sh can call the script in code-generator. Once done, execute the code-generator command.
1
2
3
4
5
6
7
8
9
|
# bash hack/update-codegen.sh
Generating clientset for test:v1 at github.com/opskumu/crd-example/generated/clientset
Generating listers for test:v1 at github.com/opskumu/crd-example/generated/listers
Generating informers for test:v1 at github.com/opskumu/crd-example/generated/informers
# crd-example tree -L 1 generated
generated
├── clientset
├── informers
└── listers
|
This completes all the operations to add ConfigMapHistory CRD and related client code. If you need to generate other Kind types, repeat the above operations (the above created files do not need to be created again).
6. Create CRD
1
|
kubectl create -f config/crd/bases/test.opskumu.com_configmaphistories.yaml
|
Create test resources.
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# cat test.yaml
apiVersion: "test.opskumu.com/v1"
kind: ConfigMapHistory
metadata:
name: test
configMap: "test"
data:
test: test
# kubectl create -f ./test.yaml
configmaphistory.test.opskumu.com/test created
# kubectl get configmaphistory.test.opskumu.com
NAME AGE
test 27s
|
See crd-example for full example code.