## Preface

The CRD API is now available in Kubernetes v1.16 GA, so I’m going to talk about some features and some common issues about CRD v1; I’ll probably mention the basics of CRD briefly, but not too much. The basics can be found in the official Kubernetes documentation. You can use this article more as a quick reference manual when you encounter problems.

CRD is the most common and convenient way to extend Kubernetes. In addition to providing the basic ability to define custom resource (CR), CRD also provides many extensions, including schema, multiple versions, conversion between versions, subresource subresources, etc. This article focuses on the following four parts Contents.

• metadata: defines the basic information of CR, such as API group and name
• versions: defines the version information of the CR
• schema: defines the structure and field types of the CR.
• subresource: defines the subresource information of the CR

The most basic but most important part, metadata contains the following main elements

• The name of the custom resource CR
• The API group the CR belongs to
• scope: whether the resource is at the Namespace or Cluster level

Here is a more complete example (the comments are basically taken from the official K8s documentation, so I won’t translate them), assuming we want to define a new VolumeSnapshot resource.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25  apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: # the name of this CR, must match the spec fields below, and be in the form: .. name: volumesnapshots.xinzhao.me spec: # group is the API group of the defined custom resource. # The custom resources are served under /apis//.... # Must match the name of the CustomResourceDefinition (in the form .). group: xinzhao.me names: # kind is normally the CamelCased singular type. Your resource manifests use this. kind: VolumeSnapshot # listKind is the serialized kind of the list for this resource. Defaults to "kindList". listKind: VolumeSnapshotList # plural name to be used in the URL: /apis/// plural: volumesnapshots # singular name to be used as an alias on the CLI and for display singular: volumesnapshot # shortNames allow shorter string to match your resource on the CLI # kubectl get volumesnapshot/vss shortNames: - vss # either Namespaced or Cluster scope: Namespaced 

where group must be a domain name, otherwise an exception will be thrown when it is created.

 1  The CustomResourceDefinition "volumesnapshots.test" is invalid: spec.group: Invalid value: "test": should be a domain with at least one dot 

## versions

defines the version information of CR, including which versions are available, the structure of each version, the conversion strategy between versions, etc.

There are currently two version conversion policies.

• None: The default policy is to change only apiVersion, other fields will be ignored, data in fields that are available in the old version but not in the new version will be lost directly, fields with the same name but mismatched structures will be reported as errors.
• Webhook: you can configure a custom conversion, API Server will call to an external webhook to do the conversion

let’s look at the overall structure through an example.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: ... spec: ... # list of versions supported by this CustomResourceDefinition versions: - name: v1beta1 served: true storage: true additionalPrinterColumns: - name: PVC type: string jsonPath: .spec.pvcName schema: ... # The conversion section is introduced in Kubernetes 1.13+ with a default value of # None conversion (strategy sub-field set to None). conversion: # None conversion assumes the same schema for all versions and only sets the apiVersion # field of custom resources to the proper value strategy: None 

• Each version can be enabled/disabled via the served field; one and only one version can be marked as a stored version (storage = true)

• Versions that get served as false will throw an exception directly.

 1  Error from server (NotFound): Unable to list "xinzhao.me/v1, Resource=volumesnapshots": the server could not find the requested resource (get volumesnapshots.xinzhao.me) 
• The version of the created CR will end up being the version with storage = true, and the previous version will be converted to the new version when it is created.

• If there is a previous version of the object, it will be converted to the storage = true version when the object is updated.

• The conversion is done through the strategy of the conversion configuration

• When you get a resource (e.g. using kubectl get), if the specified version is different from the version of the object when it was stored, k8 will handle the conversion between multiple versions (by conversion) and return the version you specified, but this is only a display conversion, in fact the data in etcd is still the old version, and only updates will actually convert. If something goes wrong with the conversion, an exception will be thrown.

 1  Error from server: conversion webhook for xinzhao.me/v1beta1, Kind=VolumeSnapshot failed: Post service "example-conversion-webhook-server" not found 

### schema

Each version has its own separate schema structure that describes what fields the CR has and what type each field is, etc.; the schema uses the JSON Schema standard, a simple example is as follows.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19  apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: ... spec: ... versions: - name: v1beta1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: pvcName: type: string 

For this CR we have defined only spec and only one field pvcName, so create a YAML file for this CR as follows.

 1 2 3 4 5 6 7  apiVersion: xinzhao.me/v1beta1 kind: VolumeSnapshot metadata: name: s1beta1 namespace: default spec: pvcName: pvc1 

Each field can be set to a default value, and the value of default in yaml should be whatever type the field is. For example, I have the following three types of fields.

  1 2 3 4 5 6 7 8 9 10 11 12 13  properties: type: type: array items: type: string default: - "test" name: type: string default: "test" size: type: number default: 1 

Fields also support validation, for example int can specify a range of max/min values, string supports regular expression validation and enum enumeration. See JSON Schema Validation for an example of what each type supports.

  1 2 3 4 5 6 7 8 9 10 11 12 13  properties: type: type: array items: type: string minItems: 2 name: type: string pattern: '\\w+' size: type: number minimum: 1 maximum: 10 

Some common problems with schema.

• All fields must be defined in schema or an exception will be thrown.

 1  error: error validating "snapshot-v1beta1.yaml": error validating data: ValidationError(VolumeSnapshot): unknown field "spec" in ...; if you choose to ignore these errors, turn validation off with --validate=false 
• If you want to store fields that are not previously defined, you can add the x-kubernetes-preserve-unknown-fields: true attribute to the corresponding parent field

• Defined fields can be left blank, or you can configure required to require certain fields to be required.

  1 2 3 4 5 6 7 8 9 10 11 12  schema: openAPIV3Schema: type: object properties: spec: type: object properties: pvc: type: object required: ["size"] properties: ... 
• An exception will be thrown if the spec.pvc.size field is not filled in
• If validation is turned off (kubectl create --validate=false ...), undefined fields will be discarded and not stored

• type does not have a map type, you can use type: object with x-kubernetes-map-type or x-kubernetes-preserve-unknown-fields to act as a map type (any field is allowed)

## subresource

Two types of subresource are supported, a Status subresource and a Scale subresource. In the example below, both status and scale subresource are enabled.

  1 2 3 4 5 6 7 8 9 10 11 12  # subresources describes the subresources for custom resources. subresources: # status enables the status subresource. status: {} # scale enables the scale subresource. scale: # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas. specReplicasPath: .spec.replicas # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas. statusReplicasPath: .status.replicas # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector. labelSelectorPath: .status.labelSelector 

### status subresource

With status subresource enabled, the entire CR content will be divided into spec and status parts, representing the expected status of the resource and the actual status of the resource, respectively. The resource’s API will then expose a new /status interface, and this CR creation/update interface will verify the entire content (including the status). But status is ignored in its entirety, and status can only be updated via the /status interface. The /status interface will only update/check the contents of status, everything else will be ignored.

For client-go, the UpdateStatus method will be generated whenever Status is defined by type.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37  // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // Snapshot is the snapshot object of the specified PVC. type Snapshot struct { metav1.TypeMeta json:",inline" // Standard object's metadata. // More info: // +optional metav1.ObjectMeta json:"metadata,omitempty" Spec SnapshotSpec json:"spec,omitempty" Status SnapshotStatus json:"status,omitempty" } // SnapshotInterface has methods to work with Snapshot resources. type SnapshotInterface interface { Create(*v1beta1.Snapshot) (*v1beta1.Snapshot, error) Update(*v1beta1.Snapshot) (*v1beta1.Snapshot, error) UpdateStatus(*v1beta1.Snapshot) (*v1beta1.Snapshot, error) ... } // UpdateStatus was generated because the type contains a Status member. // Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus(). func (c *snapshots) UpdateStatus(snapshot *v1beta1.Snapshot) (result *v1beta1.Snapshot, err error) { result = &v1beta1.Snapshot{} err = c.client.Put(). Namespace(c.ns). Resource("snapshots"). Name(snapshot.Name). SubResource("status"). Body(snapshot). Do(). Into(result) return } 

### scale subresource

Similar to status subresource, a new /scale interface will be exposed when enabled. But I’ve never used this, so I won’t go into it too much. You can see from the official documentation that the scale subresource, when enabled, also allows you to control the number of copies of a resource individually using kubectl scale.