Preface

The CRD API is now available in Kubernetes v1.16 GA, so I’m going to talk about some features and some common issues about CRD v1; I’ll probably mention the basics of CRD briefly, but not too much. The basics can be found in the official Kubernetes documentation. You can use this article more as a quick reference manual when you encounter problems.

CRD is the most common and convenient way to extend Kubernetes. In addition to providing the basic ability to define custom resource (CR), CRD also provides many extensions, including schema, multiple versions, conversion between versions, subresource subresources, etc. This article focuses on the following four parts Contents.

  • metadata: defines the basic information of CR, such as API group and name
  • versions: defines the version information of the CR
  • schema: defines the structure and field types of the CR.
  • subresource: defines the subresource information of the CR

metadata

The most basic but most important part, metadata contains the following main elements

  • The name of the custom resource CR
  • The API group the CR belongs to
  • scope: whether the resource is at the Namespace or Cluster level

Here is a more complete example (the comments are basically taken from the official K8s documentation, so I won’t translate them), assuming we want to define a new VolumeSnapshot resource.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # the name of this CR, must match the spec fields below, and be in the form: <plural>.<group>.
  name: volumesnapshots.xinzhao.me
spec:
  # group is the API group of the defined custom resource.
  # The custom resources are served under `/apis/<group>/...`.
  # Must match the name of the CustomResourceDefinition (in the form `<names.plural>.<group>`).
  group: xinzhao.me
  names:
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: VolumeSnapshot
    # listKind is the serialized kind of the list for this resource. Defaults to "`kind`List".
    listKind: VolumeSnapshotList
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: volumesnapshots
    # singular name to be used as an alias on the CLI and for display
    singular: volumesnapshot
    # shortNames allow shorter string to match your resource on the CLI
    # kubectl get volumesnapshot/vss
    shortNames:
    - vss
  # either Namespaced or Cluster
  scope: Namespaced

where group must be a domain name, otherwise an exception will be thrown when it is created.

1

The CustomResourceDefinition "volumesnapshots.test" is invalid: spec.group: Invalid value: "test": should be a domain with at least one dot

versions

defines the version information of CR, including which versions are available, the structure of each version, the conversion strategy between versions, etc.

There are currently two version conversion policies.

  • None: The default policy is to change only apiVersion, other fields will be ignored, data in fields that are available in the old version but not in the new version will be lost directly, fields with the same name but mismatched structures will be reported as errors.
  • Webhook: you can configure a custom conversion, API Server will call to an external webhook to do the conversion

let’s look at the overall structure through an example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  # list of versions supported by this CustomResourceDefinition
  versions:
  - name: v1beta1
    served: true
    storage: true
    additionalPrinterColumns:
    - name: PVC
      type: string
      jsonPath: .spec.pvcName
    schema:
      ...
  # The conversion section is introduced in Kubernetes 1.13+ with a default value of
  # None conversion (strategy sub-field set to None).
  conversion:
    # None conversion assumes the same schema for all versions and only sets the apiVersion
    # field of custom resources to the proper value
    strategy: None

Frequently asked questions about multiple versions.

  • Each version can be enabled/disabled via the served field; one and only one version can be marked as a stored version (storage = true)

  • Versions that get served as false will throw an exception directly.

    1

    Error from server (NotFound): Unable to list "xinzhao.me/v1, Resource=volumesnapshots": the server could not find the requested resource (get volumesnapshots.xinzhao.me)

  • The version of the created CR will end up being the version with storage = true, and the previous version will be converted to the new version when it is created.

  • If there is a previous version of the object, it will be converted to the storage = true version when the object is updated.

  • The conversion is done through the strategy of the conversion configuration

  • When you get a resource (e.g. using kubectl get), if the specified version is different from the version of the object when it was stored, k8 will handle the conversion between multiple versions (by conversion) and return the version you specified, but this is only a display conversion, in fact the data in etcd is still the old version, and only updates will actually convert. If something goes wrong with the conversion, an exception will be thrown.

    1

    Error from server: conversion webhook for xinzhao.me/v1beta1, Kind=VolumeSnapshot failed: Post  service "example-conversion-webhook-server" not found

schema

Each version has its own separate schema structure that describes what fields the CR has and what type each field is, etc.; the schema uses the JSON Schema standard, a simple example is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1beta1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              pvcName:
                type: string

For this CR we have defined only spec and only one field pvcName, so create a YAML file for this CR as follows.

1
2
3
4
5
6
7

apiVersion: xinzhao.me/v1beta1
kind: VolumeSnapshot
metadata:
  name: s1beta1
  namespace: default
spec:
  pvcName: pvc1

Each field can be set to a default value, and the value of default in yaml should be whatever type the field is. For example, I have the following three types of fields.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

properties:
  type:
    type: array
    items:
      type: string
    default:
    - "test"
  name:
    type: string
    default: "test"
  size:
    type: number
    default: 1

Fields also support validation, for example int can specify a range of max/min values, string supports regular expression validation and enum enumeration. See JSON Schema Validation for an example of what each type supports.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

properties:
  type:
    type: array
    items:
      type: string
    minItems: 2
  name:
    type: string
    pattern: '\\w+'
  size:
    type: number
    minimum: 1
    maximum: 10

Some common problems with schema.

  • All fields must be defined in schema or an exception will be thrown.

    1

    error: error validating "snapshot-v1beta1.yaml": error validating data: ValidationError(VolumeSnapshot): unknown field "spec" in ...; if you choose to ignore these errors, turn validation off with --validate=false

  • If you want to store fields that are not previously defined, you can add the x-kubernetes-preserve-unknown-fields: true attribute to the corresponding parent field

  • Defined fields can be left blank, or you can configure required to require certain fields to be required.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    schema:
openAPIV3Schema:
    type: object
    properties:
    spec:
        type: object
        properties:
        pvc:
            type: object
            required: ["size"]
            properties:
            ...
    • An exception will be thrown if the spec.pvc.size field is not filled in

  • If validation is turned off (kubectl create --validate=false ...), undefined fields will be discarded and not stored

  • type does not have a map type, you can use type: object with x-kubernetes-map-type or x-kubernetes-preserve-unknown-fields to act as a map type (any field is allowed)

subresource

Two types of subresource are supported, a Status subresource and a Scale subresource. In the example below, both status and scale subresource are enabled.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# subresources describes the subresources for custom resources.
subresources:
  # status enables the status subresource.
  status: {}
  # scale enables the scale subresource.
  scale:
    # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
    specReplicasPath: .spec.replicas
    # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
    statusReplicasPath: .status.replicas
    # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
    labelSelectorPath: .status.labelSelector

status subresource

With status subresource enabled, the entire CR content will be divided into spec and status parts, representing the expected status of the resource and the actual status of the resource, respectively. The resource’s API will then expose a new /status interface, and this CR creation/update interface will verify the entire content (including the status). But status is ignored in its entirety, and status can only be updated via the /status interface. The /status interface will only update/check the contents of status, everything else will be ignored.

For client-go, the UpdateStatus method will be generated whenever Status is defined by type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Snapshot is the snapshot object of the specified PVC.
type Snapshot struct {
    metav1.TypeMeta `json:",inline"`
    // Standard object's metadata.
    // More info: <https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata>
    // +optional
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   SnapshotSpec   `json:"spec,omitempty"`
    Status SnapshotStatus `json:"status,omitempty"`
}

// SnapshotInterface has methods to work with Snapshot resources.
type SnapshotInterface interface {
    Create(*v1beta1.Snapshot) (*v1beta1.Snapshot, error)
    Update(*v1beta1.Snapshot) (*v1beta1.Snapshot, error)
    UpdateStatus(*v1beta1.Snapshot) (*v1beta1.Snapshot, error)
    ...
}

// UpdateStatus was generated because the type contains a Status member.
// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
func (c *snapshots) UpdateStatus(snapshot *v1beta1.Snapshot) (result *v1beta1.Snapshot, err error) {
    result = &v1beta1.Snapshot{}
    err = c.client.Put().
        Namespace(c.ns).
        Resource("snapshots").
        Name(snapshot.Name).
        SubResource("status").
        Body(snapshot).
        Do().
        Into(result)
    return
}

scale subresource

Similar to status subresource, a new /scale interface will be exposed when enabled. But I’ve never used this, so I won’t go into it too much. You can see from the official documentation that the scale subresource, when enabled, also allows you to control the number of copies of a resource individually using kubectl scale.