1. Installation method

1
2

kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml
kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

OpenEBS is mainly used as the default storage for the Tekton pipeline. I’ve tried Longhorn before, but the pipeline kept Pending at peak times, and there were residuals after uninstalling Longhorn, which caused kube-apiserver to be abnormal, and it took a lot of effort to remove it.

2. OpenEBS is not available after the Kubernetes cluster certificate expires

Kubernetes cluster and OpenEBS components were installed on the same day. after the Kubernetes certificate expired, it was quickly renewed by kubeadm certs renew all; the OpenEBS certificate, which I had not paid much attention to before, also expired.

  • Tekton Controller Error

    1
2
3
4

    {"level":"info","ts":"2022-09-08T07:58:39.882Z",
"logger":"tekton-pipelines-controller.event-broadcaster",
"caller":"record/event.go:282",
"msg":"Event(v1.ObjectReference{Kind:\"PipelineRun\", Namespace:\"qsearch\", Name:\"p-cccq1buj5i3oh0tp2ueg\", UID:\"a1e15eee-4c44-4867-ac0e-decc16a1a0c8\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"230683178\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* failed to create PVC pvc-6dc4355ffe: Internal error occurred: failed calling webhook \"admission-webhookopenebs.io\": Post \"https://admission-server-svc.openebs.svc:443/validate?timeout=5s\": x509: certificate has expired or is not yet valid: current time 2022-09-08T07:58:39Z is after 2022-09-08T07:17:40Z\n\n","commit":"7ca5d61"}

  • OpenEBS Admission Server Error

    1

    kubectl delete ValidatingWebhookConfiguration openebs-validation-webhook-cfg

3. Solutions

  • Backup openebs-validation-webhook-cfg

    1

    kubectl get ValidatingWebhookConfiguration openebs-validation-webhook-cfg -o yaml > openebs-validation-webhook-cfg.yaml

  • Remove openebs-validation-webhook-cfg

    1

    kubectl delete ValidatingWebhookConfiguration openebs-validation-webhook-cfg

This is a solution given by the community Issues, reference: https://github.com/openebs/openebs/issues/3329.

It looks like the OpenEBS community hasn’t reproduced the problem yet and hasn’t had time to fix it. I just recently upgraded OpenEBS to openebs/admission-server:2.12.1 version.

4. Why

  • When we delete openebs-validation-webhook-cfg, what is deleted?

    View the deleted objects.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    cat openebs-validation-webhook-cfg.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: openebs-validation-webhook-cfg
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
    caBundle: xxx

    Decode the certificate to Base64.

    1

    echo xxx | base64 -d > openebs.crt

    View certificate details.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

    openssl x509 -noout -text -in openebs.crt

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = admission-server-svc-ca
        Validity
            Not Before: Sep  8 07:17:40 2021 GMT
            Not After : Sep  6 07:17:40 2031 GMT
        Subject: CN = admission-server-svc-ca

    In fact, the certificate in openebs-validation-webhook-cfg has not expired, so deleting openebs-validation-webhook-cfg means that no admission control is performed when calling the OpenEBS service, and no data legitimacy is verified. kube-apiserver will not call admission-server-svc.openebs.svc, and no errors will be reported.

  • What exactly is the expired certificate?

    In admission-server-secret, I found two certificates, one for app.crt and one for ca.crt.

    1
2
3
4
5
6
7

    kubectl -n openebs get secret admission-server-secret   -o yaml

apiVersion: v1
data:
app.crt: xxxx
app.pem: xxxx
ca.crt: xxx

    The ca.crt certificate is the same as the one above, a 10-year certificate. And app.crt is a one-year certificate.

    View certificate details.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    openssl x509 -noout -text -in app.crt 

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 389184800153601983 (0x566a983852307bf)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = admission-server-svc-ca
        Validity
            Not Before: Sep  8 07:17:40 2021 GMT
            Not After : Sep  8 07:17:40 2022 GMT

  • How to renew your certificate

    In the latest installation of OpenEBS, the admission-server-secret object is no longer found, nor is the openebs-validation-webhook-cfg. Also, there is no reference to the certificate on the upgraded OpenEBS cluster, which is very strange.

    1
2
3
4

    kubectl -n openebs get all,sa,secret -o yaml|grep admission-server-secret

    name: admission-server-secret
    selfLink: /api/v1/namespaces/openebs/secrets/admission-server-secret

    My suspicion is that the OpenEBS installation from a year ago had admission-server-secret for ValidatingWebhookConfiguration. A recent OpenEBS upgrade left behind the configuration of the previous version, leading to this strange phenomenon.

    Since there is no place to refer to this certificate, there is no need to update it, just backup it and delete it.