This article introduces cert-mangager, which is somewhat similar to certbot, but the difference is that it works in Kubernetes.

cert-manager is an automated certificate management tool that automates the issuance and management of digital certificates from various sources and for various purposes in a Kubernetes cluster. It will ensure that certificates are valid and automatically renew them at the appropriate time.

Note: The object managed by cert-manager is “certificate”. If you only need to use asymmetric encrypted public-private key pairs for JWT signing, data encryption and decryption, consider using secrets management tool Vault directly.

1. Deployment

reference: https://cert-manager.io/docs/installation/helm/

Officially, a variety of deployment methods are available, and the method of installation using helm3 is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Check the version number
helm search repo jetstack/cert-manager -l | head
# Download and unzip the chart for the purpose of gitops versioning
helm pull jetstack/cert-manager --untar --version 1.8.2
helm install \
  cert-manager ./cert-manager \
  --namespace cert-manager \
  --create-namespace \
  # This will cause all CRDs to be deleted when uninstalling with helm, which may result in the total loss of all CRDs resources! Be extra careful with
  --set installCRDs=true

2. Creating Issuers

cert-manager supports a variety of issuers, and you can even create your own Issuer through its standard API.

But in general, there are only three types of Issuer.

  • Public Trusted Certificate signed by authoritative CA: this type of certificate will be trusted by browsers, applets and other third-party applications/service providers
  • Locally signed certificates: i.e. digital certificates signed by local CA certificates
  • Self-signed certificate: i.e. the certificate is signed by the private key of the certificate itself

The following describes how to apply for a public certificate and a locally signed certificate.

1. Create public network trusted certificate through authority

A public trusted certificate created by an authority can be directly applied to the border gateway to provide TLS encrypted access services to public network users, such as various HTTPS sites and APIs. This is the most widely demanded type of digital certificate service.

cert-manager supports two ways to apply for public network trusted certificates.

Here we mainly introduce the application of public network certificate using ACMEv2 protocol. The authorities that support the application of certificate using this open protocol are

Here is also an introduction to the hierarchy of certificates by fee-based certificate services and how they should be selected.

  • Domain Validated (DV) certificates
    • Only validates domain ownership, with the fewest validation steps, lowest price, and takes only a few minutes to issue.
    • The advantage is that they are easy to issue and are great for automation.
    • The free certificates provided by various cloud vendors (AWS/GCP/Cloudflare, and Vercel/Github’s site services) for their own services are DV certificates, and Let’s Encrypt’s certificates are of this type.
      • Obviously these certificates are all very easy to issue and only verify domain ownership.
      • But the DV certificates provided by AWS/GCP/Cloudflare/Vercel/Github all work only on their cloud services and do not provide private key functionality!
  • Organization Validated (OV) Certificates
    • It is the first choice for enterprise SSL certificate and ensures the authenticity of enterprise SSL certificate through enterprise certification.
    • In addition to domain ownership, CA authorities also review the authenticity of the organization and enterprise, including registration status, contact information, malware, etc.
    • If you want to be more compliant, you probably have to use at least the OV level of certification as well.
  • Extended Validation (EV) certificate
    • The most stringent form of certification where CA organizations review in-depth information on all aspects of the organization and business.
    • Considered suitable for organizations or businesses such as large corporations, financial institutions, etc.
    • And it only supports issuing single-domain and multi-domain certificates, and does not support issuing pan-domain certificates. The highest security.

ACME supports two types of domain name authentication methods, HTTP01 and DNS01, among which DNS01 is the easiest method.

The following is an example of how to apply a Let’s Encrypt certificate with AWS Route53. (Please refer to the official documentation for the configuration of other DNS providers)

1.1 AWS IAM Authorization

First you need to create an OIDC provider for the EKS cluster.

The cert-manager needs access to query and update Route53 records, so you need to create an IAM Policy using the following configuration, which can be named <ClusterName>CertManagerRoute53Access (note the replacement of <ClusterName>).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "route53:GetChange",
      "Resource": "arn:aws:route53:::change/*"
    },
    {
      "Effect": "Allow",
      "Action": [
 "route53:ChangeResourceRecordSets",
 "route53:ListResourceRecordSets"
      ],
      "Resource": "arn:aws:route53:::hostedzone/*"
    },
    {
      "Effect": "Allow",
      "Action": "route53:ListHostedZonesByName",
      "Resource": "*"
    }
  ]
}

For example, use awscli to create this policy.

1
2
3

aws iam create-policy \
  --policy-name XxxCertManagerRoute53Access \
  --policy-document file://cert-manager-route53-access.json

The above configuration then creates an IAM Role and automatically adds a trust relationship to the EKS cluster where the cert-manager is located.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

export CLUSTER_NAME="xxx"
export AWS_ACCOUNT_ID="112233445566"

# Use eksctl to automatically create a corresponding role and add a trust relationship
# eksctl needs to be installed first
eksctl create iamserviceaccount \
  --cluster "${CLUSTER_NAME}" --name cert-manager --namespace cert-manager \
  --role-name "${CLUSTER_NAME}-cert-manager-route53-role" \
  --attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/<ClusterName>CertManagerRoute53Access" \
  --role-only \
  --approve

After that, you need to add an annotation to the ServiceAccount of the cert-manager to bind the IAM Role you just created above.

First create the helm values file cert-manager-values.yaml as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Changing this to false will also cause all CRDs and related resources of the cert-manager to be deleted!
installCRDs: true

serviceAccount:
  annotations:
    # Note the changes to ${AWS_ACCOUNT_ID} and ${CLUSTER_NAME} here
    eks.amazonaws.com/role-arn: >-
                        arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-cert-manager-route53-role

securityContext:
  enabled: true
  # According to the official documentation, this also has to be modified to allow the cert-manager to read the ServiceAccount Token and thus obtain authorization
  fsGroup: 1001

Then redeploy the cert-manager:

1

helm upgrade -i cert-manager ./cert-manager -n cert-manager -f cert-manager-values.yaml

This completes the authorization.

1.2 Creating an ACME Issuer

Create an Iusser in the xxx namespace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
  namespace: xxx
spec:
  acme:
    # Email address for receiving domain expiration alerts
    email: user@example.com
    # ACME servers, such as let's encrypt, Digicert, etc.
    # Test URLs for let's encrypt, which can be used to test the correctness of the configuration
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # The official URL of let's encrypt, with a rate limit
    # server: https://acme-v02.api.letsencrypt.org/directory

    # The name of the secret used to store the private key of the ACME account, which is automatically generated when Issuer is created.
    privateKeySecretRef:
      name: letsencrypt-staging
    
    # DNS authentication settings
    solvers:
    - selector:
        # In the case of multiple solvers, the priority is determined based on the selector of each solver, and the appropriate solver is selected to handle the certificate request event
        # In the case of dnsZones, for example, the longer the Zone, the higher the priority
        # For example, when applying for a certificate for www.sys.exapmle.com, sys.example.org has a higher priority than example.org
        dnsZones:
        - "example.org"
      dns01:
        # Use route53 for authentication
        route53:
          region: us-east-1
          # cert-manager already has an IAM Role bound via ServiceAccount
          # No additional IAM authorization-related information is needed here!

1.3 Creating a certificate through ACME and troubleshooting problems

https://cert-manager.io/docs/usage/certificate/#creating-certificate-resources

Create the certificate using the following configuration and save the certificate to the specified Secret.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: xxx
spec:
  # Secret names are always required.
  # The Istio Gateway/Ingress/Gateway APIs can all add TLS encryption by directly referencing this secret.
  secretName: tls-example.com

  # secretTemplate is optional. If set, these annotations and labels will be
  # copied to the Secret named tls-example.com. These labels and annotations will
  # be re-reconciled if the Certificate's secretTemplate changes. secretTemplate
  # is also enforced, so relevant label and annotation changes on the Secret by a
  # third party will be overwriten by cert-manager to match the secretTemplate.
  secretTemplate:
    annotations:
      my-secret-annotation-1: "foo"
      my-secret-annotation-2: "bar"
    labels:
      my-secret-label: foo

  duration: 2160h # 90d
  renewBefore: 360h # 15d
  # https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.CertificatePrivateKey
  privateKey:
    algorithm: ECDSA  # RSA/ECDSA/Ed25519, with RSA being the most widely used and Ed25519 being considered the most secure
    encoding: PKCS1  # For TLS encryption, the PKCS1 format is usually used
    size: 256  # RSA defaults to 2048, ECDSA defaults to 256, and Ed25519 does not use this attribute!
    rotationPolicy: Always  # Always recreate a new private key when renewing
  # The use of the common name field has been deprecated since 2000 and is
  # discouraged from being used.
  commonName: example.com
  # At least one of a DNS Name, URI, or IP address is required.
  dnsNames:
    - example.com
    - '*.example.com'
  isCA: false
  usages:
    - server auth
    - client auth
  # uris:  # If you want to add a URI to the subjectAltNames of the certificate, add it here
  #   - spiffe://cluster.local/ns/sandbox/sa/example
  # ipAddresses:  # 如If you want to add the ip address to the subjectAltNames of the certificate, add it here
  #   - 192.168.0.5
  subject:
    # Additional information on certificates
    # Field Index: https://cert-manager.io/docs/reference/api-docs/#cert-manager.io/v1.X509Subject
    organizations:
      - xxx
  # Issuer references are always required.
  issuerRef:
    name: letsencrypt-prod
    # We can reference ClusterIssuers by changing the kind here.
    # The default value is Issuer (i.e. a locally namespaced Issuer)
    kind: Issuer
    # This is optional since cert-manager will default to this value however
    # if you are using an external issuer, change this to that issuer group.
    group: cert-manager.io

After deploying the Certificate, describe it to see the current progress.

1
2
3
4
5
6
7

Events: 
  Type    Reason     Age   From    Message 
  ----    ------     ----  ----    ------- 
  Normal  Issuing    117s  cert-manager-certificates-trigger   Issuing certificate as Secret does not exist      
  Normal  Generated  116s  cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "example.com-f044j"     
  Normal  Requested  116s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "example.com-unv3d"   
  Normal  Issuing    20s   cert-manager-certificates-issuing   The certificate has been successfully issued

If you find that the certificate has not been ready for a long time, you can refer to Official Document - Troubleshooting Issuing ACME Certificates and conduct a layer-by-layer check according to the certificate application process.

  • First, cert-manager finds that the Secret described in the Certificate does not exist, and starts the certificate application process.
  • Firstly, generate the private key and store it in a temporary Secret
  • Then generate the CSR certificate request file with the private key and other information in the Certificate resource
    • This is also a CRD resource, which can be viewed by kubectl get csr -n xxx.
  • Then submit the CSR file to the ACME server and request the certificate issued by the authority.
    • This corresponds to the CRD resource kubectl get order.
  • For the above ACME certificate application process, Order will actually generate a DNS1 Challenge resource
    • You can check this resource through kubectl get challenge.
  • After the challenge is verified, it will go backward layer by layer, and the Order CSR status in front will become valid immediately.
  • Finally, the certificate is issued successfully, the Certificate status becomes Ready, and all Order CSR challenge resources are automatically cleaned up.

1.4 Creating a certificate via csi-driver

https://cert-manager.io/docs/projects/csi-driver/

Certificates created directly using the Certificate resource are stored in Kubernetes Secrets and are not considered secure enough. The cert-manager csi-driver avoids this pitfall, specifically, it improves security by doing the following.

  • Ensuring that the private key is stored only on the corresponding node and mounted to the corresponding Pod, completely preventing the private key from being transmitted over the network.
  • Each copy of the application uses its own generated private key, and ensures that the certificate and private key always exist during the life of the Pod.
  • Automatic certificate renewal
  • When a copy is deleted, the certificate is destroyed.

In a nutshell, csi-driver is mainly used to improve security, you can read the documentation yourself if you need, so I won’t introduce more.

2. Certificate issuance through private CA

Private CA is a kind of CA certificate generated by enterprises themselves, which is usually used by enterprises to build their own PKI infrastructure.

In the application scenario of TLS protocol, the certificate issued by Private CA is only suitable for internal use in the enterprise and must be installed on the client to access the Web API or site encrypted by its signed digital certificate normally. Note: Private CA-signed digital certificates are not trusted on the public network!

The Private CA services provided by cert-manager are.

  • Vault: The big name, Vault is a password-as-a-service tool that can be deployed in K8s clusters and provides many password and certificate related features.
    • Open source and free
  • AWS Certificate Manager Private CA: The same CA functionality as Vault, except it is hosted and maintained by AWS.
    • Each Private CA certificate: $400/month
    • One-time fee per issued certificate (only after reading the private key and certificate content) in a ladder, $0.75 per certificate for 0-1000 or less
  • See the documentation for the rest…

This is not used for the time being, so it has not been studied, after there is research to add.

3. cert-manager integrates with istio/ingress and other gateways

The Certificate resource provided by cert-manager will store the generated public and private keys in Secret, and Istio/Ingress both support this format of Secret, so it is quite simple to use.

Take Istio Gateway as an example, just specify the Secret name directly on the Gateway resource.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: example-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 8080
      name: http
      protocol: HTTP
    hosts:
    - product.example.com
    tls:
      httpsRedirect: true # sends 301 redirect for http requests
  - port:
      number: 8443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE # enables HTTPS on this port
      credentialName: tls-example.com # This should match the Certificate secretName
    hosts:
    - product.example.com # This should match a DNS name in the Certificate
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: product
spec:
  hosts:
  - product.example.com
  gateways:
  - example-gateway
  http:
  - route:
    - destination:
        host: product
        port:
          number: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: product
  name: product
  namespace: prod
spec:
  ports:
  - name: grpc
    port: 9090
    protocol: TCP
    targetPort: 9090
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: product
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: product
spec:
  host: product
  # Two subsets are defined
  subsets:
  - labels:
      version: v1
    name: v1
  - labels:
      version: v2
    name: v2
---
# Other configurations such as deployment

Then, with resources such as VirtualService, you can combine Istio with cert-manager.

4. Mount the cert-manager certificate to the custom gateway

Note, do not use subPath mount, according to the official documentation The Secret file is not automatically updated when mounted this way!

Since the certificate is stored in the Secret, it can be mounted directly to Pods as a data volume, as shown in the example below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:latest
    volumeMounts:
    - name: tls-example.com
      mountPath: "/certs/example.com"
      readOnly: true
  volumes:
  - name: tls-example.com
    secret:
      secretName: tls-example.com
      optional: false # default setting; "mysecret" must exist

For nginx, you can simply get a sidecar to monitor and reload nginx when there is a configuration change to achieve automatic certificate updates.

Or you can consider writing a k8s informer to monitor changes to the secret and reload all nginx instances when there are changes.

5. Notes

There are many optimization points for server-side TLS protocol configuration, and some configurations are obvious for performance improvement, so it is recommended to search relevant information online by yourself, and only some relevant information is listed here.

OCSP certificate verification protocol will greatly slow down the response speed of HTTPS protocol

If the client directly requests the OCSP server of CA organization to verify the certificate status through OCSP protocol, this will significantly slow down the response speed of HTTPS protocol, and the performance of the OCSP site of CA organization itself will also be greatly affected.

Solution: The HTTPS server must enable OCSP stapling function, which enables the server to access OCSP in advance to obtain certificate status information and cache it locally, so that when the client accesses using TLS protocol, the cached OCSP information will be sent to the client directly during the handshake phase, thus completing the certificate status verification. Because the OCSP information will carry the signature and validity of CA certificate, it is impossible for the server to forge it, which can also ensure the security.