K8s provides a Secret resource to store and set sensitive information such as API endpoint addresses, various user passwords or tokens, and so on. When you are not using K8s, this information may be set at deployment time through a configuration file or environment variable.
However, Secret is not really secure, as anyone who has looked at Secret with kubectl knows, we can easily see the original text of Secret, as long as we have the relevant permissions, even though its contents are base64 encoded, which is basically equivalent to plaintext.
Therefore, K8s native Secret is very simple, not particularly suitable for direct use in large companies, and is a challenge for RBAC, as many people who should not see plaintext information may be able to see it.
This is especially true now that many companies have adopted the so-called GitOps philosophy, where many things need to be put into a VCS, such as git, and this problem becomes more pronounced because the VCS also needs to set the necessary permissions.
In short, there are probably several places where the Secret contents can be made available to people who should not see them: the
- etcd storage
- via the API server
- Directly viewing the file on the node
Let’s take this example to see how Secret is used in K8s.
Secret definition, username and password are
Pod definition, here we mount Secret as volume to the container.
After the pod is started, we can go to the container to see the contents of the files after Secret is mounted to the container as a volume.
The resources in the API server are stored in etcd, and we can see the contents directly from the file.
As you can see, the contents of the basic yaml are stored in plaintext, and are decoded in base64.
A similar result can be obtained with the following command.
etcd originally stored plaintext data, but it seems that encrypted storage has been supported since 1.7. And direct access to etcd is not that easy physically.
The API server is much simpler, as long as you have access to the API server from any node, you can get the plaintext contents of the secret.
On the node
You can also see the contents of the Secret file on the node.
To find the mount point of the foo volume.
View the contents of the file under this volume.
For the points mentioned above that could compromise the Secret, it is easy to think of several solutions.
- etcd encryption
- Strict permission design for API server
- Strengthen node node user rights management and system security
However, to ensure the absolute security of the Secret, all of the above solutions are necessary, and one of them is indispensable.
The community and public cloud providers have a number of products and solutions that we can refer to.
- shyiko/kubesec: Secure Secret management for Kubernetes (with gpg, Google Cloud KMS and AWS KMS backends)
- bitnami-labs/sealed-secrets: A Kubernetes controller and tool for one-way encrypted Secrets
- Vault by HashiCorp
- Kubernetes External Secrets
kubesec only encrypts/decrypts data in Secret and supports the following key management services or software.
- AWS Key Management Service
- Google Cloud KMS
Bitnami is also a well known company in the K8s space, exporting many technologies and best practices.
SealeSecret encrypts and stores the entire secret resource as a
SealedSecret resource, and decryption can only be done by the controller in the cluster.
SealeSecret provides a kubeseal tool to encrypt the secret resource, which requires a public key, which is obtained from the SealeSecret controller.
However, just from the documentation, the key that SealeSecret controller relies on for encryption and decryption is also saved by a common Secret, isn’t that a problem? It also increases the operation and maintenance cost of SealeSecret controller.
Technically speaking, sops is not necessarily related to K8s, it is just an encrypted file editor supporting YAML/JSON/ENV/INI and other file formats, it supports AWS KMS, GCP KMS, Azure Key Vault, age, and PGP and other services and applications.
If interested, you can see its home page.
Kubernetes External Secrets
Kubernetes External Secrets is an open source software developed by godaddy, a well-known domain name service provider, which can pass confidential information stored in external KMS directly to K8s. Currently supported KSMs include.
- AWS Secrets Manager
- AWS System Manager
- Hashicorp Vault
- Azure Key Vault
- GCP Secret Manager
- Alibaba Cloud KMS Secret Manager
It is implemented through a custom controller and CRD with the following architecture diagram.
Specifically the user needs to create a resource of type ExternalSecret to map the external KMS data to the K8s Secret.
However, there are probably only two advantages to this approach.
- Unified key management, or using existing key assets
- key information does not want to be placed on VCS, etc.
For preventing the leakage of Sercet information, it is not very useful because the plaintext resources can still be seen on the API server/etcd.
Or, what External Secrets really does is to map the keys from external KMS to Secret resources in K8s, which is of little use to ensure the security of data in K8s clusters.
Kamus also provides a way to encrypt the key (a command line tool) and decrypt it only through the controller in K8s. However, the Secret stored in K8s is in an encrypted state, and the user cannot directly access the plaintext content of the Secret, as with External Secrets.
Kamus consists of 3 components, which are
- Encrypt API
- Decrypt API
- Key Management System (KMS)
KMS is a wrapper for external encryption services and currently supports the following services: - AES - AWS KMS - Azure KeyVault - Google Cloud KMS
Kamus encrypts the secret with a service account, and the Pod requests Kamus’ decryption service through the service account to decrypt the secret.
For K8s, decrypting the secret can be done by the init container: define a memory-based emptyDir and use the same volume for the service container and the init container, and store the data under the volume after the init container is decrypted, and then the service container can use the decrypted secret data.
Vault by HashiCorp
HashiCorp is one of the top companies in the cloud/DevOps space.
Vault itself is a KMS-like service for managing confidential data. For K8s’ native secret, support is provided in roughly two ways.
Agent Sidecar Injector
This approach is similar to Kamus above, and also requires two components.
- Mutation webhook: responsible for modifying pod definition and injecting init/sidecar
- agent-sidecar: responsible for fetching and decrypting data and saving it to the specified volume/path
Vault agent sidecar injector not only provides init container to initialize the secret, but also updates the secret periodically via sidecar, which is very close to the native secret implementation.
The application only needs to read the specified file on the filesystem and does not have to worry about how to get the encrypted information from outside.
Here is an example from the official blog.
In this definition,
vault agent into the pod and initializes it with
secrets/helloworld. After the pod is run, a file named
helloworld can be found under
Of course this data is raw data and is not formatted. If you want to specify the format of the output to the file, you can use vault’s template function.
Vault CSI Provider
This section can be found in the Community Solutions section below.
Of course, there is no reason why the community can’t be aware of the problems with native secrets, so the community also has the Kubernetes Secrets Store CSI Driver, a solution for integrating secrets into K8s via the CSI interface to K8s.
The Secrets Store CSI driver (
secrets-store.csi.k8s.io) allows K8s to mount multiple secrets as a volume from an external KMS into the Pod.
To use the Secrets Store CSI Driver, the general process is as follows:
Configuring Volume for Pods
kind: Pod apiVersion: v1 metadata: name: secrets-store-inline spec: containers: - image: k8s.gcr.io/e2e-test-images/busybox:1.29 name: busybox command: - "/bin/sleep" - "10000" volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "my-provider"
Once the Pod is started, you can confirm the decrypted data.
The above summary is based on publicly available information on the Internet, and has not been personally experienced, so some parts may be misunderstood, and you need to confirm the best for a deeper understanding by yourself.
But overall, the community solution is probably the easiest and not very troublesome to deploy, except that it has little to do with the native secret.
Vault solution is also very mature, worthy of attention.