We’ve talked about GitOps practices before, and we know that GitOps advocates managing all of your configuration through Git and versioning your environment configuration and infrastructure through declarative code.
In Kubernetes, we know that you can use resource manifest files to manage a cluster’s resource objects, but it’s not a good idea to store Kubernetes Secrets data in a Git repository, which is also very insecure.
Kubernetes Secrets are resource objects used to help us store sensitive information, such as passwords, keys, certificates, OAuth Token, SSH KEY, etc. Administrators can create Secrets objects, and then developers can very easily reference Secrets objects in resource manifest files without having to directly hardcode this sensitive information.
While this may seem convenient, the problem with Secrets is that they simply encode the sensitive information once in base64, and anyone can easily decrypt it to get the original data. So we say that Secrets manifest files can’t be stored directly in a Git repository, but if we had to create them manually every time, it would make our GitOps a lot less fluid.
Bitnami Labs has created an open source tool called Sealed Secrets to solve this problem.
Sealed Secrets consists of two main components.
- One is the Kubernetes Operator within the cluster
- A client tool called kubeseal
kubeseal allows us to encrypt Kubernetes Secrets objects using asymmetric encryption algorithms, and
SealedSecret is a Kubernetes CRD resource object containing an encrypted Secret that only the controller can decrypt, so it is very secure even if SealedSecret is stored in a public code repository.
When we create a SealedSecret resource object in a Kubernetes cluster, the corresponding Operator will read it and generate the corresponding Secret object, which we can then use directly in the Pod. The following is an example of a SealedSecret resource object.
When the Operator decrypts the above object, it generates the Secret object as follows.
Only the Operator can decrypt SealedSecret, and it is generally a better practice to not allow users to read Secret directly. We can prohibit low privilege users from reading Secret by creating RBAC rules, or we can restrict users to read Secret objects from their namespace only. Although SealedSecret is designed not to read them directly, users can still bypass this process and gain access to Secret objects that they are not allowed to see.
The SealedSecret resource provides several ways to prevent this behavior; it is namespace scoped by default, and once a SealedSecret is restricted to a namespace, the object cannot be used in other namespaces.
For example, if a Secret object named foo with the value bar is created under the web namespace, we cannot use this Secret in other namespaces, even if the same Secret is required. although SealedSecret’s controller does not use a separate decryption password for each namespace, it does Although SealedSecret’s controller does not use a separate decryption password for each namespace, it takes namespaces and names into account during encryption, so the effect is similar to each namespace having its own separate decryption key.
Another situation is that we may have a user on the web namespace above who can only view some Secrets but not all, which SealedSecret also allows. When we generate a SealedSecret object for a Secret named foo in the web namespace, a user on the web namespace who only has read access to the Secret object named bar cannot change the name of the Secret to bar in the SealedSecret resource object and use it to view the Secret.
Although these methods can help us prevent Secrets from being abused, they are still a bit tricky to manage. In the default configuration, there is no way to define a generic Secret to be used in multiple namespaces. And it’s likely that we have a very small team and the Kubernetes cluster is only accessed and maintained by operations staff, so we may not need this RBAC approach to permission control.
If we want to define SealedSecrets objects across namespaces, we can use scopes to achieve this functionality.
There are 3 scopes we can use to create SealedSecrets.
- strict (default): In this case, we need to consider the name and namespace of the Secret to encrypt it, and once we create the corresponding SealedSecret, we cannot change its name and namespace.
- namespace-wide: This scope allows us to rename the SealedSecret object within the namespace of the encrypted Secret.
- cluster-wide: This scope allows us to freely rename SealedSecret within the namespace of the encrypted Secret, allowing us to move the Secret to any namespace at will and name it at will.
We can use the
--scope argument when using kubeseal to specify the scope.
You can also use annotation in Secret to use scopes before passing the configuration to kubeseal: *
sealedsecrets.bitnami.com/namespace-wide: "true" means
If no annotation is specified, then kubeseal uses the strict scope by default, and if two annotations are set, then the scope with the larger scope takes precedence.
We mentioned earlier that SealedSecrets consists of a client-side kubeseal and a cluster-side Operator. Let’s start by installing the client-side kubeseal tool.
Select the latest release from the GitHub Release page and download the latest binaries at.
Then install the Operator controller on the Kubernetes cluster side at
The controller will be installed under the kube-system namespace by default:
Once the controller is running successfully, the installation is proven to be successful. Next we can use SealedSecret to encrypt our Secret object.
In order to create the SealedSecret object, we first need to create a Secret file.
Note that we used the
-dry-run argument above, so it’s not really created, but the Secret object is just a base64-encoded string, so it’s not suitable for direct placement in the source repository.
Next, use the kubeseal tool to encrypt the object.
We can see that the Secret information has been encrypted, so we can now store it in the source repository with confidence.
Next, let’s use this Secret object in a sample Pod to see if we can get its data correctly.
Create the SealedSecret object directly from above.
Once created, we can also find a Secret object named mysecret under the default namespace.
Use the resource list shown below to create a Pod for testing.
After a successful run we can look at the password data in the container to verify that it is correct:
You can see that the password message we defined
This is a secret is printed correctly.
Since there is no scope information specified in our Secret above, this Secret can only be used under the specified default (default) namespace. For example, here we modify the above SealedSecret object to a namespace named test.
Once created, we check to see if the corresponding Secret object has been generated.
We can see that the corresponding Secret object is not generated, check whether the SealedSecret object is created successfully.
We can see that the SealedSecret object exists, but the corresponding Secret is not generated, so let’s check the controller’s logs.
You can see the error message
no key could decrypt secret (secret), this is because we use the default strict scope, so changing the namespace of SealedSecret is not effective.
Changing the Secret name
Let’s also test the effect of changing the name of the Secret under the Secret namespace.
Once created, check to see if the Secret generates a new name.
You can see that no modified Secret object named anothersecret has been generated either, and you can also view the Controller’s log message.
The error is basically the same as the one above for modifying the namespace, which is what we expected because we are creating a Secret in strict mode, so we can’t cross namespaces or modify names. We can create a cluster-wide Secret object to see what the effect is.
Creating Cluster-Wide SealedSecrets
Generate a list of resources for the Secret object using the command shown below.
Then use the
cluster-wide scope to encrypt the Secret object under the test namespace.
We can see that the corresponding Secret object has been created successfully.
Now let’s rename the Secret and see if it still works after re-creation.
Once created, we can see that the Secret behind the rename is also automatically generated:
What about modifying the namespace? For example, if we create the above encrypted Secret object under the default namespace.
After successful creation, we can also see that the corresponding Secret object is created under the default namespace.
This is also as expected, since we are creating SealedSecrets with a cluster-wide scope, so we can modify the namespace and name as we wish.
More information on the use of SealedSecrets can be found in the official documentation for more information.