As a small company, our infrastructure is not complete enough, the project manager informed us in the mid-autumn festival that our system will be on the second-to-last stage environment and production environment in the near future, so from the consideration of the deployment efficiency of the operation and maintenance staff, we urgently developed a one-click installation script generation tool, so that the operation and maintenance staff can use the tool combined with the actual target environment to generate a one-click installation script, the principle of this tool is very simple, as shown in the following diagram.
As you can see from the image above, our tool is based on a template to customize the final configuration and installation script.
templates/confdirectory for the service configuration.
templates/manifestsdirectory holds the k8s yaml scripts for the service
custom/configurefile stores customized configuration data for the service configuration under templates/conf
custom/manifestsfile stores customized configuration data for the k8s yaml under templates/manifests.
templates/install.shis the installation script.
The two files in the custom directory that store the customized configuration are closely related to the target environment .
When it comes to templates, the first thing that comes to Gopher’s mind is the Go text/template technique, which uses template syntax to write template configuration files in the templates directory above. However, text/template requires us to identify all the variables that need to be customized in advance, which is a bit too much and not flexible enough.
So what other technical solutions can we use? I finally chose the yaml file merging (including overwriting and appending) scheme , which is schematically shown below.
This example contains both override and (append) merge cases. Let’s first look at override.
- The configuration in custom/manifests.yml overrides the configuration in templates/manifests/*.yaml
templates/manifests/a.yml as an example. The default value of
metadata.name in this template is default, but the operations staff customizes (customizing) the
custom/manifests.yml file according to the target environment. In this file,
a.yml file name as the value of the key, and then the full path of the configuration items to be overridden configured into the file (here the full path is
The change to
namespace name in the custom/manifests.yml file,
foo, will override
default in the original template, which will be reflected in the final
- Append the configuration from
For the configuration that is not in the original template file but added in custom, it will be appended to the final generated configuration file, take b.yml as an example. The contents of b.yml in the original template directory are as follows.
Here there are only three sub-configurations under log: type, level and compress.
And the operation and maintenance staff in
custom/manifests.yml for log added several other kinds of configuration, such as access_log, error_log, etc..
Thus, except for level and compress which will overwrite the values in the original template, the rest of the added configuration will be appended to the generated
xx_install/manifests/b.yml will be reflected.
OK! The solution is determined, so how to implement the merging of yaml files?
The Go community’s yaml package is best known as https://github.com/go-yaml/yaml. This package implements the YAML 1.2 specification, which facilitates marshal and unmarshal between Yaml and go struct.
However, the interface provided by the yaml package is rather rudimentary, and to merge yaml files, you need to do more extra work yourself, which we don’t have time for anymore. So is there a ready-to-use tool? The answer is yes, and it’s the Go community’s famous viper!
viper is an open source Go application configuration framework developed by Steve Francia, author of gohugo and former product manager of the Go language project team. viper not only supports command line arguments to pass in configuration, but also supports getting configuration from various types of configuration files, environment variables, remote configuration systems (etc.) and so on. In addition , viper also supports configuration file merge and write operations to the configuration file .
Can we just use viper’s Merge series of operations? The answer is no! Why? Because it has to do with our design above. We put all the environment-related configuration into the
custom/manifests.yml file, which will cause the configuration data in
custom/manifests.yml to appear in each of the final
templates/xx.yml configuration files once we merge.
Then we’ll implement a set of merge (overwrite and append) operations ourselves!
Let’s start by looking at the main function that drives merge.
We see that the main package uses the standard library flag package to create two command line arguments -s and -d, which represent the source path where
templates/custom is stored and the target path where the generated files are stored, respectively. After entering the main function, we first create
conf directories under the target path to store the relevant configuration files respectively (in this case, no files are generated under the conf directory).
Then the main function calls
mergeManifestsFiles to merge the yml files in
templates/manifests under the source path with
We see that
mergeManifestsFiles iterates through the template files and calls mergeConfig, the function that actually does the yml file merge, once for each file.
We see that the
mergeConfig function creates two viper instances (
viper.New()) for the file under
templates/manifests and the manifests.yml file under custom and loads the configuration data for each of them. Then we iterate through the keys in manifests.yml under custom and set the values of the matching configuration items to the viper instance representing the file under
templates/manifests, and finally we write the data of the merged viper instance to the target file.
Compile and run the generator tool.
With the default command line arguments, the files are generated in the k8s-install path, so let’s take a look at the generated files.
We see that the results of merge are consistent with our expectations (it does not matter if the order of the fields is inconsistent, this is related to the use of go map when storing key-value inside viper, the traversal order of go map is random).
But careful people may find a problem: that is, the original apiVersion in a.yml in the results file became lowercase apiversion, which will cause a.yml in the submission to k8s when the verification failed!
Why is this so? The official explanation given by viper is as follows.
Viper merges configuration from various sources, many of which are either case insensitive or uses different casing than the rest of the sources (eg. env vars). In order to provide the best experience when using multiple sources, the decision has been made to make all keys case insensitive.
There has been several attempts to implement case sensitivity, but unfortunately it’s not that trivial. We might take a stab at implementing it in Viper v2, but despite the initial noise, it does not seem to be requested that much.
Well, since it’s official that it might be supported in v2, but v2 is out of reach, let’s use the fork version of viper to solve the problem! The developer lnashier has forked a viper code and fixed the problem due to this case problem, although it is rather old (and may not be comprehensive), but it can meet our requirements! Let’s try replacing spf13/viper with lnashier/viper, and rebuild and execute generator.
We see that after replacing it with
lnashier/viper, the key apiVersion in
a.yml is not changed to lowercase anymore.
The tool basically works now. But is this tool problem-free? Unfortunately not! The generator generates the wrong file when it faces the following two forms of configuration files.
These two problems are a bit tricky, and lnashier/viper can’t solve them. I can only fork lnashier/viper to bigwhite/viper to solve this problem by myself, and the configuration form like d.yml is very specific and not universal, so bigwhite/viper is not universal, so I won’t go into details here, you can read the code (commit diff) by yourself if you are interested to see how to solve the above problem.
The code covered in this article can be downloaded from here.
Some other stuff.
kustomize is an official k8s tool that allows you to customize YAML files for multiple purposes based on k8s resource template YAML files and in combination with kustomization.yaml, the original YAML will not be changed in any way.
However, it targets only k8s-related yaml files and may not be able to do anything for our business service configuration.
CUE is a powerful declarative configuration language that has gained popularity in the past two years, created by former Go core team member Marcel van Lohuizen, who co-founded the Borg Configuration Language (BCL) - the language used to deploy all applications at Google. cUE is the result of Google’s years of experience writing configuration languages designed to improve the developer experience while avoiding some of the pitfalls. It is a superset of JSON with additional features, and is heavily used by Solomon Hykes, the father of Docker, for his new startup dagger, and by kubevela, the enterprise cloud native application management platform promoted by Ali.
An in-depth study on how to use CUE instead of the scheme I described above is yet to follow.