The Go language added support for modular programming and a built-in module-based dependency management tool in version 1.11, released in August 2018. modules in the Go language are collections of packages in a file tree, where the go.mod file contained in the module root directory defines the module’s import path, the Go language version, and other dependency requirements for the module. Each module’s dependency requirements are listed as a separate module path and the corresponding module version is specified, and only modules that meet all dependency requirements can be successfully built.

With the modular programming capabilities that come with Go, there is no need to put Go code into $GOPATH/src, which is the old GOPATH mode; in fact, we can create Go projects and initialize Go modules using go mod init in any directory outside of $GOPATH/src.

Note: For compatibility, Go commands can still be run in the old GOPATH mode in Go 1.11 and 1.12. Starting with Go 1.13, module mode (GO111MODULE=on) will be the default mode.

The history of GOPATH

Before the Go language supported modular programming, our Go projects generally needed to use the GOPATH pattern, which means that the Go code needed to be placed in $GOPATH/src. A typical GOPATH directory structure contains the following three subfolders.

1
2
3
4
5
6
7

GOPATH
├── bin              // binaries
├── pkg              // cache
|── src              // go source code
    ├── github.com
    ├── rsc.io
    ...

Using the go get command to get the dependencies will also automatically download them to $GOPATH/src.

1

$ go get rsc.io/quote # will be doloaded to $GOPATH/src/rsc.io/quote

However, there is a problem with GOPATH mode. When we use go get to get a dependency without specifying a version, the dependency code downloaded by default will be the latest version, and if project A and B depend on two incompatible versions of project C, the GOPATH path with only one version of C will not be able to satisfy both project A and B’s dependency needs. This is a tricky flaw, and in fact, the GOPATH pattern has not been officially recommended since Go 1.13.

In addition, as the Go language becomes more and more popular, dependency packages become more and more abundant, and the issue of dependency management becomes a focus for developers.

The GOPATH pattern has given rise to many version management tools, but the basic idea is to maintain a separate copy of the dependencies for each project.

  1. the vendor feature was first officially introduced in Go 1.5: a vendor/ directory is created for each Go project to store copies of the project’s required version dependencies
  2. The community has developed various version management tools based on the vendor feature. Some popular ones are govendor, and godep, which was officially recognized before.

Go dependency management tools are abundant, but there are incompatibilities between different versions of tools, and there are also learning costs for all kinds of tools. So the Go community proposed the vgo scheme, and with the gradual development and maturity of vgo, Go 1.11 released the Go module function based on this scheme, and integrated it into the official Go language tools.

Using the built-in modularity features of the Go language, there are six environment variables that developers need to care about, which can be listed using the go env command.

1
2
3
4
5
6

GO111MODULE="auto"
GOPROXY="https://goproxy.io,direct"
GONOPROXY=""
GOSUMDB="sum.golang.org"
GONOSUMDB=""
GOPRIVATE=""

1. GO111MODULE

This environment variable controls whether the use of the Go module feature is enabled. The optional values and descriptions are shown in the table below.

value description
auto Projects in $GOPATH/src continue to use GOPATH mode; projects outside of $GOPATH/src automatically use Go modularity
on Enable Go modularity mode, recommended
off Use GOPATH mode, disable Go modularity mode, not recommended

All you need to do to enable Go modular mode is to set the GO111MODULE environment variable to on.

1

$ go env -w GO111MODULE=on

2. GOPROXY

This environment variable is used to set a proxy for Go module dependency downloads so that Go can quickly pull module dependencies directly through the proxy site when pulling them. The default value is: https://proxy.golang.org,direct/.

Due to the Chinese government’s network blocking, you can’t access go’s official services directly in China. So you need to set the proxy address to enable Go modular mode, and the common mirror proxy addresses in China are as follows.

  • https://goproxy.io
  • https://mirrors.aliyun.com/goproxy/
  • https://goproxy.cn

The value of the GOPROXY environment variable is a comma-separated list of Go module proxies, allowing multiple module proxies to be set; if you want to turn off proxies, you can set it to off.

The default value of direct followed by a comma is used to tell Go that it will download directly from the source address of the dependency. For example, when a Go mirror proxy in the list of values returns 404 or 410, Go automatically tries the next one in the list, and if the next value is direct goes to the source address to download the dependent module.

Set up the Go module proxy.

1

$ go env -w GOPROXY=https://mirrors.aliyun.com/goproxy/,direct

3. GOSUMDB

The full name of this environment variable is GO checkSUM DataBase, and as it literally says, it is used to ensure that the module version data is not tampered with when pulling module dependencies, and to abort the pull immediately and report an error if inconsistencies are found. The default value of the GOSUMDB environment variable is: sum.golang.org. If it is set to off, Go commands will be prevented from verifying module dependencies when pulling them later, which is not recommended.

It is also inaccessible in China. The good thing is that after setting the proxy address GOPROXY, GOSUMDB can also be accessed

4. GONOPROXY & GONOSUMDB & GOPRIVATE

These three environment variables are mainly used when a Go project depends on a private module, for example, some dependencies exist in a private git repository, and the mirror proxy set directly with GOPROXY or the check site set with GOSUMDB will not be able to access the corresponding private repository. In this case, you need to set these dependency modules to pull checksums directly without going through the mirror proxy site. In fact, for private dependency modules, the best practice is to set the GOPRIVATE environment variable directly, whose value will be the default for GONOPROXY and GONOSUMDB.

Their value is a module path prefix separated by an English comma ,, i.e. more than one can be set, e.g.

1

$ go env -w GOPRIVATE="*.my.com,github.com/eabc/def"

This setting means that module dependencies prefixed with *.my.com and github.com/eabc/def will be considered private and will not be pulled directly from the mirror proxy site.

Go Modular Manipulation Commands

  1. go mod init creates a new module and initializes the go.mod file that describes it

  2. go build & go test and other package build commands add new dependencies to go.mod as needed

  3. go list -m all prints all dependencies of the current module

  4. go get to change the version of the required dependency (or add a new one)

  5. go mod tidy Remove unused dependencies

  6. go mod vendor Copy the correct version of the module dependency from the module to the vendor directory of the project

  7. Initialize the module

    We can create Go projects and initialize Go modules using go mod init in any directory other than $GOPATH/src.

    1
2

    $ go mod init example.com/greetings
go: creating new go.mod: module example.com/greetings

    The parameter example.com/greetings is not only the module identifier, but also the import path of the module, which will be used as a common prefix when other Go projects refer to a package under this module, plus the relative path of the package to the root of the module.

go.mod file

The above command will generate a go.mod file in the current directory after successful execution.

1
2
3

module example.com/greetings

go 1.12

In fact, in Go modularity mode, the dependency information is automatically recorded in the go.mod file after the module dependency is fetched using the go get command.

1

$ go get -u rsc.io/quote

The go get command downloads the latest dependency version by default, but you can also specify the tag or commit id for versioning with @, for example.

1

$ go get -u rsc.io/quote@v1.5.2

Pulling module dependencies with the go get command will cache the results in the $GOPATH/pkg/mod and $GOPATH/pkg/sumdb directories, where the module dependencies are stored in the $GOPATH/pkg/mod directory in the github.com/foo/bar format.

After pulling the module dependencies, the go.mod file will look like this.

1
2
3
4
5

module example.com/greetings

go 1.12

require rsc.io/quote v1.5.2 // indirect

Note: The indirect comment indicates that the module is an indirect dependency, meaning that no explicit reference to the module is found in the import statement in the current application. Use the go get command to pull module dependencies directly, rather than using go build to automatically pull module dependencies based on go.mod.

Syntax keywords that can be used in the go.mod file and their meanings.

  • module defines the module path of the current project
  • go Identifies the Go language version of the current module
  • require indicates the version of the module that depends on it
  • exclude to exclude a specific module version from use, if a version of the module dependency has a serious bug, you need to explicitly exclude a version, e.g. exclude github.com/google/uuid v1.1.0 means don’t use v1.1.0 version
  • replace replaces the module dependency declared in require, using another module dependency and its version

Usage scenarios for the replace keyword.

  • Usage scenario 1: replacing the dependency package of require

    The go.mod file for the current project is as follows.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10

    module example.com/greetings

go 1.12

require (
    github.com/google/uuid v1.1.1
    rsc.io/quote v1.5.2 // indirect
)

exclude github.com/google/uuid v1.1.0

    Execute the command go list -m all to list all the dependencies of the current module.

    1
2
3

    example.com/greetings
github.com/google/uuid v1.1.1
rsc.io/quote v1.5.2

    Add this line of configuration under the go.mod file.

    1

    replace github.com/google/uuid v1.1.1 => github.com/google/uuid v1.1.0

    Execute the command go list -m all again to list all the dependencies of the current module.

    1
2
3

    example.com/greetings
github.com/google/uuid v1.1.1 => github.com/google/uuid v1.1.0
rsc.io/quote v1.5.2

    Found that the module dependency that finally takes effect has changed from github.com/google/uuid v1.1.1 to github.com/google/uuid v1.1.0. Generally speaking, this scenario is not used much, and it works the same as modifying require directly, with the following prerequisites.

    • the currently referenced module dependency is valid
    • The package name and version on the left side of the replace command must be the corresponding package name and version contained in require.

  • Usage scenario 2: Replacing Unable-to-Download Packages

    Due to network limitations in China, some module dependencies cannot be downloaded, such as all of the dependencies under golang.org. The good thing is that these dependencies have mirrors on GitHub, so you can use the mirrors on GitHub to replace the dependencies under the golang.org import path.

    For example, if you use the golang.org/x/net package in your project.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    module example.com/greetings

go 1.12

require (
    github.com/google/uuid v1.1.1
    golang.org/x/net v0.3.2
    rsc.io/quote v1.5.2 // indirect
)

replace golang.org/x/net v0.3.2 => github.com/golang/net v0.3.2

    This way the project will download the golang.org/x/net package from GitHub when it is compiled, and the import path golang.org/x/net/xxx will not need to be changed in the source code.

  • Usage scenario 3: Debugging Dependency Packages

    When debugging a dependency package you can use replace to modify the dependency as follows.

    1
2
3

    replace (
    github.com/google/uuid v1.1.1 => ../uuid
)

    Use local uuid to replace the dependency package, at this time, we can modify . /uuid directory to debug. Besides using relative paths, you can also use absolute paths, or even use your own forked repositories.

  • Usage scenario 4: Prohibition of being dependent

The last scenario where the replace directive is used is when you don’t want your module to be directly referenced, such as in the go.mod file of a k8s project, where the require section has a large number of v0.0.0 dependencies.

1
2
3
4
5
6
7
8
9

module k8s.io/kubernetes

require (
  ...
  k8s.io/api v0.0.0
  k8s.io/apiextensions-apiserver v0.0.0
  k8s.io/apimachinery v0.0.0
  ...
)

Since none of the above dependencies exist for v0.0.0, other projects that depend directly on k8s.io/kubernetes will not be able to pull it successfully because the version cannot be found.

Because the k8s core repository does not want to be used directly as a module dependency, other projects can use other subcomponents of the k8s project. k8s hides the dependency version number from the outside, and its real dependencies are specified by replace.

1
2
3
4
5
6

replace (
  k8s.io/api => ./staging/src/k8s.io/api
  k8s.io/apiextensions-apiserver => ./staging/src/k8s.io/apiextensions-apiserver
  k8s.io/apimachinery => ./staging/src/k8s.io/apimachinery
  ...
)

go.sum file

After we build and execute commands like go build or go test we will also find a go.sum file generated in the root of the project. Its main purpose is to check for downloaded module dependencies. In practice, module dependencies can be tampered with during download, and dependencies cached locally can also be tampered with, so a single go.mod file does not guarantee a consistent build. The go.sum file was introduced on top of go.mod to record the hash value of each dependency package. When building a Go project, if the local dependency package hash checksum does not match the one recorded in the go.sum file, the build will be rejected.

A typical go.mod file is shown below.

1
2

github.com/google/uuid v1.0.0 h1:b4Gk+7WdP/d3HZH8EJsZpvV7EtDOgaZLtnaNGIu1adA=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=

Normally, each dependency version will contain two records, the first being the hash of the dependency version as a whole, and the second being the hash of the go.mod file in the dependency version only, or the first record if the dependency version does not have a go.mod file.

So how is go.sum generated?

Execute the go get command in the root of the Go project and it will update the go.mod and go.sum files simultaneously. go.mod records the dependency names and their versions, e.g.

1

github.com/google/uuid v1.1.1

The go.sum file records the hash of the dependency package (along with the hash of go.mod in the dependency package). Before updating go.sum, the Go command also queries the server indicated by the GOSUMDB environment variable to get an authoritative hash of the dependency version after downloading the dependency to ensure that the downloaded dependency is real and reliable. If the dependency package version hash computed by the Go command does not match the hash given by the GOSUMDB server, the Go command will refuse to execute further and will not update the go.sum file.

Use and update of Go modules

We first initialize the Go project with the go mod init command and set the module import path. Then, as we write the code, we write the imported packages, etc. Commands like go build or go mod tidy will automatically download the imported packages and update the go.mod and go.sum files, and then use a version management tool like Git to commit and manage the project source code and the updated go.mod and go.sum files. This ensures that you get consistent dependencies for each build.

  1. minor version update

    Using the go get -u command will update a module to depend on the latest minor version, for example, it will update 1.0.0 to 1.0.1 or 1.1.0 something like that; the go get -u=patch command will get the latest patch update, for example, it will update 1.0.0 to 1.0.1 instead of 1.1.0 version.

    If our Go project is using dependency package version 1.0.0 and the module dependency has just been updated to version 1.0.1, any of the following commands will update our to module dependency version 1.0.1.

    1
2
3

    go get -u
go get -u=patch
go get github.com/example/testmod@v1.0.1

  2. Major version update

    In general, major versions are completely different from minor versions, and major versions can break backwards compatibility. Thus, from the perspective of Go modules, a major version is a completely different package. Two incompatible versions of a library are essentially two different libraries. When you encounter a major version update, you can still use tag to update from 1.0.0 to 2.0.0, but be sure to pay attention to the backwards compatibility issue.