Figuring out the Golang plugin

To count the features in Go that I haven’t used yet, the go plugin introduced in Go 1.8 is one of them. Recently I wanted to design a plugin system for a gateway-type platform, so I thought of the go plugin ^_^.

Go plugin supports compiling Go packages to be distributed separately as shared libraries (.so), and the main application can dynamically load these go plugins compiled as dynamic shared library files at runtime, extracting the symbols of exported variables or functions from them and using them in the main application’s packages. this feature of go plugin provides Go developers with more We can use it to implement a plugin system that supports hot-plugging.

But it’s important to mention the fact that go plugin has been around for more than 4 years now, but it’s still not widely used. The reason for this is that (I guess) on the one hand Go itself supports static compilation, which allows you to compile an application into an executable file that does not depend on the OS runtime library (usually libc) at all, which is an advantage of Go, while supporting the go plugin means that you can only compile the main application dynamically, which is contrary to the advantage of static compilation; and on the other hand, the other reason accounts for a larger proportion, which is that Go plugin itself has too many constraints on the user, which makes many Go developers discouraged.

Only by experiencing it can you appreciate what it’s like . In this article, we’ll take a look at what the go plugin is, what constraints it imposes on users, and whether we should use it or not.

1. Basic usage of go plugin

As of Go 1.16, the official Go documentation clearly states that go plugin only supports Linux, FreeBSD and macOS, which is the first constraint of the go plugin. At the processor level, the go plugin mainly supports amd64 (x86-64), and support for arm series chips does not seem to be explicitly stated (I didn’t see it in the release notes for each Go version, so maybe I missed it), but I used Go 1.16.2 (for arm64) version of Go 1.16.2 (for arm64) to build the plugin package and load the dynamic shared library.so file of the main program were compiled smoothly, and everything ran fine.

The main program loads the .so package and extracts the symbols from the .so file in the same way that a C application loads a dynamic link library and calls the functions in the library at runtime. Let’s look at a visual example below.

The following is the structural layout of this example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// github.com/bigwhite/experiments/tree/master/go-plugin

├── demo1
│   ├── go.mod
│   ├── main.go
│   └── pkg
│       └── pkg1
│           └── pkg1.go
└── demo1-plugins
    ├── Makefile
    ├── go.mod
    └── plugin1.go

Where demo1 represents the main program project and demo1-plugins is the plugins project of the main program. The following is the code of the plugins project.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

// github.com/bigwhite/experiments/tree/master/go-plugin/demo1-plugins/plugin1.go

package main

import (
    "fmt"
    "log"
)

func init() {
    log.Println("plugin1 init")
}

var V int

func F() {
    fmt.Printf("plugin1: public integer variable V=%d\n", V)
}

type foo struct{}

func (foo) M1() {
    fmt.Println("plugin1: invoke foo.M1")
}

var Foo foo

The plugin package is not much different from the normal Go package, except that the plugin package has a constraint: its package name must be main, and we compile the plugin with the following command.

1

$go build -buildmode=plugin -o plugin1.so plugin1.go

If the plugin source code is not placed under the main package, we will encounter the following compiler error when compiling the plugin.

1

-buildmode=plugin requires exactly one main package

Next, let’s look at the main program (demo1).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

package main

import (
    "fmt"

    "github.com/bigwhite/demo1/pkg/pkg1"
)

func main() {
    err := pkg1.LoadAndInvokeSomethingFromPlugin("../demo1-plugins/plugin1.so")
    if err != nil {
        fmt.Println("LoadAndInvokeSomethingFromPlugin error:", err)
        return
    }
    fmt.Println("LoadAndInvokeSomethingFromPlugin ok")
}

The following is the key code in the main demo1 project.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

// github.com/bigwhite/experiments/tree/master/go-plugin/demo1/main.go
package main

import (
    "fmt"

    "github.com/bigwhite/demo1/pkg/pkg1"
)

func main() {
    err := pkg1.LoadAndInvokeSomethingFromPlugin("../demo1-plugins/plugin1.so")
    if err != nil {
        fmt.Println("LoadAndInvokeSomethingFromPlugin error:", err)
        return
    }
    fmt.Println("LoadAndInvokeSomethingFromPlugin ok")
}

We call the LoadAndInvokeSomethingFromPlugin function of the pkg1 package in the main function, which loads the go plugin passed in by the main function, finds the corresponding symbols in the plugin and uses the exported variables, functions, etc. in the plugin through these symbols. The following is the implementation of the LoadAndInvokeSomethingFromPlugin function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

// github.com/bigwhite/experiments/tree/master/go-plugin/demo1/pkg/pkg1/pkg1.go

package pkg1

import (
    "errors"
    "plugin"
    "log"
)

func init() {
    log.Println("pkg1 init")
}

type MyInterface interface {
    M1()
}

func LoadAndInvokeSomethingFromPlugin(pluginPath string) error {
    p, err := plugin.Open(pluginPath)
    if err != nil {
        return err
    }

    // 导出整型变量
    v, err := p.Lookup("V")
    if err != nil {
        return err
    }
    *v.(*int) = 15

    // 导出函数变量
    f, err := p.Lookup("F")
    if err != nil {
        return err
    }
    f.(func())()

    // 导出自定义类型变量
    f1, err := p.Lookup("Foo")
    if err != nil {
        return err
    }
    i, ok := f1.(MyInterface)
    if !ok {
        return errors.New("f1 does not implement MyInterface")
    }
    i.M1()

    return nil
}

In the LoadAndInvokeSomethingFromPlugin function, we use the Lookup method provided by the Plugin type provided by the plugin package to find the corresponding exported symbols in the loaded .so, such as V, F and Foo above, etc. The Lookup method returns the plugin.Symbol type, and the Symbol type is defined as follows.

1
2

// $GOROOT/src/plugin/plugin.go
type Symbol interface{}

We see that the underlying type of Symbol is interface{}, so it can carry any type of variable, function (thanks to the fact that functions are first-class citizens) symbol found in plugin. The types defined in the plugin are not looked up by the main program, and usually the main program does not rely on the types defined in the plugin.

Once the lookup succeeds, we can take the symbols by type assert to get instances of their real types, and with these instances (variables or functions) we can call the logic implemented in plugin. After compiling the plugin and running the main program above, we can see the following results.

1
2
3
4
5
6
7

$go run main.go
2021/06/15 10:05:22 pkg1 init
try to LoadAndInvokeSomethingFromPlugin...
2021/06/15 10:05:22 plugin1 init
plugin1: public integer variable V=15
plugin1: invoke foo.M1
LoadAndInvokeSomethingFromPlugin ok

So, how does the main program know whether the exported symbol is a function or a variable? This depends on the design of the main program plug-in system, because there must be some kind of “contract” or “agreement” between the main program and the plug-in. For example, the MyInterface interface type defined by the main program above is a convention between the main program and the plugin. As long as the plugin exposes an instance of the type that implements the interface, the main program can establish a connection with it and call the implementation in the plugin through the MyInterface interface type instance.

2. Initialization of packages in the plugin

In the above example, we see that the initialization of the plugin (plugin1 init) happens when the main program opens the .so file. According to the official documentation: " When a plugin is opened for the first time, the init functions of all packages in the plugin that are not part of the main program will be called, but a plugin is only initialized once and cannot be closed “.

Let’s verify that the same plugin is loaded multiple times in the main program, this time we upgrade the program to demo2 and demo2-plugins:.

The main program code is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

// github.com/bigwhite/experiments/tree/master/go-plugin/demo2/main.go

package main

import (
    "fmt"

    "github.com/bigwhite/demo2/pkg/pkg1"
)

func main() {
    fmt.Println("try to LoadPlugin...")
    err := pkg1.LoadPlugin("../demo2-plugins/plugin1.so")
    if err != nil {
        fmt.Println("LoadPlugin error:", err)
        return
    }
    fmt.Println("LoadPlugin ok")
    err = pkg1.LoadPlugin("../demo2-plugins/plugin1.so")
    if err != nil {
        fmt.Println("Re-LoadPlugin error:", err)
        return
    }
    fmt.Println("Re-LoadPlugin ok")
}

package pkg1

import (
    "log"
    "plugin"
)

func init() {
    log.Println("pkg1 init")
}

func LoadPlugin(pluginPath string) error {
    _, err := plugin.Open(pluginPath)
    if err != nil {
        return err
    }
    return nil
}

Since it only verifies the initialization, we removed the lookup symbols and calls. plugin’s code is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// github.com/bigwhite/experiments/tree/master/go-plugin/demo2-plugins/plugin1.go
package main

import (
    "log"

    _ "github.com/bigwhite/common"
)

func init() {
    log.Println("plugin1 init")
}

In demo2’s plugin, we also keep only the initialization-related code, and here we also add an external dependency in demo2’s plugin1: github.com/bigwhite/common.

Run the above code.

1
2
3
4
5
6
7

$go run main.go
2021/06/15 10:50:47 pkg1 init
try to LoadPlugin...
2021/06/15 10:50:47 common init
2021/06/15 10:50:47 plugin1 init
LoadPlugin ok
Re-LoadPlugin ok

With this output, we verify two claims.

• repeated loading of the same plugin does not trigger the initialization of the plugin package multiple times, as in the above result only output once : “plugin1 init”.
• Packages that are dependent in the plugin but not in the main application, these packages will be initialized when the plugin is loaded, e.g.: “commin init”.

If the main program also depends on the github.com/bigwhite/common package, we add a line to the main program’s main package.

1
2
3
4
5
6
7

// github.com/bigwhite/experiments/tree/master/go-plugin/demo2/main.go
import (
    "fmt"   

    _ "github.com/bigwhite/common"    // 增加这一行
    "github.com/bigwhite/demo2/pkg/pkg1"
)

Then we execute demo2 again and output the following result.

1
2
3
4
5
6

2021/06/15 11:00:00 common init
2021/06/15 11:00:00 pkg1 init
try to LoadPlugin...
2021/06/15 11:00:00 plugin1 init
LoadPlugin ok
Re-LoadPlugin ok

We see that the common package is already initialized in the main demo2 application, so that when the plugin is loaded, the common package will not be initialized again.

3. go plugin usage constraints

As we mentioned in the beginning, one of the main reasons why the go plugin is not widely used is because of its many constraints, so let’s look at what constraints the go plugin has.

1) The version of the common dependency package of the main program and plugin must be the same

In demo2 above, the github.com/bigwhite/common package that the main application and plugin depend on is a local module, and we use replace in go.mod to point to the local path.

1
2
3
4
5
6
7
8
9

// github.com/bigwhite/experiments/tree/master/go-plugin/demo2/go.mod

module github.com/bigwhite/demo2

replace github.com/bigwhite/common => /Users/tonybai/go/src/github.com/bigwhite/experiments/go-plugin/common

require github.com/bigwhite/common v0.0.0-20180202201655-eb2c6b5be1b6 // 这个版本号是自行"伪造"的

go 1.16

If I clone a copy of the common package, put it in the common1 directory, and point the replace github.com/bigwhite/common statement in the plugin’s go.mod to the common1 directory, and we recompile the main program and the plugin and run the main program, we will get the following Result.

1
2
3
4
5

$go run main.go
2021/06/15 14:09:07 common init
2021/06/15 14:09:07 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo2-plugins/plugin1"): plugin was built with a different version of package github.com/bigwhite/common

We see that the plugin fails to load because the common version is different. This is a constraint used by the plugin: The main application and the plugin must have the same version of the common dependency package .

Let’s look at an example where the main application and the plugin have a common package. We create demo3, in which both the main program and plugin depend on the logrus logging package, but the main program uses logrus version 1.8.1 and plugin uses logrus version 1.8.0. After compiling separately, we run the main program as follows.

1
2
3
4
5

// github.com/bigwhite/experiments/tree/master/go-plugin/demo3

2021/06/15 14:18:35 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo3-plugins/plugin1"): plugin was built with a different version of package github.com/sirupsen/logrus

We see that the main program runs with the same error as the previous example suggests, both because it uses a third-party package with an inconsistent version. To solve this problem, we just need to keep the version of the logrus package used by both consistent, for example by downgrading the logrus of the main program from v1.8.1 to v1.8.0.

1
2
3
4
5
6
7

$go get github.com/sirupsen/logrus@v1.8.0
go get: downgraded github.com/sirupsen/logrus v1.8.1 => v1.8.0
$go run main.go
2021/06/15 14:19:09 pkg1 init
try to LoadPlugin...
2021/06/15 14:19:09 plugin1 init
LoadPlugin ok

We see that after downgrading the logrus version, the main program can load the plugin normally.

There is another case, that is, the main program and plugin use different major versions of the same module, because the major version is different, although it is the same module, but in fact two different packages, this will not affect the main program to load plugin. However, the problem is that the module that is co-dependent also has its own dependency package, and when the version of a package that it depends on differs from one major version to another, it will also cause the main program to have problems loading the plugin. For example, if the main application depends on v6.15.9+incompatible version of go-redis/redis, and the plugin depends on go-redis/redis/v8, when we use such a main application to load the plugin, we will encounter the following error.

1
2
3
4
5
6

// github.com/bigwhite/experiments/tree/master/go-plugin/demo3

$go run main.go
2021/06/15 14:32:11 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo3-plugins/plugin1"): plugin was built with a different version of package golang.org/x/sys/unix

We see that the redis version is not wrong, but the problem is that redis and redis/v8 depend on different versions of golang.org/x/sys, this indirect dependency on the version of the module inconsistency will also cause the go plugin to fail to load, this is also one of the constraints of the use of the go plugin.

2) If mod=vendor build is used, then the main application and plugin must be built from the same vendor directory

The build based on vendor is a feature introduced in go 1.5. After go 1.11 introduced the go module build mode, the vendor build was retained. So the question is, if either the main application or the plugin uses vendor build or both, will the main application be able to load the plugin properly? Let’s verify this with the sample demo4. (demo4 and demo3 is more or less the same, here will not list the specific code).

First we generate the vendor directory for the main program (demo4) and the plugins (demo4-plugins) respectively.

1
2
3
4
5

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go mod vendor

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4-plugins
$go mod vendor

We test the following three cases (go version 1.16 gives preference to build with vendor when available by default. So to build based on mod you need to explicitly pass in -mod=mod).

• the main application is built based on mod, the plugin is built based on vendor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4-plugins
$go build -mod=vendor -buildmode=plugin -o plugin1.so plugin1.go

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go build -mod=mod -o main.mod main.go

$main.mod
2021/06/15 15:41:21 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo4-plugins/plugin1"): plugin was built with a different version of package golang.org/x/sys/unix

• The main program is built on vendor, the plug-in is built on mod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4-plugins
$go build -mod=mod -buildmode=plugin -o plugin1.so plugin1.go

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go build -mod=vendor -o main.vendor main.go

$./main.vendor
2021/06/15 15:44:15 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo4-plugins/plugin1"): plugin was built with a different version of package golang.org/x/sys/unix

• The main application and the plug-in are built on their own vendor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4-plugins
$go build -mod=vendor -buildmode=plugin -o plugin1.so plugin1.go

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go build -mod=vendor -o main.vendor main.go

$./main.vendor
2021/06/15 15:45:11 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo4-plugins/plugin1"): plugin was built with a different version of package golang.org/x/sys/unix

From the above test, we see that the main program will fail to load the plugin no matter which side uses vendor build or both sides are based on their respective vendor build. How to solve this problem? Make the main program and plugin build based on the same vendor !

We copy plugin1.go to demo4, and then build the main program and plugin1.go with separate vendor builds:.

1
2
3
4
5

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go build -mod=vendor -o main.vendor main.go

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go build -mod=vendor -buildmode=plugin -o plugin1.so plugin1.go

Copy the compiled plugin1.so into demo4-plugins, and run main.vendor.

1
2
3
4
5
6
7

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$cp plugin1.so ../demo4-plugins
$main.vendor
2021/06/15 15:48:56 pkg1 init
try to LoadPlugin...
2021/06/15 15:48:56 plugin1 init
LoadPlugin ok

We see that main programs and plugins based on the same vendor are compatible. The following table summarizes the compatibility between the main program and the plugin when they are built in different build modes.

In vendor build mode, the plugin can only be loaded successfully by the main program if it is built based on the same vendor directory !

3) The compiler version used by the main program and the plugin must be the same

If we use different versions of the Go compiler to compile the main program and the plugin separately, will they be compatible? Let’s also take demo4 to verify it. I have go 1.16.5 and go 1.16 compilers on the host, go 1.16.5 is the patch maintenance version of go 1.16, the difference is not of the same magnitude compared to go 1.16 and go 1.15, we compile the main program with go 1.16 and the plugin with go 1.16.5.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4-plugins
$go version
go version go1.16.5 darwin/amd64
$go build -buildmode=plugin -o plugin1.so plugin1.go

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go version
go version go1.16 darwin/amd64

$go run main.go
2021/06/15 15:58:44 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin.Open("../demo4-plugins/plugin1"): plugin was built with a different version of package runtime/internal/sys

We see that the plugin is not compatible with the main program even when compiled with the patch version. We upgrade demo4 to compile with go version 1.16.5.

1
2
3
4
5
6
7

$go version
go version go1.16.5 darwin/amd64
$go run main.go
2021/06/15 15:59:05 pkg1 init
try to LoadPlugin...
2021/06/15 15:59:05 plugin1 init
LoadPlugin ok

We have seen that only when the main program and the plugin are compiled with the exact same version (the patch version should also be the same), they are compatible and the main program can load the plugin properly.

So does the OS version affect the compatibility of the main program and plugin? There are no official instructions for this, so I tested it myself. I built demo4-plugin (based on mod=mod) on centos 7.6 (amd64, go 1.16.5) and then copied it to a host with ubuntu 18.04 (amd64, go 1.16.5), the demo4 main program on the ubuntu host was compatible with the plugin.

4) Master programs with plugins can only use dynamic links

Go is known for being statically compiled for ease of distribution and deployment, but main programs using plugins can only use dynamic links. Don’t believe me? Then let’s challenge the main program in demo4 to compile statically.

First, let’s look at the default compilation.

1
2
3
4
5
6
7
8

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$go build main.go
$ldd main
    linux-vdso.so.1 (0x00007ffc05b73000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6a9fa3f000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6a9f820000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6a9f42f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6a9fc43000)

We see that the demo4 main program is compiled by default as an executable that needs to be dynamically linked at runtime, and it relies on a number of linux runtime libraries, e.g. libc.

The reason for all this is that we use some standard libraries implemented via cgo in demo4, such as the plugin package.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// $GOROOT/src/plugin/plugin_dlopen.go

// +build linux,cgo darwin,cgo freebsd,cgo

package plugin

/*
#cgo linux LDFLAGS: -ldl
#include <dlfcn.h>
#include <limits.h>
#include <stdlib.h>
#include <stdint.h>

#include <stdio.h>

static uintptr_t pluginOpen(const char* path, char** err) {
    void* h = dlopen(path, RTLD_NOW|RTLD_GLOBAL);
    if (h == NULL) {
        *err = (char*)dlerror();
    }
    return (uintptr_t)h;
}
... ...
*/

We see that plugin_dlopen.go has the build indicator in the header, which is only compiled if cgo is on. If we remove cgo, for example, using the following command line.

1
2
3
4

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4
$ CGO_ENABLED=0 go build main.go
$ ldd main
    not a dynamic executable

We do compile a statically linked executable, but when we execute the file, we get the following results.

1
2
3
4

$ ./main
2021/06/15 17:01:51 pkg1 init
try to LoadPlugin...
LoadPlugin error: plugin: not implemented

We see that some functions of the plugin package were not compiled into the final executable because cgo was turned off, so the error “not implemented” was reported!

With CGO turned on, we can still let the external linker use static links, so let’s try again.

1
2
3
4
5
6
7
8

// github.com/bigwhite/experiments/tree/master/go-plugin/demo4

$ go build -o main-static -ldflags '-linkmode "external" -extldflags "-static"' main.go
# command-line-arguments
/tmp/go-link-638385712/000001.o: In function `pluginOpen':
/usr/local/go/src/plugin/plugin_dlopen.go:19: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
$ ldd main-static
    not a dynamic executable

We do get a statically compiled binary, but the compiler also gives a WARNING.

To execute this file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ ./main-static
2021/06/15 17:02:35 pkg1 init
try to LoadPlugin...
fatal error: runtime: no plugin module data

goroutine 1 [running]:
runtime.throw(0x5d380a, 0x1e)
    /usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc000091b50 sp=0xc000091b20 pc=0x435712
plugin.lastmoduleinit(0xc000076210, 0x1001, 0x1001, 0xc000010040, 0x24db1f0)
    /usr/local/go/src/runtime/plugin.go:20 +0xb50 fp=0xc000091c48 sp=0xc000091b50 pc=0x466750
plugin.open(0x5d284c, 0x18, 0xc0000788f0, 0x0, 0x0)
    /usr/local/go/src/plugin/plugin_dlopen.go:77 +0x4ef fp=0xc000091ec0 sp=0xc000091c48 pc=0x4dad8f
plugin.Open(...)
    /usr/local/go/src/plugin/plugin.go:32
github.com/bigwhite/demo4/pkg/pkg1.LoadPlugin(0x5d284c, 0x1b, 0xc000091f48, 0x1)
    /root/test/go/plugin/demo4/pkg/pkg1/pkg1.go:13 +0x35 fp=0xc000091ef8 sp=0xc000091ec0 pc=0x4dbbb5
main.main()
    /root/test/go/plugin/demo4/main.go:12 +0xa5 fp=0xc000091f88 sp=0xc000091ef8 pc=0x4ee805
runtime.main()
    /usr/local/go/src/runtime/proc.go:225 +0x256 fp=0xc000091fe0 sp=0xc000091f88 pc=0x438196
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x46a841

The warning eventually evolved into a runtime panic, and it seems that the main program using the plugin can only be compiled into a dynamically linked executable. The current go project has several issues related to this.

• https://github.com/golang/go/issues/33072
• https://github.com/golang/go/issues/17150
• https://github.com/golang/go/issues/18123

4. plugin version management

One of the bigger issues in implementing a plugin system using dynamic linking is the versioning of the plugin.

Dynamic link libraries on linux are versioned using soname. soname’s key feature is that it provides compatibility criteria when upgrading a library on the system, and the soname of the new library is the same as the soname of the old library, so that programs generated by linking with the old library will still work fine with the new library. This feature makes it very easy to upgrade programs that make shared libraries and locate errors under Linux.

What is soname? In directories like /lib and /usr/lib where shared libraries are centrally located, you will always see something like the following.

1
2
3

2019-12-10 12:28 libfoo.so -> libfoo.so.0.0.0*
2019-12-10 12:28 libfoo.so.0 -> libfoo.so.0.0.0*
2019-12-10 12:28 libfoo.so.0.0.0*

Regarding libfoo.so there are actually three file entries, of which libfoo.so.0.0.0 is the real shared library file, while the other two file entries are symbolic links to libfoo.so.0.0.0. Why is this the case? It has to do with the naming convention and versioning of shared libraries.

The shared library convention has multiple name attributes for each shared library, including real name, soname, and linker name.

• real name

real name refers to the name of the file that actually contains the shared library code (e.g. libfoo.so.0.0.0 in the above example), and is the argument that follows -o on the shared library compile command line.

• soname

soname is the abbreviation for shared object name, and is the most important of the three names. The system linker uniquely identifies a shared library by its soname (e.g., libfoo.so.0 in the above example), both at the compilation stage and at the runtime stage. Even if the real name is the same but the soname is different, it will be considered by the linker as two different libraries. The soname of the shared library can be specified during compilation by passing parameters to the linker, e.g. we can specify it by “gcc -shared -Wl,-soname -Wl,libfoo.so.0 -o libfoo.so.0.0.0 libfoo.o” libfoo.so.0.0.0 has a soname of libfoo.so.0. ldconfig -n directory_with_shared_libraries will automatically generate a symbolic link named soname to the real name file based on the soname of the shared library, but you can also You can also create this symbolic link yourself by using the ln command. Also in linux we can check the soname of the shared libraries with readelf -d. The list of shared libraries that the lldd file depends on shows the soname and the path of the shared libraries.

• linker name

linker name is the name provided to the compiler at compilation stage (e.g. libfoo.so in the above example). If you build a shared library with a real name that looks like libfoo.so.0.0.0 in the example above with a version number, then you cannot get the linker to find the corresponding shared library file by using -L path -lfoo directly in the compiler command unless you provide a linker name for libfoo.so.0.0.0 (e.g. libfoo.so, a symbolic link to libfoo.so.0.0.0). linker names are usually created manually during shared library installation.

So can the go plugin be versioned in the same way as soname? Let’s create demo5 based on demo1 and experiment with it.

In demo5-plugins, we add version information to the .so build.

1
2
3
4
5
6
7

// github.com/bigwhite/experiments/tree/master/go-plugin/demo5-plugins

$go build -buildmode=plugin -o plugin1.so.1.1 plugin1.go
$ln -s plugin1.so.1.1 plugin1.so.1
$ls -l
lrwxr-xr-x  1 tonybai  staff       14  7 16 05:42 plugin1.so.1@ -> plugin1.so.1.1
-rw-r--r--  1 tonybai  staff  2888408  7 16 05:42 plugin1.so.1.1

We created a symbolic link plugin1.so.1 for the built plugin1.so.1 with the ln command, and plugin1.so.1 was passed to demo5 as the soname of our plugin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// github.com/bigwhite/experiments/tree/master/go-plugin/demo5/main.go

func main() {
    fmt.Println("try to LoadAndInvokeSomethingFromPlugin...")
    err := pkg1.LoadAndInvokeSomethingFromPlugin("../demo5-plugins/plugin1.so.1")
    if err != nil {
        fmt.Println("LoadAndInvokeSomethingFromPlugin error:", err)
        return
    }
    fmt.Println("LoadAndInvokeSomethingFromPlugin ok")
}

Running demo5.

1
2
3
4
5
6
7
8
9

// github.com/bigwhite/experiments/tree/master/go-plugin/demo5

$go run main.go
2021/07/16 05:58:33 pkg1 init
try to LoadAndInvokeSomethingFromPlugin...
2021/07/16 05:58:33 plugin1 init
plugin1: public integer variable V=15
plugin1: invoke foo.M1
LoadAndInvokeSomethingFromPlugin ok

We see that the plugin passed in as soname is loaded and extracts the symbols without any problems.

Later, if the plugin changes, such as patch, we only need to upgrade the plugin to plugin1.so.1.2, then soname remains the same, and the main program does not need to change.

Note: If the plugin name is the same and the content is the same, the main program will not have problems loading multiple times; but if the plugin name is the same but the content is different, the main program will cause a runtime panic when running multiple loads, and it is a panic that cannot be recovered. So be sure to do a good job of plug-in version management .

5. Summary

go plugin is a go plugin solution provided natively by the go language (non-go plugin solution, you can use c shared library, etc.). However, after the above experiments and learning, we have seen many constraints on the use of plugins, which does create a big obstacle to the promotion of the use of go plugin, resulting in the current go plugin is not very widely used.

According to the constraints seen above, if you want to apply go plugin, you have to do:

• Consistent build environment
• Consistent versioning of third-party packages.

Therefore, the industry uses builder containers to ensure that the main application and the plugin use the same build environment when using the go plugin.

Among the few users of the go plugin, there are three well-known open source projects that deserve careful follow-up study.

• gosh
• tyk api gateway
• tidb

In particular, tidb, also gives its plug-in system using go plugin’s complete design scheme, which is worth everyone read carefully.

All the source code involved in this article can be downloaded from here.