Go Subtest

Unit testing is a crucial part of software development, and its existence includes but is not limited to the following aspects.

Improving code quality: Unit testing ensures correctness, reliability and stability of code, thus reducing code defects and bugs.
Reduce regression testing costs: When modifying code, unit testing can quickly check whether it affects the functionality of other modules, avoiding the cost of regression testing the entire system.
Promote teamwork: Unit testing can help team members better understand and use the code written by each other, improving the readability and maintainability of the code.
Improving development efficiency: Unit testing can automate the execution of tests and reduce the time and workload of manual testing, thus improving development efficiency.

The designers of the Go language placed a lot of emphasis on language features and environment features from the very beginning of Go’s design, and it turns out that Go’s success is due to its focus on engineering the overall environment of a software project. The fact that Go has a built-in lightweight testing framework is a reflection of Go’s focus on the environment. The Go team continues to invest in this built-in testing framework, and new, more convenient and flexible features are constantly being added to the Go testing framework to help Gopher better organize test code, execute tests more efficiently, etc.

Go’s introduction of subtest in Go 1.7 is a typical example. The addition of subtest allows Gopher to apply the built-in go test framework in a more flexible way.

In this article, I will talk about subtest with you in the context of what I have learned about subtest awareness, understanding and usage in my daily development.

1. Go Unit Testing Review

In the Go language, unit tests are considered first-class citizens, and combined with Go’s built-in lightweight testing framework, Go developers can easily write unit test cases.

Go’s unit tests are usually placed in the same package as the code being tested, and the source file where the unit tests are located ends with _test.go, as required by this Go testing framework. The test functions are prefixed with Test, accept an argument of type *testing.T, and use methods such as t.Error, t.Fail, and t.Fatal to report test failures. All the test code can be run using the go test command. If the test passes, a message is output indicating that the test was successful; otherwise, an error message is output indicating which tests failed.

Note: Go also supports benchmarking, example testing, fuzzy testing, etc. for performance testing and documentation generation, but these are not what this article will focus on. Note: t.Error <=> t.Log+t.Fail

Usually when writing Go test code, we first consider top-level test.

2. Go top-level test

The above-mentioned functions in *_test.go, which are in the same directory as the source code under test, start with Test and are Go top-level tests. For example.

/ https://github.com/bigwhite/experiments/blob/master/subtest/add_test.go

func Add(a, b int) int {
    return a + b
}

func TestAdd(t *testing.T) {
    got := Add(2, 3)
    if got != 5 {
        t.Errorf("Add(2, 3) got %d, want 5", got)
    }
}

func TestAddZero(t *testing.T) {
    got := Add(2, 0)
    if got != 2 {
        t.Errorf("Add(2, 0) got %d, want 2", got)
    }
}

func TestAddOppositeNum(t *testing.T) {
    got := Add(2, -2)
    if got != 0 {
        t.Errorf("Add(2, -2) got %d, want 0", got)
    }
}

Note: “got-want” is a common naming convention in Go test in Errorf

The execution of the top-level test has the following characteristics.

go test places each TestXxx in a separate goroutine for execution, maintaining mutual isolation.
a TestXxx use case fails, and outputting an error result via Errorf, or even Fatalf, will not affect the execution of other TestXxx.
a TestXxx use case in which a result judgment has not passed, if an error result is output via Errorf, the TestXxx will continue to execute.
However, if the TestXxx is using Fatal/Fatalf, this will cause the execution of that TestXxx to end immediately at the point where Fatal/Fatalf is called, and subsequent test code within the TestXxx function will not be executed.
Default execution of individual TestXxx one by one in the declared order, even if they are executed in their respective goroutines.
The go test -shuffle=on allows each TestXxx to be executed in random order, which detects whether there is an execution order dependency between the individual TestXxx, which we want to avoid in the test code.
The “go test -run=regular” method allows you to select certain TestXxx to be executed.
Each TestXxx function can call the t.Parallel method (i.e., testing.T.Parallel method) to add the TestXxx to the set of use cases that can be executed in parallel; note that the order of execution of these TestXxx is not determined once they are added to the parallel execution set.

In combination with the table-driven tests that are part of Go’s best practices (as shown in the following code TestAddWithTable), we can achieve the equivalent of the above three TestXxx without having to write many TestXxx, using the following TestAddWithTable.

func TestAddWithTable(t *testing.T) {
    cases := []struct {
        name string
        a    int
        b    int
        r    int
    }{
        {"2+3", 2, 3, 5},
        {"2+0", 2, 0, 2},
        {"2+(-2)", 2, -2, 0},
        //... ...
    }

    for _, caze := range cases {
        got := Add(caze.a, caze.b)
        if got != caze.r {
            t.Errorf("%s got %d, want %d", caze.name, got, caze.r)
        }
    }
}

Go top-level test can meet most of Gopher’s regular single test requirements, and the table-driven convention is easy to understand.

However, top-level test + table-driven tests simplify the writing of test code, but also bring some shortcomings.

Sequential execution of cases in a table, not shuffle;
all cases in the table are executed in the same goroutine, poor isolation.
If fatal/fatalf is used, then once a case fails, the subsequent test table entries (cases) will not get executed.
test cases within tables cannot be executed in parallel.
The organization of test cases can only be tiled, which is not flexible enough to build up a hierarchy.

For this reason Go version 1.7 introduced subtest!

3. Subtest

Go language’s subtest is a function that divides a test function (TestXxx) into multiple small test functions, each of which can be run independently and report the test results. This way of testing allows for more fine-grained control of test cases, making it easier to locate problems and debug.

The following is a sample code that uses subtest to transform TestAddWithTable to show how to write subtest in Go.

// https://github.com/bigwhite/experiments/blob/master/subtest/add_sub_test.go

func TestAddWithSubtest(t *testing.T) {
    cases := []struct {
        name string
        a    int
        b    int
        r    int
    }{
        {"2+3", 2, 3, 5},
        {"2+0", 2, 0, 2},
        {"2+(-2)", 2, -2, 0},
        //... ...
    }

    for _, caze := range cases {
        t.Run(caze.name, func(t *testing.T) {
            t.Log("g:", curGoroutineID())
            got := Add(caze.a, caze.b)
            if got != caze.r {
                t.Errorf("got %d, want %d", got, caze.r)
            }
        })
    }
}

In the code above, we define a test function called TestAddWithSubtest and use the t.Run() method in combination with a table test to create three subtests, so that each subtest can reuse the same error handling logic, but with different test case parameters to reflect the differences. Of course, if you don’t use table-driven tests, then each subtest can have its own independent error handling logic!

Executing the test case TestAddWithSubtest above (we intentionally changed the implementation of the Add function to be wrong), we will see the following result.

$go test  add_sub_test.go
--- FAIL: TestAddWithSubtest (0.00s)
    --- FAIL: TestAddWithSubtest/2+3 (0.00s)
        add_sub_test.go:54: got 6, want 5
    --- FAIL: TestAddWithSubtest/2+0 (0.00s)
        add_sub_test.go:54: got 3, want 2
    --- FAIL: TestAddWithSubtest/2+(-2) (0.00s)
        add_sub_test.go:54: got 1, want 0

We see that in the error message output, each failure case is identified by “TestXxx/subtestName”, and we can easily match it to the corresponding line of code. The deeper meaning is that subtest gives a sense of “hierarchy” to the entire test organization! With the -run flag, we can select one/some subtest of a top-level test to be executed in this “hierarchy”.

$go test  -v -run TestAddWithSubtest/-2 add_sub_test.go
=== RUN   TestAddWithSubtest
=== RUN   TestAddWithSubtest/2+(-2)
    add_sub_test.go:51: g: 19
    add_sub_test.go:54: got 1, want 0
--- FAIL: TestAddWithSubtest (0.00s)
    --- FAIL: TestAddWithSubtest/2+(-2) (0.00s)
FAIL
FAIL    command-line-arguments  0.006s
FAIL

Let’s see what features the subtest has (you can look at it in comparison with the previous top-level test).

go subtest is also executed in a separate goroutine, maintaining mutual isolation.
A Subtest use case fails, and the output of an error result via Errorf, or even Fatalf, will not affect the execution of other Subtest under the same TestXxx.
if a result judgment in a Subtest fails, the execution of that Subtest will continue if an error result is output via Errorf.
However, if the subtest is using Fatal/Fatalf, this will cause the execution of that subtest to end immediately at the point where Fatal/Fatalf is called, and subsequent test code within the subtest function will not be executed.
By default the subtests under each TestXxx will be executed one by one in the declared order, even if they are executed in their respective goroutines.
So far, subtest does not support shuffle-style random order execution.
By means of “go test -run=TestXxx/regular[/…]” we can choose to execute one or some subtests under TestXxx.
Each subtest can call the t.Parallel method (i.e. testing.T.Parallel method) to add the subtest to the set of use cases that can be executed in parallel. Note: the execution order of these subtests is not determined after being added to the parallel execution set.

In summary, the advantages of subtest can be summarized as follows.

More fine-grained testing: By dividing test cases into multiple small test functions, problems can be more easily located and debugged.
Better readability: subtest can make test code clearer and easier to understand.
More flexible testing: subtests can be combined and arranged as needed to meet different testing needs.
More hierarchical organization of test code: subtest allows you to design a more hierarchical organization of test code, share resources more easily and set setup and teardown at a certain organization level.

4. Subtest vs. top-level test

The top-level test itself is actually a subtest, except that its scheduling and execution is controlled by the Go test framework and is not visible to us developers.

For gopher.

Simple package testing can be satisfied in top-level test, which is straightforward, intuitive, and easy to understand.
For complex packages in larger projects, the organization, flexibility, and scalability of subtest can help us improve testing efficiency and reduce testing time once the hierarchy of test code organization is involved.

The source code covered in this article can be downloaded at here.

5. Ref

https://tonybai.com/2023/03/15/an-intro-of-go-subtest/

Table of Contents

1. Go Unit Testing Review

2. Go top-level test

3. Subtest

4. Subtest vs. top-level test

5. Ref