In Go, it is easy to write Benchmark functions to test the performance of a function point. For important functions, we can add the corresponding test process in CI/CD to be aware of the function performance changes when they occur. So the question is, how do you detect performance changes in a function?

To put it another way, you write a function and find that it is slow and you need to optimize it, and when you Google it and find a better implementation, you find that it is indeed faster through the Benchmark function. But you can’t say exactly how much faster, and you want to know the performance comparison before and after the function optimization, and how many percentage points of improvement with high confidence?

For the above requirement scenario, there is a tool that can help you, and it is benchstat.

## Benchmark Example

Let’s review the benchmark first. For ease of understanding, here is an example of computing the classical calculation of Fibonacci series values.

 ``````1 2 3 4 5 6 7 `````` ``````func FibSolution(n int) int { if n < 2 { return n } return FibSolution(n-1) + FibSolution(n-2) } ``````

The above code is a recursive implementation, and it is clear that the function becomes very time consuming as n gets larger and larger. For example, if n is 20, the Benchmark function looks like this.

 ``````1 2 3 4 5 `````` ``````func BenchmarkFib20(b *testing.B) { for i := 0; i < b.N; i++ { FibSolution(20) } } ``````

Execute `go test -bench=BenchmarkFib20` on the command line to get the performance results.

 ``````1 `````` ``````BenchmarkFib20-8 39452 30229 ns/op ``````

Where -8 represents 8 cpu, the number of function runs is 39452, and the average time spent per function is 30229ns.

If we want to get multiple samples, we can specify the `-count=N` parameter of go test. For example, if we want to get the sample data 5 times, we would run `go test -bench=BenchmarkFib20 -count=5`.

 ``````1 2 3 4 5 `````` ``````BenchmarkFib20-8 39325 30297 ns/op BenchmarkFib20-8 39216 30349 ns/op BenchmarkFib20-8 39901 30251 ns/op BenchmarkFib20-8 39336 30455 ns/op BenchmarkFib20-8 39423 30894 ns/op ``````

The iterative implementation to calculate the Fibonacci series values is as follows.

 `````` 1 2 3 4 5 6 7 8 9 10 11 12 `````` ``````func FibSolution(n int) int { if n < 2 { return n } p, q, r := 0, 0, 1 for i := 2; i <= n; i++ { p = q q = r r = p + q } return r } ``````

The plainest way to compare the performance difference between these two functions is to benchmark them separately and then analyze these benchmark results by hand, but this is not intuitive.

## benchstat

benchstat is an official Go recommended command line tool for calculating and comparing statistics related to benchmarking.

We can install it with the following command.

 ``````1 `````` ``````go install golang.org/x/perf/cmd/benchstat@latest ``````

Execute the -h parameter to see a description of the tool’s usage.

 `````` 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 `````` ``````~ \$ benchstat -h usage: benchstat [options] old.txt [new.txt] [more.txt ...] options: -alpha α consider change significant if p < α (default 0.05) -csv print results in CSV form -delta-test test significance test to apply to delta: utest, ttest, or none (default "utest") -geomean print the geometric mean of each file -html print results as an HTML table -norange suppress range columns (CSV only) -sort order sort by order: [-]delta, [-]name, none (default "none") -split labels split benchmarks by labels (default "pkg,goos,goarch") ``````

We would like to compare the performance benchmarks of FibSolution(n) from 15 to 20, for both implementations.

 ``````1 2 `````` ``````\$ go test -bench=. -count=5 | tee old.txt \$ go test -bench=. -count=5 | tee new.txt ``````

Note that the two commands are executed with recursive and iterative implementation logic corresponding to the FibSolution function, respectively.

At this point, we can compare the performance of these two function implementation logics.

 ``````1 2 3 4 5 6 7 8 `````` `````` \$ benchstat old.txt new.txt name old time/op new time/op delta Fib15-8 2.67µs ± 2% 0.01µs ± 5% -99.81% (p=0.008 n=5+5) Fib16-8 4.20µs ± 1% 0.01µs ± 2% -99.87% (p=0.008 n=5+5) Fib17-8 6.81µs ± 0% 0.01µs ± 2% -99.92% (p=0.008 n=5+5) Fib18-8 11.1µs ± 1% 0.0µs ± 1% -99.95% (p=0.008 n=5+5) Fib19-8 18.0µs ± 2% 0.0µs ± 4% -99.97% (p=0.008 n=5+5) Fib20-8 29.2µs ± 1% 0.0µs ± 3% -99.98% (p=0.008 n=5+5) ``````

As you can see, the execution time of the recursive implementation of the function increases significantly as the value of n becomes larger. The iterative implementation reduces the average time overhead by more than 99% compared to the recursive implementation, which is a very significant optimization effect.

In addition, p=0.008 indicates the confidence level of the results, and a larger p value indicates a lower confidence level. In general, 0.05 is used as the threshold value, beyond which the results are not credible. n=5+5 indicates the number of valid samples used respectively.

## Summary

benchstat is a benchmarking statistics tool that can be used to alleviate the cost of manually analyzing data when we do some optimization work.

If you have a project that deploys automated tests in your CI/CD process, you may want to add this tool to the mix. It may help you identify problems in advance when there are changes to functions that add to performance loss.