Golang and Tls1.3

This time let’s talk about a recent problem: how to speed up the connection building speed of massive connections based on TLS secure communication?

Below the TLS (Transport Layer Security) layer is the TCP layer, and the first thing we might think of is optimizing the kernel parameters related to the TCP handshake to quickly establish a TCP connection, for example.

1
2
3
4
5
6

net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_syncookies
net.ipv4.tcp_synack_retries
net.ipv4.tcp_abort_on_overflow
net.core.somaxconn
… …

In addition, in order to speed up connection building for massive connections, in addition to increasing the speed at which applications can get connections from the kernel’s accept connection queue, we can also use a multi-thread/multi-goroutine mechanism to concurrently listen to the same port and concurrently accept it, if you are using the Golang, take a look at the go-reuseport package.

After talking about the TCP layer, are there any optimizations that can be made to the TLS layer that will have an impact on connection building speed? Yes, there is, and that is the use of TLS version 1.3 to speed up the handshake process and thus the connection building speed.TLS 1.3 is a new TLS standard released in 2018 and has only started to be supported by mainstream languages, browsers and web servers in the last 2-3 years. So how does it differ from the previous TLS 1.2, and how well does Go support TLS 1.3? How do you write secure communication code using TLS 1.3 in Go, and how much faster is TLS 1.3 connection establishment than TLS 1.2?

With these questions in mind, let’s move on to the main part of this post! Let’s take a brief look at the differences between TLS 1.3 and TLS version 1.2.

1. Differences between TLS 1.3 and TLS 1.2

TLS is a secure connection protocol standard based on encryption, decryption and signature algorithms developed and published by the Internet Engineering Task Force (IETF) (https://www.ietf.org/) as an alternative to SSL, which evolved as follows.

TLS Evolutionary Process

Among them, TLS 1.0 and 1.1 versions will be deprecated in 2020 because they are no longer safe, and the mainstream version, which is also the most widely used, is the TLS 1.2 version released in 2008 (the usage share is as shown in the statistics below), while the latest version is the TLS 1.3 officially released in 2018, and the release of TLS 1.3 version also means that TLS 1.2 The release of TLS 1.3 also means that TLS 1.2 has entered the “obsolescence” period, although in practice it will take a long time for TLS 1.2 to “go offline”.

Percentage of use of each TLS version

TLS 1.3 is not incompatible with TLS 1.2. In the TLS 1.3 protocol specification, we can see some of the major changes listed for TLS 1.3 compared to TLS 1.2.

  • The non-AEAD (Authenticated Encryption with Associated Data) algorithms in the original list of symmetric encryption algorithms, including 3DES, RC4, AES-CBC, etc., are removed, and only more secure encryption algorithms are supported.

    Note: Common AEAD algorithms include: AES-128-GCM, AES-256-GCM, ChaCha20-IETF-Poly1305, etc. On CPUs (desktop, server) with AES acceleration, AES-XXX-GCM series is recommended, and ChaCha20-IETF-Poly1305 series is recommended for mobile devices.

  • Static RSA and Diffie-Hellman cipher suites (cipher suites) have been removed; all public key-based key exchange mechanisms now provide forward security.

    Note: Forward Secrecy refers to the fact that the leakage of a long-used master key does not lead to the leakage of past session keys. Forward security protects past communications from the threat of cryptographic or key exposure in the future. If a system has forward security, it can guarantee the security of historical communications in the event of a master key compromise, even if the system is actively attacked.

  • The negotiation mechanism in TLS version 1.2 has been deprecated and a new, faster key negotiation mechanism has been introduced: PSK (Pre-Shared Key), which simplifies the handshake process (the figure below shows a comparison of the TLS 1.2 and TLS 1.3 handshake processes). Also all handshake messages after ServerHello are now encrypted, and various extended messages previously sent in plaintext in ServerHello now enjoy encryption protection.

    TLS 1.2 versus TLS 1.3 handshake process

  • A zero round-trip time (0-RTT) mode has been added to save a round-trip time for some application data when resuming connection establishment. At the cost of losing certain security attributes.

    Note: When a client (such as a browser) successfully completes a TLS 1.3 handshake with the server for the first time, both the client and the server can store a pre-shared encryption key locally, which is called the recovery master key. If the client later establishes a connection with the server again, it can use this recovery key to send the encrypted application data from its first message to the server without performing a second handshake. 0-RTT mode has a security weakness. Sending data via recovery mode does not require any interaction with the server, which means that an attacker (typically a middle-man) can capture encrypted 0-RTT data and resend it to the server, or replay it. The solution to this problem is to ensure that all 0-RTT requests are idempotent.

Of these major changes, the obvious one related to initial connection speed is the change in the TLS 1.3 handshake mechanism: from 2-RTT to 1-RTT (as shown in the figure above). Let’s use Go as an example to see how TLS 1.3 actually improves on TLS 1.2 in terms of connection speed.

2. Go’s support for TLS 1.3

  • The Golang provides optional support for TLS 1.3 starting with Go 1.12. Under Go 1.12, you can enable TLS 1.3 by setting GODEBUG=tls13=1 and not explicitly setting the MaxVersion of tls Config. The 0-RTT feature of TLS 1.3 is not supported in this implementation.
  • Go version 1.13 has TLS 1.3 enabled by default, you can turn off support for TLS 1.3 using GODEBUG=tls13=0.
  • By the time Go 1.14 is released, TLS 1.3 becomes the default TLS version option and can no longer be turned off with GODEBUG=tls13=0! However, the version of TLS to be used can be configured via Config.MaxVersion.
  • Go 1.16 version, in the case that the server or client does not support AES hardware acceleration, the server side will prefer other AEAD cipher suite (cipher suite), such as ChaCha20Poly1305, instead of the AES-GCM cipher suite.
  • In Go 1.18, Config.MinVersion on the client side will default to TLS 1.2, replacing the previous default of TLS 1.0/TLS 1.1, but you can change this setting by explicitly setting Config.MinVersion on the client side. However, this change does not impact the server side.

With that understood, let’s look at a simple client-side and server-side example using Go and TLS version 1.3.

3. Go TLS 1.3 client-server communication example

This time, instead of referring to the Go standard library crypto/tls package, let’s play with the trendy one: generating a set of TLS-based client-server communication code examples with AI assistance. I’m using the AI programming assistant (AICodeHelper), and here is a screenshot of the generation process.

ChatGPT is not available to mainland China.

Sample client-server communication for tls 1.3 in go

AICodeHelper generates most of the code for us, but the server-side code has two problems: it can only handle a client-side connection and does not generate the snippet to pass in the server certificate and private key, we do a little modification based on the above framework code to get our server and client-side code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

Server-side code.

// https://github.com/bigwhite/experiments/blob/master/go-and-tls13/server.go
package main

import (
    "bufio"
    "crypto/tls"
    "fmt"
    "net"
)

func main() {
    cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        fmt.Println(err)
        return
    }

    config := &tls.Config{Certificates: []tls.Certificate{cer}}
    ln, err := tls.Listen("tcp", "localhost:8443", config)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer ln.Close()

    for {
        conn, err := ln.Accept()
        if err != nil {
            fmt.Println(err)
            continue
        }
        go handleConnection(conn)
    }
}

func handleConnection(conn net.Conn) {
    defer conn.Close()
    r := bufio.NewReader(conn)
    for {
        msg, err := r.ReadString('\n')
        if err != nil {
            fmt.Println(err)
            return
        }

        println(msg)

        n, err := conn.Write([]byte("hello, world from server\n"))
        if err != nil {
            fmt.Println(n, err)
            return
        }
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

// https://github.com/bigwhite/experiments/blob/master/go-and-tls13/client.go

package main

import (
    "crypto/tls"
    "log"
)

func main() {
    conf := &tls.Config{
        InsecureSkipVerify: true,
    }

    conn, err := tls.Dial("tcp", "localhost:8443", conf)
    if err != nil {
        log.Println(err)
        return
    }
    defer conn.Close()

    n, err := conn.Write([]byte("hello, world from client\n"))
    if err != nil {
        log.Println(n, err)
        return
    }

    buf := make([]byte, 100)
    n, err = conn.Read(buf)
    if err != nil {
        log.Println(n, err)
        return
    }

    println(string(buf[:n]))
}

For convenience, we use self-signed certificates here, and the client does not check the public key digital certificate of the server (we do not need to generate the key and certificate related to the creation of CA), we just need to use the following command to generate a pair of server.key and server.crt.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

$openssl genrsa -out server.key 2048
Generating RSA private key, 2048 bit long modulus
..........................+++
................................+++
e is 65537 (0x10001)

$openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []:localhost
Email Address []:

Run Server and Client, here I am using Go version 1.19 compiler.

1
2
3
4
5
6
7

$go run server.go
hello, world from client

EOF

$go run client.go
hello, world from server

Our example is already working! So how can we prove that the TLS connection between the client and server in the example is version 1.3? Or how to check which TLS version is used between client and server?

Some of you may say: use wireshark network tool, this works, but it is more laborious to use wireshark, especially for TLS 1.3. We have a simpler way, we can do it in our development environment by modifying the standard library. Let’s move on to the next page.

4. Selection of TLS version on server and client side

The TLS handshake process is initiated from the client side, and from the client’s point of view, when the client receives the response from serverHello it can get the TLS version to be used after the decision. So here we modify the clientHandshake method of crypto/tls/handshake_client.go and use fmt.Printf in its implementation to output information about the TLS connection (see the output in the following code starting with “====”).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

// $GOROOT/src/crypto/tls/handshake_client.go

func (c *Conn) clientHandshake(ctx context.Context) (err error) {
    ... ...
    hello, ecdheParams, err := c.makeClientHello()
    if err != nil {
        return err
    }
    c.serverName = hello.serverName

    fmt.Printf("====client: supportedVersions: %x, cipherSuites: %x\n", hello.supportedVersions, hello.cipherSuites)

    ... ...

    msg, err := c.readHandshake()
    if err != nil {
        return err
    }

    serverHello, ok := msg.(*serverHelloMsg)
    if !ok {
        c.sendAlert(alertUnexpectedMessage)
        return unexpectedMessageError(serverHello, msg)
    }

    if err := c.pickTLSVersion(serverHello); err != nil {
        return err
    }

    ... ...

    if c.vers == VersionTLS13 {
        fmt.Printf("====client: choose tls 1.3, server use ciphersuite: [0x%x]\n", serverHello.cipherSuite)
        ... ...
        // In TLS 1.3, session tickets are delivered after the handshake.
        return hs.handshake()
    }
    fmt.Printf("====client: choose tls 1.2, server use ciphersuite: [0x%x]\n", serverHello.cipherSuite)

    hs := &clientHandshakeState{
        ... ...
    }

    if err := hs.handshake(); err != nil {
        return err
    }
    ... ...
}

After modifying the standard library, let’s re-run the above client.go.

1
2
3
4

$go run client.go
====client: supportedVersions: [304 303], cipherSuites: [c02b c02f c02c c030 cca9 cca8 c009 c013 c00a c014 9c 9d 2f 35 c012 a 1301 1302 1303]
====client: choose tls 1.3, server use ciphersuite: [0x1301]
hello, world from server

Here we look at the first line of output, here the output is the client side of the construction of the clientHello handshake packet in the content, showing the client side of the supported TLS version and cipher suites (cipher suites), we see that the client supports 0 × 304, 0 × 303 two TLS version, these two numbers and the following code in the constants respectively.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// $GOROOT/src/crypto/tls/common.go
const (
    VersionTLS10 = 0x0301
    VersionTLS11 = 0x0302
    VersionTLS12 = 0x0303
    VersionTLS13 = 0x0304

    // Deprecated: SSLv3 is cryptographically broken, and is no longer
    // supported by this package. See golang.org/issue/32716.
    VersionSSL30 = 0x0300
)

And those hexadecimal numbers contained in the output cipherSuites are from the following constants.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

// $GOROOT/src/crypto/tls/cipher_suites.go
const (
    // TLS 1.0 - 1.2 cipher suites.
    TLS_RSA_WITH_RC4_128_SHA                      uint16 = 0x0005
    TLS_RSA_WITH_3DES_EDE_CBC_SHA                 uint16 = 0x000a
    TLS_RSA_WITH_AES_128_CBC_SHA                  uint16 = 0x002f
    TLS_RSA_WITH_AES_256_CBC_SHA                  uint16 = 0x0035
    TLS_RSA_WITH_AES_128_CBC_SHA256               uint16 = 0x003c
    TLS_RSA_WITH_AES_128_GCM_SHA256               uint16 = 0x009c
    TLS_RSA_WITH_AES_256_GCM_SHA384               uint16 = 0x009d
    TLS_ECDHE_ECDSA_WITH_RC4_128_SHA              uint16 = 0xc007
    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA          uint16 = 0xc009
    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA          uint16 = 0xc00a
    TLS_ECDHE_RSA_WITH_RC4_128_SHA                uint16 = 0xc011
    TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA           uint16 = 0xc012
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA            uint16 = 0xc013
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA            uint16 = 0xc014
    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256       uint16 = 0xc023
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256         uint16 = 0xc027
    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256         uint16 = 0xc02f
    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256       uint16 = 0xc02b
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384         uint16 = 0xc030
    TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384       uint16 = 0xc02c
    TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256   uint16 = 0xcca8
    TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 uint16 = 0xcca9

    // TLS 1.3 cipher suites.
    TLS_AES_128_GCM_SHA256       uint16 = 0x1301
    TLS_AES_256_GCM_SHA384       uint16 = 0x1302
    TLS_CHACHA20_POLY1305_SHA256 uint16 = 0x1303
    ... ...
}

The second line of output from client.go shows that for this build, both sides finally chose TLS version 1.3 and the cipher suite TLS_AES_128_GCM_SHA256. This is consistent with what we described earlier in our review of the history of TLS 1.3 support in Go, where TLS 1.3 was the default choice.

So can we choose which version to use when building a connection? Of course we can, we can configure it both on the server side and on the client side. Let’s see how to configure it on the server side first.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// https://github.com/bigwhite/experiments/blob/master/go-and-tls13/server_tls12.go

func main() {
    cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        fmt.Println(err)
        return
    }

    config := &tls.Config{
        Certificates: []tls.Certificate{cer},
        MaxVersion:   tls.VersionTLS12,
    }
    ... ...
}

We create server_tls12.go based on server.go, and in this new source file we add a configuration MaxVersion to tls.Config and set its value to tls.VersionTLS12, which means the maximum supported TLS version is TLS 1.2. This way when we use client.go to connect to a server-side application running on server_tls12.go, we will get the following output.

1
2
3
4

$go run client.go
====client: supportedVersions: [304 303], cipherSuites: [c02b c02f c02c c030 cca9 cca8 c009 c013 c00a c014 9c 9d 2f 35 c012 a 1301 1302 1303]
====client: choose tls 1.2, server use ciphersuite: [0xc02f]
hello, world from server

We see that both sides of the interaction end up choosing TLS version 1.2, using a cipher suite of 0xc02f, i.e. TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256.

Similarly, if you want to configure the highest support for TLS version 1.2 on the client side, you can also use the same way, you can look at the source file client_tls12.go in the corresponding code base of this article, so I will not repeat it here.

Here, some of you may have a question: we can configure the version of TLS to be used, but for TLS 1.3, can we configure the ciphersuites to be used? The answer is currently no, for the following reason: Config.CipherSuites field comment: Note that TLS 1.3 ciphersuites are not configurable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// $GOROOT/src/crypto/tls/common.go

type Config struct {
    ... ...
    // CipherSuites is a list of enabled TLS 1.0-1.2 cipher suites. The order of
    // the list is ignored. Note that TLS 1.3 ciphersuites are not configurable.
    //
    // If CipherSuites is nil, a safe default list is used. The default cipher
    // suites might change over time.
    CipherSuites []uint16
    ... ...
}

tls package will select cipher suites based on whether the system supports AES acceleration, if it supports AES acceleration, the following defaultCipherSuitesTLS13 will be used so that AES-related suites will be selected first, otherwise defaultCipherSuitesTLS13NoAES will be used and TLS_ CHACHA20_POLY1305_SHA256 will be selected first.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

// $GOROOT/src/crypto/tls/cipher_suites.go

// defaultCipherSuitesTLS13 is also the preference order, since there are no
// disabled by default TLS 1.3 cipher suites. The same AES vs ChaCha20 logic as
// cipherSuitesPreferenceOrder applies.
var defaultCipherSuitesTLS13 = []uint16{
    TLS_AES_128_GCM_SHA256,
    TLS_AES_256_GCM_SHA384,
    TLS_CHACHA20_POLY1305_SHA256,
}   

var defaultCipherSuitesTLS13NoAES = []uint16{
    TLS_CHACHA20_POLY1305_SHA256,
    TLS_AES_128_GCM_SHA256,
    TLS_AES_256_GCM_SHA384,
}

Note: joe shaw has written an article “Abusing go:linkname to customize TLS 1.3 cipher suites”, which describes a way to customize TLS 1.3 cipher suites via go:linkname. If you are interested, you can read it.

5. Benchmark of connection establishment speed

Finally, let’s see how much faster TLS 1.3 is in establishing a connection compared to TLS 1.2. Considering the difference in the number of RTTs between the two versions, i.e. the network latency has a large impact on the connection establishment speed, I deliberately chose a network with a ping of 20-30ms. We set up a Benchmark Test for TLS 1.2 and TLS 1.3 respectively.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

// https://github.com/bigwhite/experiments/blob/master/go-and-tls13/benchmark/benchmark_test.go

package main

import (
    "crypto/tls"
    "testing"
)

func tls12_dial() error {
    conf := &tls.Config{
        InsecureSkipVerify: true,
        MaxVersion:         tls.VersionTLS12,
    }

    conn, err := tls.Dial("tcp", "192.168.11.10:8443", conf)
    if err != nil {
        return err
    }
    conn.Close()
    return nil
}

func tls13_dial() error {
    conf := &tls.Config{
        InsecureSkipVerify: true,
    }

    conn, err := tls.Dial("tcp", "192.168.11.10:8443", conf)
    if err != nil {
        return err
    }
    conn.Close()
    return nil
}

func BenchmarkTls13(b *testing.B) {
    b.ReportAllocs()

    for i := 0; i < b.N; i++ {
        err := tls13_dial()
        if err != nil {
            panic(err)
        }
    }
}

func BenchmarkTls12(b *testing.B) {
    b.ReportAllocs()

    for i := 0; i < b.N; i++ {
        err := tls12_dial()
        if err != nil {
            panic(err)
        }
    }
}

server is deployed on 192.168.11.10, for each benchmark test, we give 10s of test time, the following are the results of the run.

1
2
3
4
5
6
7
8
9

$go test -benchtime 10s -bench .
goos: linux
goarch: amd64
pkg: demo
cpu: Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
BenchmarkTls13-8        216   56036809 ns/op   47966 B/op     608 allocs/op
BenchmarkTls12-8        145   82395933 ns/op   26655 B/op     283 allocs/op
PASS
ok  demo 37.959s

We see that connection establishment is indeed faster for TLS 1.3 compared to TLS 1.2. However, the implementation of Go TLS 1.3 seems to be a bit more complex in terms of memory allocation.

6. Ref

  • https://tonybai.com/2023/01/13/go-and-tls13/