I recently ran into a problem.

Our kube-apiserver is configured with OIDC authentication and the OIDC issuer is added with dns server records, but for some reason I need to override the dns server resolution and use the hostAlias IP address instead, but the actual test found that it always took DNS resolution, although the /etc/hosts file file has been added with custom hosts records. The domain names that are not registered with the dns server can still be resolved by /etc/hosts.

The reason is that the base image of kube-apiserver is busybox, and unlike centos, it does not have /etc/nsswitch.conf file, so it always gives priority to DNS resolution and ignores /etc/hosts file.

The solution is simple, just add /etc/nsswitch.conf file to the mirror to specify the resolution order, the content is as follows.

1

hosts:          files dns

That is, files takes precedence over dns.

By the way, let’s organize the complete process of golang resolving domain name in linux system.

There are two methods of domain name resolution in golang: the built-in Go resolver; and the cgo-based system resolver. Configured through the environment variable GODEBUG.

1
2

export GODEBUG=netdns=go    # force pure Go resolver
export GODEBUG=netdns=cgo   # force cgo resolver

The built-in Go resolver is used by default, because when DNS resolution blocks, the built-in Go resolver just blocks a goroutine, while cgo’s resolver blocks an OS-level thread.

1

func init() { netGo = true }

Read resolv.conf and force cgo if it fails.

1
2
3
4
5
6
7
8
9

confVal.resolv = dnsReadConfig("/etc/resolv.conf")
if confVal.resolv.err != nil && !os.IsNotExist(confVal.resolv.err) &&
    !os.IsPermission(confVal.resolv.err) {
    // If we can't read the resolv.conf file, assume it
    // had something important in it and defer to cgo.
    // libc's resolver might then fail too, but at least
    // it wasn't our fault.
    confVal.forceCgoLookupHost = true
}

When using the built-in Go parser, there are four other subdivisions depending on the parsing priority.

1
2
3
4
5
6
7
8

const (
    // hostLookupCgo means defer to cgo.
    hostLookupCgo      hostLookupOrder = iota
    hostLookupFilesDNS                 // files first
    hostLookupDNSFiles                 // dns first
    hostLookupFiles                    // only files
    hostLookupDNS                      // only DNS
)

When the /etc/nsswitch.conf file does not exist or does not specify the hosts field, hostLookupDNSFiles is used under linux, which means that dns resolution takes precedence over hosts resolution, so the problem mentioned at the beginning of the article occurs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

nss := c.nss
srcs := nss.sources["hosts"]
// If /etc/nsswitch.conf doesn't exist or doesn't specify any
// sources for "hosts", assume Go's DNS will work fine.
if os.IsNotExist(nss.err) || (nss.err == nil && len(srcs) == 0) {
    if c.goos == "linux" {
        // glibc says the default is "dns [!UNAVAIL=return] files"
        // http://www.gnu.org/software/libc/manual/html_node/Notes-on-NSS-Configuration-File.html.
        return hostLookupDNSFiles
    }
    return hostLookupFilesDNS
}

The parsing order can be specified via nsswitch.conf. The code is quite simple.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

var mdnsSource, filesSource, dnsSource bool
var first string
for _, src := range srcs {
    if src.source == "files" || src.source == "dns" {
        if !src.standardCriteria() {
            return fallbackOrder // non-standard; let libc deal with it.
        }
        if src.source == "files" {
            filesSource = true
        } else if src.source == "dns" {
            dnsSource = true
        }
        if first == "" {
            first = src.source
        }
        continue
    }
    // Some source we don't know how to deal with.
    return fallbackOrder
}

// Cases where Go can handle it without cgo and C thread
// overhead.
switch {
case filesSource && dnsSource:
    if first == "files" {
        return hostLookupFilesDNS
    } else {
        return hostLookupDNSFiles
    }
case filesSource:
    return hostLookupFiles
case dnsSource:
    return hostLookupDNS
}

So by specifying hosts: files dns, the resolution policy is hostLookupFilesDNS, i.e. /etc/hosts is used first.

See hostLookupOrder for the detailed resolution order.