When solving the 5-second timeout problem with CoreDNS, we mentioned using NodeLocal DNSCache to solve the problem, in addition to forcing tcp resolution through dnsConfig. NodeLocal DNSCache improves clusterDNS performance and reliability by running a DaemonSet on the cluster nodes. Pods in ClusterFirst’s DNS mode can connect to kube-dns’s serviceIP for DNS queries. This is converted to a CoreDNS endpoint via iptables rules added by the kube-proxy component. By running DNS caching on each cluster node, NodeLocal DNSCache reduces the latency of DNS lookups, makes DNS lookup times more consistent, and reduces the number of DNS queries sent to kube-dns.

There are several benefits to running NodeLocal DNSCache in a cluster.

• Pods with the highest DNS QPS may have to resolve to another node if there is no local CoreDNS instance, and having a local cache will help improve latency when using NodeLocal DNSCache
• Skipping iptables DNAT and connection tracking will help reduce conntrack contention and prevent UDP DNS entries from filling up the conntrack table (a common 5s timeout problem is caused by this)
• connections from the local cache proxy to the kube-dns service can be upgraded to TCP, TCP conntrack entries will be removed when the connection is closed, while UDP entries must time out (default nf_conntrack_udp_timeout is 30 seconds)
• Upgrading DNS queries from UDP to TCP will reduce the tail wait time attributed to dropped UDP packets and DNS timeouts, typically up to 30 seconds (3 retries + 10 seconds timeout)

To install NodeLocal DNSCache is also very simple, just get the official list of resources directly from

 1  $wget https://github.com/kubernetes/kubernetes/raw/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml  This resource manifest file contains several variables, among them. • __PILLAR__DNS__SERVER__ : indicates the ClusterIP of the service kube-dns, which can be obtained with the command kubectl get svc -n kube-system | grep kube-dns | awk '{ print$3 }'
• __PILLAR__LOCAL__DNS__: indicates the local IP of DNSCache, default is 169.254.20.10
• __PILLAR__DNS__DOMAIN__ : indicates the cluster domain, the default is cluster.local

There are also two parameters __PILLAR__CLUSTER__DNS__ and __PILLAR__UPSTREAM__SERVERS__, which are configured by mirroring 1.15.6 and above, and the corresponding values come from kube-dns ConfigMap and the custom Upstream Server configuration. The installation is done directly by executing the command shown below.

 1 2 3 4 5  $sed 's/k8s.gcr.io/cnych/g s/__PILLAR__DNS__SERVER__/10.96.0.10/g s/__PILLAR__LOCAL__DNS__/169.254.20.10/g s/__PILLAR__DNS__DOMAIN__/cluster.local/g' nodelocaldns.yaml | kubectl apply -f -  You can check if the corresponding Pod has started successfully by using the following command.  1 2 3 4 5 6 7 8  $ kubectl get pods -n kube-system | grep node-local-dns node-local-dns-8zm2f 1/1 Running 0 9m54s node-local-dns-dd4xg 1/1 Running 0 9m54s node-local-dns-hs8qq 1/1 Running 0 9m54s node-local-dns-pxfxn 1/1 Running 0 9m54s node-local-dns-stjm9 1/1 Running 0 9m54s node-local-dns-wjxvz 1/1 Running 0 9m54s node-local-dns-wn5wc 1/1 Running 0 7m49s 

Note that node-local-dns is deployed with DaemonSet using hostNetwork=true, which will take up port 8080 of the host, so you need to make sure that port is not taken.

If the kube-proxy component is using ipvs mode, we need to modify the -cluster-dns parameter of the kubelet to point to 169.254.20.10, Daemonset will create a NIC in each node to tie this IP, and the Pod will send a The Pod will send DNS requests to this IP, and only when the cache does not hit will it proxy to the upstream cluster DNS for queries. In iptables mode, Pod still requests to the original cluster DNS, and the node has this IP listening, which will be intercepted by the local machine and then request the upstream cluster DNS, so there is no need to change the --cluster-dns parameter.

Since I’m using a kubeadm installation of version 1.16 of the cluster here, we just need to replace the value of the parameter clusterDNS in the /var/lib/kubelet/config.yaml file on the node and restart it, or we can add an official DaemonSet resource object entirely to the initContainer in the official DaemonSet resource object to do this:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24  initContainers: # ipvs模式下需要修改dns配置，重启kubelet - name: setup image: alpine tty: true stdin: true securityContext: privileged: true command: - nsenter - --target - "1" - --mount - --uts - --ipc - --net - --pid - -- - bash - -c - | # 确保 kubelet --cluster-dns 被设置为 169.254.20.10 echo "Configuring kubelet --cluster-dns=169.254.20.10" sed -i 's/10.96.0.10/169.254.20.10/g' /var/lib/kubelet/config.yaml systemctl daemon-reload && systemctl restart kubelet 

However, it should be noted that the above approach is not recommended for online environments, as it will give priority to modifying the cluster-dns parameter of the kubelet and then installing NodeLocal, which after all has a vacuum period in which we can manually go node by node to verify.

 1 2  $sed -i 's/10.96.0.10/169.254.20.10/g' /var/lib/kubelet/config.yaml$ systemctl daemon-reload && systemctl restart kubelet 

After the node-local-dns installation and configuration is complete, we can deploy a new Pod to verify: (test-node-local-dns.yaml)

 1 2 3 4 5 6 7 8 9  apiVersion: v1 kind: Pod metadata: name: test-node-local-dns spec: containers: - name: local-dns image: busybox command: ["/bin/sh", "-c", "sleep 60m"] 

Direct Deployment.

 1 2 3 4 5 6  $kubectl apply -f test-node-local-dns.yaml$ kubectl exec -it test-node-local-dns /bin/sh / # cat /etc/resolv.conf nameserver 169.254.20.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5 

We can see that nameserver has become 169.254.20.10. Of course, if you want to use node-local-dns for the previous history of Pod, you need to rebuild it.