Redundant DNS lookups Some applications that need to resolve external DNS domains, when running in a container, if we catch packets in the container’s network namespace for dns messages (udp port 53), we may find that several redundant attempts are made before they resolve correctly. Here are the packets I grabbed while ping google.com in the container’s network namespace. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 sudo nsenter -t 3885 -n tcpdump -i eth0 udp port 53 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:09:11.

Pod prioritization, preemption Pod prioritization and preemption, introduced in kubernetes v1.8, entered beta status in v1.11, and entered GA phase in v1.14, is already a mature feature. As the name suggests, the Pod priority, preemption feature, by subdividing applications into different priorities, prioritizes resources to high-priority applications, thus improving resource availability while guaranteeing the quality of service for high-priority applications. Let’s use the Pod priority and preemption function briefly. Ibu’s

Go has better ecological support and a smaller binary size than Kotlin Native. Why Use Go Mobile Go has better ecological support and a smaller binary size than Kotlin Native. Although the Go Mobile maintainers are suspected of running away, we have support for Apple Silicon and Catalyst through third-party Fork. Here’s a SDK with the same NASA API as in Developing Cross-Platform Library with Kotlin Native to see how it works.

JSX stands for javascript XML, a way to write HTML structures in javascript code. Advantages Uses HTML-like syntax. Leverage js’ own programmability to create HTML structures. Usage Basic instructions You need to use babel for syntax conversion, and for react the following code is equivalent. 1 2 3 4 5 function render() { return (<div id='d'> <p>hello world</p> </div>) } 1 2 3 4 5 6 7 function render() {

The cross-platform principle of Kotlin Native Kotlin Native’s cross-platform is pretty much all-inclusive. JVM JS Android / Android NDK Apple Linux Windows WebAssembly In short, although Kotlin can run on JVM and call Java code, Kotlin is not Java, and with the help of LLVM, Pure Kotlin Code can be compiled into With LLVM, Pure Kotlin Code can be compiled into platform code to achieve VM-less cross-platform. It can be

When I was developing my own project, I needed to use MySQL for unit testing, so I started a MySQL container locally using docker, imitating the way of go-txdb. When I executed the test, the following error occurred. 1 2 mysql: [Warning] Using a password on the command line interface can be insecure. ERROR 2059 (HY000): Authentication plugin 'caching_sha2_password' cannot be loaded: dlopen(/usr/local/mysql/lib/plugin/caching_sha2_password.so, 2): image not found Cause Analysis Conclusion: The local client version is too low and does not support the server-side version of the authentication method.

When we want to start a series of interdependent services at the same time, it is particularly tedious to start them one by one and in strict order. This is where we can use docker compose to perform this series of operations. Compose Compose is a tool for defining and running multi-container Docker applications. With Compose you can use a YML file to configure all the services your application needs, and then create and start all the services from the YML file configuration with a single command.

We often check the DNS information of an IP or domain name at work to see which machine the current IP is bound to, or if the current domain name resolves properly. This is where the nslookup command comes into play. Command Details The nslookup command is mainly used to query the DNS information of a domain name. nslookup has two working modes: “interactive mode” and “non-interactive mode”. Enter nslookup directly in the command line, and you will enter the interactive mode without entering any parameters.

Background When upgrading cilium v1.8.1 to v1.11.1, the business pod reported a mysql authorization error, and after checking, we found that the clientIP of mysql server is the nodeIP of the business pod, not the default podIP, because mysql server only authorized the current K8s cluster The mysql server only authorizes the pod cidr of the current K8s cluster, so it reports an authorization error. The contradiction is that when using cilium v1.

Problem phenomenon One day I encountered a problem: when accessing a web service, some requests failed, and Connection Refused was returned. For some reason, the web service is a hostNetwork type, which is the same network namespace as the host, and when I logged into Node, I could see that the listening socket was still there, but when I made a direct curl request, it returned Connection Refused. Why? Usually Connection Refused means that the listening socket is not open and the corresponding port number is not opened, the kernel will return icmp-port-unreachable, but now it is obvious that the port is open.

When reading kubernetes code, sometimes you will see some code compares arrays to nil. 1 2 3 4 5 6 7 8 9 // bindAPIUpdate gets the cached bindings and PVCs to provision in podBindingCache // and makes the API update for those PVs/PVCs. func (b *volumeBinder) bindAPIUpdate(podName string, bindings []*bindingInfo, claimsToProvision []*v1.PersistentVolumeClaim) error { if bindings == nil { return fmt.Errorf("failed to get cached bindings for pod %q", podName) } if claimsToProvision == nil { return fmt.

Problem phenomenon Our kubernetes ingress controller is using ingress-nginx from kubernetes and we recently encountered a " Too many open files" problem. 1 2 3 2019/09/19 09:47:56 [warn] 26281#26281: *97238945 a client request body is buffered to a temporary file /var/lib/nginx/body/0000269456, client: 1.1.1.1, server: xxx.ieevee.com, request: "POST /api/v1/xxx HTTP/1.1", host: "xxx.ieevee.com" 2019/09/19 09:47:56 [crit] 26281#26281: accept4() failed (24: Too many open files) 2019/09/19 09:47:56 [crit] 26281#26281: *97238948 open() "/var/lib/nginx/body/0000269457" failed

Problem phenomenon We have some GPU machines and need to count GPU related information, the data is taken from Prometheus, but one day we suddenly found that some labels of some GPU node metrics are empty. Here are the metrics of normal nodes. 1 container_accelerator_duty_cycle{acc_id="GPU-d8384090-81d2-eda5-ed02-8137eb037460",container_name="nvidia-device-plugin-ctr",endpoint="https-metrics",id="/kubepods/besteffort/podd917e00f-a779-11e9-b971-6805ca7f5b2a/38a5decf96e9f007fcb0059d79017ea3b3c29ff4c9a81b7fec86cf63c06baf53",image="sha256:7354b8a316796fba0463a68da79d79eb654d741241ca3c4f62a1ef24d8e11718",instance="10.0.0.1:10250",job="kubelet",make="nvidia",model="Tesla P40",name="k8s_nvidia-device-plugin-ctr_nvidia-device-plugin-daemonset-sn2cg_lambda_d917e00f-a779-11e9-b971-6805ca7f5b2a_1",namespace="ieevee",node="x1.ieevee.com",pod_name="nvidia-device-plugin-daemonset-sn2cg",service="kubelet"} 0 Here are the metrics of the exception node. 1 container_accelerator_duty_cycle{acc_id="GPU-a7b535d0-d6ca-022c-5b23-1bff863646a4",container_name="",endpoint="https-metrics",id="/kubepods/besteffort/pod8bb25662-de9a-11e9-84e7-f8f21e04010c/cde3858becb05366e71f230e876204be586662f274dcb4a6e2b75ea404f2d5a9",instance="10.0.0.2:10250",job="kubelet",make="nvidia",model="Tesla V100-PCIE-16GB",name="",namespace="",pod_name=""} You can see that the data taken out by Prometheus, the container_name, name, namespace, and pod_name in the label are all empty.

Problem description Users can use shared memory to do some for communication (vs golang’s “Shared memory by communication”). On a kvm or physical machine, the size of shared memory available to the user is about half of the total memory. Below is the shared memory on my pve machine, /dev/shm is the size of the shared memory. 1 2 3 4 5 6 7 8 root@pve:~# free -h total used free shared buff/cache available Mem: 47Gi 33Gi 3.

A pod running on a kubernetes cluster is easy to access from within the cluster, most simply, through the pod’s ip, or through the corresponding svc. However, outside the cluster, the pod ip of the flannel-based kubernetes cluster is not accessible from outside the cluster because it is an internal address. To solve this problem, kubernetes provides several methods as follows. hostNetwork: true When hostNetwork is true, the container will use the network of the host node, so the container’s services can be accessed from outside the cluster as node-ip + port, as long as you know which node the container is running on.

Requirements Overview Previously, some services hosted on Kubernetes at home, such as portal, emby, weave scope, etc., were accessed using service ip, which was slightly troublesome to access, mainly because the ip had to be remembered. Kubernetes provides Ingress to solve the problem of load balancer type service (vip consumption, L7 load feature, etc.). For this requirement, you can set up pan domain on godaddy with type A, for example *.

Overview In Kubernetes, kube-controller-manager, kube-scheduler, and the underlying implementation of controller-rumtime using Operator all support leader election in highly available systems. This article will focus on understanding how the leader election in controller-rumtime (the underlying implementation is client-go) is implemented in the kubernetes controller. Background When running kube-controller-manager, there are some parameters provided to cm for leader election, you can refer to the official documentation parameters to understand the parameters.

Background When we add business logs to our application code, regardless of the level of logging, in addition to the information that we actively pass to Logger for it to log, it is also very important to know which function printed the line and where it is located, otherwise it is likely to be like looking for a needle in a haystack when troubleshooting. For logging, it is important to record the function name and line number of the caller of the Logger method.

Inspired by PromQL, Loki also has its own query language, called LogQL, which is like a distributed grep that aggregates views of logs. Like PromQL, LogQL is filtered using tags and operators, and has two main types of query functions. Query to return log line contents Calculating relevant metrics in the log stream by filtering rules Log queries A basic log query consists of two parts. log stream selector log pipeline Due to the design of Loki, all LogQL queries must contain a Log Stream selector.

Problem description When containerizing nginx, there is a common problem: How do I automatically set the number of nginx worker processes? In the nginx.conf configuration file of the official nginx container image, there is a worker process configuration. 1 worker_processes 1; It will configure nginx to start only 1 worker. this works well when the nginx container is 1 core. When we want nginx to give a higher configuration, for example 4c or 16c, we need to make sure that nginx can also start the corresponding number of worker processes.