Deploying an Elasticsearch stack on a Kubernetes cluster

If Logstash is used to receive log input from Filebeat centrally, it is easy to cause performance bottleneck due to single machine; if Kafka is used to receive log input from Filebeat, the timeliness of logs is not guaranteed. Here, the logs collected by Filebeat are directly output to Elasticsearch. 1. Preparation Node planning Instead of distinguishing master, data, and client nodes, three nodes of a cluster are directly multiplexed here.

Design and implementation of kube-apiserver

kube-apiserver is a component in kubernetes that interacts directly with etcd and controls changes to core resources in kubernetes. It provides the following main functions. Providing Kubernetes API, including authentication authorization, data validation, and cluster state changes, for clients and other components to call. Proxy some additional component components in the cluster, such as Kubernetes UI, metrics-server, npd, etc. Creating kubernetes services, i.e., Service that provides apiserver, kubernetes Service. conversion

Implementation of apiserver service in kube-apiserver

In kubernetes, the kubernetes API can be accessed from both outside and inside the cluster, directly outside the cluster by accessing the API provided by the apiserver, and inside the cluster by accessing the service as the ClusterIP for kubernetes. kubernetes clusters create a kubernetes service after initialization. A kubernetes service is created and maintained by kube-apiserver, as follows. 1 2 3 4 5 6 7 $ kubectl get service

knative serving component analysis

The created components can be seen under the knative-serving namespace after the deployment of knative. 1 2 3 4 5 6 7 8 9 10 $ kubectl get pod -n knative-serving NAME READY STATUS RESTARTS AGE activator-7fff689bcb-zt9pm 2/2 Running 2 28d autoscaler-5bcff95856-pr6nk 2/2 Running 3 28d autoscaler-hpa-75584dd678-fpk7w 2/2 Running 1 28d controller-bbdd78bc4-6cqm4 2/2 Running 1 28d istio-webhook-5f5794dcc4-sgzlj 2/2 Running 1 28d networking-istio-7d875675c7-gc55v 1/1 Running 0 28d storage-version-migration-f46wc 1/2 Running 2 28d webhook-68bb66b676-9xk4s 2/2 Running 11 28d Create knative serving First create a knative service for testing, the yaml file is shown below.

Blkio Cgroup

blkio cgroup basic functions blkio is a subsystem in cgroup v1. The main reason for using cgroup v1 blkio subsystem is to reduce the problem of mutual interference when processes read and write to the same disk together. The cgroup v1 blkio control subsystem can limit the IOPS and throughput of process reads and writes, but it can only limit the speed of file reads and writes for Direct I/O, but not for Buffered I/O.

How Pod Eviction Happens in kubernetes

Recently, we found that many instances on our network are in Evicted state. Through pod yaml, we can see that the instances are being evicted because of insufficient node resources. However, these instances are not automatically cleaned up, and most users of the platform will think that there is a problem with the service or the platform when they see Evicted instances under the service, which affects the user experience. The Pods in the Evicted state have been destroyed in the underlying container, which has no impact on the user’s service.

The difference between function return values and pointers in Golang

Variable memory allocation and recycling Go programs allocate memory for variables in two places, one is the global heap and the other is the function call stack. The Go language has a garbage collection mechanism, and it is up to the compiler to decide whether a variable is allocated on the heap or stack in Go, so developers don’t need to pay much attention to whether a variable is allocated

Golang program startup flow analysis

1. Flow of Golang code being run up by the OS 1.1. Compilation The go source code is first compiled into an executable file by go build, which is an ELF format executable file on linux platform, and the compilation stage will go through three processes: compiler, assembler, and linker to finally generate an executable file. Compiler: *.go source code is generated as plan9 assembly code for *.s by the

Analysis of Golang GPM Models

Differences between threads, kernel threads and user threads Threads: From the kernel’s point of view there is no such concept as threads. Linux implements all threads as processes, and the kernel has no special scheduling algorithm to handle threads. A thread is simply seen as a process that shares some resources with other processes. Like processes, each thread has its own task_struct, so in the kernel, a thread appears to

Using and Extending the Descheduler Component in Kubernetes

Descheduler component introduction When an instance is created, the scheduler can select the best node for scheduling based on the cluster state at that time, but the resource usage within the cluster is dynamically changing, and the cluster will become unbalanced over time, requiring the Descheduler to migrate the pods already running on the node to other nodes to achieve a more balanced distribution of resources within the cluster. There are several reasons why we want to migrate the instances running on a node to other nodes.

OIDC (OpenID Connect) Introduction

I recently learned about OIDC in the authentication system and put together a blog that covers the concepts, processes, and usage of OIDC. Before introducing these, we need to clarify two terms Authorization and Authentication. Authorization vs Authentication Here is OKTA’s definition of the two terms. Authentication, in the form of a key. The lock on the door only grants access to someone with the correct key in much the same way that a system only grants access to users who have the correct credentials.

Self-signed certificate

Introduction CA provides certificate to ensure the security of transmitted information. Of course, individuals can also play the role of CA, but the client is not trusted at this time, and the CA certificate, i.e. CA public key, needs to be integrated into the client. Information Security The security issues that need to be ensured during information transmission are: confidentiality of information, security of information, and identification of both parties.

Circuit Breaker Pattern

Introduction Circuit breaker mode, analogous to the circuit breaker mechanism in real circuits. When the line voltage is too high, the fuse will be broken, and the power can be restored after successful repair. Distributed scenarios also face service exceptions and network timeouts, which require a certain amount of time to recover. If the retry request is kept, it will return failure and take up resources during the unrecovered time.

WWW & Root Record

Concept WWW The World Wide Web is a system of many interlinked hypertexts accessible through the Internet. The World Wide Web was invented in 1989 by British scientist Tim Berners-Lee, who wrote the first web browser in 1990 while working at CERN in Switzerland. The Web browser was released outside CERN in 1991, first to other research institutions in January 1991, and made available to the public on the Internet in August 1991.

DNS & CoreDNS

DNS DNS (Domain Name System) is a global distributed database for storing the mapping relationship between domain names and Internet IP addresses. DNS is divided into two main categories: authoritative DNS, and recursive DNS. Authoritative DNS Authoritative DNS is a DNS server set up at the domain name registrar for a specific domain name record, and is used for the management of the specific domain name itself. It only resolves domain names that it owns, and denies access to domains that are not its own.

Task Queues

In many systems, to decouple, or to handle tasks that take a long time (for example, some network requests may be slow, or some requests are CPU-intensive and need to wait for a while), we usually introduce task queues. A typical task queue consists of the following three parts. The first part is the producer, there are two common ones, one is triggered by the user, for example, in web applications, the user needs to send a request to the mail provider when verifying the mailbox; the other one is triggered by the machine, for example, the timing task, I generally call it scheduler.

Develop a Hello World level eBPF program from scratch using C

The hottest Linux kernel technology in the last two years is none other than eBPF! Since 2019, in addition to the rapid evolution of eBPF technology itself, Observability, Security and Networking projects based on eBPF technology have sprung up. Familiar ones include cilium (bringing eBPF technology to the Kubernetes world), Falco (a de facto standard for Kubernetes threat detection engines when running cloud-native security), Katran (a high-performance four-tier load balancer), pixie (an observability tool for Kubernetes applications), and more.

Using consul as a registry for istio(intree or by service entry)

registry By default istio uses k8s as registry, k8s service, endpoint corresponds to service, instance. For some Spring Cloud services that are not yet connected to the Service Grid, the registry they use may be consul, how to make the Consumer service on the Service Grid to access the non-Service Grid Provider is a problem faced by the application during the Service Grid migration. istio itself provides some mechanisms to bring in external registries of services.

Consul Basics

consul introduction CP model. consul deployment on k8s 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 apiVersion: apps/v1 kind: Deployment metadata: labels: app: consul name: consul spec: selector: matchLabels: app: consul template: metadata: labels: app: consul spec: containers: - args: - agent - -enable-script-checks - -dev - -client - 0.0.0.0 image: consul:1.8.4 name: consul Service registration Configuration file method Specify the configuration directory via -config-dir.

Golang: Explaining container/heap

The heap container is provided in golang’s container package. What can this container be used for and how does it do it? This article explains the heap, the heap package, the uses of the heap package, and the implementation of the heap package, starting from the source code of golang 1.9.3. 1 What is heap Let’s start by explaining what a heap (Heap) is. According to Wikipedia Heap (Heap) is a generic term for a special class of data structures in computer science.