When the pod is in a crash state, the container keeps restarting and the kubelet logs may not capture the logs. Workaround:
The kubectl previous parameter works: If true, print the logs for the previous instance of the container in a pod if it exists.
-
Single-container pods:
1
|
kubectl logs pod-name --previous
|
-
Multi-container pods:
1
|
kubectl logs pod-name --previous -c container-name
|
For example:
1
2
3
4
5
|
NAME READY STATUS RESTARTS AGE
nginx-7d8b49557c-c2lx9 2/2 Running 5
kubectl logs nginx-7d8b49557c-c2lx9 --previous
Error: xxxxxxxxxxx
|
kubelet will keep the first few failed containers of the pod, which is a prerequisite for viewing. kubelet implements the principle of previous: the log of the pod is stored in /var/log/pods/podname and is a linked file to the log file of the container of docker, while kubelet will also keep the log file of the last There is also a link file to the log file of the pod’s last crashed container, which is the file that is viewed using previous
For example, viewing a pod:
1
2
3
4
5
|
ubuntu@~$ kubelet get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 2394 99d
nginx-deployment-6wlhd 1/1 Running 0 79d
redis 1/1 Running 0 49d
|
Go to the node where the pod is located and look at the two log files put in by the kubelet:
1
2
|
ls /var/log/pods/default_busybox_f72ab71a-5b3b-4ecf-940d-28a5c3b30683/busybox
2393.log 2394.log
|
The meaning of the numbers: 2393 proves to be the log after the 2393rd reboot, and 2394 represents the log after the 2394th reboot.
The two log files are actually linked files that point to the docker log files.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
/busybox# stat 2393.log
File: 2393.log -> /data/kubernetes/docker/containers/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca-json.log
Size: 173 Blocks: 8 IO Block: 4096 symbolic link
Device: fc02h/64514d Inode: 529958 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-01-31 13:32:03.751514283 +0800
Modify: 2023-01-31 13:32:03.039526838 +0800
Change: 2023-01-31 13:32:03.039526838 +0800
Birth: -
/busybox# stat 2394.log
File: 2394.log -> /data/kubernetes/docker/containers/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c-json.log
Size: 173 Blocks: 8 IO Block: 4096 symbolic link
Device: fc02h/64514d Inode: 529955 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-01-31 14:32:03.991106950 +0800
Modify: 2023-01-31 14:32:03.183119308 +0800
Change: 2023-01-31 14:32:03.183119308 +0800
Birth: -
|
See the log files that point to each of these two containers, one that is currently running in the pod and one that was last run by the pod and has now been exited.
1
2
3
4
|
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ed9ebf05852 ff4a8eb070e1 "sleep 3600" 24 minutes ago Up 24 minutes k8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2394
68a5b32c9fdb ff4a8eb070e1 "sleep 3600" About an hour ago Exited (0) 24 minutes ago k8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2393
|
When using logs, the current container file is read, and when using -previous, the log file of the last exiting container is read, since the kubelet keeps the last exiting container for pods.
We manually edit the contents of these two files to see if the kubelet is reading them.
1
2
3
4
5
6
7
8
9
10
|
/busybox# cat 2393.log
{"log":"last crash logs\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
/busybox# cat 2394.log
{"log":"now pod log\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
ubuntu@10-234-32-51:~$ k logs busybox --previous
last crash logs
ubuntu@10-234-32-51:~$ k logs busybox
now pod log
|
As it is a linked file, then it may actually be read from somewhere else, or rather directly from the container directory. As the linked file we changed was followed by the log file in the container directory, we created two files directly to do the verification.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
ubuntu@10-234-32-51:~$ k get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 2395 99d
nginx-deployment-6wlhd 1/1 Running 0 79d
redis 1/1 Running 0 49d
/busybox# ls
2394.log 2395.log
/busybox# rm 2394.log 2395.log
我们删除，然后自己创建，这时是regular file，而不是链接文件了：
/busybox# ls
2394.log 2395.log
/busybox# stat 2394.log
File: 2394.log
Size: 100 Blocks: 8 IO Block: 4096 regular file
Device: fc02h/64514d Inode: 529965 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-01-31 15:42:11.307170422 +0800
Modify: 2023-01-31 15:42:07.711225229 +0800
Change: 2023-01-31 15:42:07.711225229 +0800
Birth: -
/busybox# stat 2395.log
File: 2395.log
Size: 86 Blocks: 8 IO Block: 4096 regular file
Device: fc02h/64514d Inode: 529967 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-01-31 15:41:17.539989934 +0800
Modify: 2023-01-31 15:41:14.348038586 +0800
Change: 2023-01-31 15:41:14.352038525 +0800
Birth: -
/busybox# cat 2394.log
{"log":"previous logs create by myself\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
/busybox# cat 2395.log
{"log":"create by myself\n","stream":"stderr","time":"2022-11-05T08:11:27.31523845Z"}
ubuntu@10-234-32-51:~$ k logs busybox
create by myself
ubuntu@10-234-32-51:~$ k logs busybox --previous
previous logs create by myself
|
Conclusion: kubelet reads the log files under /var/log/pods/, -previous reads the log files under /var/log/pods/, and has a special link file to the log file of the last exiting container to get the logs before the container crashes.