Long-running clusters often face a variety of resource exhaustion problems, in addition to insufficient disk Kubelet will also actively clean up the image to increase uncertainty, this article provides some command snippets for cleanup work.

1. Kubernetes Base Object Cleanup

  • Clean up Pods in Evicted state
1

kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n
  • Clean up Pods in Error state
1

kubectl get pods --all-namespaces -o wide | grep Error | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n
  • Clearing the Completed state of Pods
1

kubectl get pods --all-namespaces -o wide | grep Completed | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n
  • Clean up unused PVs
1

kubectl describe -A pvc | grep -E "^Name:.*$|^Namespace:.*$|^Used By:.*$" | grep -B 2 "<none>" | grep -E "^Name:.*$|^Namespace:.*$" | cut -f2 -d: | paste -d " " - - | xargs -n2 bash -c 'kubectl -n ${1} delete pvc ${0}'
  • Clear PVCs that are not bound
1

kubectl get pvc --all-namespaces | tail -n +2 | grep -v Bound | awk '{print $1,$2}' | xargs -L1 kubectl delete pvc -n
  • Clear the PVs that are not bound
1

kubectl get pv | tail -n +2 | grep -v Bound | awk '{print $1}' | xargs -L1 kubectl delete pv

2. Linux Cleanup

  • View all disk space
1
2
3
4

df -hl /

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       100G   47G   54G  47% /
  • View specified directory occupancy
1
2
3

du -sh .

24G .
  • Delete the folder with the specified prefix
1
2

cd /nfsdata
ls | grep archived- |xargs -L1 rm -r
  • Clean up zombie processes
1

ps -A -ostat,ppid | grep -e '^[Zz]' | awk '{print }' | xargs kill -HUP > /dev/null 2>&1

3. Docker Cleanup

  • View disk usage
1
2
3
4
5
6
7

docker system df

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              361                 23                  178.5GB             173.8GB (97%)
Containers          29                  9                   6.682GB             6.212GB (92%)
Local Volumes       4                   0                   3.139MB             3.139MB (100%)
Build Cache         0                   0                   0B                  0B
  • Clean up none mirrors
1

docker images | grep none | awk '{print $3}' | xargs docker rmi
  • Clean up data volumes that are no longer in use
1

docker volume rm $(docker volume ls -q)

or

1

docker volume prune
  • Clear the cache
1

docker builder prune
  • Complete cleanup

Remove closed containers, useless storage volumes, useless networks, dangling images (no tag images)

1

docker system prune -f
  • Clean up the mirrors on the regular match

This is a cleanup of master-8bcf8d7-20211206-111155163 format mirrors.

1

docker images |grep -E "([0-9a-z]*[-]){3,}[0-9]{9}" |awk '{print $3}' | xargs docker rmi

4. Set Timing

  • View timed tasks
1

crontab -l
  • Set up timed tasks
1

crontab -e

Text added timed tasks

1
2

*/35 */6 * * *  /usr/bin/docker images | grep none | awk '{print $3}' | xargs /usr/bin/docker rmi
45 1 * * * /usr/bin/docker system prune -f

Here the first task is executed at the 35th minute of every six hours and the second task is executed at 1:45 every day. Note that the command needs to use an absolute path, otherwise it may fail to execute.

  • Timed task format

Set the timing format: * * * * * * shell

The first asterisk, minute, is 0-59 The second asterisk, hour, has a value from 0-23 The third asterisk, day, has a value from 1-31 The fourth asterisk, month, has a value from 1-12 months, or abbreviated English, such as Nov, Feb, etc. The fifth asterisk, week, has values from 0-6 or abbreviated English, such as Wen, Tur, etc., representing the week days, where 0 is the weekend