Sometimes you may find that K8S fails to create a Pod with the following error message.
1
2
3
|
OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:385:
applying cgroup configuration for process caused: mkdir /sys/fs/cgroup/memory/kubepods/burstable/podxxx/xxx:
cannot allocate memory: unknown"
|
You can see the same error in the docker logs.
1
2
3
4
5
6
|
$ journalctl -u docker
Mar 20 00:04:33 k8sworker06-new dockerd[3176]: time="2023-03-20T00:04:33.461528517+08:00" level=error
msg="Handler for POST /v1.40/containers/bc91b4fd862386647df69cb636f779c05eb034d9e5db2ab527b51b90f128a5df/start
returned error: OCI runtime create failed: container_linux.go:380: starting container process caused:
process_linux.go:385: applying cgroup configuration for process caused:
mkdir /sys/fs/cgroup/memory/kubepods/burstable/podxxx/xxx: cannot allocate memory: unknown"
|
Refer to this article which mentions:
- There is a bug in kmem allocation for cgroups in linux kernel 3.10.xxx that can have memory leak problem.
- If you create and destroy containers frequently and the memory leak reaches a certain level, the above situation will occur.
In fact, you can reproduce this problem like this:
1
2
|
$ mkdir /sys/fs/cgroup/memory/test
cannot allocate memory
|
Delete this directory by rmdir /sys/fs/cgroup/memory/test
So the solution is:
- CentOS 7.x kernel is 3.10.xxx, upgrade CentOS to 8.x kernel will become 4.x to solve the problem
- Or restart the server (restarting the docker service is useless, because the problem is with the cgroup)