My server (fedora) stops all podman containers after 2-3 hours since 3 days. I can start all containers again, and the same happens after a while. I do not know where to look for the problem.
In top, I found a oom message. I assume that the system runs out of memory and stops all services. How can I find the problem? I can’t find anything in the container logs.
I can see that systemctl status is always starting. It doesn’t become “running”. But I do not know how to proceed.
If you’re seeing an OOM killer messsage note that it doesn’t necessarily kill the problem process, by default the kernel hands out memory upon requestt, regardless of whether it has ram to back the allocation. When a process then writes to the memory (at some later time) and the kernel determines that there is no physical ram to store that write, it then invokes OOM Killer. This then selects a process and kills it. MySQL (and MariaDB) use large quantities of ram for cache, and by default the kernel lies about how much is available, so they often end up using more than the system can handle.
If you have many databases in containers, set memory limits for those containers, that should make all the databases play nicer together. Additionally , you may want to disable
overcommitin the kernel, this will cause the kernel to return out of memory to a process attempting to allocate ram and stop lying about free ram to processes that ask, often greatly increasing stability.