Questions tagged [oom-killer]
oom-killer Out-Of-Memory killer in the Linux kernel
43
questions
0
votes
1
answer
19
views
Adjust Logging Level for the OOM Killer
Is there a way to configure Linux so that the OOM killer will report/post only which process is killed, but not dump the entire machine state with it?
The reason I ask that is, that on our production ...
0
votes
0
answers
47
views
OOM Killer killing processes at 60% usage
Whenever my host hits 60% usage, OOM killer kicks in and kills one of my processes that has the highest oom score.
However, the memory on the host only hits 60%! Why is it kicking in so early? It is ...
- 1
1
vote
1
answer
298
views
Interpreting OOM killer logs when triggered on a Java thread
When Linux OOM Killer interrupts a process, the kernel logs usually provide enough information about the culprit's memory consumption (even it is not killed eventually). For example, when snmpd ...
- 113
0
votes
1
answer
686
views
Linux server loses network connectivity after an OOM event
We have a server runnnig Linux 5.15 and we've had multiple verified situations where a process is killed by OOM and that leaves the whole system inaccessible on the network, both for inbound and ...
- 101
0
votes
0
answers
391
views
Implications of CPU limit on a docker run Java (tomcat) application?
I have had an ongoing problem running a Tomcat java web application in a docker container (which I refer to as a 'task' in this post) hosted in an ECS (elastic container service) on AWS.
We notice ...
- 21
0
votes
1
answer
203
views
Out of Memory Mismatch compared to what system rerports
Oct 25 07:41:32 KVM-BOX kernel: memory: usage 255216608kB, limit 255216640kB, failcnt 28058
[root@KVM-BOX ~]# free -hm
total used free shared buff/cache available
...
3
votes
0
answers
536
views
Running out of memory with plenty of memory and swap
I have a Debian-based system that is running out of memory but there appears to be plenty of free memory. The box will run for about 6-12 days and then it will begin killing anything allocating memory ...
- 31
0
votes
0
answers
917
views
RHEL 8. linux OOM killer invoked to kill postgres process
Database is going to recovery mode whenever the OOM killer is invoked on the postgres process.
/var/log/messages output with respect to the issue
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),...
0
votes
1
answer
823
views
How do I track down a mysql memory issue on a server running a single WordPress site?
I'm having a weird issue. A few times a week, a website I'm running on Amazon Lightsail comes to a crashing halt and becomes IO bound (I can't usually use ssh when this happens) and the server runs ...
- 534
3
votes
4
answers
6k
views
mysqld crashing almost every hour - mysqld invoked oom-killer
I am running wordpress on a centOS digital ocean droplet
my mysql keeps crashing almost every hour -- with an out of memory error
manually restarting mysqld brings it back up via sudo service mysqld ...
- 31
0
votes
0
answers
297
views
Monitoring apache workers that are being terminated by oomkiller (kubernetes)
I have a docker container based on php:7.2-apache-stretch running in kubernetes/rancher. The current k8s memory limit is excessively low, so users get errors and truncated pages when worker processes ...
0
votes
1
answer
1k
views
Slab SReclaimable memory can not be recycled?
CentOS Linux release 7.2.1511 (Core)
Linux version 3.10.0-514.26.2.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP TueJul 4 15:04:05 UTC ...
0
votes
1
answer
985
views
Kill the most memory-hungry process when memory gets low [closed]
Sometimes my Linux desktop, laptop, or server run out of RAM because the experimental software I work with occasionally tries to use more RAM than I have available.
When this happens, the system and ...
- 281
0
votes
2
answers
1k
views
Kubernetes DaemonSet Pod exceeds memory limit but is not killed
We use kubernetes (specifically: openshift) to run our infrastructure in our team.
One of the daemonsets, fluentd, is currently causing a lot of trouble, frequently taking full nodes down with huge ...
- 360
2
votes
0
answers
85
views
What abbreviations in OOM Killer memory statistics report means?
When OOM Killer or kernel reports memory state it uses the next abbreviations
Node 0 DMA: 26*4kB (M) 53*8kB (UM) 33*16kB (ME) 23*32kB (UME) 6*64kB (ME) 7*128kB (UME) 1*256kB (M) 2*512kB (ME) 0*...
- 173
1
vote
0
answers
575
views
Linux out of memory killer kills process when there is more than enough memory available
We are running a 64-Bit Ubuntu and our 32GB of physical memory is split into 3 zones (DMA: 16MB, DMA32: 4GB and Normal: 30GB). According to the dmesg logs printed below our system ran out of free ...
2
votes
1
answer
7k
views
Why total-vm is is different than show in table?
I am running into oom_killer activity and cannot undesrand dmesg output
[Fri Jan 11 02:11:48 2019] lowmem_reserve[]: 0 2811 64249 64249
[Fri Jan 11 02:11:48 2019] Node 0 DMA32 free:248704kB min:...
- 123
8
votes
3
answers
6k
views
Please help interpret OOM-Killer
We were testing our java application and it was killed by the OOM Killer, logs below.
A couple of points: this is a java web application. It runs with a 190G heap. The server is a 240G total memory ...
- 211
0
votes
1
answer
424
views
What are the specific column headers for the "Mem-Info" dump after "oom-killer"
I'm running a Debian 8 Jessie development server.
16GB mem.
I've looked at the other posts about oom-killer, but none are really pertinent to what I'm asking here
I'm trying to track down what ...
- 535
0
votes
0
answers
422
views
Strange numbers in Oom killer log
First, the log:
[Wed aug 22 01:11:31 2018] java invoked oom-killer: gfp_mask=0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
[Wed aug 22 01:11:31 2018] java ...
- 123
2
votes
3
answers
844
views
Stop a process (script) from invoking oom killer
I had a very unfortunate situation, where a bash script that contained a subtle error gone wild and took all available memory and then started killing other tasks (production apps!) to get even more.
...
- 123
3
votes
2
answers
4k
views
OOM killer shut down MySQL although ram usage seems low
Setup
EC2 t2.large instance with Ubuntu 16.04 (2 Cores, 8GB RAM)
Apache/2.4.18 (Ubuntu)
5.7.20-0ubuntu0.16.04.1 (Ubuntu)
Currently this instance hosts multiple Drupal 6 & 7 sites, around 150 ...
- 31
7
votes
1
answer
14k
views
Is it recommended to set OOMScoreAdjust=1000 in systemd for a replaceable service?
systemd has the OOMScoreAdjust option, which allows to adjust the oom-killer score of the started process.
To quote from the systemd documentation:
OOMScoreAdjust=
Sets the adjustment level ...
- 561
1
vote
0
answers
779
views
Processes is getting killed randomly on REdHat
We have around java virtual machines with limited memory, and MongoDB.
Deployed processes are randomly getting killed/terminated abruptly in time. No matter java process, nodejs or mongodb process. We ...
- 11
0
votes
1
answer
384
views
Number of httpd processes and prefork.c
Hi I got OOM killer problems which caused:
[mpm_prefork:error] [pid 18103] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
I had to restart the server to get it working.
In /...
- 103
0
votes
1
answer
1k
views
VmRSS used to only about 25%, yet oom-killer strikes
I have a dedicated MySQL server equipped with 128 GB RAM. MySQL recently gets killed by the oom-killer, although MySQL is configured to use 95 GB in the worst case. In my research I came across this:
...
- 105
0
votes
1
answer
302
views
Kernel killing processes with memory available
We have a CentOS5.8 server with 64Gb RAM running normally around 700Mb RAM usage. When I try to do a tar -czf into about a 22Gb archive, the pagecache climbs up to about 22Gb, then the kernel starts ...
- 3
7
votes
2
answers
12k
views
How does Docker deal with OOM killer and memory limits?
I have a docker container which starts a simple java (jgroups-based) application through a bash script. The java process is limited through Xmx by 128m, the container is allowed to use 256m (swap is ...
- 239
3
votes
1
answer
2k
views
Adjust oom_score at process launch
I've got a few servers that have begun oom-killing their backup processes and, while I understand that encountering the oom condition is quite bad in itself, I need this process to not die so that ...
- 2,121
3
votes
1
answer
5k
views
Oracle invoked oom-killer even when RAM and SWAP is plenty free?
Can you please tell me what cause this Oracle process killed? Seems like plenty of RAM free, and plenty of SWAP free. There followed few other oracle processes killed.
The VM has the 16G of vMem and 8 ...
- 33
-2
votes
1
answer
155
views
Is there a way for Apache to control how much memory it reserves?
Apache seems to have no control over how much memory it reserves. In MPM prefork you control the maximum number of processes you want to allow, but whatever number you use there, there is always a ...
- 115
1
vote
1
answer
949
views
OOM score adjustements
On Ubuntu 12.04/Java 1.7
I have been reading/searching about OOM killer and generally understood what "proc" man page says especially oom_adj part. It all started because my important "MQ listeners" ...
- 185
67
votes
3
answers
8k
views
Rsync triggered Linux OOM killer on a single 50 GB file
I have a single 50 GB file on server_A, and I'm copying it to server_B. I run
server_A$ rsync --partial --progress --inplace --append-verify 50GB_file root@server_B:50GB_file
Server_B has 32 GB ...
- 833
0
votes
0
answers
585
views
OOM Killer not triggering?
I'm working in an environment which has just been upgraded to Amazon Linux 2015.03 from Debian 6, quite a big jump.
On the new platform a server instance has locked up a few times due to running out ...
- 300
2
votes
1
answer
15k
views
Understanding OOM killer logs
I run some processes inside docker container and I use memory limitation for this container. Sometimes some processes inside docker container got killed by OOM killer. I see that in syslog file:
...
- 223
10
votes
2
answers
3k
views
View full program command line arguments in OOM killer logs
Is it possible to view full program command line arguments in OOM killler logs ?
What I see now in /var/log/syslog is
Memory cgroup out of memory: Kill process 29187 (beam.smp) score 998 or sacrifice ...
- 223
5
votes
3
answers
7k
views
Mysql crashing, oom-killer, out of memory, tuning issues?
I just moved all of my websites to a new server with 4GB RAM. Almost immediately, mysql started crashing, and at one point, didn't reboot which caused a major outage (since I didn't notice until ...
- 287
8
votes
1
answer
3k
views
linux OOM-kill why?
.
Greetings, fellow admins. I post in the hope of shedding light on the OOM-kills that haunt one of my company's machines. I cannot decide if they are legit OOMs or not.
It's a Centos 6.x with kernel ...
- 81
20
votes
4
answers
15k
views
Linux oom situation (32 bit kernel)
I have continous oom&panic situation unresolved. I am not sure system fills up all the ram (36GB). Why this system triggered this oom situation? I suspect it as related to lowmem zone in 32 bit ...
- 698
1
vote
1
answer
143
views
On SmartOS how to tell if process was killed for OOM
A daemon process of mine gets killed almost immediately, as soon as it starts receiving requests. The application log has absolutely no error output, even when I try to run it in foreground.
I have ...
- 113
7
votes
1
answer
11k
views
MongoDB getting OOM killed
we are running a mongodb replicaset on three machines. All three machines have around 16GB but only 255MB Swap. Swappiness is left on it's default value 60. The machines are running CentOS 6.4. The ...
- 171
4
votes
1
answer
3k
views
Troubleshooting oom-killer using atop: Is it fixed or not?
I have a Debian system with 4GB of RAM, 2GB of swap and 2 CPUs. I've been experiencing a number of system failures involving oom-killer. The system will be working fine one minute, and the next will ...
- 115
16
votes
1
answer
14k
views
Scaling Logstash (with redis/elasticsearch)
Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server.
We tried using rsyslogd as the ...
- 27.5k