Questions tagged [oom]
Linux Out-of-memory Killer
108
questions
0
votes
1
answer
19
views
Adjust Logging Level for the OOM Killer
Is there a way to configure Linux so that the OOM killer will report/post only which process is killed, but not dump the entire machine state with it?
The reason I ask that is, that on our production ...
0
votes
0
answers
47
views
OOM Killer killing processes at 60% usage
Whenever my host hits 60% usage, OOM killer kicks in and kills one of my processes that has the highest oom score.
However, the memory on the host only hits 60%! Why is it kicking in so early? It is ...
- 1
-1
votes
1
answer
92
views
AWS EC2 instance running out of memory running NodeJs application, cannot identify root cause because of log suppression
As stated in the title we have a NodeJS application running on an EC2 instance on AWS which sporadically runs out of memory and kills the process.
Business wise it is not really a hindrance since ...
1
vote
0
answers
317
views
OOM killer beeing invoked by snapd systemd timer, CPU usage locks at 100% (snapd, mount, certbot)
Im having trouble debugging why my nodeJS webapp gets killed by the OOM killer in Debian 11. Im working on a small VPS (VMware ESX 7.0.3 build-21313628) by IONOS with single core 1GHz cpu, 512MB RAM, ...
0
votes
0
answers
58
views
How to find memory consuming request in mongodb
There is a small(4 replicas) mongodb cluster(replicaset). Every node is 24GB ram with 8GB wiredtiger cache size.
Cluster is used by many (about 100) applications which do few hundred requests per ...
- 1,056
0
votes
0
answers
283
views
Why does OOM killer works although memory is enough?
On one of the worker nodes of the Kubernetes cluster, a situation occurs periodically (several times a day) when the OOM killer is triggered, killing the "manager" process
I assume it's ...
- 11
3
votes
3
answers
789
views
How to run DNF without going OOM on GCP e2-micro
I'm trying to run Fedora Cloud Base 36 on an e2-micro in the US central region of Google Cloud Platform (GCP). I haven't upgraded packages in a couple months so I tried to do that today. When I run:
$ ...
google-cloud-platform
2
votes
0
answers
3k
views
Explain of OOM killer logs
I have a question about the OOM killer logs. We are expecting a lot of OOM kills.
The ecosystem
My ecosystem looks like below:
I have a server with 4 cores and 8 GB of RAM.
I am running there the ...
- 152
0
votes
1
answer
789
views
Why MongoDB occurs out of memory (OOM) every few days
My Server Info
AWS Ubuntu 18.04
MongoDB 4.2.17
4 Cores
16 GRAM
This server only running mongodb, and allocated 10G memory to mongodb
wiredTiger:
engineConfig:
cacheSizeGB: 10
But this ...
- 1
3
votes
1
answer
2k
views
How to inspect why my service process be killed by Linux?
I have a vital service running a Ubuntu(20.04.1) server. Recently it is always killed by OS.
At first I guess that probably is resulted by the OOM(out of memory) operation of OS, so I modified the ...
- 179
0
votes
0
answers
297
views
MariaDB memory much higher than innodb_buffer_pool_size
I have a backend server with 1G RAM for my HTTP server and my MariaDB.
I noticed the database keeps getting killed by OOM once or twice a day. Most of the time the OOM is triggered by the HTTP server, ...
2
votes
1
answer
368
views
oom_reaper: MongoDb out of memory
We run a small mongodb replica set on three bare metal servers (no virtualization, no docker/kubernetes) with Debian 11 and mongodb 5.0.6:
machineA: 128GB RAM, 1TB disk, PRIMARY
machineB: 128GB RAM, ...
- 247
0
votes
1
answer
203
views
Out of Memory Mismatch compared to what system rerports
Oct 25 07:41:32 KVM-BOX kernel: memory: usage 255216608kB, limit 255216640kB, failcnt 28058
[root@KVM-BOX ~]# free -hm
total used free shared buff/cache available
...
3
votes
0
answers
536
views
Running out of memory with plenty of memory and swap
I have a Debian-based system that is running out of memory but there appears to be plenty of free memory. The box will run for about 6-12 days and then it will begin killing anything allocating memory ...
- 31
0
votes
1
answer
622
views
mariadb oom-killer on CentOS8 in EC2 t2.micro instance
I seem to be having memory problems in my t2.micro instance (1GB) running nginx, mariadb, php and WordPress.
I can see that mariadb.service is being killed regularly (I have used grep -e kill /var/log/...
- 103
0
votes
0
answers
917
views
RHEL 8. linux OOM killer invoked to kill postgres process
Database is going to recovery mode whenever the OOM killer is invoked on the postgres process.
/var/log/messages output with respect to the issue
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),...
0
votes
1
answer
756
views
Daily Apt Upgrade causing oom-killer to kill my java server
My server has died several times in the middle of the night because of this. How much memory is apt-upgrade asking for? Is it being reasonable? I'm trying to decipher the logs here. Any help or ...
- 51
0
votes
2
answers
416
views
Why cannot use buff/cache?
It seems mysqld allocates too much memory as buff/cache:
free -m
total used free shared buff/cache available
Mem: 990 448 96 36 ...
- 99
0
votes
1
answer
768
views
When the kernel says "Out of memory: Killed proces..." does it mean real memory or virtual memory?
Looking at journal -xe for the reason(s) a process is killed, the message comes up in red "Out of memory: Killed process..." but it doesn't say whether the limiting resource is virtual ...
- 158
6
votes
1
answer
788
views
fio 3.23 core dumps when bench-marking many small files
I have been asked to come up fio benchmark results for this test dataset: 1048576x1MiB. So, overall size is 1TiB. The set contains 2^20 1MiB files. The server runs CentOS Linux release 7.8.2003 (Core)...
- 63
0
votes
1
answer
823
views
How do I track down a mysql memory issue on a server running a single WordPress site?
I'm having a weird issue. A few times a week, a website I'm running on Amazon Lightsail comes to a crashing halt and becomes IO bound (I can't usually use ssh when this happens) and the server runs ...
- 534
3
votes
4
answers
6k
views
mysqld crashing almost every hour - mysqld invoked oom-killer
I am running wordpress on a centOS digital ocean droplet
my mysql keeps crashing almost every hour -- with an out of memory error
manually restarting mysqld brings it back up via sudo service mysqld ...
- 31
2
votes
2
answers
2k
views
NFS server with out of memory
I have multiple machines sharing home directory via NFS share used by
6-10 users. All machines are used to run computational experiments including
the one with NFS server. Although it is very rare but ...
- 29
0
votes
1
answer
2k
views
Kubernetes Pod OOMKilled Issue
The scenario is we run some web sites based on an nginx image in kubernetes cluster. When we had our cluster setup with nodes of 2cores and 4GB RAM each. The pods had the following configurations, cpu:...
0
votes
1
answer
985
views
Kill the most memory-hungry process when memory gets low [closed]
Sometimes my Linux desktop, laptop, or server run out of RAM because the experimental software I work with occasionally tries to use more RAM than I have available.
When this happens, the system and ...
- 281
0
votes
0
answers
1k
views
Out of memory in Linux server killing my tomcat server
From last night I am getting below unexpected behaviour.
Out of memory: Kill process 1740 (java) score 128 or sacrifice child
Apr 8 19:14:20 ubuntu-s-4vcpu-8gb-sgp1-01 kernel: [17644.393549] Out of ...
5
votes
1
answer
5k
views
apache httpd process be killed by kernel oom
some apache httpd process's memory and cpu usage too high, be killed by kernel, but I don't know why httpd use such many memory. I need some method to analyse or some method to reproduct.
some ...
- 51
0
votes
0
answers
523
views
Why is systemd-resolve killing my java server?
I've been running into this problem where every few days one of my java servers is just dying. To figure out why, I ran cat /var/log/syslog which produces:
Sep 21 06:36:24 ip-172-31-42-5 systemd-...
- 51
0
votes
1
answer
438
views
OOM despite the large amount of free SWAP
During some ML training htop shows usage of all RAM (16GB) and only 2G (out of 16GB) of SWAP when OOM happens.
dmesg shows:
[pon sie 12 11:53:44 2019] Purging GPU memory, 0 bytes freed, 131072 ...
- 103
0
votes
1
answer
1k
views
What Java -Xmx setting can I reasonably expect to work on an AWS EC2 Ubuntu server?
What is your experience? Can you confirm my experimental findings?
Can I generally use total RAM - 600 MB, or 0.4 * total RAM?
Or is it always trial and error, and hoping that it is low enough?
...
- 421
2
votes
0
answers
85
views
What abbreviations in OOM Killer memory statistics report means?
When OOM Killer or kernel reports memory state it uses the next abbreviations
Node 0 DMA: 26*4kB (M) 53*8kB (UM) 33*16kB (ME) 23*32kB (UME) 6*64kB (ME) 7*128kB (UME) 1*256kB (M) 2*512kB (ME) 0*...
- 173
1
vote
0
answers
575
views
Linux out of memory killer kills process when there is more than enough memory available
We are running a 64-Bit Ubuntu and our 32GB of physical memory is split into 3 zones (DMA: 16MB, DMA32: 4GB and Normal: 30GB). According to the dmesg logs printed below our system ran out of free ...
1
vote
3
answers
2k
views
OOM kills a process while there is still a lot of available memory
I am aware that there are many threads like this one, but none of them actually give a methodology on how to read an OOM message.
I have a db that is being monitored through NRPE for available ...
- 253
1
vote
1
answer
3k
views
Java application gets killed in kubernetes while the resource limits and heap size are specified
Background
A spring boot Java application is deployed in a kubernetes cluster and gets killed several times per day.
I'm using openjdk:8u181-jre for my Java apps.
Kubernetes version: v1.11.5
Node ...
- 113
7
votes
2
answers
30k
views
Enough memory but OOM killer invoked?
It has been a long time since my old friend the OOM killer came to visit me. Now it is coming by once a week and I cannot figure out why it’s triggered. It doesn’t seem that I’m missing memory. What ...
- 93
4
votes
2
answers
381
views
Out of memory at 18% usage : where the ram goes?
This is related to : Out of memory at 72% usage
It looks to be the same problem but the question is slightly different : Where my memory goes ? I have 18% memory usage and my OOM Killer is killing ...
- 629
3
votes
2
answers
3k
views
Linux OOM-killer acting despite plenty available memory
Once a week or so the OOM-killer shoots down a postgres process in my server, despite that 'free' states it has plenty of available memory.
I have read several threads here and there, but cant see ...
1
vote
0
answers
718
views
OOM killer kills process though page cache is never evicted
Our servers in production sometimes under load runs into the following issue (Linux 4.10):
Application handling the load (webserver) is running at high load.
A new job starts (e.g. cron). This ...
- 11
1
vote
0
answers
4k
views
"bash: fork: Cannot allocate memory" for any command - how to prevent this at OS level?
I was testing how an Ubuntu 16.04 machine handles low memory conditions and had several suspended jobs that that used up almost all the RAM on the server. I was still able to run normal bash commands ...
- 369
3
votes
1
answer
6k
views
Linux OOM: Trying to understand memory statistics / mismatch with expected values
I am trying to understand why the OOM killer has reaped a process on a KVM virtual machine running with 1 gigabyte of memory. While I do already know that running Linux machines without swap is not ...
- 33
1
vote
0
answers
453
views
Is there any legitimate reason to mlock all of a process's mmaped files?
I'm administering some Linux servers that are running an httpd based application developed by one of our apps teams. This is in EC2 and each host has 32GB of RAM. I've noticed that there is about once ...
- 2,665
3
votes
2
answers
9k
views
Long page allocation stalls on Linux – why does this happen?
I have a problem (which I can reliably reproduce) on a bunch of Linux hosts, where the system becomes completely unresponsive after a process aggressively consumes memory. I see things like this in ...
- 304
3
votes
2
answers
1k
views
GKE's Container-Optimized OS Out-Of-Memory freezes
I have issue with Container-Optimized OS on GKE. If I run this simple command https://pastebin.com/raw/0WPAnAzn to consume all the RAM, at some point host freezes and doesn't respond to anything. ...
0
votes
1
answer
384
views
Number of httpd processes and prefork.c
Hi I got OOM killer problems which caused:
[mpm_prefork:error] [pid 18103] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
I had to restart the server to get it working.
In /...
- 103
0
votes
1
answer
1k
views
VmRSS used to only about 25%, yet oom-killer strikes
I have a dedicated MySQL server equipped with 128 GB RAM. MySQL recently gets killed by the oom-killer, although MySQL is configured to use 95 GB in the worst case. In my research I came across this:
...
- 105
1
vote
4
answers
15k
views
Ansible throws an “ ERROR! A worker was found in a dead state ” error
When I run a playbook that simply copies a directory from one place to another, ansible throws
ERROR! A worker was found in a dead state
Error. After some googling, it looks like this is caused by ...
- 179
2
votes
1
answer
9k
views
OOM killer invoked despite tons of memory available? [duplicate]
I have a server where the OOM killer has been invoked once or twice almost each night since two weeks ago. The server should have more than enough memory it could free (cache/buffers, inactive_file ...
- 121
10
votes
1
answer
8k
views
Linux process killed even though enough memory available
I am investigating why two of our processes were killed by the Linux OOM killer - even though there seems to have been enough RAM and plenty of SWAP available at both times.
When I interpret it as by ...
- 323
3
votes
2
answers
4k
views
Slab reclaimable memory is not freed when needed
Correct me if I am wrong, but to my understanding slab reclaimable holds cached kernel objects which can be freed if needed. So if application needs to allocate more space, even if the 'free' memory ...
- 161
0
votes
1
answer
533
views
OOM Killer on xenserver - 2832MB memory free - killing xapi
I am trying to understand why OOM killer starts killing processes when there is plenty of memory free on the server:
Result of uname -a:
Linux hostname 2.6.32.43-0.4.1.xs1.8.0.835.170778xen #1 SMP ...
- 1