Almalinux 9 Failed to allocate memory

Question

I have SingleStore DB CLI (Cluster in a Box) deployed in a AWS EC2 Instance r6i.4xlarge with Almalinux 9 OS. At the moment of load data by pipelines the following error appears:

line 34: ER_MEMSQL_OOM (1712), Leaf Error (127.0.0.1:3307): The operating system failed to allocate memory (MemSQL memory use 18658.88 Mb). The request was not processed. See https://docs.memsql.com/troubleshooting/latest/memory-errors

The table I need to load is almost 5GB.

The Server is a 32CPU/128GB and never reach this amount.

Here is my /etc/sysctl.conf file:

vm.max_map_count = 1000000000

vm.min_free_kbytes = 4194304

net.core.wmem_max = 8388608

net.core.rmem_max = 8388608

vm.overcommit_memory = 0

#vm.overcommit_ratio = 99

vm.swappiness = 10

From Singlestore everything seems to be ok, does anyone experimented this kind of errors?

Please help

Answer 1

First of all, "cannot allocate memory" does not mean "out of memory". What it means is that the kernel is not willing to allocate (i.e. "reserve") the memory the process requested. This is only loosely related to the actual memory usage, as most processes use a lot less memory than they allocate. For that the defaults are to overcommit memory (i.e. to allocate more memory than the system has available).

Usually when people run into such issues, the easiest solution is to just add swap space so the kernel will allocate more memory. It won't really use the swap a lot, it just needs to know that it could.