KVM: qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Question

When trying to start a VM (1GB allocated) I have run into memory allocation issue even though I have more than enough memory on the system (16GB):

error: Failed to start domain exampleVM
error: internal error: qemu unexpectedly closed the monitor: 2023-09-10T11:49:20.853205Z qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

I have three VMs running with 6GB total allocated:

I have not had swap allocated then (have added it now to make system working), but why should it be the problem? Shouldn't the system just swap out in-memory pages to make place for VM's memory?

Please explain this problem because it is rather mysterious to me. Google only came up with inadequate answers and maybes.

P.S.

% free -g
              total        used        free      shared  buff/cache   available
Mem:             15           6           2           5           7           3
Swap:             7           3           4

% ps_mem.py 
 Private  +   Shared  =  RAM used   Program

 44.0 KiB +  82.0 KiB = 126.0 KiB   dnsmasq
120.0 KiB +  79.0 KiB = 199.0 KiB   xrdp
292.0 KiB +  81.0 KiB = 373.0 KiB   cron
280.0 KiB + 115.0 KiB = 395.0 KiB   agetty (2)
540.0 KiB + 100.5 KiB = 640.5 KiB   auditd
600.0 KiB +  58.5 KiB = 658.5 KiB   mdadm
868.0 KiB + 150.5 KiB =   1.0 MiB   xrdp-sesman
996.0 KiB + 157.5 KiB =   1.1 MiB   exim4
200.0 KiB +   1.0 MiB =   1.2 MiB   ha_logd (2)
984.0 KiB + 333.5 KiB =   1.3 MiB   systemd-timesyncd
  1.2 MiB + 368.0 KiB =   1.5 MiB   systemd-logind
  1.0 MiB + 564.0 KiB =   1.5 MiB   upowerd
  1.6 MiB + 154.0 KiB =   1.7 MiB   dbus-daemon
  1.8 MiB + 154.0 KiB =   1.9 MiB   smartd
  1.5 MiB + 462.0 KiB =   1.9 MiB   unattended-upgr
  2.0 MiB +  94.5 KiB =   2.1 MiB   systemd-udevd
  1.6 MiB + 583.5 KiB =   2.2 MiB   virtlogd
  1.7 MiB + 559.5 KiB =   2.2 MiB   polkitd
  1.9 MiB + 314.0 KiB =   2.2 MiB   systemd-journald
  2.2 MiB + 126.5 KiB =   2.3 MiB   rsyslogd
  1.5 MiB + 854.5 KiB =   2.3 MiB   ssh
  1.4 MiB + 933.5 KiB =   2.4 MiB   (sd-pam)
  3.3 MiB + 617.5 KiB =   3.9 MiB   udisksd
  4.4 MiB +   1.2 MiB =   5.6 MiB   tuned
  4.2 MiB +   1.5 MiB =   5.7 MiB   bash (2)
  2.6 MiB +   3.5 MiB =   6.2 MiB   sshd (3)
  4.0 MiB +   3.2 MiB =   7.2 MiB   systemd (2)
  9.4 MiB +   1.2 MiB =  10.6 MiB   libvirtd
 13.6 MiB +   1.1 MiB =  14.7 MiB   fail2ban-server
 17.5 MiB + 300.5 KiB =  17.8 MiB   zabbix_agent2
 12.4 MiB +  40.5 MiB =  52.9 MiB   heartbeat (6)
  7.4 GiB +   6.2 MiB =   7.5 GiB   qemu-system-x86_64 (5)
---------------------------------
                          7.6 GiB
=================================

% cat /proc/meminfo 
MemTotal:       16298980 kB
MemFree:         2471024 kB
MemAvailable:    5498208 kB
Buffers:         2161672 kB
Cached:          3251996 kB
SwapCached:        10692 kB
Active:          8700828 kB
Inactive:        4551496 kB
Active(anon):    7763376 kB
Inactive(anon):  2282256 kB
Active(file):     937452 kB
Inactive(file):  2269240 kB
Unevictable:       59280 kB
Mlocked:           59280 kB
SwapTotal:       8388604 kB
SwapFree:        2057932 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       7888728 kB
Mapped:            83356 kB
Shmem:           2164944 kB
Slab:             236904 kB
SReclaimable:     144204 kB
SUnreclaim:        92700 kB
KernelStack:        4448 kB
PageTables:        23336 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16538092 kB
Committed_AS:   19768952 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:            11904 kB
HardwareCorrupted:     0 kB
AnonHugePages:   7198720 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     3027424 kB
DirectMap2M:    13621248 kB

Answer 1

This is not KVM/QEMU/LIBVIRT bug, since it is the kernel running out of system memory by allocating of 1G.

This can be:

• impossibility to free cached blocks by the FS (depending on type of FS, e. g. zfs can handle it differently to normal behavior of kernel). Try to drop caches, set overcommit_memory etc, and check whether it'd help and take a look at similar issue.
• memory fragmentation, see this or this related issues.