I'm having an issue where I'm running pm2 as pm2-root under systemd (this is on Amazon Linux2023). The daemon runs fine, and if a process dies, it restarts it fine. However, if the server reboots, pm2-root restarts, and it says it's restarted all of the processes, but it hasn't. For example:
# pm2 ls
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name │ mode │ ↺ │ status │ cpu │ memory │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 3 │ IDAP Dev │ fork │ 0 │ stopped │ 0% │ 0b │
│ 4 │ IDAP Website │ fork │ 0 │ online │ 0% │ 0b │
│ 10 │ Mirovoy NextJS │ fork │ 0 │ online │ 0% │ 0b │
│ 5 │ Reanthropize │ fork │ 0 │ online │ 0% │ 0b │
│ 6 │ Reanthropize Stage │ fork │ 0 │ online │ 0% │ 0b │
│ 1 │ lp-api │ fork │ 0 │ online │ 0% │ 0b │
│ 9 │ url-shortener │ fork │ 0 │ online │ 0% │ 0b │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
I've already manually intervened with ids 3, 9, and 10, but let's look at 1 (full desc at end):
# pm2 desc 1
Describing process with id 1 - name lp-api
┌───────────────────┬────────────────────────────────────────────────────────────┐
│ status │ online │
│ restarts │ 0 │
│ uptime │ 23h │
│ pid path │ /root/.pm2/pids/lp-api-1.pid │
│ watch & reload │ ✘ │
│ unstable restarts │ 0 │
│ created at │ 2022-08-23T14:26:17.285Z │
└───────────────────┴────────────────────────────────────────────────────────────┘
There's absolutely nothing in the logs:
# pm2 logs 1
[TAILING] Tailing last 15 lines for [1] process (change the value with --lines option)
/root/.pm2/logs/lp-api-out.log last 15 lines:
/root/.pm2/logs/lp-api-error.log last 15 lines:
But it's definitely not running:
# ll /root/.pm2/pids/
total 8
-rw-r--r--. 1 root root 5 Sep 26 10:09 Mirovoy-NextJS-10.pid
-rw-r--r--. 1 root root 5 Sep 26 00:01 url-shortener-9.pid
# lsof -i -P -n | grep LISTEN | grep 9292
#
Yet if I now manually restart it:
# pm2 stop 1
[PM2] Applying action stopProcessId on app [1](ids: [ '1' ])
[PM2] [1](1) ✓
# pm2 delete 1
[PM2] Applying action deleteProcessId on app [1](ids: [ '1' ])
[PM2] [lp-api](1) ✓
# cd /mnt/data/www/vhosts/chrisrichardson.info/landing-page-api // cd to exec cwd
# pm2 start npm --name lp-api -- run prod // run script args
all is good
# ll /root/.pm2/pids/
total 12
-rw-r--r--. 1 root root 5 Sep 26 10:09 Mirovoy-NextJS-10.pid
-rw-r--r--. 1 root root 5 Sep 26 11:19 lp-api-11.pid // <- PID now here
-rw-r--r--. 1 root root 5 Sep 26 00:01 url-shortener-9.pid
# lsof -i -P -n | grep LISTEN | grep 9292
node 79177 root 19u IPv6 410747 0t0 TCP *:9292 (LISTEN)
How do I get it to actually start the processes on system reboot, instead of just pretending/thinking it does?
Note: I've also included the full desc of it after manual intervention, and the only thing I see is that there's some env divergence, but I don't see how that's causing the problem.
Appendix
Full output of pm2 desc 1:
Describing process with id 1 - name lp-api
┌───────────────────┬────────────────────────────────────────────────────────────┐
│ status │ online │
│ name │ lp-api │
│ namespace │ default │
│ version │ N/A │
│ restarts │ 0 │
│ uptime │ 24h │
│ script path │ /bin/npm │
│ script args │ run prod │
│ error log path │ /root/.pm2/logs/lp-api-error.log │
│ out log path │ /root/.pm2/logs/lp-api-out.log │
│ pid path │ /root/.pm2/pids/lp-api-1.pid │
│ interpreter │ node │
│ interpreter args │ N/A │
│ script id │ 1 │
│ exec cwd │ /mnt/data/www/vhosts/chrisrichardson.info/landing-page-api │
│ exec mode │ fork_mode │
│ node.js version │ 17.9.0 │
│ node env │ N/A │
│ watch & reload │ ✘ │
│ unstable restarts │ 0 │
│ created at │ 2022-08-23T14:26:17.285Z │
└───────────────────┴────────────────────────────────────────────────────────────┘
Divergent env variables from local env
┌──────────────┬──────────────────────┐
│ HOSTNAME │ ip-10-0-4-227.eu-cen │
│ SUDO_COMMAND │ /bin/su │
│ LANG │ en_GB.UTF-8 │
│ LS_COLORS │ rs=0:di=38;5;27:ln=3 │
│ PATH │ /sbin:/bin:/usr/sbin │
│ _ │ /bin/pm2 │
└──────────────┴──────────────────────┘
Add your own code metrics: http://bit.ly/code-metrics
Use `pm2 logs lp-api [--lines 1000]` to display logs
Use `pm2 env 1` to display environment variables
Use `pm2 monit` to monitor CPU and Memory usage lp-api
Full output of desc after manual intervention:
# pm2 desc 11
Describing process with id 11 - name lp-api
┌───────────────────┬────────────────────────────────────────────────────────────┐
│ status │ online │
│ name │ lp-api │
│ namespace │ default │
│ version │ N/A │
│ restarts │ 0 │
│ uptime │ 7m │
│ script path │ /root/.nvm/versions/node/v18.17.1/bin/npm │
│ script args │ run prod │
│ error log path │ /root/.pm2/logs/lp-api-error.log │
│ out log path │ /root/.pm2/logs/lp-api-out.log │
│ pid path │ /root/.pm2/pids/lp-api-11.pid │
│ interpreter │ node │
│ interpreter args │ N/A │
│ script id │ 11 │
│ exec cwd │ /mnt/data/www/vhosts/chrisrichardson.info/landing-page-api │
│ exec mode │ fork_mode │
│ node.js version │ 18.17.1 │
│ node env │ N/A │
│ watch & reload │ ✘ │
│ unstable restarts │ 0 │
│ created at │ 2023-09-26T11:19:35.717Z │
└───────────────────┴────────────────────────────────────────────────────────────┘
Actions available
┌────────────────────────┐
│ km:heapdump │
│ km:cpu:profiling:start │
│ km:cpu:profiling:stop │
│ km:heap:sampling:start │
│ km:heap:sampling:stop │
└────────────────────────┘
Trigger via: pm2 trigger lp-api <action_name>
Code metrics value
┌────────────────────────┬───────────┐
│ Used Heap Size │ 15.67 MiB │
│ Heap Usage │ 90.12 % │
│ Heap Size │ 17.38 MiB │
│ Event Loop Latency p95 │ 1.13 ms │
│ Event Loop Latency │ 0.23 ms │
│ Active handles │ 5 │
│ Active requests │ 0 │
└────────────────────────┴───────────┘
Divergent env variables from local env
Add your own code metrics: http://bit.ly/code-metrics
Use `pm2 logs lp-api [--lines 1000]` to display logs
Use `pm2 env 11` to display environment variables
Use `pm2 monit` to monitor CPU and Memory usage lp-api
