CMS update almost killed this blog...

So today I received a message telling me that a new version of Ghost (the blogging system that I'm running now) is available. Bug fixes, dependency updates, all those crap that they always use. So I decided to upgrade.

But the thing is, they didn't offer a auto-update option, which meant I had to start from the console and do everything by myself. And obviously I started...

root@mingjie-info:/var/www# service ghost stop
root@mingjie-info:/var/www# wget https://ghost.org/zip/ghost-latest.zip
....
ghost-latest.zip    100%[===================>]   4.00M  7.79MB/s    in 0.5s
2017-01-15 22:45:44 (7.79 MB/s) - ‘ghost-latest.zip’ saved [4189954/4189954]
root@mingjie-info:/var/www# rm -rf ghost/core
root@mingjie-info:/var/www# unzip -uo ghost*.zip -d ghost
....
root@mingjie-info:/var/www# chown -R ghost:ghost ghost/*
root@mingjie-info:/var/www# cd /var/www/ghost

Everything seemed to be working perfectly. But here came the problem.

root@mingjie-info:/var/www/ghost# npm install --production
extract:moment → gunzTarP ▐ ╢████████████████████████████████████████░░░░░░░░░╟
Killed
root@mingjie-info:/var/www/ghost# npm install --production
loadDep:rewire → network  ▐ ╢██████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░╟
Killed

For that one second I was like, wait? What? Node rebuild just killed itself (XD). And fortunately I backed my database up ahead of time, so I'm not afraid to try anything now. All I want, is to fix this thing without rebuilding my server.

So I started from the basic commands. Clearing the cache, cleaning up the modules --

root@mingjie-info:/var/www/ghost# npm cache clean
root@mingjie-info:/var/www/ghost# rm -rf node_modules
root@mingjie-info:/var/www/ghost# npm install --production
extract → gunzTarPerm     ▐ ╢██████████████████████████████████░░░░░░░░░░░░░░░╟
Killed
root@mingjie-info:/var/www/ghost# npm install --production
KilledurrentTree → gunzTa ▐ ╢████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░╟

Again. I tried this over and over again, but it just did not work. When I almost gave up trying and decided to rebuild the entire thing, I noticed something - the process got killed at different spots.

Think about this. If something went wrong in Node, would't it get stuck at the same spot every time?

That means, it was not the node process that's causing this issue - something's wrong with the server itself. Memory was the only thing I could think of. Now it explains everything, the process ran, and server memory soon got filled up, so the system decided to kill the node process to maintain the kernel working.

Here's how I fixed it: simply expand the swap.

root@mingjie-info:/var/www/ghost# swapoff -a
root@mingjie-info:/var/www/ghost# sudo swapon -s
root@mingjie-info:/var/www/ghost# sudo dd if=/dev/zero of=/swapfile bs=1024 coun                                                                                        t=256k
262144+0 records in
262144+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 1.3216 s, 203 MB/s
root@mingjie-info:/var/www/ghost# sudo mkswap /swapfile
Setting up swapspace version 1, size = 256 MiB (268431360 bytes)
no label, UUID=de8d2b65-203d-4698-aedc-ba54bc35f16e
root@mingjie-info:/var/www/ghost# sudo swapon /swapfile
swapon: /swapfile: insecure permissions 0644, 0600 suggested.
root@mingjie-info:/var/www/ghost# swapon -s
Filename                                Type            Size    Used    Priority
/swapfile                               file            262140  32      -1
root@mingjie-info:/var/www/ghost# sudo nano /etc/fstab
root@mingjie-info:/var/www/ghost# echo 10 | sudo tee /proc/sys/vm/swappiness
10
root@mingjie-info:/var/www/ghost# echo vm.swappiness = 10 | sudo tee -a /etc/sys                                                                                        ctl.conf
vm.swappiness = 10
root@mingjie-info:/var/www/ghost# sudo chown root:root /swapfile
root@mingjie-info:/var/www/ghost# sudo chmod 0600 /swapfile
root@mingjie-info:/var/www/ghost# npm install --production
> dtrace-provider@0.6.0 install /var/www/ghost/node_modules/gscan/node_modules/ghost-ignition/node_modules/bunyan/node_modules/dtrace-provider
> node scripts/install.js
root@mingjie-info:/var/www/ghost# service ghost start

Excellent.

What lesson did this story give me? When you counter an issue while running a server, ALWAYS, ALWAYS, check the log. Because the log, literally said, "out of memory, killed by system".


Mingjie Jiang

Maryland, United States
https://www.mingjie.info

Comments

comments powered by Disqus