You've built a node application and now it's time to deploy and run it. But how do you make sure that if your app crashes, it restarts automatically? Or if the host machine goes down, how do you ensure that your app comes back up? Ive seen a number of people across the internet suggest things like node-supervisor, forever, nodemon, and hell, even gnu screen. These might be fine for running a server locally or in a testing environment, but they have a (pretty large) drawback; the underlying process is (by default) managed by the user that ran it. This means that the process that is managing your node app (supervisor, forever, nodemon, screen, etc) isn't managed by anything else and if it goes down, then what? I thought we were going for uptime here...
For whatever reason, it seems that people forget that the underlying operating system (we're assuming Linux here) has an init system that is designed to do exactly what we want. Now that the majority of the major linux distros come with systemd it's easier than ever to make sure that your node app is properly managed, not to mention it can handle logging for you as well.
Setting up your machine
We're going to be using Fedora 23 for the purpose of this article and installing node directly on it.
curl https://nodejs.org/download/release/v4.2.1/node-v4.2.1-linux-x64.tar.gz | sudo tar xvz --strip-components=1 -C /usr/local
node -v
# v4.2.1What is systemd?
systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic.
tl;dr - at a high level, systemd gives you:
- process management
- logging management
- process/service dependency management via socket activation
Our test node application
We're going to use this stupidly simple HTTP server as our test application. It's so simple, that it doesn't even depend on any external packages.
server.js
var http = require('http');
var server = http.createServer(function(req, res){
res.end(new Date().toISOString());
});
server.listen(8000);We'll build on this as we go to demonstrate the features of systemd.
Running your application
To run our application, we need to write out a unit file that describes what to run and how to run it. For this, we're going to want to look at:
Here's a super simple unit file to get us started:
node-server.service
[Unit]
Description=stupid simple nodejs HTTP server
[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/node server.js
Type=simplePlace this file in /etc/systemd/system and run:
sudo systemctl start node-server.servicesystemctl is the utility to manage systemd-based services. When given a unit name that isn't a direct path, it looks in /etc/systemd/system, attempting to match the provided name to unit file names. Now, let's check the status of our service:
systemctl status node-server.service
● node-server.service - stupid simple nodejs HTTP server
Loaded: loaded (/etc/systemd/system/node-server.service; static; vendor preset: disabled)
Active: active (running) since Mon 2015-11-30 11:40:18 PST; 3s ago
Main PID: 17018 (node)
CGroup: /system.slice/node-server.service
└─17018 /usr/local/bin/node server.js
Nov 30 11:40:18 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 11:40:18 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...Awesome, it looks to be running! Let's curl our HTTP server and see if it actually is:
curl http://localhost:8000/
2015-11-30T19:43:17.102ZManaging logs with systemd-journald
Now that we have everything running, let's modify our script to print something to stdout when a request comes in.
var http = require('http');
var server = http.createServer(function(req, res){
var date = new Date().toISOString();
console.log('sending date: ', date);
res.end(date);
});
server.listen(8000);Edit your server to look like the code above. For logging, all we need to do is log directly to stdout and stderr; systemd-journald will handle everything else from here. Now, let's restart our server and tail the log:
sudo systemctl restart node-server.service
journalctl -f -u node-server.service
-- Logs begin at Mon 2015-10-19 17:41:06 PDT. --
Nov 30 11:40:18 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 11:40:18 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...
Nov 30 11:46:30 localhost.localdomain systemd[1]: Stopping stupid simple nodejs HTTP server...
Nov 30 11:46:30 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 11:46:30 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...Close out journalctl (ctrl-c) and curl your HTTP server again. You should now see a new line added to the log:
-- Logs begin at Mon 2015-10-19 17:41:06 PDT. --
Nov 30 11:40:18 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 11:40:18 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...
Nov 30 11:46:30 localhost.localdomain systemd[1]: Stopping stupid simple nodejs HTTP server...
Nov 30 11:46:30 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 11:46:30 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...
Nov 30 11:47:40 localhost.localdomain node[17076]: sending date: 2015-11-30T19:47:40.319ZHandling crashes and restarting
What if your application crashes; you probably want it to restart, otherwise you wouldn't need an init system in the first place. Systemd provides the Restart= property to specify when your application should restart, if at all. We're going to use Restart=always for simplicity sake, but all of the options can be found in a table on the systemd.service docs page.
Our updated unit file:
[Unit]
Description=stupid simple nodejs HTTP server
[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/node server.js
Type=simple
Restart=always
RestartSec=10Note that we also added RestartSec=10. This is just so that we can easily see in the logs the restart. Now that our unit file is updated, we need to tell systemd:
sudo systemctl daemon-reloadBefore we restart everything, let's modify our server so that it crashes:
var http = require('http');
var server = http.createServer(function(req, res){
var date = new Date().toISOString();
console.log('sending date: ', date);
throw new Error('crashing');
res.end(date);
});
server.listen(8000);Now we can restart everything:
sudo systemctl restart node-server.serviceNow when you curl your server, it will crash and restart itself. We can verify this by checking the logs as we did above:
journalctl -f -u node-server.service
Nov 30 12:01:38 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 12:01:38 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...
Nov 30 12:02:20 localhost.localdomain node[17255]: sending date: 2015-11-30T20:02:20.807Z
Nov 30 12:02:20 localhost.localdomain systemd[1]: node-server.service: Main process exited, code=exited, status=1/FAILURE
Nov 30 12:02:20 localhost.localdomain systemd[1]: node-server.service: Unit entered failed state.
Nov 30 12:02:20 localhost.localdomain systemd[1]: node-server.service: Failed with result 'exit-code'.
Nov 30 12:02:30 localhost.localdomain systemd[1]: node-server.service: Service hold-off time over, scheduling restart.
Nov 30 12:02:30 localhost.localdomain systemd[1]: Started stupid simple nodejs HTTP server.
Nov 30 12:02:30 localhost.localdomain systemd[1]: Starting stupid simple nodejs HTTP server...Starting your app on boot
Often times, you may want your application or service to start when the machine boots (or reboots for that matter). To do this, we need to add an [Install] section to our unit file:
[Unit]
Description=stupid simple nodejs HTTP server
[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/node server.js
Type=simple
Restart=always
RestartSec=10
[Install]
WantedBy=basic.targetNow, we can enable it:
sudo systemctl enable node-server.service
Created symlink from /etc/systemd/system/basic.target.wants/node-server.service to /etc/systemd/system/node-server.service.When control is handed off to systemd on boot, it goes through a number of stages:
local-fs-pre.target
|
v
(various mounts and (various swap (various cryptsetup
fsck services...) devices...) devices...) (various low-level (various low-level
| | | services: udevd, API VFS mounts:
v v v tmpfiles, random mqueue, configfs,
local-fs.target swap.target cryptsetup.target seed, sysctl, ...) debugfs, ...)
| | | | |
\__________________|_________________ | ___________________|____________________/
\|/
v
sysinit.target
|
____________________________________/|\________________________________________
/ | | | \
| | | | |
v v | v v
(various (various | (various rescue.service
timers...) paths...) | sockets...) |
| | | | v
v v | v rescue.target
timers.target paths.target | sockets.target
| | | |
v \_________________ | ___________________/
\|/
v
basic.target
|
____________________________________/| emergency.service
/ | | |
| | | To do this, we first need to add an [In v
v v v emergency.target
display- (various system (various system
manager.service services services)
| required for |
| graphical UIs) v
| | multi-user.target
| | |
\_________________ | _________________/
\|/
v
graphical.targetAs you can see in the chart, basic.target is the first target hit after the core components of the system come online. This flow chart is how systemd orders services and even resolves service dependencies. When we ran systemctl enable it creates a symlink into /etc/systemd/systemd/basic.target.wants. Everything that is symlinked there will be run as part of the basic.target step in the boot process.
Wrap up
As you can see, it's pretty simple to get everything up and running with systemd and you don't have to mess around with user space based process managers. With a small config and like 10 minutes of time, you now no longer have to worry about your entire app crashing down and not being able to restart itself.
Now that you have a basic understanding of systemd and what it offers, you can start really digging into it and exploring all of the other features that it offers to make your application infrastructure even better.