I recently had to think about changing my cron jobs that run every 2 minutes to something more reliable.
Basically, running a cron job every other minute has the disadvantage of restarting the script from the beginning even when the previous instance is still running. Say for example you have a script that needs to check if there are new records in a database table. You run that script every minute to “poll” the database, edit the records and move them to a new table.
What happens if you run the script at 2:00am and it doesn’t stop, then your cron runs again at 2:02am while the first one is still running? You will get two copies of the data in the database table.
This could easily corrupt your data and lead to duplicate results as seen above.Now this is where daemons and queue managers come in handy.
In this scenario I decided it was time for me to take the gauntlet of doing something different with queued jobs. While I’m still implementing this I thought it would be useful to keep a reference to the article that helped me make that decision. Find a piece of it below. Source link at the bottom.
Writing an upstart script
Turns out, writing your own upstart scripts is way easier than building init.d files based on the
Ok so here’s how it looks like; You should store the script in
/etc/init/yourprogram.conf, create one for each Node program you write.
description "node.js server" author "kvz - http://kevin.vanzonneveld.net" # used to be: start on startup # until we found some mounts weren't ready yet while booting: start on started mountall stop on shutdown # Automatically Respawn: respawn respawn limit 99 5 script # Not sure why $HOME is needed, but we found that it is: export HOME="/root" exec /usr/local/bin/node /where/yourprogram.js >> /var/log/node.log 2>&1 end script post-start script # Optionally put a script here that will notifiy you node has (re)started # /root/bin/hoptoad.sh "node.js has started!" end script
Wow how easy was that? Told you, upstart scripts are childsplay. In fact they’re so compact, you may find yourself changing almost every line cause they contain specifics to our environment.
Node can do a lot of stuff. Or break it if you’re not careful. So you may want to run it as a user with limited privileges. We decided to go conventional and chose
We found the easiest way was to prepend the Node executable with a sudo like this:
exec sudo -u www-data /usr/local/bin/node
Don’t forget to change your export HOME accordingly.
Restarting your Node.js daemon
This is so ridiculously easy..
$ start yourprogram $ stop yourprogram
And yes, Node will already:
- automatically start at boottime
- log to
..that’s been defined inside our upstart script.
stop are just shortcuts. Who’s really behind the wheel here, is
initctl. You can play around with the command to see what other possibilities there are:
$ initctl help $ initctl status yourprogram $ initctl reload yourprogram $ initctl start yourprogram # yes, this is the same start # etc
Update from October 30th, 2012
The basic idea has not changed since 2009, but we did add some tricks to our upstart script. Here’s what we now use in production at transloadit.com:
# cat /etc/init/transloaditapi2.conf # http://upstart.ubuntu.com/wiki/Stanzas description "Transloadit.com node.js API 2" author "kvz" stop on shutdown respawn respawn limit 20 5 # Max open files are @ 1024 by default. Bit few. limit nofile 32768 32768 script set -e mkfifo /tmp/api2-log-fifo ( logger -t api2 </tmp/api2-log-fifo & ) exec >/tmp/api2-log-fifo rm /tmp/api2-log-fifo exec sudo -u www-data MASTERKEY=`cat /transloadit/keys/masterkey` /transloadit/bin/server 2>&1 end script post-start script /transloadit/bin/notify.sh 'API2 Just started' end script
Starting & stopping your daemon:
Usually, the suggested method for starting and stopping daemon process is to use the system’s “service” command. So here we would use:
sudo service transloaditapi2 start sudo service transloaditapi2 stop
Check if your daemon is running:
sudo service transloaditapi2 status