Running a Daemon (self restarting) Process in Ubuntu – the easy way

I recently had to think about changing my cron jobs that run every 2 minutes to something more reliable.

Basically, running a cron job every other minute has the disadvantage of restarting the script from the beginning even when the previous instance is still running. Say for example you have a script that needs to check if there are new records in a database table. You run that script every minute to “poll” the database, edit the records and move them to a new table.

What happens if you run the script at 2:00am and it doesn’t stop, then your cron runs again at 2:02am while the first one is still running? You will get two copies of the data in the database table.

This could easily corrupt your data and lead to duplicate results as seen above.Now this is where daemons and queue managers come in handy.

In this scenario I decided it was time for me to take the gauntlet of doing something different with queued jobs. While I’m still implementing this I thought it would be useful to keep a reference to the article that helped me make that decision. Find a piece of it below. Source link at the bottom.

 

Writing an upstart script

Turns out, writing your own upstart scripts is way easier than building init.d files based on the /etc/skeleton file.

Ok so here’s how it looks like; You should store the script in /etc/init/yourprogram.conf, create one for each Node program you write.

description "node.js server"
author      "kvz - http://kevin.vanzonneveld.net"

# used to be: start on startup
# until we found some mounts weren't ready yet while booting:
start on started mountall
stop on shutdown

# Automatically Respawn:
respawn
respawn limit 99 5

script
    # Not sure why $HOME is needed, but we found that it is:
    export HOME="/root"

    exec /usr/local/bin/node /where/yourprogram.js >> /var/log/node.log 2>&1
end script

post-start script
   # Optionally put a script here that will notifiy you node has (re)started
   # /root/bin/hoptoad.sh "node.js has started!"
end script

Wow how easy was that? Told you, upstart scripts are childsplay. In fact they’re so compact, you may find yourself changing almost every line cause they contain specifics to our environment.

non-root

Node can do a lot of stuff. Or break it if you’re not careful. So you may want to run it as a user with limited privileges. We decided to go conventional and chose www-data.

We found the easiest way was to prepend the Node executable with a sudo like this:

exec sudo -u www-data /usr/local/bin/node

Don’t forget to change your export HOME accordingly.

Restarting your Node.js daemon

This is so ridiculously easy..

$ start yourprogram
$ stop yourprogram

And yes, Node will already:

  • automatically start at boottime
  • log to /var/log/node.log

..that’s been defined inside our upstart script.

initctl

But wait, start and stop are just shortcuts. Who’s really behind the wheel here, is initctl. You can play around with the command to see what other possibilities there are:

$ initctl help
$ initctl status yourprogram
$ initctl reload yourprogram
$ initctl start yourprogram # yes, this is the same start
# etc

Update from October 30th, 2012

The basic idea has not changed since 2009, but we did add some tricks to our upstart script. Here’s what we now use in production at transloadit.com:

# cat /etc/init/transloaditapi2.conf
# http://upstart.ubuntu.com/wiki/Stanzas

description "Transloadit.com node.js API 2"
author      "kvz"

stop on shutdown
respawn
respawn limit 20 5

# Max open files are @ 1024 by default. Bit few.
limit nofile 32768 32768

script
  set -e
  mkfifo /tmp/api2-log-fifo
  ( logger -t api2 </tmp/api2-log-fifo & )
  exec >/tmp/api2-log-fifo
  rm /tmp/api2-log-fifo
  exec sudo -u www-data MASTERKEY=`cat /transloadit/keys/masterkey` /transloadit/bin/server 2>&1
end script

post-start script
   /transloadit/bin/notify.sh 'API2 Just started'
end script

Starting & stopping your daemon:

Usually, the suggested method for starting and stopping daemon process is to use the system’s “service” command. So here we would use:

sudo service transloaditapi2 start
sudo service transloaditapi2 stop

Check if your daemon is running:

sudo service transloaditapi2 status

via kvz.io.

Advertisements

Switching From Apache MPM Prefork to Worker

My very first experience of setting up a live cloud server was one I had looked forward to with optimism.

In the past I was comfortable with the shared hosting and semi dedicate solutions which provided the basic tools for managing a website. But then I needed shell access and hosting my applications using the dedicated solutions provided by the shared hosting companies was costing a lot more than I bargained for.

A little overcommitted?

After getting several warning emails from my shared hosting company at the time for exceeding resource usage, there was just one option left – move.

Almost everyone is used to the localhost setup of a WAMP or LAMP stack. However, these implementations are built to run on a single computer without the rigors that a constantly active webserver experiences.

In the case of a live webserver on the internet you need to tweak and tune stuff in order to ensure your server doesn’t get overrun on this highway. Crawlers, Spiders, and SPAM bots all contribute to traffic on your server and you need to plan to manage them.

Setting up your LAMP stack is somewhat straight forward (on Debian, Ubuntu type: sudo apt-get install apache2 php5 libapache2-mod-php5).

I wouldn’t go into the details of setting that up here. What I want to share on the other hand is the effect of going along with the default implementation of the LAMP stack, and particularly the implementation of the Apache server that it comes bundled with.

By default, most new installations of Apache install the prefork memory management module (mpm-prefork) which handles each connection using a single process.

Sort of like having multiple instances of Apache running at the same time on the server. However, after several cases of server thrashing (endless memory swapping), I was left with two choices (other than increasing the RAM again). Either switch to Nginx which by default is built to work with PHP as a Fast CGI script, or configure Apache to run PHP using FastCGI with mpm-worker.

Due to time constraints I had to make the switch quick and I was sure I didn’t have enough time to debug a faulty new Nginx install. So I opted to go with the least intensive and more familiar option of changing Apache’s MPM to from prefork to worker.

Highlighted below are the simple steps I took to make this work in less than thirty minutes.

Note that the commands are for Debian distributions of Linux.

  1. Install the FPM CGI binary for PHP. apt-get install php5-fpm
  2. This will run as a daemon and you need to configure to run as a socket instead of it listening on a local port.
    • Open the file located at /etc/php5/fpm/pool.d/www.conf
    • Comment the line with listen = 127.0.0.1:9000 by adding a semicolon to the beginning of that line so it becomes ;listen = 127.0.0.1:9000
    • Now add a line after it with the following: listen = /var/run/php5-fpm.sock
      which makes PHP-FPM run as a socket
    • Restart the service by entering: service php5-fpm restart
  3. Now you need to install the FastCGI module for Apache’s worker MPM which would replace prefork. So you enter apt-get install libapache2-mod-fastcgi
  4. Once the installation is complete, Apache would have replaced your default prefork MPM with the new FastCGI module for worker.
  5. Finally you need to configure your sites to use the new FASTCGI module. It is preferable to make the changes to your main Apache configuration file located at /etc/apache2/apache2.conf
    • Use this config below making sure the directory path below exists or create a path that can be accessed on the server file system
    • <IfModule mod_fastcgi.c>
             AddHandler php5-fcgi .php
             Action php5-fcgi /php5-fcgi
             Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
             FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /var/run/php5-fpm.sock -pass-header Authorization
       </IfModule>
    • If you are using virtual hosts it is still preferable to make the changes in the main Apache config file in /etc/apache2/apache2.conf instead of adding it to each config file for each site you have enabled.
  6. Now, all that is left is for you to restart your web server typing the command: service apache2 restart

And that’s all to it. Your server should now work faster and suffer fewer memory outages than when it was configured to use prefork.

Best of luck in your adventure towards better performance …

Related Posts:

Using PHP5-FPM With Apache2 On Ubuntu 12.04 LTS

Setting up Apache with PHP-FPM

Installing Apache 2 with PHP FastCGI on Ubuntu 12.10

HOWTO: Building IPTables rules