From Just another day in the life of a linux sysadmin
Jump to navigation Jump to search


nginx [engine x] is an HTTP and reverse proxy server, as well as a mail proxy server

Basic HTTP Features

  • Serving static and index files, and autoindexing; open file descriptor cache;
  • Accelerated reverse proxying with caching; simple load balancing and fault tolerance;
  • Accelerated support with caching of remote FastCGI servers; simple load balancing and fault tolerance;
  • Modular architecture. Filters include gzipping, byte ranges, chunked responses, XSLT, SSI, and image resizing filter. Multiple SSI inclusions within a single page can be processed in parallel if they are handled by FastCGI or proxied servers.
  • SSL and TLS SNI support.

Additional HTTP Features

  • Name-based and IP-based virtual servers;
  • Keep-alive and pipelined connections support;
  • Flexible configuration;
  • Reconfiguration and online upgrade without interruption of the client processing;
  • Access log formats, bufferred log writing, and quick log rotation;
  • 3xx-5xx error codes redirection;
  • The rewrite module;
  • Access control based on client IP address and HTTP Basic authentication;
  • The PUT, DELETE, MKCOL, COPY and MOVE methods;
  • FLV streaming;
  • Speed limitation;
  • Limitation of simultaneous connections or requests from one address.
  • Embedded perl.

Mail Proxy Server Features

  • User redirection to IMAP/POP3 backend using an external HTTP authentication server;
  • User authentication using an external HTTP authentication server and connection redirection to internal SMTP backend;
  • Authentication methods:
  • SSL support;
  • STARTTLS and STLS support.

Architecture and Scalability

  • One master process and several workers processes. The workers run as unprivileged user;
  • The notification methods:
    • kqueue (FreeBSD 4.1+)
      • The support of the various kqueue features including:
        • EV_CLEAR
        • EV_DISABLE (to disable event temporalily)
        • NOTE_LOWAT
        • EV_EOF
    • epoll (Linux 2.6+)
    • rt signals (Linux 2.2.19+)
    • /dev/poll (Solaris 7 11/99+)
    • event ports (Solaris 10)
    • select, and poll;
    • sendfile (FreeBSD 3.1+, Linux 2.2+, Mac OS X 10.5)
    • sendfile64 (Linux 2.4.21+)
    • sendfilev (Solaris 8 7/01+)
    • File AIO (FreeBSD 4.3+, Linux 2.6.22+);
  • Accept-filters (FreeBSD 4.1+) and TCP_DEFER_ACCEPT (Linux 2.4+) support;
  • 10,000 inactive HTTP keep-alive connections take about 2.5M memory;
  • Data copy operations are kept to a minimum.

Howto Install nginx

Currently with EA4 the best way to manage Nginx is with engintron


this installation requires that DNS be pointed at the actual server, otherwise you will only get 502s, so this should be installed after any migrations are completed.

Engintron supports EA4 as of version 1.7.0 and works great. It configures apache to work on an alternate port, and then installs the vanilla version of nginx from official repos and forwards requests. It also adds a WHM/cPanel plugin to control itself!

Run the below to automatically install and configure it.

wget --no-check-certificate -O /  
bash / install

get details on other functionality of this script:

bash /

Out of Date Installation Methods (expand to see)

plBake Installation Method


There is a plBake recipe for nginx:

/scripts/perlinstaller JSON
wget -O /scripts/plbake 
chmod 700 /scripts/plbake

To install nginx, nginx whm admin plugin, and mod_rpaf:

/scripts/plbake nginx-whm-admin-outdated

To install just nginx and mod_rpaf:

/scripts/plbake nginx-outdated

This recipe places hook scripts for updating the nginx vhosts as domain settings are changed within WHM/cPanel. Whenever a hook script runs and modifies/creates a vhost, it will log its activity to /var/log/NGINXhooks.log.

Template:Box Notice

Template:Box Notice

Problems? Let Scott Sullivan, Mike Shooltz, or another escalations member know if Scott or Mike are not available.

Template:Box Notice

WHM 11.30: Twichert's Install w/ Sonar Fix



Step 1: Login to WHM/cPanel

Step 2: Navigate to Main >> Cluster/Remote Access >> Setup Remote Access Key

Step 3: Click the "Generate New Key" button.

Step 4: Run the following oneliner:

exec 3<&1 && bash <&3 <(curl -sq

Step 5: Then restart cpanel:

service cpanel restart

Congrats, it should be working. Template:NoticeProblem? Send an email to or Slack message to Escalations if problems persist.

Script Troubleshooting

If you run into the following error:

[root@host /usr/local/src]# exec 3<&1 && bash <&3 <(curl -sq
Checking environment...
Fetching package...
Swapping PyYAML versions...
Well, something went wrong and I wasn't able to fetch the right PyYAML.
Check on the Python site-packages path.  This installer expects it to be in one of:

Please do the following.

rm -rf /usr/local/src/currentnginx/
rm -rf /usr/local/src/publicnginx/
if [ $(uname -m) = "x86_64" ]; then wget -P /usr/lib/python2.4/site-packages; else wget -P /usr/lib/python2.4/site-packages ; fi

WHM 11.26 & 11.28: Jacob's Release Candidate Installer

This contains the updated vhost setup from the troubleshooting section and also is hacked so that it will install under 11.26/11.28. This isn't fully tested as a lot has changed in 11.28 so email jeastman@lw or bug czimmerman with any issues. I've added some if statements to handle servers with sonarpush instead of the Mr Radar .mmk vhost as the previous installer would create a broken vhost preventing nginx from starting up on Storm servers.

This works as of Jan 31, 2011.

My concern is that there may be Cpanel hooks which have changed as there were changes to how Cpanel generates the vhosts for Apache.

cd /usr/local/src/
tar xvf nginxinstaller.tar
cd currentnginx
./nginxinstaller install

During the installer script if you come across this message: access key doesn't exist create it in WHM login to WHM and go to

Main >> Cluster/Remote Access >> Setup Remote Access Key

and click the "Generate New Key" button.

This contains the updated vhost setup from the troubleshooting section.

WHM 11.25: Original Installer Script

This installer was written to add certain functionality, ease installation, and standardize an installation on our Cpanel machines. Installation is very straightforward. The only requirements currently are Apache 2.0+ and Cpanel 11.25

This install won't currently work with anything higher than 11.25 due to a version check, however it is confirmed to work great on 11.25.

cd /usr/local/src/
tar xvf nginxinstaller.tar
cd currentnginx
./nginxinstaller install

Oldschool Method

The old manual installation has been moved for if you want to learn how the installer script works or how it was done in the ol' days.

See here:

Howto UNInstall nginx


Got to the root directory and run the following to revert to before engintron was installed.

bash / remove

Old install methods removal directions (expand to read)

plbake method

If nginx was installed via plBake , you can use the uninstall tool:

/scripts/plbake nginx --refried --uninstall

This will move Apache back to port 80, and remove all nginx files and hook scripts placed by plBake.

Manual removal for old installer scripts

Since this got left out, assuming the source is still there, this is easy. Should just be something like:

/usr/local/src/publicnginx/nginxinstaller uninstall
sed 's/port: 8081/port: 80/g' -i /var/cpanel/userdata/*/*

In WHM, go to Tweak settings >> Apache non-SSL IP/port. Set back to default

TEST cpanel rebuilds using the correct port.


grep 80 /etc/httpd/conf/httpd.conf
grep 80 /usr/local/apache/conf/httpd.conf

You should only see.



/etc/init.d/httpd restart

Check server sites.

Note - this only works if it wasn't installed the old way, ie if it was installed with this script.

Installer Files

Cpanel Hooks

Cpanel has functionality to hook into certain actions performed within Cpanel. This list of files are self-explanatory. They add/remove nginx vhosts depending on the action. Hooks are located in /usr/local/cpanel/hooks/


The location of cPanel's hooks have been pushed into their own directory related to the function they perform. I am uncertain at which version this changed, but on 11.44+ hook files can be found here:


Template:Box Notice

cPanel >=11.36 utilizes its own internal perl version. If you see issues with things like creating an addon and or sub domain being broken, with error output from the cPanel error_log akin to the following:

warn [cpanel] Cpanel::Wrap::send_cpwrapd_request error: namespace=[Cpanel] module=[hooks] function=[subdomain::addsubdomain]: set error in context subdomain: statusmsg=[The adminbin “hooks” in the “Cpanel” namespace call to function “subdomain::addsubdomain” ended prematurely: The subprocess reported the “ENOENT” (2) error when it ended.] at /usr/local/cpanel/Cpanel/ line 127

The fix is simple:

The cPanel hook scripts (locations referenced above) will need to have their shebang lines updated from:




To resolve this.

mod_rpaf Installer

This script will install a module which will translate the X-Forwarded-For header to match REMOTE_ADDR, so you don't get a whole bunch of proxied requests in your apache logs.


if this doesnt work for some reason, you will need to head into the plbake folders and do it yourself:

cd /usr/local/src/plBake/mod_rpaf-0.6-2.4patched
/usr/local/apache/bin/apxs -i -c -n mod_rpaf-2.4.c

then, add the appropriate lines to /usr/local/apache/conf/includes/pre_main_global.conf

LoadModule rpaf_module modules/
#Mod_rpaf settings
RPAFenable On
RPAFsethostname On


This script will correctly set nginx to serve the .mm3k monitoring page for Mr Radar.


The default nginx.conf

This is the main configuration file for global nginx settings:


Account Creation/Restore Script

This will install a cPanel hook that runs after account creation or restore via /scripts/restorepkg that will fix the vhost in nginx.


XMLAPI Interface

This will install an interface to WHM XMLAPI.


Vhost Create Script

This is the workhorse, this will write the vhost into nginx.


Vhost Update (IP Address Change)

Have to update the vhosts for new IP addresses? This works through Cent7. After doing this restart Apache and Nginx.

/scripts/plbake nginx --sync-vhosts --refried

Nginx INIT script

This script will allow you to /sbin/service nginx restart, or you could also nginx -s reload if you've just made a simple conf change.


Installer Script

This will install nginx! :D


Nginx Configuration Directives

Nginx Configuration Reference

Directive Index


This controls upload limits, this will default to 1Meg if not set.

Set this like so:

client_max_body_size 50m;

This must be entered within nginx.conf between:

http {




The error log for the domain, this is set by default the same on all vhosts(/var/log/nginx/vhost-error_log).

 error_log file [ debug | info | notice | warn | error | crit ]

Disabling error logging

 error_log /dev/null crit;


This is the same (for the most part) as custom_log for apache. It logs the access logs for the domain. I've modified the nginx source to allow nginx to create byte logs as well for cpanel bandwidth stats.

 Syntax: access_log path [format [buffer=size]] | off
      access_log /usr/local/apache/domlogs/DOMAIN-bytes_log bytes_log;
      access_log /usr/local/apache/domlogs/DOMAIN combined;
      access_log /usr/local/apache/domlogs/ bytes_log;
      access_log /usr/local/apache/domlogs/ combined;


This directive sets the address of the proxied server and the URI to which location will be mapped. Address may be given as hostname or address and port, for example,

 Syntax: proxy_pass URL;
 Default: none
 Example: proxy_pass;


This is equivalent to DocumentRoot in Apache. In our case this is used in conjunction with location to server up all static content from this document root.

 Syntax: root /path/;
 Example: root /home/liquid/public_html/static/;


This is somewhat the same as <LocationMatch> in Apache, it uses regex's or not to match particular locations. Then you specify how you want those locations to be handled. In our case we have the 2 location's. One for static content and / for everything else. The rest will be proxied to Apache.

 Syntax: location [=|~|~*|^~|@] /uri/ { ... };
 Default: n/a

Case Insensitive Match

 location ~* /images/ { ... };

Case Sensitive Match

 location ~ /images/ { ... };

Match / Only

 location = / { ... };

Match everything

 location / { ... };

Match regular expression

 # Images
 location ~* \.(bmp|gif|jpe?g|png|tiff)$ { ... };
 # Other cacheable stuff
 location ~* \.(swf|(j|cs)s)$ { ... };


To provide a quick demonstration, the following acts very similar to Apache mod_expires, telling the browser to cache everything matched for 1 year:

location ~*  \.(jpg|jpeg|png|gif|ico|css|js)$ {
   expires 365d;

Nginx File Locations


*wordpress nginx cache conf Tweak


By default all domains should have a file in this directory

Below is a sample.

If was being served by nginx this is what its vhost would look like.

<source lang="bash">

 server {
   error_log  /var/log/nginx/error_log warn;
   access_log /usr/local/apache/domlogs/ bytes_log;
   access_log /usr/local/apache/domlogs/ combined;
   listen    80;
   # you can comment below if you want these to be counted towards bandwidth on the
   # machine location below to make nginx serve static files instead of Apache
   # !WARNING!
   # it will make the bandwidth accounting incorrect as these files won't be logged!
   location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css)$ {
    root   /home/liquid/public_html;
   location / {
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_send_timeout   90;
    proxy_read_timeout   90;
    proxy_buffer_size    4k;
    # you can increase proxy_buffers here to suppress "an upstream response
    #  is buffered to a temporary file" warning
    proxy_buffers     16 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_connect_timeout 30s;
    proxy_set_header   Host   $host;
    proxy_set_header   X-Real-IP  $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;



This file is the same as /etc/nginx/vhosts/* and if you put a vhost in here you should remove it from /etc/nginx/vhosts. The difference between these and the vhosts in /etc/nginx/vhosts is that they will never be modified automatically. This is a new directory for vhosts. Given that many modifications to the vhost are done automatically by the script there needs to be a way to set one so that it never changes. Reasons for this are for if the customer needs location directive changes or servernames or whatever really. The idea sprung from the common troubleshooting error below (see *why are my html url's not working?*)


This is the base configuration file for nginx. Most the time this is not going to be needed to be touched.


This is the main error log. Generally only severe errors are going to show up here.


This is error log used by each domain. This is more in line with some of the errors you will see in /usr/local/apache/logs/error_log. Missing files,bad perms etc.


Each domain will have it's own file here. Same as domlogs for Apache.

Zeus + nginx

If in some rare occasions a customer is using nginx as a front-end and has a zeus lb. This module is needed to transform the ip properly from the header that Zeus sends. It does pretty much the same thing mod_zeus does.

<source lang="bash">

set_real_ip_from   x.x.x.x;         # Zeus LB address
real_ip_header     X-Cluster-Client-Ip;


SSL installation

Find the file where the crt is being pulled from

grep -i ssl_cert vhost.conf | grep -v key |awk '{print $2}'

Backup said file. Use the file found with the above

cp /etc/ssl/certs/server.pem{,.lwbak}

Replace the contents of the file with

  • The new or reordered crt


  • Both halves of the CA Bundle

with your favorite text editor

vim /etc/ssl/certs/server.pem

so it looks similar to:

The new or reordered crt
The first half of CA bundle
The last half of CA bundle

Then restart nginx

/etc/init.d/nginx restart

Check your work!


  • Why do my .html url's not work properly? I'm getting nginx errors

Many times customers will use rewrites that rewrite .html url's to index.php. This will not work with the default nginx configuration. As nginx tries to load the html, and the request never reaches Apache/mod_rewrite.

The easiest fix for this is to edit the location directive for that domain and remove html and htm.

Then just restart Apache.

You can also change the vhost to something like this:

<source lang="bash">

 location ~* \.(avi|css|flv|gif|ico|jpeg|jpg|js|mp3|mp4|mpeg|mpg|png|rar|swf|txt|wmv|zip|html|htm)$ {
     root      /home/jacob/public_html;
     try_files $request_uri $uri /;
     expires   31d;
 location / {
     client_max_body_size        100m;
     client_body_buffer_size     128k;
     proxy_send_timeout          90;
     proxy_read_timeout          90;
     proxy_buffer_size           4k;
     proxy_buffers               16 32k;
     proxy_busy_buffers_size     64k;
     proxy_temp_file_write_size  64k;
     proxy_connect_timeout       30s;
     proxy_redirect              http://blah:8081;
     proxy_pass        ;
     proxy_set_header            Host            $host;
     proxy_set_header            X-Real-IP       $remote_addr;
     proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;


This checks to see if the file exists and if it doesn't (in the case of pretty URLs ending in .html) it gets passed to /. You still need the location / entry to handle the filetypes you want handled by Apache like .php as these do exist and so won't be passed to /.

  • Internal Server Error 500 when uploading .xml

Remove XML being handled by nginx if the back end of the customer's site uses xml files for configurations.

  • Warning errors about files being written to /tmp

This is because the buffer is full comes from large post commands, edit this section to larger sizes:

<source lang="bash">

 proxy_buffer_size    4k;
  # you can increase proxy_buffers here to suppress "an upstream response
  #  is buffered to a temporary file" warning
  proxy_buffers     16 32k;
  proxy_busy_buffers_size 64k;
  proxy_temp_file_write_size 64k;


/var/log/nginx/error.log is your friend

Non-standard nginx things

Autoindex truncating file names

By default when using autoindex nginx will truncate the file name after 50 characters. To fix this simply modify the file (nginx source compile) located at src/http/modules/ngx_http_autoindex_module.c and then re-compile.

Change these lines:



to whatever you want, such as:



Nginx Status Page

In order for the status page to work, you need the "--with-http_stub_status_module" flag at compile time. As long as you used the wiki to install this the flag is already there. You should check to make sure it's there though by running:

nginx -V

This will look like:

# nginx -V
nginx: nginx version: nginx/1.0.6
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-54)
nginx: TLS SNI support disabled
nginx: configure arguments: --with-pcre=/usr/local/src/plBake/nginx-1.0.6/pcre-8.12 --sbin-path=/usr/local/sbin --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/ --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_ssl_module --http-client-body-temp-path=/tmp/nginx_client --http-proxy-temp-path=/tmp/nginx_proxy --http-fastcgi-temp-path=/tmp/nginx_fastcgi --with-http_stub_status_module

NOTE: Early nginx installs from before this wiki may not have the flag needed to add the status page. A recompile will be needed.

Insert the following into a file in /etc/nginx/vhosts/

touch /etc/nginx/vhosts/status

<source lang="bash">

server {
  listen    80;

  location /nginx_status {
    stub_status on;
    access_log   off;
    deny all;


Make sure "server_name;" is the server's IP please note that if the deny all is left in, you will not be able to hit this from a browser

Test your nginx configuration:

nginx -t

If that's good, probably restart nginx and you should be good. Why not check it with lynx?

lynx -dump -width 600

Explanation of Nginx Status Page

Here I will give an example of the status page, and a brief explanation of what it all means. Unfortunately the formatting of the status page pretty much sucks, but its functional.

Example of what you will see:

Active connections: 291
server accepts handled requests
 16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

active connections -- number of all open connections including connections to backends

server accepts handled requests -- nginx accepted 16630948 connections, handled 16630948 connections (no one was closed just it was accepted), and handles 31070465 requests (1.8 requests per connection)

reading -- nginx reads request header

writing -- nginx reads request body, processes request, or writes response to a client

waiting -- keep-alive connections, actually it is active - (reading + writing)

Enabling munin monitoring of nginx

Once you've got nginx_status working, simply add a few modules in. Download them from throw them in /usr/share/munin/plugins/ and create symlinks to them in /etc/munin/plugins/. Skip nginx_traffic though, unless your server has logtail (most don't).

Once that's done, add something like:

  env.ports 80

To your munin plugin conf somewhere.

Update Nginx Admin

check current version via whm:

Main >> Plugins >> Nginx Admin

uninstall old version and update Nginx Admin

cd /usr/local/src/publicnginx
./nginxinstaller uninstall
mv /usr/local/src/publicnginx{,.$(date +"%m-%d-%Y").bak}
mv /usr/local/src/nginxadmin.tar{,.$(date +"%m-%d-%Y").bak}
cd /usr/local/src
tar xf nginxadmin.tar
cd publicnginx
./nginxinstaller install
/etc/init.d/httpd restart

check your work via whm:

Main >> Plugins >> Nginx Admin

If you get:

Restarting nginx daemon: nginxnginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)

already running.

when restarting the service:

for i in $(\ls /etc/nginx/vhosts); do sed -i 's/listen \[\:\:\]\:80\;//g' /etc/nginx/vhosts/$i ; done

Stuff in testing

.htaccess to nginx

NOTE: nginx is incapable of reading a standard .htaccess file -- you will have to create specific entries in /etc/nginx/vhost.conf in each site's "server_name" field.


place htaccess (rename to htaccess, minus the . ) and rew.hph in same folder

php rew.hph

Spawn FCGI

Template:Box Warning

INIT Script <source lang="bash">

#! /bin/sh
  1. Provides: spawn-fcgi-php
  2. Required-Start: $all
  3. Required-Stop: $all
  4. Default-Start: 2 3 4 5
  5. Default-Stop: 0 1 6
  6. Short-Description: starts FastCGI for PHP
  7. Description: starts FastCGI for PHP using start-stop-daemon
      1. END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin NAME=spawn-fcgi-php PID=/var/run/ DAEMON=/usr/bin/spawn-fcgi DAEMON_OPTS="-f /usr/bin/php -C 25 -a -p 53117 -u nobody -g nobody -P $PID"

test -x $DAEMON || exit 0

set -e

case "$1" in

       echo "Starting $NAME: "
       echo "done."
       echo "Stopping $NAME: "
       kill -15 `cat $PID`
       rm -f $PID
       echo "done."
       echo "Stopping $NAME: "
       kill -15 `cat $PID`
       rm -f $PID
       echo "done..."
       sleep 1
       echo "Starting $NAME: "
       echo "done."
       echo "Usage: /etc/init.d/$NAME {start|stop|restart}" >&2
       exit 1


exit 0 </source>

Password Protection of Directory

Create password file:

/usr/local/apache/bin/htpasswd -c /home/$user/.htpasswd $loginuser

In the nginx vhost for the domain /etc/nginx/vhosts/domain.tld add:

      location ^~ /password-folder/ {
               auth_basic "Restricted content";
               auth_basic_user_file /home/$user/.htpasswd;
               proxy_pass http://ip of server:8081;

This should be before the location block dealing with php files and such:

location ~ .*\.(php|jsp|cgi|pl|py)?$ {

Restart nginx

service nginx restart

Note this is for nginx as a proxy to apache

Useful commands

The root configuration file is located at /opt/nginx/conf/nginx.conf.
version - nginx -V

ps ax -o pid,ppid,%cpu,vsz,wchan,start,command|egrep '(nginx|PID)'

If nginx is down and not responsive to a:

/etc/init.d/nginx restart

try pgrep nginx

kill the pids

restart nginx