Varnish

From Just another day in the life of a linux sysadmin
Jump to navigation Jump to search

At this point I would avoid using Varnish. Recommend cache-control tweaks plus cloudflare instead, nginx also has a proxy cache that will support ssl and there is now a cpanel plugin that handles ssl termination. Also keep in mind that php7 + opcache often reduces php cpu usage well below previous php versions usage too.

What it is

Varnish is a caching program and sits on port 80 and passes to the web server, much like how our Nginx installs pass dynamic content to Apache. Varnish uses a config file to control its myriad of caching options. Some basic explanation of the config file can be found here:

Varnish Introduction

Offical LW Varnish Support Policy

Keep in mind, I said port 80. Varnish does not (and will not) cache data over HTTPS. To get this functionality, you will have to have an SSL terminator in front of varnish, like nginx, configured to pass requests to varnish (which will then pass requests to apache (yo dawg))

Varnish 3 / Centos 5, 6, and 7

Automated Installer

not EA4 compatible! use the manual method.

Before using the below plBake, First! Do this: 

 yum install readline-devel

Desired Result (this will differ slightly depending on your OS version): 

Installed:
  readline-devel.x86_64 0:6.2-9.el7                                                              
Complete!


There is a plBake recipe for varnish which also installs mod_rpaf: 

wget -O /scripts/plbake http://layer3.liquidweb.com/scripts/plBake/plBake 
chmod 700 /scripts/plbake
/scripts/perlinstaller JSON

To install varnish to port 80, and proxy to Apache on 8081: 

/scripts/plbake varnish --install

For Ea4 you may have to link this file for mod_rpaf to complete its install: 

ln -s /usr/bin/apxs /usr/local/apache/bin/apxs

If this worked stop here, don't continue on through the lower setup portion of this wiki.
*** Please be sure to also install Mod_RemoteIP or mod_rpaf something similar to allow being able to see the correct IP addresses in the Apache logs!

Uninstall

To remove varnish, and place Apache back on port 80:

Even if it was not installed by plbake, you can run the plbake install then run the line below; it will properly remove varnish 

/scripts/plbake varnish --uninstall --refried

Please send feature requests and bug reports to ssullivan@liquidweb.com/mshooltz@liquidweb.com


CENTOS 7 Unistall plbake output:Sorry, this recipe does not work on CentOS 7 yet

Manual Install

Current Version is 3.0.5 as of 01/19/2014. Varnish is distributed in the EPEL (Extra Packages for Enterprise Linux) package repositories.

Varnish Cache Installation on RedHat

If you are on RHEL 5 or a compatible distribution, use 

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release/varnish-release-3.0-1.el5.centos.noarch.rpm

For RHEL 6 and compatible distributions, use 

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el6/noarch/varnish-release/varnish-release-3.0-1.el6.noarch.rpm

and then run 

yum install varnish


Setup (Manual Install ONLY) Short and Sweet Version, Single IP Server

1. Switch Apache to port 8080 via WHM -> "Tweak Settings" -> "Apache non-SSL IP/port"

2. Set the port Varnish runs on: 

vim /etc/sysconfig/varnish
change VARNISH_LISTEN_PORT=6081 to VARNISH_LISTEN_PORT=80

3. Set the host and port: 

vim /etc/varnish/default.vcl
.host = "se.rv.er.ip"
.port = 8080

4. Start Varnish: 

service varnish start
chkconfig --add varnish
chkconfig varnish on

Setup (Manual Install ONLY)

Template:Box Warning

Because Varnish listens on port 80, we need to switch Apache to an alternate port. The easiest way to do this is through WHM. Go to Tweak Settings and near the bottom you should see this set: 

0.0.0.0:80

Change that to something like 0.0.0.0:8080. Once you save, Apache will be reconfigured to listen on port 8080 and not port 80. But browsers still try to connect on port 80! Time to get Varnish up and running so it can redirect port 80 traffic to Apache! 

varnishd -a :80 -b localhost:8080

Template:Notice

This starts the Varnish daemon listening on port 80 while passing it's traffic to a webserver at localhost on port 8080. No need to open the port, this is all internal. 

varnishd -a :80 -b localhost:8080 -s malloc,400MB -T 127.0.0.1:2000

The same as above but -s malloc,400MB specifies a maximum of 400MB to be used for memory storage caching. -T 127.0.0.1:2000 specifies which port to run the Varnish management on.



  • An alternate spot for configuration of ports can be found under /etc/sysconfig/varnish. Look at that file with a text editor, and you will see there are 3 alternative options for configuration, and they will look something like this:
  1. Alternative 2, Configuration with VCL
  3. Listen on port 6081, administration on localhost:6082, and forward to
  4. one content server selected by the vcl file, based on the request. Use a
  5. fixed-size cache file.

DAEMON_OPTS="-a :6081 \ 

            -T localhost:6082 \
            -f /etc/varnish/default.vcl \
            -u varnish -g varnish \
            -S /etc/varnish/secret \
            -s file,/var/lib/varnish/varnish_storage.bin,1G"

This was set to be using 6081 as its listening port, although I had already ran the command from above (varnishd -a :80 -b localhost:8080) to specify ports.

Change the ' -a :6081 / ' to the port needed, for whatever Varnish needs to be at, in our instance it is 80, then save.

ALSO

Under /etc/varnish/default.vcl, when you open that with a text editor you are going to see, by default, two settings near the top:

  1. This is a basic VCL configuration file for varnish. See the vcl(7)
  2. man page for details on VCL syntax and semantics.
  4. Default backend definition. Set this to point to your content
  5. server.

backend default { 

 .host = "127.0.0.1";
 .port = "80";

}

Here you can set timeout limits and a bunch of other nice variables, but thats what google is for and not this. Change that port of "80" to whatever you now have Apache set to, which if you are following this, should be 8080:

backend default { 

 .host = "127.0.0.1";
 .port = "8080";

}

Once this is all done and saved, do /etc/init.d/varnish restart to get Varnish all restarted and have the settings take hold.

You can check if apache and varnish are listening on the proper ports by entering the command:

netstat -lntpe |grep 80

Which will show what is listening to port 80. Change the 80 to whichever port number you need to check.

  • Still working on this - dmichael
  • alternative configuration information added 10/19/10 tweaver


Sometimes Varnish wont start. There is a plethora of reasons why Varnish wont start.

Starting Varnish in debug mode to see what is going on.

Try to start varnish by:

  1. varnishd -f /etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000 -a 0.0.0.0:8080 -d


change the port number if the configuration is set with a different port.

Once Varnish is installed, add it to chkconfig so that it starts back up when the server reboots. Otherwise, all websites on the server will be down until Varnish is manually started: 

chkconfig --add varnish
chkconfig varnish on

PURGE requests

Some varnish compatible software will attempt to issue PURGE requests to varnish. These requests simply instruct Varnish to delete the file from cache. The one I've dealt with is W3 Total Cache for WordPress. In order to make varnish compatible with these requests, you'll need to update the varnish vcl file with the instructions on how to handle PURGE events as well as an access list for hosts that are allowed to issue these requests. It's a rather simple change, once you have the information you need.

Modify the following file: /usr/etc/varnish/default.vcl

Somewhere before sub vcl_recv, you'll need to setup the purge access list, make sure to supply the correct IP/Hostname information needed for your setup: 

acl purge {
    "127.0.0.1";
}

Now the next change needs to inside of the sub vcl_recv section. I just put it in at the top of the routine: 

if (req.request == "PURGE") {
    if (!client.ip ~ purge) {
        error 405 "Not allowed.";
    }
    return (lookup);
}

Finally, at the bottom of the file we'll need to create two new subs to handle the purge requests. These should be somewhere after the close } for sub vcl_recv: 

sub vcl_hit {
    if (req.request == "PURGE") {
        purge;
        error 200 "OK";
    }
}
sub vcl_miss {
    if (req.request == "PURGE") {
        purge;
        error 404 "Not cached";
    }
}

Now that your changes are in place, just do a quick restart of varnish and it should now handle PURGE requests properly.

A side effect of a varnish that does not support PURGE requests is that it'll pass the requests onto Apache, which doesn't support them either. Resulting in the apache error_log filling up with errors like this: 

 [Sat Jun 06 12:08:23 2015] [error] [client 127.0.0.1] Invalid method in request PURGE /.* HTTP/1.1

Performance Tweaking

http://www.varnish-cache.org/trac/wiki/Performance

chkserv.d

Create a file: /etc/chkserv.d/varnishd And put the following in it: <source lang='text'> service[varnish]=x,x,x,killall -TERM varnishd;sleep 2;killall -9 varnishd;/etc/rc.d/init.d/varnish stop;/etc/rc.d/init.d/varnish start,varnishd,varnish </source> Then, add the following to the end of /etc/chkserv.d/chkservd.conf <source lang='text'> varnishd:1 </source> This checks to see if there is a varnishd process running that is owned by the user varnish (using the "ps" command), and if there is not issues the following commands to restart varnish:

<source lang='text'> killall -TERM varnishd sleep 2 killall -9 varnishd /etc/rc.d/init.d/varnish stop /etc/rc.d/init.d/varnish start </source>

A failure and successful restart will look like the following in /var/log/chkservd.log:

<source lang='text'> Service Check Started Loading services .....cpanellogd....cpsrvd....ftpd....imap....lfd....named....queueprocd....varnish..Done [2016-02-24 14:55:25 -0500] OOM check ....Done [2016-02-24 14:55:25 -0500] Service check ....varnish [][check command:-][socket connect:N/A][fail count:1]Restarting varnish.... system: killall -TERM varnishd system: sleep 2 system: killall -9 varnishd system: /etc/rc.d/init.d/varnish stop system: /etc/rc.d/init.d/varnish start [notify:failed service:varnish]]...queueprocd [[check command:+][socket connect:N/A]]...p0f [[check command:N/A][socket connect:N/A]]...nscd [[check command:N/A][socket connect:N/A]]...named [[check command:+][socket connect:N/A]]...lfd [[check command:+][socket connect:N/A]]...imap [[socket_service_auth:1][check command:+][socket connect:+]]...ftpd [[socket_service_auth:1][check command:+][socket connect:+]]...dnsadmin [[check command:N/A][socket connect:N/A]]...cpsrvd [[http_service_auth:1][check command:N/A][socket connect:+]]...cpgreylistd [[check command:N/A][socket connect:N/A]]...cpanellogd [[check command:+][socket connect:N/A]]...Done Service Check Finished </source>

Useful Commands

Test varnish conf before restarting. Should output the full conf with no errors. 

varnishd -C -f  /usr/etc/varnish/default.vcl

Real time backend (apache) usage. Use this to figure out what is not being cached. 

varnishtop -i txurl

Filter logs for request from a single ip. 

varnishlog -c -m ReqStart:10.0.1.5

Disable Varnish Per Domain

In order to disable Varnish on a per domain basis you can use the following. 

vim /usr/etc/varnish/default.vcl

Find the following section 

# Example conf to bypass a domain you do not want to cache. 
#if (req.http.host ~ "$domain.com") {
#return (pass);
#}

Uncomment rows 2,3, and 4 and modify $domain.com

Save and restart varnish 

/etc/init.d/varnish restart

Disable Varnish Per Location

To disable Varnish on a per domain/path basis you can use the following. 

vim /usr/etc/varnish/default.vcl

Find the section of the config which holds the "sub vcl_recv" sections, and add in a new block below them: 

sub vcl_recv {
  if (req.http.host ~ "domain.com" && req.url ~ "^/your-path-location") {
    return(pass);
  }
}

Make sure to replace "domain.com" and "^/your-path-location" with the updated versions for the site in mention.

Then, save and restart varnish 

/etc/init.d/varnish restart

troubleshooting

Template:Notice It would seem that with Magento you tend to need to add a value to default.vcl or some pages will 503. 

vim /usr/etc/varnish/default.vcl

insert: 

.connect_timeout = 30s;

That may not be a high enough value, I've had to raise it much higher to get past the 503 in the past.

Multiple IP's and Vhosts? Varnish does not play well with multi vhosts/IP configurations. To correct you must edit /etc/varnish/default.vcl.

Include a backend for each IP on the server. In the vcl_recv() function you will want to match on the domain name and send the request to the appropriate varnish backend.

This exact configuration may seem a bit dramatic but with magento sometimes things take a long time. 

.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;


Varnish Fails to Start After Editing default.vlc

This is most likely cause by a version mismatch or syntax error. Make sure that the configuration you are attempting to use is intended for the Varnish version.

Is Varnish serving cached content?

If you are wanting to ensure that the content that Varnish is indeed caching, and serving from its cache, an HTTP header can be added that indicates a HIT or MISS. The following can be added to Varnish's default.vcl, or any other .vcl file that you may have as an include. Location will vary based on install method, but these locations are based on the plBake installation method, and using the default.vcl. As far as I know, the default location for an install performed via yum is located: /etc/varnish/default.vcl. 

vim /usr/etc/varnish/default.vcl

After vim'ing the file, you'll need to add the following vcl_deliver. Ensure that this is not within another "sub" section as you will get errors. 

sub vcl_deliver {
        if (obj.hits > 0) {
                set resp.http.X-Cache = "HIT";
        } else {
                set resp.http.X-Cache = "MISS";
        }
}

This will add an HTTP header of X-Cache, named X-Cache, that checks for "obj.hits", if it's not cached, you'll see MISS, if it is cached, you'll see HIT. 

curl -I -L myvpsdomain.com

Unless the domain is busy and is actively getting traffic, you'll likely see a miss based on the last visit and the time that you have Varnish set to cache for. Just run the curl again and see if it gets a HIT.

varnishncsa - Varnish requests displayed in Apache format

If you'd like to see what requests Varnish is actively serving, in the same format as Apache's domlogs, run: 

varnishncsa

You can also add a flag to add a "hit" or "miss" to the beginning of each request: 

varnishncsa -F  "%{Varnish:hitmiss}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i""

WordPress - Varnish 3

https://wiki.int.liquidweb.com/articles/Wordpress_-_Varnish

Drupal 7 - Varnish 3

https://wiki.int.liquidweb.com/articles/Drupal7-_varnish

Varnish 4 / Centos 7

Must install manually. jemalloc is a prerequisite. 

rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-4.0.el7.rpm
rpm -i http://cbs.centos.org/kojifiles/packages/jemalloc/3.6.0/8.el7.centos/x86_64/jemalloc-3.6.0-8.el7.centos.x86_64.rpm
yum install varnish
echo "    Varnish is installed from yum on this server" >> /etc/motd
systemctl enable varnish
systemctl restart varnish.service

Files in /etc/varnish.

WordPress - Varnish 4

https://wiki.int.liquidweb.com/articles/Wordpress_-_Varnish#Varnish_4_-_wordpress

Apache varnish confs

To apply to an individual virtualhost, 2_4 can be 2 and 2_2 on older servers. 

mkdir -p /usr/local/apache/conf/userdata/ssl/2_4/$user/$domain/ && mkdir -p /usr/local/apache/conf/userdata/std/2_4/$user/$domain/

To apply to all virtualhosts, the 2_4 2_2 and 2 is not needed in this case. 

mkdir -p /usr/local/apache/conf/userdata/ && mkdir -p /usr/local/apache/conf/userdata/ssl/

To apply to all virtualhosts for a specific user, 2_4 can be 2 and 2_2 on older servers. 

mkdir -p /usr/local/apache/conf/userdata/std/2_4/$user/ && mkdir -p /usr/local/apache/conf/userdata/ssl/2_4/$user/

Create a $anything.conf file in one of the about folders and add. 

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined

Rebuild httpd.conf and restart apache to apply the above vhost includes. 

/scripts/rebuildhttpdconf

Troubleshooting

Varnish & new IPs

When you add a new IP through WHM, nothing happens to the varnish config, you need to add a new stanza manually to /usr/etc/varnish/default.vcl, in the following format: 

backend b123_45_67_89 {
        .host = "123.45.67.89";
        .port = "8080";
}
acl a123_45_67_89 {
        "123.45.67.89";
}

Make sure the backend port is whatever it should be for your system (nginx/apache/tomcat/etc). Then, inside the subroutine vcl_recv, add:

VARNISH 3: 

        if (server.ip ~ a123_45_67_89) {
                set req.backend = b123_45_67_89;
        }

VARNISH 4: 

        if (server.ip ~ a123_45_67_89) {
                set req.backend_hint = b123_45_67_89;
        }

It should be clear where to add these alongside the others. Then, restart varnish. This update method is the same for any varnish install method.

If this is the first additional IP, then change the 'default' backend to match the above format, and add an acl and if statement for it.

Varnish & .htpasswd

Varnish was not accepting the user/pw for an .htpasswd set up but going directly to Apache worked - fixed by adding the following

in sub vcl_recv() put the following: 

if ( req.url ~ ".*htaccess.*" ) {

   return(pass);

}

Varnish & Nginx

Ran into a server that had varnish & nginx installed via PlBake. While this is fine, it looks like one domain was setup on the server, when other accounts were moved over though nginx couldn't start as it tried to bind those new accounts to port 80. Services were managed for the following ports: 

varnish: 80
nginx:   8081
apache:  8088

The varnish config looked to be the default, pointed to 8081 

 backend default {
     .host = "127.0.0.1";
.port = "8081";
}

Varnish was already configured to proxy through to nginx, however I had to update the ports in the nginx conf files (/etc/nginx/vhosts) for each domain to the proper port. Sample file: 

[root@exp /etc/nginx/vhosts]# cat tung.show2work.com 
server {
        error_log /var/log/nginx/vhost-error_log warn;
        listen 67.227.228.146:8081;
        server_name tung.show2work.com www.tung.show2work.com ;
        access_log /usr/local/apache/domlogs/tung.show2work.com-bytes_log bytes_log;
        access_log /usr/local/apache/domlogs/tung.show2work.com combined;
        root /home/tungshow2work/public_html;
        location / {
                location ~.*\.(3gp|gif|jpg|jpeg|png|ico|wmv|avi|asf|asx|mpg|mpeg|mp4|pls|mp3|mid|wav|swf|flv|html|htm|txt|js|css|exe|zip|tar|rar|gz|tgz|bz2|uha|7z|doc|docx|xls|xlsx|pdf|iso)$ {
                        expires 7d;
                        try_files $uri @backend;
                }
                error_page 405 = @backend;
                add_header X-Cache "HIT from Backend";
                proxy_pass http://67.227.228.146:8088;
                include proxy.inc;
        }
        location @backend {
                internal;
                proxy_pass http://67.227.228.146:8088;
                include proxy.inc;
        }
        location ~ .*\.(php|jsp|cgi|pl|py)?$ {
                proxy_pass http://67.227.228.146:8088;
                include proxy.inc;
        }
        location ~ /\.ht {
                deny all;
        }
}

Also to avoid this problem in the future I updated the Hooks used to create the nginx conf files to the proper ports in these files, same changes as the nginx conf file above: 

https://wiki.int.liquidweb.com/articles/Nginx#Cpanel_Hooks

ref https://liquidweb.zendesk.com/agent/tickets/508037

Varnish & Cloudflare

You may modify vcl_recv to strip the __cfduid cookies set by CloudFlare so Varnish can cache the response. The following VCL will strip all cookies starting with two underscores or including 'has_js', such as CloudFlare and Google Analytics cookies:

Add to /usr/etc/varnish/default.vcl 

    sub vcl_recv {
      # Remove has_js and CloudFlare/Google Analytics __* cookies.
      set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
      # Remove a ";" prefix, if present.
      set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
    }

Source: Can I use CloudFlare and Varnish together?

Varnish & WordFence

For customer's that want to use WordFence they need to first enable the X-Forwarded-For header in Varnish, and then configure WordFence to look for it. https://docs.wordfence.com/en/Wordfence_options#Use_the_X-Forwarded-For_HTTP_header._Only_use_if_you_have_a_front-end_proxy_or_spoofing_may_result.

Retrieved from "https://jparks.work/index.php?title=Varnish&oldid=166"