Varnish + HTTP Cache: An intro guide for web developers – Part 1
Varnish, the web application accelerator:
Varnish development was initiated as a project within VG as a direct response to increasing demand hitting our servers hard. Existing caching systems were simply not fast or flexible enough to deal with VGs needs; and so Varnish was born with it’s first official release in 2006.
Varnish is now developed and maintained under a separate company (https://www.varnish-cache.org). It is open source (two-clause BSD license) and used among many large web applications including VG.
Varnish accelerates webapps by sitting in between the application server and your visitors. It intelligently keeps responses from your application server in memory to give blazing-fast responses to the clients that request the same information; giving the application server less workload.
Varnish as a developer:
I believe that all professional web developers should have an understanding of HTTP caching mechanisms that aid in creating high performance webapps, and have the skillset needed to work closely with sysops.
This blogpost will initiate a series of posts that will help you as a developer to get up to speed with relevant aspects of HTTP and server-side accelerators like Varnish.
You are familiar with the common *nix shells, server side web development and have a basic understanding of the HTTP protocol concepts.
Goal of part #1:
- Initial Varnish configuration and starting it
- Start a PHP 5.4+ development webserver and serve it’s content through Varnish. This is just a simple alternative, Apache is preferred if you are comfortable with setting that up.
Installation of Varnish, to get started:
We are assuming a Linux based environment for these posts.
A lot of things will be distribution-agnostic, but for the sake of convenience I will cover the Debian based distros like Ubuntu and Arch Linux for the things that are not. To get started learning I recommend simply installing Varnish on your workstation (or virtual machine image) and play around with it as you follow these posts. When installing Varnish on a dedicated server the process will be similar, but usually automated and managed by your ops team using something like Chef.
Most modern distributions have Varnish in their official package repositories. These packages will include Varnish itself (3.x) as well as the dependencies needed for it to run. That makes installation super-easy.
Here we see som information on the varnish package from Arch Linux package manager:
Name : varnish Version : 3.0.5-1 Description : High-performance HTTP accelerator Architecture : x86_64 URL : http://www.varnish-cache.org/ Licenses : BSD ... Depends On : gcc libedit pcre ... Installed Size : 1340.00 KiB ...
As you can see Varnish has a very small number of dependencies.
Go ahead and install it (Arch Linux):
pacman -S varnish
apt-get install varnish
Varnish is now installed. If we try to start it now it will fail as it is currently unconfigured (Arch Linux example output):
sudo systemctl start varnish Jan 08 11:07:37 arch-work systemd: Starting Web Application Accelerator... Jan 08 11:07:37 arch-work systemd: Started Web Application Accelerator. Jan 08 11:07:37 arch-work varnishd: Message from VCC-compiler: Jan 08 11:07:37 arch-work varnishd: No backends or directors found in VCL program, at least one is necessary. Jan 08 11:07:37 arch-work varnishd: Running VCC-compiler failed, exit 1 Jan 08 11:07:37 arch-work varnishd: VCL compilation failed Jan 08 11:07:37 arch-work systemd: varnish.service: main process exited, code=exited, status=2/INVALIDARGUMENT Jan 08 11:07:37 arch-work systemd: Unit varnish.service entered failed state.
We start with the basics, we simply just want to get Varnish up and running with a default configuration that is easy to modify onwards as we learn.
1. Open /etc/varnish/default.vcl in your favourite editor. This is the default configuration loaded by Varnish and is currently incomplete.
2. Uncomment (remove # in front of the respective lines) so that the default configuration template is exposed. (vim: ctrl+v, s-G, d)
3. We now have a valid Varnish Configuration (VCL) that contains some informative boilerplate! Take a few minutes to read through the syntax to get a feel for how this works.
On Ubuntu you will have to specify the startup parameters for Varnish in /etc/default/varnish, for example:
DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s malloc,256m"
Starting Varnish (Arch Linux):
sudo systemctl start varnish
Starting Varnish (Ubuntu):
sudo service varnish start
Now we are up and running! Varnish is now listening to port 80.
To confirm you can visit “localhost” in your browser to see the content, or simply perform a HEAD request using curl to see the response headers:
curl -I localhost HTTP/1.1 503 Service Unavailable Server: Varnish Content-Type: text/html; charset=utf-8 Retry-After: 5 Content-Length: 419 Accept-Ranges: bytes Date: Wed, 08 Jan 2014 10:22:28 GMT X-Varnish: 1205105390 Age: 0 Via: 1.1 varnish Connection: close
The 503 is expected and correct at this time. This is simply Varnish telling us that it cannot find our backend. The important thing here is that we verify that we got an actual HTTP response from Varnish.
Note: Varnish does not care about what you use to serve your content because it operates on the HTTP level. Apache, NodeJS, Tomcat are all fine examples of servers!
Caching your web application:
We have Varnish running now, and the default VCL has a “backend” defined at 127.0.0.1:8080.
Feel free to revisit /etc/varnish/default.vcl to confirm.
Let’s start a webserver serving a super-simple webapp on 127.0.0.1:8080. The rest of this article assumes that PHP 5.4+ is installed, and we will use PHP without any framework and the built-in webserver onwards.
Install php (if not already installed) (Arch Linux):
sudo pacman -S php
apt-get install php5 php5-cli
Create a directory for our little project:
mkdir cached_webapp && cd cached_webapp
Create a file in your favourite editor, index.php:
<?php echo 'Many http such speed many cache wow';
Start the PHP development webserver on the backend that Varnish is currently connected to. Remember to change the port if you did something different in varnish.vcl:
php -S 127.0.0.1:8080
NOTE: If you use Apache instead, just ignore the information regarding php -S
Check that the webserver serves content:
curl -I 127.0.01:8080 HTTP/1.1 200 OK Host: localhost:8080 Connection: close X-Powered-By: PHP/5.5.7 Content-Type: text/html
Awesome. Now let’s see if Varnish manages to serve us some content via that backend (Varnish runs on port 80):
curl -I 127.0.0.1 HTTP/1.1 200 OK Host: 127.0.0.1 X-Powered-By: PHP/5.5.7 Content-Type: text/html Content-Length: 35 Accept-Ranges: bytes Date: Wed, 08 Jan 2014 11:02:27 GMT X-Varnish: 1205105394 Age: 0 Via: 1.1 varnish Connection: keep-alive
All good. Varnish is now serving the content from our backend (PHP 5.4 dev http server). As you can see Varnish adds some header information. This is also configurable to developers and sysops through the Varnish configuration language that you have already seen briefly in /etc/varnish/default.vcl.
Let’s request the same content again:
curl -I 127.0.0.1 HTTP/1.1 200 OK Host: 127.0.0.1 X-Powered-By: PHP/5.5.7 Content-Type: text/html Content-Length: 35 Accept-Ranges: bytes Date: Wed, 08 Jan 2014 11:02:29 GMT X-Varnish: 1205105395 1205105394 Age: 2 Via: 1.1 varnish Connection: keep-alive
Notice the “Age” header. Varnish now served the response to us straight from memory without bothering php! You may also notice that the accesslog for PHP webserver running in terminal or Apache accesslog did not generate any output for this request. Varnish handled it all without asking PHP for anything.
Terminology you should be comfortable with now since used in the later parts:
- Update Varnish configuration: Refers to changing the VCL configuration file and reloading it by restarting varnish
- Change webapplication: Refers to updating the PHP code in our example project
Summary part #1:
We now have a basic environment set up for testing and learning Varnish and HTTP cache concepts.
In part #2 we will get started with HTTP headers and how to use these to get Varnish to behave exactly how we want.