VGTech is a blog where the developers and devops of Norways most visited website share code and tricks of the trade… Read more



Are you brilliant? We're hiring. Read more

Varnish + HTTP Cache: An intro guide for web developers – Part 1

DevOps

Varnish, the web application accelerator:

varnish

Varnish development was initiated as a project within VG as a direct response to increasing demand hitting our servers hard. Existing caching systems were simply not fast or flexible enough to deal with VGs needs; and so Varnish was born with it’s first official release in 2006.

Varnish is now developed and maintained under a separate company (https://www.varnish-cache.org). It is open source (two-clause BSD license) and used among many large web applications including VG.

Concept:
Varnish accelerates webapps by sitting in between the application server and your visitors. It intelligently keeps responses from your application server in memory to give blazing-fast responses to the clients that request the same information; giving the application server less workload.

Varnish as a developer:
I believe that all professional web developers should have an understanding of HTTP caching mechanisms that aid in creating high performance webapps, and have the skillset needed to work closely with sysops.

This blogpost will initiate a series of posts that will help you as a developer to get up to speed with relevant aspects of HTTP and server-side accelerators like Varnish.

Assumptions:
You are familiar with the common *nix shells, server side web development and have a basic understanding of the HTTP protocol concepts.

Goal of part #1:

  • Initial Varnish configuration and starting it
  • Start a PHP 5.4+ development webserver and serve it’s content through Varnish. This is just a simple alternative, Apache is preferred if you are comfortable with setting that up.

Installation of Varnish, to get started:
We are assuming a Linux based environment for these posts.
A lot of things will be distribution-agnostic, but for the sake of convenience I will cover the Debian based distros like Ubuntu and Arch Linux for the things that are not. To get started learning I recommend simply installing Varnish on your workstation (or virtual machine image) and play around with it as you follow these posts. When installing Varnish on a dedicated server the process will be similar, but usually automated and managed by your ops team using something like Chef.

Most modern distributions have Varnish in their official package repositories. These packages will include Varnish itself (3.x) as well as the dependencies needed for it to run. That makes installation super-easy.

Here we see som information on the varnish package from Arch Linux package manager:

Show code
Name : varnish
Version : 3.0.5-1
Description : High-performance HTTP accelerator
Architecture : x86_64
URL : http://www.varnish-cache.org/
Licenses : BSD
...
Depends On : gcc libedit pcre
...
Installed Size : 1340.00 KiB
...

As you can see Varnish has a very small number of dependencies.

Go ahead and install it (Arch Linux):

Show code
pacman -S varnish

Ubuntu:

Show code
apt-get install varnish

Varnish is now installed. If we try to start it now it will fail as it is currently unconfigured (Arch Linux example output):

Show code
sudo systemctl start varnish
Jan 08 11:07:37 arch-work systemd[1]: Starting Web Application Accelerator...
Jan 08 11:07:37 arch-work systemd[1]: Started Web Application Accelerator.
Jan 08 11:07:37 arch-work varnishd[1169]: Message from VCC-compiler:
Jan 08 11:07:37 arch-work varnishd[1169]: No backends or directors found in VCL program, at least one is necessary.
Jan 08 11:07:37 arch-work varnishd[1169]: Running VCC-compiler failed, exit 1
Jan 08 11:07:37 arch-work varnishd[1169]: VCL compilation failed Jan 08 11:07:37 arch-work systemd[1]: varnish.service: main process exited, code=exited, status=2/INVALIDARGUMENT Jan 08 11:07:37 arch-work systemd[1]: Unit varnish.service entered failed state.

Configuration:
We start with the basics, we simply just want to get Varnish up and running with a default configuration that is easy to modify onwards as we learn.

1. Open /etc/varnish/default.vcl in your favourite editor. This is the default configuration loaded by Varnish and is currently incomplete.
2. Uncomment (remove # in front of the respective lines) so that the default configuration template is exposed. (vim: ctrl+v, s-G, d)
3. We now have a valid Varnish Configuration (VCL) that contains some informative boilerplate! Take a few minutes to read through the syntax to get a feel for how this works.

On Ubuntu you will have to specify the startup parameters for Varnish in /etc/default/varnish, for example:

Show code
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"

Starting Varnish (Arch Linux):

Show code
sudo systemctl start varnish

Starting Varnish (Ubuntu):

Show code
sudo service varnish start

Now we are up and running! Varnish is now listening to port 80.
To confirm you can visit “localhost” in your browser to see the content, or simply perform a HEAD request using curl to see the response headers:

Show code
curl -I localhost
HTTP/1.1 503 Service Unavailable
Server: Varnish
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 419
Accept-Ranges: bytes 
Date: Wed, 08 Jan 2014 10:22:28 GMT
X-Varnish: 1205105390
Age: 0
Via: 1.1 varnish
Connection: close

The 503 is expected and correct at this time. This is simply Varnish telling us that it cannot find our backend. The important thing here is that we verify that we got an actual HTTP response from Varnish.
Note: Varnish does not care about what you use to serve your content because it operates on the HTTP level. Apache, NodeJS, Tomcat are all fine examples of servers!

Caching your web application:
We have Varnish running now, and the default VCL has a “backend” defined at 127.0.0.1:8080.
Feel free to revisit /etc/varnish/default.vcl to confirm.

Let’s start a webserver serving a super-simple webapp on 127.0.0.1:8080. The rest of this article assumes that PHP 5.4+ is installed, and we will use PHP without any framework and the built-in webserver onwards.

Install php (if not already installed) (Arch Linux):

Show code
sudo pacman -S php

Ubuntu:

Show code
apt-get install php5 php5-cli

Create a directory for our little project:

Show code
mkdir cached_webapp && cd cached_webapp

Create a file in your favourite editor, index.php:

Show code
<?php echo 'Many http such speed many cache wow';

Start the PHP development webserver on the backend that Varnish is currently connected to. Remember to change the port if you did something different in varnish.vcl:

Show code
php -S 127.0.0.1:8080

NOTE: If you use Apache instead, just ignore the information regarding php -S

Check that the webserver serves content:

Show code
curl -I 127.0.01:8080
HTTP/1.1 200 OK
Host: localhost:8080
Connection: close
X-Powered-By: PHP/5.5.7
Content-Type: text/html

Awesome. Now let’s see if Varnish manages to serve us some content via that backend (Varnish runs on port 80):

Show code
curl -I 127.0.0.1
HTTP/1.1 200 OK
Host: 127.0.0.1
X-Powered-By: PHP/5.5.7
Content-Type: text/html
Content-Length: 35
Accept-Ranges: bytes 
Date: Wed, 08 Jan 2014 11:02:27 GMT
X-Varnish: 1205105394
Age: 0
Via: 1.1 varnish
Connection: keep-alive

All good. Varnish is now serving the content from our backend (PHP 5.4 dev http server). As you can see Varnish adds some header information. This is also configurable to developers and sysops through the Varnish configuration language that you have already seen briefly in /etc/varnish/default.vcl.

Let’s request the same content again:

Show code
curl -I 127.0.0.1
HTTP/1.1 200 OK
Host: 127.0.0.1
X-Powered-By: PHP/5.5.7
Content-Type: text/html
Content-Length: 35
Accept-Ranges: bytes 
Date: Wed, 08 Jan 2014 11:02:29 GMT
X-Varnish: 1205105395 1205105394
Age: 2
Via: 1.1 varnish
Connection: keep-alive

Notice the “Age” header. Varnish now served the response to us straight from memory without bothering php! You may also notice that the accesslog for PHP webserver running in terminal or Apache accesslog did not generate any output for this request. Varnish handled it all without asking PHP for anything.

Terminology you should be comfortable with now since used in the later parts:

  • Update Varnish configuration: Refers to changing the VCL configuration file and reloading it by restarting varnish
  • Change webapplication: Refers to updating the PHP code in our example project

Summary part #1:
We now have a basic environment set up for testing and learning Varnish and HTTP cache concepts.

In part #2 we will get started with HTTP headers and how to use these to get Varnish to behave exactly how we want.

Full-stack developer. Linux/PHP/JavaScript/Databases @VG I currently work as Developer / Techlead @ Core


1 comments

  • arshad

    i have setup varnish for php + json web services for an ios app, could you please explain how to configure varnish for php web services files.


Leave your comment