Musings on software development…

Banning URLs from Varnish using Apache Camel and RabbitMQ – Part 1


Hello and Welcome! :-)

Over the course of three postings, I would like to present a tutorial on using RabbitMQ and Apache Camel to BAN (their parlance for removing) URLs (objects) held within Varnish Cache. This proposed approach allows for a complete decoupling of application logic from the caching system thus promoting greater flexibility, scalability and resiliance – in effect creating a sophisticated event driven architecture that would grow with business needs. Think about the enormous benefits and potential here – an application-initiated (and as-close-to-possible) realtime event driven mechanism for removing objects from within Varnish!

I hope you find this tutorial useful to you and I welcome any feedback you have. Please do drop me a email or comment if you have anything you wish to add :-)

Before I go any further I would like to sincerely thank the members of my team for doing the initial work on this – Vinay, Diego, Craig, Fred and Manju – your contributions are gratefully appreciated – thanks guys! :-)

In today’s posting I will introduce the key players and describe the  technologies that will be used.

Basic Requirements and Assumptions

These articles are written for Java developers and application support personnel in mind. The language used is Java 7 and the target platform is Unix based (I personally use Debian testing). There are comprehensive instructions on each application website for Varnish and RabbitMQ on how to download, install and do an initial configuration – please have a read and please install each component. The project source code and associated files are hosted on BitBucket as a Mercurial repository.

NOTE: Although the target audience is anyone with a familiarity of Java, the information presented here can be adapted to *ANY* architecture. All you need is a way to send message to a queue, for something to pick the message off the queue and process it, then for something to send on the processed message to a running instance of Varnish. An easy recipe to follow :-)

Architecture Overview

Shown below is what we will be aiming towards for our BAN solution:

The Key Players

Without further ado, let me introduce the core components (the versions in brackets are those that I had when I did this proof of concept):

Varnish (3.0.2)

Varnish Cache is a free sophisticated modern web accelerator that is growing in adoption and use throughout the world. It sits in front of any server that talks HTTP  and caches the response from the backend server. In our experience, Varnish has proven itself to be very reliable, very fast and very easy to configure. If you have a need to dramatically speed up responses to the client, I would highly recommend having a serious look at Varnish to see if it will help you (chances are it will! :-)).

RabbitMQ (2.7.1)

RabbitMQ is an implementation of an enterprise messaging system that talks AMQP (it can support other protocols such as STOMP). RabbitMQ is highly reliable and scalable and a good choice if you are looking for a way to send messages between systems – it will be the “backbone” that facilitates our event driven architecture.

Apache Camel (2.8.3)

A very popular integration framework that knows how to route, split, aggregate and do lots more with messages that flow through it. We will be using Apache Camel to pick messages from RabbitMQ and process them for sending on to Varnish.

Camel-Spring-AMQP [and Spring AMQP] (1.0)

A recently developed component for Apache Camel that speaks Spring-AMQP natively (it uses Spring AMQP underneath the covers). We will be using this component to connect to RabbitMQ.

Configuring Varnish

The default out-of-the-box configuration for Varnish will suffice for now. We will reconfigure Varnish in another posting to support BANing of URLs via a customised HTTP requests. An example Varnish configuration file is included in the project source code (in the conf/varnish directory).

Configuring RabbitMQ

It is important that RabbitMQ is installed with the web management console included – although not a requirement to use RabbitMQ, it really really makes life a lot simplier (and we will be using the web management console to send messages). The simplest thing is to install directly from the download on the website. We will require a Queue to be configured that will hold our HTTP BAN messages. I’m a big fan of images to explain what to do, so please find instructions below:

Log into RabbitMQ Web Management console:

Click on the Queues tab and create the following Queue:

You should see this when you click Add queue:

We now have to associate the Queue to an Exchange (along with a Routing Key):

You should see this when you click Bind:

After achieving this step, you now will have a Queue (banQueue) bound to an Exchange ( with the Routing Key (banQueue). We can now use this queue when we send ban messages.

Configuring Apache Camel

No configuration at the moment is required since the project contains the an example Camel configuration for you to use (in the conf/spring directory). You will probably need to adapt this for your own use later.

Configuring the Varnish-Ban Project

If you have not done so already, please clone the project to your machine. It is a very simple project. Please do have a look around the structure and the source files to learn how I’ve put things together. I’m a big fan of Apache Ivy as the dependency manager and I like to keep things easy using Ant. Instructions are included in the README.MD on how to configure Ivy to work so that you can download the required project dependencies.

That’s all for now!

Hopefully you will now have installed the following:

  • Varnish Cache
  • RabbitMQ with a queue called banQueue ready to receive messages
  • Cloned the Varnish-Ban project and resolved dependencies

In the next posting we will walk through the source code and configure Varnish to BAN URLs based on a customised HTTP Request. Until then, have fun!


Written by dharrigan

December 28, 2011 at 3:43 pm

Posted in development, java, software, varnish

Tagged with , , , ,