Code and Snow and Stuff

Musings on software and the quality of the white stuff…

Posts Tagged ‘rabbitmq

Banning URLs from Varnish using Apache Camel and RabbitMQ – Part 2

Welcome Back!

I hope you found Part 1 on this tutorial useful. You should by now have a running instance of Varnish cache along with a running instance of RabbitMQ. You should also have cloned the Varnish-Ban project from Bitbucket and perhaps had a look through the project structure and source code. I hope there is nothing too unusual in there :-).

In today’s posting we will be covering the following topics:

  • The Varnish-Ban Camel Component
  • Configuring Varnish to respond to HTTP BAN requests.

I hope you enjoy the continuing adventure! :-)

The Varnish-Ban Component

Writing a component to hook into Apache Camel is really quite simple. There are various ways of doing it, but I choose a very expicit and straightforward way to achieve the goal of working with Camel. The main requirements were to:

  1. Create a POJO which implements the Component interface.
  2. Create the Service class that will handle the sending of the BAN request to Varnish.
  3. Add a file called varnish-ban into the folder META-INF/services/org/apache/camel/component. This will allow Camel to auto-register the component.
  4. Create a Camel XML file describing the route and the processing requirements that Camel with respond with.

These steps are described below.

Creating the POJO

Writing the POJO was very simple. Below is a screen shot of the the actual class:

A component in Camel is responsible for creating the Endpoints – in effect it is a Factory.  In my configuration, the Component calls the Endpoint which creates a Producer that invokes the VarnishBanServiceImpl class. I like decoupling of code, so it seemed sensible to me to externalise the actual work of the banning mechanism into a service class that does the work. The service class has the responsibility of sending the BAN request to Varnish. The varnishServerUrl is given to us by Camel when it processes the XML configuration file (see below). The main thing here is we don’t have to do any extra work to obtain the varnishServerUrl – it’s all externalised into the XML file.

Creating the Service Class

The VarnishBanService does all the real work. Fortunately for us, even this class is quite small and very straightforward in its functionality. It simply creates an instance of a HTTP Client (from Apache HTTP Components) and sends off our customised HTTP request (a BAN request) to Varnish:

Our customised HttpRequest – the HttpBan class is very simple:

All that is happening here is that we are extending a base HTTP Class (provided by  HTTP Client) and overriding the getMethod invocation to return our customised HTTP method – cleverly called BAN :-). The toString is a simple helper when we are printing out debug/logging messages. You can create your own particular HTTP Method (SUSHI anyone?) if you have different needs. We could have called our method “LOLBAN” if we wanted to :-)

The remainder of the VarnishBanService class just handles the response back from Varnish and prints out some debug/logging information. Please have a look over to understand how it works. There shouldn’t be any surprises. I’m not handling any exceptions here, but what you could do is wrap up the exception into an AMQP message and shove it back into another Queue for another system to process (a monitoring application for example).

Enabling Auto-Discovery of our Component by Camel

If one creates a file with the same name as our chosen URI (see below in the Camel XML  route configuration section to discover what this is all about), then Camel will automagically register our newly created component and make it ready for use. Like so:

The file has one line in it:

This is all that is required to enable auto discovery in Camel. Pretty neat.

Creating the Camel XML Route Configuration

There are several ways to configure Routes in Camel – one is to use Java DSL to wire things together – another way is to use an XML configuration. I choose to use the XML configuration way just to keep things separate. Underneath the hood, Camel uses Spring, so using an XML configuration file seemed like a nice fit as well.

The file consists of the following elements:

  1. The Source Route. This is our connection to RabbitMQ using the Camel-Spring-AMQP component (see the file applicationContext-beans.xml) in the source code.
  2. What do do when a message comes in (send it on the varnish route)
  3. Splitting the XML payload from RabbitMQ using XPath to obtain the URLs that we wish to BAN
  4. Invoking our Varnish-Ban component against a running varnish instance (http://localhost:6081)
  5. Handling any exceptions that may occur. In this example nothing is done, but we could choose to invoke another Camel component to drop an error message into another queue (banQueueError?)

Configuring Varnish for HTTP BAN Requests

Varnish by default does not permit BANs to occur via HTTP requests. To help encourage Varnish to do so, we need to write a bit of VCL (Varnish Control Language). I’ve put the recipe (a complete VCL file) below (this example is also contained with the conf/varnish/default.vcl file in the Varnish-Ban project):

backend default {
    .host = "";
    .port = "8080";

acl purge {

sub vcl_fetch {
    set beresp.ttl = 5m;

sub vcl_recv {
    unset req.http.Cookie;
    if (req.request == "BAN") {
        if (client.ip !~ purge) {
            error 401 "Not allowed";
        error 200 "Banned " + req.url;

Let’s walk through each section:


This is the backend service that Varnish is fronting – in most cases this will be a webserver. Here I’m instructing Varnish to cache requests from a server running on my local machine and listening on port 8080 (Varnish by default listens on port 6081, so if I hit http://localhost:6081 what will actually be served up is content coming from http://localhost:8080).

acl purge

In this section I’m defining an ACL (Access Control List) list of authorised machines that will be allowed to execute a PURGE (an invented name – I could have called it BANNERS if I wanted to). The ACL is used in the VCL_RECV section.


A FETCH is the response from the backend – in the sense that Varnish has “fetched” the response and potentially cached it. Here I’m saying to Varnish to cache all backend responses for 5 minutes.


A REC(ei)V(e) is the request coming into Varnish from a client. The important things to note here are:

  • I’m removing Cookies. By default Varnish does not cache any requests that contain Cookies.
  • We will do something special if the type of the request (from the HTTP HEADER) is a “BAN” type . I invented this type – it could be called another name.
  • We will only allow those clients defined in our ACL the authority to BAN URLs from Varnish – otherwise we return back a 401 (Not Authorised) to the client.
  • Finally we return a 200 back to the Client once we have finished processing the BAN request.

The example VCL should be put into your “default.vcl” and Varnish restarted. When this is done we are ready to move to the final part of this tutorial!

That’s all for now!

Hopefully by now you will have a running application. In the third and last article of this tutorial we will be sending BAN messages to Varnish and observing the results. Until then, have fun!


Written by dharrigan

January 5, 2012 at 9:23 am

Posted in development, java, software, varnish

Tagged with , , , ,

Banning URLs from Varnish using Apache Camel and RabbitMQ – Part 1


Hello and Welcome! :-)

Over the course of three postings, I would like to present a tutorial on using RabbitMQ and Apache Camel to BAN (their parlance for removing) URLs (objects) held within Varnish Cache. This proposed approach allows for a complete decoupling of application logic from the caching system thus promoting greater flexibility, scalability and resiliance – in effect creating a sophisticated event driven architecture that would grow with business needs. Think about the enormous benefits and potential here – an application-initiated (and as-close-to-possible) realtime event driven mechanism for removing objects from within Varnish!

I hope you find this tutorial useful to you and I welcome any feedback you have. Please do drop me a email or comment if you have anything you wish to add :-)

Before I go any further I would like to sincerely thank the members of my team for doing the initial work on this – Vinay, Diego, Craig, Fred and Manju – your contributions are gratefully appreciated – thanks guys! :-)

In today’s posting I will introduce the key players and describe the  technologies that will be used.

Basic Requirements and Assumptions

These articles are written for Java developers and application support personnel in mind. The language used is Java 7 and the target platform is Unix based (I personally use Debian testing). There are comprehensive instructions on each application website for Varnish and RabbitMQ on how to download, install and do an initial configuration – please have a read and please install each component. The project source code and associated files are hosted on BitBucket as a Mercurial repository.

NOTE: Although the target audience is anyone with a familiarity of Java, the information presented here can be adapted to *ANY* architecture. All you need is a way to send message to a queue, for something to pick the message off the queue and process it, then for something to send on the processed message to a running instance of Varnish. An easy recipe to follow :-)

Architecture Overview

Shown below is what we will be aiming towards for our BAN solution:

The Key Players

Without further ado, let me introduce the core components (the versions in brackets are those that I had when I did this proof of concept):

Varnish (3.0.2)

Varnish Cache is a free sophisticated modern web accelerator that is growing in adoption and use throughout the world. It sits in front of any server that talks HTTP  and caches the response from the backend server. In our experience, Varnish has proven itself to be very reliable, very fast and very easy to configure. If you have a need to dramatically speed up responses to the client, I would highly recommend having a serious look at Varnish to see if it will help you (chances are it will! :-)).

RabbitMQ (2.7.1)

RabbitMQ is an implementation of an enterprise messaging system that talks AMQP (it can support other protocols such as STOMP). RabbitMQ is highly reliable and scalable and a good choice if you are looking for a way to send messages between systems – it will be the “backbone” that facilitates our event driven architecture.

Apache Camel (2.8.3)

A very popular integration framework that knows how to route, split, aggregate and do lots more with messages that flow through it. We will be using Apache Camel to pick messages from RabbitMQ and process them for sending on to Varnish.

Camel-Spring-AMQP [and Spring AMQP] (1.0)

A recently developed component for Apache Camel that speaks Spring-AMQP natively (it uses Spring AMQP underneath the covers). We will be using this component to connect to RabbitMQ.

Configuring Varnish

The default out-of-the-box configuration for Varnish will suffice for now. We will reconfigure Varnish in another posting to support BANing of URLs via a customised HTTP requests. An example Varnish configuration file is included in the project source code (in the conf/varnish directory).

Configuring RabbitMQ

It is important that RabbitMQ is installed with the web management console included – although not a requirement to use RabbitMQ, it really really makes life a lot simplier (and we will be using the web management console to send messages). The simplest thing is to install directly from the download on the website. We will require a Queue to be configured that will hold our HTTP BAN messages. I’m a big fan of images to explain what to do, so please find instructions below:

Log into RabbitMQ Web Management console:

Click on the Queues tab and create the following Queue:

You should see this when you click Add queue:

We now have to associate the Queue to an Exchange (along with a Routing Key):

You should see this when you click Bind:

After achieving this step, you now will have a Queue (banQueue) bound to an Exchange ( with the Routing Key (banQueue). We can now use this queue when we send ban messages.

Configuring Apache Camel

No configuration at the moment is required since the project contains the an example Camel configuration for you to use (in the conf/spring directory). You will probably need to adapt this for your own use later.

Configuring the Varnish-Ban Project

If you have not done so already, please clone the project to your machine. It is a very simple project. Please do have a look around the structure and the source files to learn how I’ve put things together. I’m a big fan of Apache Ivy as the dependency manager and I like to keep things easy using Ant. Instructions are included in the README.MD on how to configure Ivy to work so that you can download the required project dependencies.

That’s all for now!

Hopefully you will now have installed the following:

  • Varnish Cache
  • RabbitMQ with a queue called banQueue ready to receive messages
  • Cloned the Varnish-Ban project and resolved dependencies

In the next posting we will walk through the source code and configure Varnish to BAN URLs based on a customised HTTP Request. Until then, have fun!


Written by dharrigan

December 28, 2011 at 3:43 pm

Posted in development, java, software, varnish

Tagged with , , , ,


Get every new post delivered to your Inbox.

Join 54 other followers