dnsmasq goodness
At present, I’m working from home. In order to access certain services, I normally VPN into the office network and leave the VPN running all day. Unfortunately, doing so cuts off DNS resolution from my machine to other computers running on my home network (I run my own bind9 server at home). This is because, on connection, the office VPN (which I configured 
) pushes to each client, routes, ntp servers and also the office DNS server – all well and good – but how on earth do I now access my own home machines (without resorting to lame-ass /etc/hosts entries?) since the office DNS knows nothing about my home network machine names!
What to do? The answer is the marvelous dnsmasq program and a very simple configuration! I assume, of course, you have (or will about to have) dnsmasq installed.
First step is a simple change to my local resolv.conf, to read:
nameserver 127.0.0.1
search home.network office.network
This causes all DNS resolution to be forwarded to dnsmasq (which we will shortly configure to listen on 127.0.0.1). Additionally, I configure two search domains so that simple hostnames like “bob” or “bamboo” or “fish” will be changed to “bob.home.network” or “fish.office.network” depending on the results of queries to DNS.
Second step is to setup dnsmasq and because I’m using the wonderful, amazing, beautiful and oh-so-loveable Debian, I create a dnsmasq configuration file in /etc/dnsmasq.d and name it 00default (the name doesn’t matter at all – could be called wibble for all it cares).
The contents of this dnsmasq configuration file are:
server=192.168.75.3
server=/home.network/192.168.75.3
server=/office.network/10.10.10.3
listen-address=127.0.0.1
bind-interfaces
Whenever the VPN is up and running:
Firstly, all queries for all addresses are sent to my home DNS, i.e., http://www.google.com is looked up from the DNS listening on 192.168.73.3. This is the default behaviour and allows me to access external resources as normal.
Secondly, if I query for a machine called bob (which exists on my home network), my computer first tacks on home.network (from my resolv.conf), and queries the nameserver at 127.0.0.1 – which is where dnsmasq is listening. Dnsmasq in turn hands off to 192.168.75.3 since the query now matches the domain name of bob.home.network (notice the home.network in the configuration file). My locally running DNS responds saying “Yup, I know that address for that computer and returns with 192.168.75.2.” All honkydorey and sparkles.
Finally, if I query for a machine called bamboo (which does not exist on my home network), my computer first tacks on home.network, queries my local DNS (same rules as before), but that fails to return an address. An error is returned, resolv now tacks on office.network and dnsmasq works it’s magic by now querying the DNS server at 10.10.10.3, since the query now matches the domain, i.e., bamboo.office.network. That DNS server in the office knows about bamboo.office.network and responds with the correct IP address. Ah, all rainbows and unicorns.
Lovely! Thank you dnsmasq!
You can leave dnsmasq running all the time. It won’t resolve any office network names if the VPN is not up and running (since it cannot connect to the office DNS at 10.10.10.3). Oh, and my domain names (home.network and office.network) are only examples folks. They are not the real domain names. Change those to suit your particular setup.
I hope this is of some service to people!
-=david=-
Blag Blag :-)
w00t!
Delegating local subdomain BIND9 queries to Amazon Route53
Hello!
Recently we have started to use Amazon AWS heavily. It’s a truely great service. One of the tasks we need to accomplish is to join our locally running BIND9 DNS server to instances within our Amazon Virtual Private Cloud. We wanted to basically allow Amazon Route53 (their flexible DNS service) to resolve instances running with Amazon. So, how do we do this? In the end, it turned out to be really simple!
Let’s say you have an instance at 172.16.100.5
And you wanted it to have the name like funkychicken.development.domain.com.
But you still wanted your local BIND9 server to be the authority for all *.domain.com addresses.
Firstly create your hosted zone in Amazon Route53
development.domain.com
Find the nameservers that this domain uses (double click on the new entry on the Amazon Route53 panel)
Then find out the IP addresses of each of the name servers associated with your new hosted zone (Amazon use 4 nameservers for redundancy).
Finally
Then here is the magic sauce for your BIND9 configuration…
zone "development.domain.com" {
type forward;
forwarders {
205.251.193.46;
205.251.198.50;
205.251.194.151;
205.251.197.158;
};
};
Result!
This allows all local requests in your company for *.development.domain.com to be sent out to Amazon Route53 nameservers for resolution!
You can test this easily by creating an instance, finding out it’s IP, creating a new A record in Amazon Route53 to point to that A record and ping it from inside your company!
i.e.,
funkychicken.development.domain.com (i-a3b613aa), A record on Route53.
-=david=-
Java Thread Sleep Tip
Hi,
If you often find yourself writing code similar to this:
...
try {
Thread.sleep(10 * 1000); // sleep for 10 seconds
} catch(final InterruptedException e) {
...
}
...
You can make your intent a lot clearer (and IMHO cleaner) by doing this instead:
...
try {
Thread.sleep(TimeUnit.SECONDS.toMillis(10)); // sleep for 10 seconds
} catch(final InterruptedException e) {
...
}
...
The really nice thing about this approach is that you can substitute the SECONDS to anything else, i.e.,
...
try {
Thread.sleep(TimeUnit.HOURS.toMillis(10)); // sleep for 10 hours!
} catch(final InterruptedException e) {
...
}
...
Neato!
-=david=-
Banning URLs from Varnish using Apache Camel and RabbitMQ – Part 2
Welcome Back!
I hope you found Part 1 on this tutorial useful. You should by now have a running instance of Varnish cache along with a running instance of RabbitMQ. You should also have cloned the Varnish-Ban project from Bitbucket and perhaps had a look through the project structure and source code. I hope there is nothing too unusual in there .
In today’s posting we will be covering the following topics:
- The Varnish-Ban Camel Component
- Configuring Varnish to respond to HTTP BAN requests.
I hope you enjoy the continuing adventure!
The Varnish-Ban Component
Writing a component to hook into Apache Camel is really quite simple. There are various ways of doing it, but I choose a very expicit and straightforward way to achieve the goal of working with Camel. The main requirements were to:
- Create a POJO which implements the Component interface.
- Create the Service class that will handle the sending of the BAN request to Varnish.
- Add a file called varnish-ban into the folder META-INF/services/org/apache/camel/component. This will allow Camel to auto-register the component.
- Create a Camel XML file describing the route and the processing requirements that Camel with respond with.
These steps are described below.
Creating the POJO
Writing the POJO was very simple. Below is a screen shot of the the actual class:
A component in Camel is responsible for creating the Endpoints – in effect it is a Factory. In my configuration, the Component calls the Endpoint which creates a Producer that invokes the VarnishBanServiceImpl class. I like decoupling of code, so it seemed sensible to me to externalise the actual work of the banning mechanism into a service class that does the work. The service class has the responsibility of sending the BAN request to Varnish. The varnishServerUrl is given to us by Camel when it processes the XML configuration file (see below). The main thing here is we don’t have to do any extra work to obtain the varnishServerUrl – it’s all externalised into the XML file.
Creating the Service Class
The VarnishBanService does all the real work. Fortunately for us, even this class is quite small and very straightforward in its functionality. It simply creates an instance of a HTTP Client (from Apache HTTP Components) and sends off our customised HTTP request (a BAN request) to Varnish:
Our customised HttpRequest – the HttpBan class is very simple:
All that is happening here is that we are extending a base HTTP Class (provided by HTTP Client) and overriding the getMethod invocation to return our customised HTTP method – cleverly called BAN . The toString is a simple helper when we are printing out debug/logging messages. You can create your own particular HTTP Method (SUSHI anyone?) if you have different needs. We could have called our method “LOLBAN” if we wanted to
The remainder of the VarnishBanService class just handles the response back from Varnish and prints out some debug/logging information. Please have a look over to understand how it works. There shouldn’t be any surprises. I’m not handling any exceptions here, but what you could do is wrap up the exception into an AMQP message and shove it back into another Queue for another system to process (a monitoring application for example).
Enabling Auto-Discovery of our Component by Camel
If one creates a file with the same name as our chosen URI (see below in the Camel XML route configuration section to discover what this is all about), then Camel will automagically register our newly created component and make it ready for use. Like so:
The file has one line in it:
This is all that is required to enable auto discovery in Camel. Pretty neat.
Creating the Camel XML Route Configuration
There are several ways to configure Routes in Camel – one is to use Java DSL to wire things together – another way is to use an XML configuration. I choose to use the XML configuration way just to keep things separate. Underneath the hood, Camel uses Spring, so using an XML configuration file seemed like a nice fit as well.
The file consists of the following elements:
- The Source Route. This is our connection to RabbitMQ using the Camel-Spring-AMQP component (see the file applicationContext-beans.xml) in the source code.
- What do do when a message comes in (send it on the varnish route)
- Splitting the XML payload from RabbitMQ using XPath to obtain the URLs that we wish to BAN
- Invoking our Varnish-Ban component against a running varnish instance (http://localhost:6081)
- Handling any exceptions that may occur. In this example nothing is done, but we could choose to invoke another Camel component to drop an error message into another queue (banQueueError?)
Configuring Varnish for HTTP BAN Requests
Varnish by default does not permit BANs to occur via HTTP requests. To help encourage Varnish to do so, we need to write a bit of VCL (Varnish Control Language). I’ve put the recipe (a complete VCL file) below (this example is also contained with the conf/varnish/default.vcl file in the Varnish-Ban project):
backend default {
.host = "127.0.0.1";
.port = "8080";
}
acl purge {
"localhost";
}
sub vcl_fetch {
set beresp.ttl = 5m;
}
sub vcl_recv {
unset req.http.Cookie;
if (req.request == "BAN") {
if (client.ip !~ purge) {
error 401 "Not allowed";
}
ban_url(req.url);
error 200 "Banned " + req.url;
}
}
Let’s walk through each section:
backend
This is the backend service that Varnish is fronting – in most cases this will be a webserver. Here I’m instructing Varnish to cache requests from a server running on my local machine and listening on port 8080 (Varnish by default listens on port 6081, so if I hit http://localhost:6081 what will actually be served up is content coming from http://localhost:8080).
acl purge
In this section I’m defining an ACL (Access Control List) list of authorised machines that will be allowed to execute a PURGE (an invented name – I could have called it BANNERS if I wanted to). The ACL is used in the VCL_RECV section.
vcl_fetch
A FETCH is the response from the backend – in the sense that Varnish has “fetched” the response and potentially cached it. Here I’m saying to Varnish to cache all backend responses for 5 minutes.
vcl_recv
A REC(ei)V(e) is the request coming into Varnish from a client. The important things to note here are:
- I’m removing Cookies. By default Varnish does not cache any requests that contain Cookies.
- We will do something special if the type of the request (from the HTTP HEADER) is a “BAN” type . I invented this type – it could be called another name.
- We will only allow those clients defined in our ACL the authority to BAN URLs from Varnish – otherwise we return back a 401 (Not Authorised) to the client.
- Finally we return a 200 back to the Client once we have finished processing the BAN request.
The example VCL should be put into your “default.vcl” and Varnish restarted. When this is done we are ready to move to the final part of this tutorial!
That’s all for now!
Hopefully by now you will have a running application. In the third and last article of this tutorial we will be sending BAN messages to Varnish and observing the results. Until then, have fun!
-=david=-
Automatically Adding JAXB2 Classes to Spring using Annotations
Hi Again!
An annoying aspect of using the built-in Jaxb2Marshaller that is bundled with Spring is the fact that you have to manually add each class to be bound into the XML. This is sooooo boring and old skool Thankfully, it’s not that hard to get Spring to scan all classes in a package for properly annotated classes and have those added to the JAXBContext automatically. I provide below, for your pleasure, an example bit of code that does the job.
Please note: Until (if?) Spring changes Jaxb2Marshaller, you will have to do some xml fudgery to get this to work:
<bean id="marshaller" class="AnnotationJaxb2Marshaller">
<property name="classesToBeBound">
<list><value>a.valid.jaxb2.annotated.class.ThatWillBeIgnored</value></list>
</property>
<property name="packagesToScan">
<list><value>my.package.with.jaxb.classes</value></list>
</property>
</bean>
The reason is that in Jaxb2Marshaller afterPropertiesSet is marked as final. However, in that method the JAXBContext is initialised, the code borks out if no classesToBeBound is set – therefore I set a dummy (but valid!) entry. The list that holds the set of classes to use (containing our dummy class) will be overwritten by the packagesToScan. Okay enough rabbling, here is the code:
import java.io.IOException;
import java.util.Arrays;
import java.util.ArrayList;
import java.util.List;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.annotation.XmlEnum;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlSeeAlso;
import javax.xml.bind.annotation.XmlType;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import org.springframework.core.type.classreading.CachingMetadataReaderFactory;
import org.springframework.core.type.classreading.MetadataReader;
import org.springframework.core.type.classreading.MetadataReaderFactory;
import org.springframework.core.type.filter.AnnotationTypeFilter;
import org.springframework.core.type.filter.TypeFilter;
import org.springframework.oxm.UncategorizedMappingException;
import org.springframework.oxm.jaxb.Jaxb2Marshaller;
import org.springframework.util.Assert;
import org.springframework.util.ClassUtils;
/**
* An extension to the Jaxb2Marshaller that scans the classpath for classes annotated
* with the @XmlRootElement (and others) annotation. I don't like typing in the class in
* classesToBeBound since I often forget which ones I've done!
*/
public class AnnotationJaxb2Marshaller extends Jaxb2Marshaller {
private static final String RESOURCE_PATTERN = "/**/*.class";
private String[] packagesToScan;
private final ResourcePatternResolver resourcePatternResolver = new PathMatchingResourcePatternResolver();
private final TypeFilter[] jaxb2TypeFilters = new TypeFilter[]{
new AnnotationTypeFilter(XmlRootElement.class, false),
new AnnotationTypeFilter(XmlType.class, false),
new AnnotationTypeFilter(XmlSeeAlso.class, false),
new AnnotationTypeFilter(XmlEnum.class, false),
};
/**
* Scan packages looking for any classes annotated with the @XmlRootElement annotation.
*/
protected List<Class<?>> scanPackages() {
final List<Class<?>> annotatedClasses = new ArrayList<Class<?>>();
try {
if (packagesToScan != null) {
for (final String pkg : packagesToScan) {
final String pattern = ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX + ClassUtils.convertClassNameToResourcePath(pkg) + RESOURCE_PATTERN;
final Resource[] resources = resourcePatternResolver.getResources(pattern);
final MetadataReaderFactory metadataReaderFactory = new CachingMetadataReaderFactory(resourcePatternResolver);
for (final Resource resource : resources) {
final MetadataReader metadataReader = metadataReaderFactory.getMetadataReader(resource);
final String className = metadataReader.getClassMetadata().getClassName();
if (matchesFilter(metadataReader, metadataReaderFactory)) {
final Class<?> jaxb2AnnotatedClass = resourcePatternResolver.getClassLoader().loadClass(className);
annotatedClasses.add(jaxb2AnnotatedClass);
}
}
}
}
} catch (final IOException ex) {
throw new UncategorizedMappingException("Failed to scan classpath for unlisted classes", ex);
} catch (final ClassNotFoundException ex) {
throw new UncategorizedMappingException("Failed to load annoted classes from classpath", ex);
}
return annotatedClasses;
}
/**
* Determine if any of the classes matches our list of acceptable annotations.
*
* @param metadataReader for the resource.
* @param metadataReaderFactory for the resource.
* @return true if the class contains the annotation.
* @throws IOException if anything goes wrong.
*/
protected boolean matchesFilter(final MetadataReader metadataReader, final MetadataReaderFactory metadataReaderFactory) throws IOException {
if (jaxb2TypeFilters != null) {
for (final TypeFilter typeFilter : jaxb2TypeFilters) {
if (typeFilter.match(metadataReader, metadataReaderFactory)) {
return true;
}
}
}
return false;
}
/**
* Why oh why is the afterPropertiesSet in the super class final???
*/
@Override
public synchronized JAXBContext getJaxbContext() {
if (packagesToScan.length > 0) {
// We will try *my* way
final List<Class<?>> annotatedClasses = scanPackages();
if (annotatedClasses.size() > 0) {
setClassesToBeBound(annotatedClasses.toArray(new Class<?>[0]));
}
}
return super.getJaxbContext();
}
/**
* Set packages to scan.
*/
public void setPackagesToScan(final String[] packagesToScan) {
Assert.notEmpty(packagesToScan, "'packagesToScan' must not be empty");
this.packagesToScan = Arrays.copyOf(packagesToScan, packagesToScan.length);
}
}
Enjoy!
-=david=-
Banning URLs from Varnish using Apache Camel and RabbitMQ – Part 1
Introduction
Hello and Welcome!
Over the course of three postings, I would like to present a tutorial on using RabbitMQ and Apache Camel to BAN (their parlance for removing) URLs (objects) held within Varnish Cache. This proposed approach allows for a complete decoupling of application logic from the caching system thus promoting greater flexibility, scalability and resiliance – in effect creating a sophisticated event driven architecture that would grow with business needs. Think about the enormous benefits and potential here – an application-initiated (and as-close-to-possible) realtime event driven mechanism for removing objects from within Varnish!
I hope you find this tutorial useful to you and I welcome any feedback you have. Please do drop me a email or comment if you have anything you wish to add
Before I go any further I would like to sincerely thank the members of my team for doing the initial work on this – Vinay, Diego, Craig, Fred and Manju – your contributions are gratefully appreciated – thanks guys!
In today’s posting I will introduce the key players and describe the technologies that will be used.
Basic Requirements and Assumptions
These articles are written for Java developers and application support personnel in mind. The language used is Java 7 and the target platform is Unix based (I personally use Debian testing). There are comprehensive instructions on each application website for Varnish and RabbitMQ on how to download, install and do an initial configuration – please have a read and please install each component. The project source code and associated files are hosted on BitBucket as a Mercurial repository.
NOTE: Although the target audience is anyone with a familiarity of Java, the information presented here can be adapted to *ANY* architecture. All you need is a way to send message to a queue, for something to pick the message off the queue and process it, then for something to send on the processed message to a running instance of Varnish. An easy recipe to follow
Architecture Overview
Shown below is what we will be aiming towards for our BAN solution:
The Key Players
Without further ado, let me introduce the core components (the versions in brackets are those that I had when I did this proof of concept):
Varnish (3.0.2)
Varnish Cache is a free sophisticated modern web accelerator that is growing in adoption and use throughout the world. It sits in front of any server that talks HTTP and caches the response from the backend server. In our experience, Varnish has proven itself to be very reliable, very fast and very easy to configure. If you have a need to dramatically speed up responses to the client, I would highly recommend having a serious look at Varnish to see if it will help you (chances are it will! ).
RabbitMQ (2.7.1)
RabbitMQ is an implementation of an enterprise messaging system that talks AMQP (it can support other protocols such as STOMP). RabbitMQ is highly reliable and scalable and a good choice if you are looking for a way to send messages between systems – it will be the “backbone” that facilitates our event driven architecture.
Apache Camel (2.8.3)
A very popular integration framework that knows how to route, split, aggregate and do lots more with messages that flow through it. We will be using Apache Camel to pick messages from RabbitMQ and process them for sending on to Varnish.
Camel-Spring-AMQP [and Spring AMQP] (1.0)
A recently developed component for Apache Camel that speaks Spring-AMQP natively (it uses Spring AMQP underneath the covers). We will be using this component to connect to RabbitMQ.
Configuring Varnish
The default out-of-the-box configuration for Varnish will suffice for now. We will reconfigure Varnish in another posting to support BANing of URLs via a customised HTTP requests. An example Varnish configuration file is included in the project source code (in the conf/varnish directory).
Configuring RabbitMQ
It is important that RabbitMQ is installed with the web management console included – although not a requirement to use RabbitMQ, it really really makes life a lot simplier (and we will be using the web management console to send messages). The simplest thing is to install directly from the download on the website. We will require a Queue to be configured that will hold our HTTP BAN messages. I’m a big fan of images to explain what to do, so please find instructions below:
Log into RabbitMQ Web Management console:
Click on the Queues tab and create the following Queue:
You should see this when you click Add queue:
We now have to associate the Queue to an Exchange (along with a Routing Key):
You should see this when you click Bind:
After achieving this step, you now will have a Queue (banQueue) bound to an Exchange (amq.direct) with the Routing Key (banQueue). We can now use this queue when we send ban messages.
Configuring Apache Camel
No configuration at the moment is required since the project contains the an example Camel configuration for you to use (in the conf/spring directory). You will probably need to adapt this for your own use later.
Configuring the Varnish-Ban Project
If you have not done so already, please clone the project to your machine. It is a very simple project. Please do have a look around the structure and the source files to learn how I’ve put things together. I’m a big fan of Apache Ivy as the dependency manager and I like to keep things easy using Ant. Instructions are included in the README.MD on how to configure Ivy to work so that you can download the required project dependencies.
That’s all for now!
Hopefully you will now have installed the following:
- Varnish Cache
- RabbitMQ with a queue called banQueue ready to receive messages
- Cloned the Varnish-Ban project and resolved dependencies
In the next posting we will walk through the source code and configure Varnish to BAN URLs based on a customised HTTP Request. Until then, have fun!
-=david=-
