The web is made of messages, e-mail is the first class citizen of them. Instant messaging protocols (like the beloved XMPP) and now Twitter and every twitter-like applications are collecting and sending back tons of messages. Let’s see how this concept can be applied and becoming very useful at a smaller scale too.
Before going further, Matt Biddulph (CTO of Dopplr) made a very interesting presentation explaining how Dopplr is using messages that transit between the different parts of the application. Take a look at it! Dopplr: It's made of messages - Matt Biddulph
How messages can help your web application to become better? Like every good manager do, by delegating. Instead of doing everything yourself, let other people doing that for you. The drawback is that you’ll have to wait on that person to do his job, the good ones are that you can quickly answering your customer (sending back a HTML page in our case).
I wrote a simple application called up! that is an image uploader. The cool thing about this image uploader is that, like Flickr for example, once the pictures are uploaded the application tells you that the thumbnails are currently processed and that you’ll very soon get them without telling anything before its done (which is rude, but common)
Let’s dive with the image upload.
When John uploads his picture, the file is saved into a temporary directory and a message of this kind is sent to a queue called resize.
("/tmp/foobar.jpg", "jpeg")
And respond directly with a HTML page containing a placeholder for the thumbnail.
<p id="preview"> <span>foobar.jpg</span> sent. </p> <p id="loading"><em> Please wait for the preview to be generated. </em></p>
Now, the client will try to obtain the expected thumbnails.
On the client-side a JavaScript code will ping that expecte picture every 2 seconds as long as the server doesn’t find it and therefor answers with an error 404.
During that time, a consumer (a different process, independant from the web server) will get that message, resize the picture in 3 different sizes and put them at the right place. The arrow that goes back to the queue is a message acknowledgement saying that everything went find and that the message can be removed from the queue. The goal is to not lose any messages, if possible.
Once the thumbnails are available, the page displays the picture and the polling is stopped.
Different messaging mecanisms exist, some of them are very simple, like STOMP, but most of them are transcient only. It means no persistancy, if nobody listens; the messages are lost. Because shit happens, I prefer having some kind of persistency and acknowledgement that a message was well processed. Persistance means slowliness too. For that, this demo uses RabbitMQ and the AMQP protocol. The first one is powered by Erlang and both of them are open source. Installing RabbitMQ is as simple as (once downloaded and extracted): make run. Learn more about RabbitMQ reading: Rabbits and warrens.
As libraries, it uses both txamqp (tx stands for twisted matrix) and amqp-lib. I find amqp-lib handy for the client side, to just send a message and prefer twisted matrix for a server that has to live (if possible) for ever and without causing too many trouble. Using only amqp-lib is of course possible.
Resizing image is one use case, maybe not the best one. In another application, I’m also using a queue for sending e-mail notifications, or it could be IM notification (AMQP instead of XML-RPC in my twootr article for example), ... Separating them from the web server logic as the great advantage that you can tune both of them separately, instanciate more processes to handle image resizing the one you have have too much work without affecting anyone else. Small pieces of software collaborating together.
I’ve read once, that once you’ve gone in that path, you cannot go back. You’ve been warned. I think that big monolithics web apps aren’t agile enough those days where data are growing bigger and data are going faster to many people, mediums and third-parties.
Download it from bitbucket and feel free to hack it or to send me any questions.
