Yoan Blanc’s weblog

I had a long chat this morning with David about the post of yesterday and apart his question about using Comet instead of Ajax polling, he made me realizing that it wasn’t that clear. So, there the big picture of the final architecture:

the big picture.

In green: the web Server, Twootr and the Database are using web.py; in red the bot which is composed of the Jabber bot and a RPC bot that are using Twisted. And in blue you from your any web browsers or IM clients (such as GTalk, Gaim, Adium, ...). The communication protocols are HTTP between you and the web server (like usual) and XMPP between your IM client and the JabberBot (called TwootrBot). The web server can ask the bot to performs some actions using XML-RPC. The TwootrBot is using directly twootr to access to the data inside the database (which isn’t perfect, but useful).

We also discussed why XML-RPC and not XMPP, which would also make sense if we consider the web site as an agent that want to talk to the bot. XMPP is somehow heavier than XML-RPC but offer many advantages like the authentification, the status of the agent you may want to talk to is known and it’s extensible by nature. I decided that both are part of a same thing and thus thought that a simple IPC would be enough. You to decide.

And last but not least, Comet, also known as the opposite of Ajax or “the server PUSH”. In short, the client isn’t polling every n seconds for an update but with different mechanism, will wait that the server gives it the information it need. I did a previous experiment with web.py (as well) and the “callback-polling” trick that involves an iframe. This time, I used the “long-polling” which is a more for system when you don’t know when an event will occurs. The “callback-polling” is perfect for highly updated information like quotes on financial websites.

On the server side, it’s quite easy:

class LongPolling(object):
 def GET(self):
  # the time the long poll
  # will run at max
  seconds = 60
  # here the wait loop
  while True:
   seconds -= 1
   # look for fresh data
   data = self.getData()
   if data or not(seconds):
    break
  
   # wait a sec
   time.sleep(1)
  
  # do something with your data
  pass

Of course, you may imagine that you’ll have some POST parameters and so one. For twootr, I’ve putted the number of seconds inside the arguments that are given by the XMLHttpRequest (XHR) call which makes it usual either in Ajax or Comet mode. And that’s cool.

Now, on the Ajax side (reading here JavaScript): I had to upgrade the Ajax implementation I’ve made for Base2 to have a timeout property, just in the case the server won’t answer in the time we asked it to. A basic polling call can look like this (if we assume it exists a class called XHR that behave like that):

function update() {
 new XHR("POST",
  "/ajax/",
  function(response) {
   document.body.innerHTML =
    response.responseText;
  });
}

setInterval(update, 30000);

Every 30 seconds, this script will retreive the HTML from /ajax/ and update the body of the page with it. The important thing is the setInterval which says that it’s a normal polling behaviour. Then, a long-polling will look this way.

function update() {
 new XHR("POST",
  "/ajax/",
  function(response) {
   document.body.innerHTML =
    response.responseText;
     
   # this does a chaining
   # similar to setInterval
   setTimeout(update, 1000);
 });
}

update();

Do you see the different? As we expect that the XHR call to be long (or very short), the next call to it is done after the last one is finished. This code is very simplified because, you have to handle the mentioned timeout thing as well, which this time become very important.

A long-polling system built this way is more sensitive to errors because the calls are chained and not sent at regular interval. So either your code his perfect or I was wondering about putting a similar mechanism to a watchdog. A function that is executed every n seconds and checks that an Ajax call is currently running, if their isn’t for a certain amount of time it will re-call update.

The issue with Comet is that your server has to handle very well a large amount of parallel connections. Servers, like Apache for example are built to deliver pages as fast as possible and not really to have many people waiting for something to happen. This is way, most of the time, smaller servers are choosen like lighttpd, Jetty, Yaws, …

From a JavaScript point of view, it would be easily doable to offer the ability of a client to switch from Comet to Ajax, from long-polling to simple polling in order to reduce the server load. But if the server is configured to support many simultaneous connection, it might better by doing more calls, very frequently.

The final impression, with long-polling instead of simple polling is a great reactivity. From hitting enter in my IM client to seeing the page updating appears to be as fast as an eyeblink (maybe one from someone who’s tired, but still). That speed is very enjoyable, as a user. A web application that offers directly of feedback of any performed action will appear as faster (even if it’s not the case).

You can find an updated version of twootr and if you want to chat about it, drop a comment bellow (in the “most non-accessible commenting system I’ve ever seen” according to David, but spammers are blind) or ping me. And if you like Comet, please take a look at the Bayeux protocol which is client server JSON protocol that aims to be used for Comet things.

I’ve already did some experiments here with a Jabber bot that was kinda basic, an interface to Tumblr. It toke me a consequent amount of time to make it more (or finally) useful and there it is. A web application composed of a website and a IM bot that aims to be a Twitter-like. You have your buddies, and are able to interact with them using either the web interface or the IM (with a GTalk or Jabber account). This is a proof of concept and you can do whatever you want with it at your own risk.

First, JabberBot (src) received some precious enhancement especially to manage the roster (formerly the buddy list). I was wondering about using the status message of the users but finally forget it. In order to get the status of the user, you have to make it on of your buddies. So the current bot, can had someone to his buddy roster and ask him to become friends which is called subscribe.

myBot.add("you@jabber.org")
myBot.sendPresence("you@jabber.com", "subscribe")

The default behavior of the current bot is to keep a mutual relation either we are friends (both) or I don’t know you. Jabber permits more states I’m your friend (from) , you’re mine (to) or we aren’t friends at all (none) but I’m keeping you in my buddy list (which is useful for your ex-girlfriends).

Upon this bot, there is a Twootrbot (src) that will publish the messages that he gets from his buddies to the website and will broadcast the message entered inside the website. The main question was how having a bot (which is like an infinite process) and a website (which is like lots of processes) that communicates together.

I tried the named pipes but they implies that you’re on the same machine and how you create one, open it, write in it has to be done in a specific order. Find here a little Python code that plays with named pipe (for Unices only). And finally used XML-RPC (via xmlrpclib) with is very simple to use and permits the bot to be distant from the web server. It’s a kinda heavy IPC system but it does the job.

A sample on the client side, that send a “Hello World!” to the echo method on the given server. XML-RPC is very very simple.

from xmlrpclib import ServerProxy
xmlrpc = ServerProxy('http://example.org/')
xmlrpc.echo('Hello World!')

The purpose of the bot is to get messages from IM and write them into the database. Or receive a XML-RPC call asking it to add a new buddy or to send a message to the recipients list. In fact the Jabber and XML-RPC bots are separate classes but runs in the same process and communicate together by simple function calls.

Now the big part, the website. I’ve chosen web.py to get something very malleable and that you don’t have to understand that much before doing something. To make simple, web.py offers you to build easily a web site but, like PHP (sorry for the comparison), which means you have to clean it up afterwards. You may like that or preferring having a true framework that, in a way, forces you a way of working. Basically, I’m telling here that every tool that has a create_project first operation falls into this second category.

It’s an endless troll but it seems that the MVC doesn’t fit that well the web, and for example Django ended up with a different acronym: MTV. I liked being able to try finding what kind of approach I would prefer. I’m still not convinced by what I’ve done but is happy of been able to try something in that field.

What I like about web.py:

• simple;
• the documentation fit in one page;
• good with the bottom up principle;
• SQL inside, kind of (I still like doing some);
• small and friendly community.

To resume that, as I like doing small things, quickly, to test an idea and then throw away (or blog about it). But Django have, from what I know, a lot more of good gems for a long term and more that one guy work. And as I need my daily programming.reddit.com, using web.py (which was built for it) came naturally.

Last but not least, the JavaScript part is built upon Base2 like my previous JavaScript blog posts. Base2 is still young (like web.py) and needs some attention. What is interesting with it is that it doesn’t provide you a framework to work with like most of the popular JavaScript tools (YUI, jQuery, Dojo, Prototype, MooTools). So no animations, no Ajax, no XPath selectors only methods that are parts of some specifications and not yet part of all the browsers (like querySelectorAll which only exists in Webkit at the moment), a packaging system (which seems to me very similar to the Perl ones) or a decent Event system (you may be aware that both MSIE and the W3C still think differently on that one).

I had to build an Ajax implementation (src) in order to do some of it because I didn’t find any. Anyways it was fun, it’s usage is very similar to the interface that Prototype offers. XmlHttpRequest works in a way that is close to that: some parameters like the url, the method, the body, ... and a lot of callbacks.

I’m inviting you to download it and test it. You need a jabber account, a GTalk one may not work for the bot. The python dependencies are: web.py, sqlite3 and twisted.words. Or if you are lazy, ping me: greut@swissjabber.ch, I should be able to run it on my computer for you.

I’m sure that the CSS Validator has complained on you about a background-color or a color that are half set.

I toke it into account and tends to forget it by the time until recently. The new (1.4.5) Audacious comes with a nice black GTK theme and I decided to use it all the time to embrace the dark theme mood.

And then the whole web collapse, because a lot (and the famous ones) aren’t respecting that basic rule: always set the background and the color, not only one of them. I can do a little recap of which applications aren’t usable anymore (without selecting the text to effectively see it):

• Google: Groups, Finance, Docs, News, login page, Code, GTalk in GMail, Calendar, ... way to many;
• Facebook is more or less affected, but still;
• Yahoo! Mail, Finance (very few), Del.icio.us, Flickr;
• Wordpress.com;
• Microsoft;
• Canonical;
• AOL;
• … and this weblog.

Usually the reset.css from YUI, Eric Meyer or others don’t contain a color reset which can prevent from that kind of massive failure. Blueprint does it right as long as you are using typography.css at the same time.

And there is also some issues (with a dark theme) in native apps like Pidgin which displays the extended away contacts in black despite the fact the background is already black. The Firefox status bar is also sadly black on black. A funny thing is that the Opera's behaviour is quite different regarding the GTK theme so it also breaks but not the same way.

Whatever we do, how many times things are repeated it’s hard to not forget that we aren’t using the same tools, configuration; that we don’t have all the same abilities to see and read (small text or weak contrasts). The best you can do is to try (for a while) to use an uncommon configuration: white on black, big fonts, text-only browser, no CSS, no JavaScript and so forth. I know that I’m not doing that enough either. Thank you to be a little more aware of that now and so forth. I know that I’m not doing that enough either. Thank you to be a little more aware of that now.

So, I ended up fixing it with Stylish assuming we "all" are considering that a default background is black and a default foreground color is white. And I warmly encourage you to put, for test purposes only, the following rules in Stylish to have a white on black rule or choose some fantasy colors like pink and purple if you prefer.

html, input,
button, textarea,
select, option {
 background:#000;
 color:#fff }

Now what to do? Should I have a white on black enabled website in order to be nice to people that prefer that kind of strong contrast? I assume they are using a kind of mechanism that Opera is providing, which simply overrides all existing rules but is that because the web is broken and it’s the only way to achieve that or because it’s the more convenient way to get an usable content?