Random Acts of Senseless Blogging

100,000 tasklets: Erlang and Go

2009-11-15T23:27:00.007Z

Purely out of interest, and to see how Erlang and Google's Go compare on my puny laptop (a Dell Mini 10v), I wrote an Erlang version of the example application on 100,000 tasklets: Stackless and Go.

Basically, it creates a chain of 100,000 microthreads (tasklet, goroutine, process ... take your pick), sends a value in one end and waits for the result at the other end. The number is incremented by each microthread it passes through.

The code comes in two parts. Firstly, a chain.erl module:

-module(chain).
-export([run/1]).

run(Num) ->
Tail = chain(Num, self()),
Tail ! 0,
receive Result -> Result end.

chain(0, Tail) ->
Tail;

chain(Num, Tail) ->
chain(Num-1, spawn(fun() -> f(Tail) end)).

f(Tail) ->
receive
Num -> Tail ! Num+1
end.

And secondly a simple escript to start it off (mostly to make it easy to run under time):

#!/usr/bin/env escript
%%! +P 1000000 -smp disable
-export([main/1]).

main([]) ->
Result = chain:run(100000),
io:format("~p~n", [Result]).

And the run times:

$ time ./chain 
100000

real 0m1.520s
user 0m1.012s
sys 0m0.468s
$ time ./go-chain 
100000

real 0m3.371s
user 0m1.672s
sys 0m1.000s

A couple of things to point out/mention:

The Erlang code is just beautiful (well, maybe not the escript so much ;-)). To me, it's more readable than either the Python or Go versions.
I turned SMP off. Yes, it's an optimisation but then the tasks are running in series so it's never going to help.
The Go version was compiled and linked using 8g and 8l.
The Go version didn't always complete and sometimes took a *very* long time ... just not when running under time for some reason.
Go seemed to use about 3x as much memory.

What does this prove? Absolutely nothing! Firstly, it's an unrealistic application. Also, Go is really quite new and I'm sure performance and memory use will improve in the coming months.

So why did I do this? Simply, because I really enjoy playing with Erlang and Go is interesting and a hot topic. (In my opinion anything with concurrency built into the runtime is onto a good thing ... I sure wish we didn't have to resort to Twisted, Stackless, greenlets or generator hacks in Python.)

CouchDB and XML

2009-10-22T11:53:00.003+01:00

Everyone knows by now that CouchDB is a JSON document store, right? What's not quite so obvious is that it handles XML very nicely too via E4X. Strictly speaking, it's the Mozilla Spidermonkey JavaScript engine that provides the E4X support but that's CouchDB's default view server.

I'm calling out to a 3rd party SOAP service and the result is returned as an XML string (don't blame me for that, it's what their WSDL file says). I've been tossing that result into a CouchDB doc - it's so easy, you might as well - in case I needed to refer to it later.

That time has come as I now need to be a bit more selective about how I handle the result. So, I fired up CouchDB's Futon and wrote a view that emitted bits of the JSON together with the interesting parts of the XML blob. Now I have a simple view of the data without all the XML "noise". Very nice :).

What's best about this is that using CouchDB, XML and E4X together is trivial and E4X is compact and quick to learn.

Build CouchDB on Ubuntu 9.10 (Karmic Koala)

2009-09-30T22:48:00.005+01:00

Ubuntu 9.10 includes CouchDB in the standard desktop installation ... perhaps that's all you need?

Personally, I like to track CouchDB trunk but building on the latest Ubuntu is not quite as easy as it used to be. The following is mostly from memory so may not be quite right ...

The list of packages in the CouchDB README to compile from source does not work (at least, not for me). Instead, ensure your apt sources are up to date and install the following packages:

$ sudo apt-get install build-essential erlang-nox erlang-dev libicu-dev xulrunner-dev libcurl4-openssl-dev

Next, configure CouchDB to use xulrunner's headers and libraries, using a custom LD_RUN_PATH:

$ LD_RUN_PATH=/usr/lib/xulrunner-1.9.1.5 ./configure --with-js-lib=/usr/lib/xulrunner-devel-1.9.1.5/lib/ --with-js-include=/usr/lib/xulrunner-devel-1.9.1.5/include

And then make with the same LD_RUN_PATH:

$ LD_RUN_PATH=/usr/lib/xulrunner-1.9.1.5 make

Finally, make install CouchDB as usual and you're ok to start the CouchDB server.

Note that if Ubuntu's standard CouchDB package is also installed you may need to avoid your install clashing with it. Edit your install's etc/couchdb/local.ini and set port in the httpd section to something different. I use 15984 but it doesn't really matter as long as it's unique.

Tornado ... first thoughts

2009-09-18T15:26:00.004+01:00

It's quite amusing to see the furore that's surrounded Tornado's release, especially how it compares to Twisted.

I like Twisted a lot although I'm far from a fan boy. Deferreds are not ideal but without proper coroutines (stackless, greenlet, etc) or message dispatch (see Erlang) in the core language they're a reasonable way to model async processes.

Anyway, a colleague asked me about my opinion on Tornado the day it was released. I'd only looked at it very briefly by then (scanned the docs and scanned the code) but I'm never short on opinion ;-).

So, mostly for posterity (there are far more in-depth posts elsewhere), below is my response almost verbatim. I'm sure to be wrong about some points, and I have no problem admitting that, but I've since seen comments that confirm at least some.

Only had a quick look at Tornado yesterday; I intend to look more
closely sometime. However, a few thoughts did spring to mind ...

Simple web page looks nice and simple ... as it should.

Can't get too excited about the url dispatch and request handler.
Nothing new to see, afaict.

Performance is quite impressive but I was surprised that 4 cores
wasn't even close to 4x the performance of a single core. Probably
something else limiting it but not very clear from the chart.

No Twisted performance comparison? I suspect Tornado is a little
faster but then it's much more focussed on what it's trying to
support.

Tornado's a callback-style framework. However, they don't seem to
bother handling errors (at least not in any consistent way). I think
that's a really bad idea. Sure, Twisted's Deferred is not ideal but I
think it's necessary. Sure, they could add an errback to every
function too ... but they don't seem to have bothered yet so they're
going to end up with an inconsistent way of reporting errors at best.
(The iostream module, for instance, appears to silently dump errors).

Their database module sucks big time. As far as I can see it blocks.
What good is that in an async framework!? It's MySQL only. Also ...
autocommit(True) ... argh!!!!

There's absolutely nothing in the docs about how to call those pesky
blocking libraries, you know like the entire Python stdlib or MySQL,
and there's no threading support afaict.

template library ... hohum.

locale support ... "Loads translations from CSV files in a directory"
... enough said.

OK, too many negatives so far ...

auth module looks very cool. i'd love to be able to support all those
out-of-the-box.

web "UI modules". looks interesting, need to take a closer look.

XSRF. nice, and now I know how to handle it without sessions ... use a
cookie instead, duh!

static file serving ... i like the automatic far-future expires + file
version. i've been meaning to write something to do that for a while
now only I was going to use the file's timestamp instead of the
content hash (much better).

Ten Tracks - 10 music tracks for £1

2009-03-10T13:00:00.003Z

Looks like Ten Tracks only launched late last year ... but wow, what a fantastic site for music lovers.

Basically, there are a bunch of channels, each channel publishes 10 tracks per month (almost, it's early days yet) and we, the music buying public, get to buy all 10 tracks for just £1.

Thank you Toob for telling me about the site and also for allowing Ten Tracks to distribute one of your tracks in this month's Open Ear channel.

World of Goo on Linux ... hurray!

2009-02-13T20:29:00.004Z

2D Boy have just released the Linux version of World of Goo. As if that wasn't good enough, there are even .deb packages available.

A wonderful game just got a bit more wonderfuller (and yes, that is a real word ;-)).

restish resources ... from the ground up

2009-01-07T00:36:00.017Z

I've been working on a WSGI web framework called restish. Yes, another one ... sorry! Only, this one's got a serious preference for trying to stay close to HTTP and the way of the web, and therefore encourages REST principles.

restish is really simple and, in my opinion, a pleasure to use. It's also extremely light-weight compared to some other web frameworks, mostly because it doesn't actually attempt to do that much :).

One feature that will hopefully be of interest is that there is no reliance on threads. There's no thread local use anywhere. In fact, they're banned, I despise the things! That allows a restish app to run happily inside a threaded Paste Deploy server or a Spawning server in greenlet mode, i.e. with threads switched off), or potentially any other web server. Basically, *you* choose how you want to deploy it. If you decide to use a threaded model then fine, but why should the web framework dictate to you form the start?

Anyway enough, let's see some code. I thought I'd try to demonstrate the basic idea of restish resources. (I do intend to move all this to the restish documentation at some point but that will take longer than an informal blog post.)

Oh, all the following bits of code *should* run. If you want to try them out, the 'application' is actually a WSGI application. Running under Spawning is as easy as:

$ spawn module_name.application

The simplest resource imaginable

from restish import app, http

def hello_world(request):
return http.ok([('Content-Type', 'text/plain')], 'Hello, world!')

application = app.RestishApp(hello_world)

OK, so there's this thing called a RestishApp. It's just a WSGI application that kicks of the request handling process. Nothing too interesting there. When it's created it's passed the root resource for the site.

A resource, at its simplest, is something callable that takes a http.Request instance as its only arg and returns a http.Response instance. You can build a http.Response yourself but the http module provides some response factories to simplify application code and save a bit of typing.

The Resource class

from restish import app, http, resource

class HelloWorld(resource.Resource):
def __call__(self, request):
return http.ok([('Content-Type', 'text/plain')], 'Hello, world!')

root_resource = HelloWorld()
application = app.RestishApp(root_resource)

You weren't really expecting anything interesting so soon were you? ;-)

Most of the time using a function as a resource is too limiting so restish provides a Resource class. It has some magic abilities as we'll see later but, just like the hello_world(request) function above, it's basically something callable.

Request parameters

from restish import app, http, resource

class Users(resource.Resource):
def __call__(self, request):
username = request.GET.get('username') or 'anonymous'
return http.ok([('Content-Type', 'text/plain')], 'Hello, %s!'%(username,))

root_resource = Users()
application = app.RestishApp(root_resource)

I hope noone's impressed by that code. In fact, I'm not even going to describe it but would like to point out that passing the username as a URL segment is almost certainly a nicer way to do things. So, moving swifly on ...

Resource children

from restish import app, http, resource

class Users(resource.Resource):

def __call__(self, request):
doc = "matt: %s" % (request.path.child('matt'),)
return http.ok([('Content-Type', 'text/plain')], doc)

@resource.child()
def matt(self, request, segments):
return user

def user(request):
return http.ok([('Content-Type', 'text/plain')], 'Hello, matt!')

root_resource = Users()
application = app.RestishApp(root_resource)

The Users resource (the root of the site) returns a document that looks like, "matt: /matt", where "/matt" is the URL of the "matt" resource. Notice how the URL for the matt resource is created? 'request.path' is a url.URL instance - a smart string that knows how to parse and manipulate URLs, e.g. by adding a child segment. http.Request has a few URL instance attributes.

The 'matt' method has a @child decorator to expose it as a child resource factory. By default @child() uses the name of the decorated method as the name of the segment it matches so here it will be called to create a resource for the 'matt' child, i.e the thing at the URL '/matt'.

(You can pass an explicit segment name to @child instead, e.g. @child('matt'), allowing you to call your method whatever you want.)

Statically-named children are useful and quite common but dynamically-named children are more interesting.

Dynamically named resource children

from restish import app, http, resource

USERS = ['alice', 'matt', 'rebecca']

class Users(resource.Resource):

def __call__(self, request):
doc = '\n'.join(['%s: %s' % (username, request.path.child(username)) \
for username in USERS])
return http.ok([('Content-Type', 'text/plain')], doc)

@resource.child('{username}')
def child_user(self, request, segments, username):
if username in USERS:
return User(username)

class User(resource.Resource):

def __init__(self, username):
self.username = username

def __call__(self, request):
return http.ok([('Content-Type', 'text/plain')], 'Hello, %s!'%(self.username,))

root_resource = Users()
application = app.RestishApp(root_resource)

We now have a "database" of users. OK, so it's just a list of username but you get the idea. The User resource returns a document containing a list of users each with their URL.

This time, the @child decorator has been passed a segment match template. '{username}' means match a single URL segment, extract the segment and pass it to the method as the username keyword arg.

The child_user method returns a User resource instance, giving it the username the resource represents, or None to signal a 404.

If you think the User class is bit "heavy" then, no problem, use a partial function instead, or a lambda it you prefer:

class Users(resource.Resource):
[...]
@resource.child('{username}')
def child_user(self, request, segments, username):
if username in USERS:
return functools.partial(user, username)

def user(username, request):
return http.ok([('Content-Type', 'text/plain')], 'Hello, %s!'%(username,))

Request methods

So far, every resource would respond in exactly the same way for all HTTP methods. It doesn't differentiate between GET, POST, PUT, DELETE, etc. Let's fix that now.

from restish import app, http, resource

USERS = ['alice', 'matt', 'rebecca']

class Users(resource.Resource):

@resource.GET()
def text(self, request):
doc = '\n'.join(['%s: %s' % (username, request.path.child(username)) \
for username in USERS])
return http.ok([('Content-Type', 'text/plain')], doc)

@resource.child('{username}')
def child_user(self, request, segments, username):
if username in USERS:
return User(username)

class User(resource.Resource):

def __init__(self, username):
self.username = username

@resource.GET()
def text(self, request):
return http.ok([('Content-Type', 'text/plain')], 'Hello, %s!'%(self.username,))

root_resource = Users()
application = app.RestishApp(root_resource)

The only difference here is that we've replaced the resource's __call__ method with a nicely named method decorated with @resource.GET(). Now the resources only respond to a HTTP GET; anything else returns a "405 Method Not Allowed" response.

$ curl -X GET http://localhost:8080/
alice: /alice
matt: /matt
rebecca: /rebecca
$ curl -X POST http://localhost:8080/
405 Method Not Allowed

I've actually just sneakily introduced some content negotation too. Not only does @resource.GET() match the HTTP method but it also matches the request's "Accept" header. However, GET defaults to an "Accept" match '*/*', i.e. any content type the client asks for.

Content negotiation ... at last

I mentioned above that decorating with @GET also performs '*/*' content negotiation. We can easily configure a resource to handle requests for different content types.

import simplejson
from restish import app, http, resource

USERS = ['alice', 'matt', 'rebecca']

class Users(resource.Resource):

@resource.GET(accept='text/plain')
def text(self, request):
doc = '\n'.join(['%s: %s' % (username, request.path.child(username)) \
for username in USERS])
return http.ok([], doc)

@resource.GET(accept='application/json')
def json(self, request):
users = [{'username': username, 'url': request.path.child(username)} \
for username in USERS]
return http.ok([], simplejson.dumps(users))

@resource.child('{username}')
def child_user(self, request, segments, username):
if username in USERS:
return User(username)

class User(resource.Resource):

def __init__(self, username):
self.username = username

@resource.GET(accept='text')
def text(self, request):
return http.ok([], 'Hello, %s!'%(self.username,))

@resource.GET(accept='json')
def json(self, request):
doc = simplejson.dumps({'username': self.username, 'url': request.path})
return http.ok([], doc)

root_resource = Users()
application = app.RestishApp(root_resource)

This time we have 'text' and 'json' methods, decorated with @GET(accept='text/plain') and @GET('application/json') respectively. Now we have a resource that will look at the Accept header, find the best matching method and call it. No match results in a "406 Not Acceptable" error.

$ curl -H "Accept: text/plain" http://localhost:8080/
alice: /alice
matt: /matt
rebecca: /rebecca
$ curl -H "Accept: application/json" http://localhost:8080/
[{"username": "alice", "url": "/alice"}, {"username": "matt", "url": "/matt"}, {"username": "rebecca", "url": "/rebecca"}]
$ curl -H "Accept: text/html" http://localhost:8080/
406 Not Acceptable

Note that the resource no longer has to specify Content-Type headers. That's because the Accept matching process knows what it found and fills it in for you ... how kind :). (Don't worry, you can still include the Content-Type in the response headers if you want to handle it yourself.)

Note also that the User resource uses shorthand in the form of @GET(accept='text') and @GET(accept='json'). They're expanded to the full MIME type on your behalf and so work just the same. Frankly, typing 'application/json' is tedious and 'application/xhtml+xml' is perverse ;-).

Well, that's all for now although there's a few other things I wanted to mention. A quick list will have to do for now:

wildcard accept matching, e.g. 'image/*'

PUT, POST, DELETE, etc

Content-Type header matching (basically the same as Accept matching but for data sent from the client

Handling multiple content types with one method, e.g. @GET(accept=['html', 'xhtml'])

@child URL matching in general

@child that matches any URL

Consuming additional URL segments during traversal

Hope someone finds this post interesting!

The Royal Institution Christmas Lectures

2009-01-03T00:48:00.003Z

I really enjoyed watching the Royal Institution Christmas Lectures, presented by Chris Bishop. I don't think I've actually watched them properly since I was a kid!

We sat down as a family every night to watch and it clearly sparked some interest from the kids. We've had the cover off the computer looking at its innards, we've been playing with Phun (the kids remembered it after seeing the multi-touch displays), we played with GNOME's Dasher. Heck, we even touched on public key encryption.

Sure, there were a couple of bits that weren't so good. In particualr Bill Gates was seriously dull and the programme on software, The Ghost in the Machine, was not exactly great (the kids still don't know what I do ;-)) but over all, fantastic.

If only all TV was that interesting!

GET and idempotence

2009-01-03T00:11:00.004Z

I was reminded today of what seems to be a common misunderstanding of the idempotent requirements of a GET in a REST-ful architecture. GET doesn't mean the response must be the same each time; only that it must have no side effects, i.e. it should never cause the server's state to change.

For instance, CouchDB includes a resource, /_uuids, that returns a number of server-generated UUIDs. As far as I know, it only exists to support languages without a decent UUID library. It has no effect on the server.

However, CouchDB will only respond when /_uuids is POST'ed to:


$ curl -X GET http://localhost:5984/_uuids?count=2
$ curl -X POST http://localhost:5984/_uuids?count=2
{"uuids":["d267c530c9591eeeed72f589fcef5599","f4c12de5147ec97e7a770440cc7a69f2"]}

One suggestion on the mailing list (although not from one of CouchDB's core developers) for the use of POST is to, "comply with REST as it returns a different output each time".

A GET would be just fine here. In fact, it would be more in keeping with the intended use of the HTTP methods.

It's easy to come up with examples of a REST-ful resource that sends a different response every request, with probably the most obvious being some sort of time server. Consider the following URLs:

/time
/time/BST
/time/EST
etc

You would GET the current time from an appropriate resource and I sure hope there will be a different response each time ;-). You could also PUT a time to one of those resource to set the time.

Flow Charts (from xkcd)

2008-12-17T23:11:00.003Z

xkcd is so often amusing but occassionally they post something that quite literally makes me laugh out loud. Today, is one of those times ...

(from http://xkcd.com/518/)

Selenium, give me back my desktop!

2008-12-05T19:21:00.004Z

Selenium is a great tool for testing web applications but, when running it locally, it does tend to take over your desktop with Firefox windows popping up constantly, leaving you twiddling your thumbs until the test suite is finished.

A beautifully lightweight and simple solution on Linux is to display the Firefox windows under test in an Xnest window.

So, here's my Selenium server startup script:


# Configure the display we'll run Xnest on and the location of the Firefox
# binary.
XNEST_DISPLAY=:10
FIREFOX_BIN="/usr/lib/firefox-3.0.4/firefox"

# Start Xnest and capture its pid.
Xnest $XNEST_DISPLAY -ac &
xnest_pid=$!

# From now on we want all X11 apps to start on Xnest's display.
DISPLAY=$XNEST_DISPLAY

# Let's run a nice, simple window manager on it.
openbox-session &

# Run the selenium server.
java -jar selenium-server.jar -forcedBrowserMode "*firefox3 $FIREFOX_BIN"

# Close the Xnest session.
kill $xnest_pid

Doctor Who Confidential

2008-04-19T22:07:00.000+01:00

Pah, it's nothing to do with a, "Behind-the-scenes look at the making of Doctor Who". It exists purely to show my 4 year old that the nasty monsters aren't actually real so they can sleep at night. ;-).

Adobe (Hot) Air

2008-04-05T00:01:00.002+01:00

Came across this fantastic statement in some Adobe marketing video, :

"So, it's actually the first time that you've ever been able to double-click on an FLV file on your desktop and actually watch it. Before you've had to basically watch Flash video through the browser."

Silly me. I can't possibly have been able to play FLV files on my computer for quite some time and all these projects must be completely deluded too.

2nd Kyu, Aikido

2008-03-11T22:53:00.005Z

Yay, I just got 2nd Kyu Aikido!

I know I made a number of mistakes (like not avoiding and getting a tanto in the stomach :-/) but I obviously did well enough.

Recently I've been practicing randori-no-kata, shichi-hon-no-kuzushi, randori-no-kata-no-ura-waza and koryu-dai-san; all a long way from perfect, of course. And what did they ask me to do .... left-handed Randori-no-kata, tanto hikitategeiko and some free play. Oh well, it's all good stuff.

The general comments about the grading, and I know some apply to me, was that it needed to be more flowing with more movement and "life".

Comments on my personal performance was that I was getting a reasonable amount of movement in the tanto hikitategeiko, and that my kata was quite good. They obviously weren't watching the whole time ;-).

Virding's First Rule of Programming

2008-02-27T21:00:00.006Z

"Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang."

-- Virding's First Rule of Programming

I'm really no Erlang expert but, from what I've seen so far, there's a fair amount of truth to that statement.

Earthquake rocks house ... a bit

2008-02-27T01:04:00.002Z

Just sitting here playing around with Erlang and Mnesia's database replication when I felt what can only have been an earthquake.

It wasn't very big (this is the UK, after all) but it made the house shudder quite strongly.

I've felt one here before. That was more like a very low rumble, quite a strange sensation, but I don't remember it making the house move to the same extent as this one.

I tried to check with the British Geological Survey Seismology Home Page but it seems to be down at the moment. Have to try again in the morning.

One site account, multiple OpenID identities

2008-02-26T17:32:00.004Z

If you're adding OpenID to your site then please allow your users to add as many of their OpenID identities as they want to their account.

I've got a few identities I use so adding a couple to sites that are important to me means I'm not reliant on a single OpenID provider being available when I need it. It's a simple form of fault tolerance.

Multiple identities can also be a way to allow an account on a particular site to be shared by multiple people, each with their own authentication credentials.

Get your MP3s now, last few left

2008-02-24T16:45:00.002Z

Spotted this on play.com.

Erm ... but ... why? Are they going to run out of them or something ;-) ?

Perhaps this time I'll keep it up

2008-02-24T16:00:00.003Z

I've kept various forms of blog over the years from personal "just to keep some notes" to a company-related journal. I've been hopeless at posting regularly to all of them and, as a result, they have withered and died.

The whole corporate journal thing really doesn't appeal to me much anyway. If I'm going to post to a blog then I really don't want to have to think too hard about it. I've also always felt very uncomfortable posting anything of personal interest to it for fear of "losing" it if I ever change job.

I doubt this blog will do much better (I've not even convinced myself that I want to use blogger.com yet) but it really is about time I posted about stuff I'm interested in, stuff I play with, etc.