Anthony Bailey's blog

Review: Anathem

Wed, 29 Apr 2009 23:14:48 GMT

After the more historic Baroque Cycle, Stephenson returns to speculative sci-fi a la Diamond Age, playing with the parallel evolution of language, culture, and quantum narratives.
I enjoyed this a lot. It oddly reminded me of the Riverworld saga, but with philosophical ideas rather than characters from history making their cameos against a densely detailed unfolding backdrop and story.

Review: The Baroque Cycle

Sat, 21 Feb 2009 20:42:36 GMT

The Baroque Cycle by Neal Stephenson

Size matters, here. The Baroque Cycle is something of an epic, at around one million words long. (Workings: its three books end on page numbers 916 + 815 + 886 = 2617 pages * 45 lines/page * 10 words/line = 1.12Mwords, then round down some whitespace.)

The trilogy is a "behind the scenes" saga spanning fifty years of scientific and economic alchemy, featuring characters from the pages of history but centered on three of their peers whose tales escaped the official record.

In many ways this is the 17th/18th century version of the same author's more contemporary Cryptonomicon, with which it shares the above approach, plenty of adventures colored by cryptography and computation, and more. However, I found its sprawling scope resulted in less value per page: its three books contain only a little more gold in total than Cryptonomicon's one.

Contacts: Let's keep all this between ourselves

Mon, 15 Dec 2008 23:38:46 GMT

I want to describe how I like contact(s) information on the Net to work.

I say "contact(s)" because I have complementary ideas regarding both the contact details for a person, and the details of who their contacts are - you can say "friends" or "social graph" for the latter if you like. To oversimplify: I want the information sources to be distributed, and to delegate to individuals as authorities for their own details. I find that most current technologies don't share this ideology; however, I can still use them to act in tune with it.

An introductory note: I'm going to talk about people "owning" a web page. What I mean is that they have significant control over the main content of the page, which is published under as permanent an URL as possible. They definitely don't have to serve the page from their own machine, or from a host that they rent - a page on a social network or in an ISP's basic web hosting area is fine. The URL can be under someone else's domain, so long as they have confidence it will last a while. (IMHO aside: own the domain.)

My suggestion about contacts assumes some kind of a homepage/identity baseline: that people own some page that is about them - such as a blog, or a presence on a social network. Their homepage URL is then a decent identifier for their identity online. For bonus points, they can prove that they are the person who owns that page through OpenID, or Facebook Connect, or similar. All this is standard for many people online already, and it can easily enough be true for anyone else who cares to make it so.

My stance is that I want to own my contact information - because I believe I'm best placed to be the central authority it. I want to publish it as a resource located on or somewhere that is discoverable from my homepage, in a standard format. (Until the world stabilizes a bit, probably in multiple formats, all views of the same underlying data.) I'd like others to do the same. It's up to them exactly which details they publish. If they care to, they can even restrict access to some information, making it "friends only". (IMHO aside: most privacy is squeamish.)

I also want to own my contacts information - the list of other people that I know. For each of them I want to store and publish a homepage URL, and also a name - given how neither or those identifiers is completely permanent. I also own how I personally would describe my relationship with these people. This perspective need not be reciprocated.

But, I don't want to own my contacts' information. I want to delegate that to them! In as much as my contacts list is an address book, it should be so only through indirection: to find out about a contact, go to the homepage I have linked for them, and discover the contact information that they themselves have published there.

Most current systems do not assume such delegation: I am expected to own other people's contact information, as well as mine. I presume this is a deliberate compromise in order to gain critical mass (of contact details published online in general and on the vendors network in particular.) My own disdain for such practicality comes partly from not finding much need for an address book per se - perhaps if my network(ing) were better this would be otherwise. But, even allowing for that, I don't see the delegation I desire called out as an ideal or an aspiration anywhere in the design or implementation of current systems. Evangelizing this, and exploring how to get there from here, is the motivation for this and my next blog entry.

There are a two main reasons I want things to work this way.

Firstly, there is the golden rule: if I want to own my contact details, I should should grant other people the right to own theirs.
Secondly, I'd rather not have the hassle of complicating my own records with other people's details. Let the person who knows best update their information in a single place, and I'll get their latest details by going there. This also avoids my having to take responsibility for choosing which of their details to publish. Given current preciousness about privacy, this side-steps a potential minefield.

There is a yet more open approach that I have refrained from advocating. In a truly distributed wiki-wiki-world, one could argue that an individual's opinions on information about themself should not be sacrosanct - that we should seek a consensus of everyone's opinions. On most truths, I'm all for this kind of emergence. But when it comes to how to contact an individual I think having them be the single authority is too useful a shortcut, and it may have psychologically benefit, too! Letting the rest of the world speak for those as yet offline, though, could be a decent strategy to cover the gaps between where we are and where I want us to be.

Talking of such strategies: next time I'll write about my current implementation of the plan I advocate using existing contacts technologies - hCard, XFN, FoaF, PoCo, et al.

I have some further opinions that are related to the discussion above but would have distracted from its core, so I've moved them down here.

Own the domain: If you're going to own a page of significance to you, I would advocate controlling the domain name of the URL you publish it under. By all means have another party host the content, just point your own URL at where the page currently is. That way you can change the service/hosting provider without losing the URL by which people find the page.

Most privacy is squeamish: Although many are nervous about publishing it, I think much contact and other "personal" information is most naturally public: going ex-directory is weird. The impact of problems that might arise from publishing e-mail, mailing address or biography is overestimated. You're getting spam anyway. Stalkers are rare. The existence of the concept of "identity theft" occuring through knowledge of simple everyday facts - like a birthday, a hometown, a mother's maiden name - is a symptom of immature authentication procedures, a laughable security through not-even-obscurity. Lastly, contact details are useless without at least some users - and trying to control how others use and copy bits is a mug's game.

Review: Implementation Patterns

Wed, 22 Oct 2008 19:50:10 GMT

Implementation Patterns (Addison-Wesley Signature Series) by Kent Beck

This book concerns the detailed choices we make minute-to-minute when implementing programs.
The first twenty pages express Beck's context and motivation in his usual fashion: some personal history and philosophy, some thoughts on cost, and a prioritized set of values (communication, simplicity and flexibility) and some principles (DRYness, symmetry, preferring declarative to procedural expression, plus three ways to group things that change together) that help in translating those values into specific practices or patterns.
The next hundred pages is a catalog of seventy-seven such patterns.
Two closing chapters offer an overview of collections, and sketch some ideas of how things change if you're coding a framework rather than a specific application.
I think those first twenty pages are excellent. Two of my previous favorite books by Beck outlined a core precept, asked "what if we run with it?", then detailed how to extract far more value than you might initially expect. (To over-simplify: XP Explained follows from keeping the cost-of-change curve as flat as possible, Test-Driven Development follows from a belief that making code testable improves its design.)
The main idea in Implementation Patterns is that making code readable is key in good implementation, and that turning those dials to eleven will provide similar unexpected insight. (For example: given that a good name for a variable describes its role in context, then if all good names seem long, you're probably missing an abstraction of context - i.e. there's a class waiting to be extracted.) Not quite as big an idea as in the other two books, but very plausible, and my appetite was whetted for the patterns where I hoped this idea, and all the values and principles, would be explored and discussed in context.
Alas, no. I was very disappointed. (The above was about the only example I could find.) There was very little exploration and discussion, and the patterns did not often refer back to the values and principles, even implicitly.
OK, since it doesn't build sufficiently on the introduction's abstractions, the meat of the book had better work well as a stand-alone catalog of patterns.
Alas, a second no. One of the useful things explicit lists of patterns do is to give a name to a concept so we can communicate about it. Beck's list does do this useful thing. But, that is about the only useful thing it does. Once named, there is rarely additional insight in the pattern descriptions: they mostly do exactly what they say on the tin. You can read the table of contents to see how simple the concepts under discussion are.
The last two chapters are fine, but unremarkable.
My recommendation: borrow the book to read the material preceding the catalog. Then buy and read Clean Code.

Minimal OpenID glue for Rails authentication

Sun, 19 Oct 2008 21:04:04 GMT

The Rails framework and culture offer a well-developed set of plug-ins and patterns for the authentication of users and the authorization of their actions. These are great for more sophisticated enterprise and social applications - but as a hobbyist, my own apps don't tend to need a database full of users and roles. Typically the only user I care about authenticating is me, for write access to content that everyone else should view read-only without any hassle. I want something very lightweight.

The app behind my ego site is a good example. It presents pages based on a few sets of resources - contacts, and various flavors of permalinks for blog entries which are hosted elsewhere. Everybody should be able to access those pages. Behind the scenes there is some simple scaffolding for managing the resources. Only I should be accessing those.

The simplest solution is security through obscurity - to keep quiet about the management URLs. It's not that bad an approach for a low-stakes app, and I did rely on it for a while. Felt wrong, though.

Having decided to do better, I still wanted to keep things as lightweight as I could. My first preference is to minimize the use of the Rails session: I don't want people who visit my public pages to get lumbered with a pointless cookie, so I switch sessions off by default.

One way to keep authenticated session management out of the way of users and apps is to place it in the HTTP layer. I am a big fan of the http_authentication Rails plug-in. The browser on the client side takes care of the session sign-on UI and state, getting users to log in once on first access to protected pages. In your app, you can use a before_filter with implementation as simple as:

authenticate_or_request_with_http_basic do |ignore_user, password| password == 'your secret' end

But, I have a second preference. I don't want arbitrary secrets in my apps, or in my brain. I want single sign-on, please, and my web app just needs to know that I am who I say I am - let someone else do the verification of that identity. And I am already enthusiastic about a technology for authenticating identity in this way - it's OpenID.

Following a "gem install ruby-openid", I still wanted as little Rails glue as possible. The open_id_authentication plug-in is pretty compact given its power, but it does tend to assume that users will live in the database, and it provides all kinds of store machinery that I don't need.

So I cut non-essentials away from the plug-in source and ended up with this minimal parent for my resource controllers.

require 'openid/consumer' class ResourceProtectingController < ApplicationController @@PROTECTED_ACTIONS = %w(new create edit update destroy) session :disabled => false, :only => @@PROTECTED_ACTIONS before_filter :protect, :only => @@PROTECTED_ACTIONS protected def protect return if session[:authenticated] realm = "#{request.protocol + request.host_with_port}" requested_url = "#{realm + request.relative_url_root + request.path}" identity_url = "http://anthonybailey.net" consumer = OpenID::Consumer.new(session, nil) if params[:open_id_complete].nil? open_id_request = consumer.begin(identity_url) open_id_request.return_to_args['open_id_complete'] = '1' redirect_to(open_id_request.redirect_url(realm, requested_url)) else returned_params = params.reject {|k, v| request.path_parameters[k] } returned_params.delete(:format) open_id_response = consumer.complete(returned_params, requested_url) if open_id_response.status != OpenID::Consumer::SUCCESS render :text => "Forbidden: got #{open_id_response.class} " + "on claiming #{identity_url}", :status => 401 else session[:authenticated] = true end end end end

Some notes on the implementation:

Because OpenID involves a stateful request dance between the relying party and the identity provider, I had to reintroduce the Rails session. However, I can still keep cookies away from the public by only enabling it for the actions needing protection.
Now I have a session, I ensure that only one dance is required, by stashing the authenticated status. This also works around the fact that OpenID can only return using the GET method: in regular use the client will GET a prelude page before it needs to POST, etc. a real change, at which point we've finished dancing with the provider.
Mostly as a conceptual indulgence, I left the read-only actions unprotected. (To keep things GET-first, I moved scaffolding UI to delete resources off index pages and onto edit pages - but I kind of feel that's where it should be anyway.)
Since if you're not me, you're not supposed to be here, I didn't friendly-up the content when access fails.
Finally, as an aside, it amuses me that my identity URL actually is the domain of the URLs I'm protecting. (I don't run my own identity server, mind - I delegate.)

Review: Little Brother

Wed, 01 Oct 2008 23:07:23 GMT

Little Brother by Cory Doctorow

This is a fun coming-of-age adventure telling a very contemporary tale of tech-savvy teenage righteous rebellion against a police state.
The plot is straightforward in the large, but the formula is well-executed, is written in a convincing voice, and features wonderful details. Doctorow regularly writes apparently futuristic science-fiction simply by basing his fiction on today's cutting edge, and the edge cuts keenly in this book - the politics and technologies are all here today or grown tomorrow.
I read the free CC-licensed download, as I did with one of his previous Eastern Standard Tribe - and the business model of this publishing approach (of which Doctorow is one of the earliest adopters and most passionate proponents) proves itself again: I've just ordered a copy of the just released UK paperback to send as geek propaganda to my own younger sibling.

Review: Clean Code

Wed, 03 Sep 2008 19:43:22 GMT

Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin

I wouldn't usually have cared to read yet another book on the generics of good programming practice. But this one was by Bob Martin, "Uncle" of the first family of thoughtful contemporary coding, and whose blog is high on my roll.
The book claims to be unusually hard work due to the substantial amount of code it contains. I disagree - you certainly want to dive deep into the examples, but I've read plenty of texts containing more code and less value. (Although, for a book that encourages reading code, and that argues for the retirement of outdated formatting conventions, I am frankly mystified by the fixed-width, unhighlighted, vanishingly thin font used to present the examples: there might as well be tractor holes down both sides of the print-out.)
Martin makes the pragmatic simplifying choice of presenting everything in Java. That's fine; apart from some concurrency specifics, most of what is promoted will slip easily back and forth to e.g. C++ and Ruby - and if you can make Java pleasant to read, you've earned my respect.
After an introduction featuring great abstract definitions of cleanliness from the near-godly, we get twelve solid chapters of concrete advice. An account of best practice should not be surprising to a decent practitioner - and I found much of Clean Code to be a reassuring mix of the obvious ("oh, I already know that") and the nicely crystallized ("oh, that's what I already know".) TDD, culling comments, careful naming, ultra-DRY, breaking dependencies, small methods, yada yada yada.
There's also sufficient challenge ("oh, you're going to push it that far?") and controversy ("er, wtf? maybe...") to keep it interesting. Uncle Bob is illuminatingly obsessive about the law of Demeter and the single responsibility and open/closed principles. He particularly surprised me with his enthusiasm for keeping plain old data structures around within the object-oriented world.
But the three large case studies that close the book are the real gold here - the summary list of smells drawn from them is no substitute for their glorious, odorous detail. They are very engaging: despite their final shininess, I bet you too will want to polish them a little further.
Recommended.

Review: Bulletproof Web Design

Sun, 20 Jul 2008 16:08:40 GMT

Bulletproof Web Design: Improving flexibility and protecting against worst-case scenarios with XHTML and CSS by Dan Cederholm

This is a slim (270 spacey pages) primer on robust CSS practice. It isn't a reference book, nor really a compendium of recipes; more a set of ransackable case studies through which runs a persuasive essay on how and why to do this stuff right. (The basic precept is a principled separation of concerns: define style for mark-up that is independently semantically sound, rather than hacking div after nested div into your page in order to kludge your target design.)
I bought it a couple of years ago to learn how to do CSS-based layout, and did so by coding through many of the examples. I recently read it through a second time and was again impressed by its clarity and cohesion, and by how its elegant solutions do not sacrifice practicality - hence this review and recommendation. (And apart from a few thankfully obsolete IE5-generation workarounds, the content had not dated at all: though it preaches graceful degradation, the book hasn't yet had to practise it.)

(If it seems strange that I broke blog silence this way - yes, usually I'd put such a brief piece on my tumblelog, but Tumblr eats the hreview mark-up.)

What does BDD offer TDD die-hards?

Mon, 14 Apr 2008 00:06:30 GMT

"As a long-serving test-driven developer, I want to understand how behavior-driven development differs, so that I can decide whether or not to join the revolution."

Introduction

A couple of years ago I was developing medical imaging software for diagnostic radiologists. The field exhibited an archetypal divide between non-coding experts who knew lots about the domain, and developers who did not. In the language of classical XP, we were coding from requirements supplied by a customer. Without a map, we explored many aspects of the customer/testing/developer relationship:

the importance of a ubiquitous language spoken across the whole team
the connection between a spec and an executable test
the automation of customer-written acceptance tests

Very near the end of that time, I attended XP Day 5 in London. Whilst there I talked some with Elizabeth Keogh, who was assisting Dan North with the birth of Behavior Driven Development. It sounded fascinating and very relevant.

But it became less of an immediate concern, because soon after I changed jobs. I moved to a new language and test suite in the context of a (simpler) domain and a culture where the devs were encouraged to internalize the expert/customer role.

Now, over the past couple of years the BDD meme and tools have been surging in popularity. Meanwhile I'm working well in a classical Test-Driven Development place some indeterminate distance away, catching occasional glimpses in my peripheral vision of what looks like a pleasant enough image in a slightly warped fun-house mirror, and trying to work out how the reflection differs.

A couple of months back Tom ten Thij gave a great BDD introduction at our local Ruby Users Group. But he had to handle heckling from many of the audience who met the following spec:

Given: I'm a software developer who loves TDD and communicating well with customers,
When: I hear a talk on BDD that seems to promise exactly what I get from TDD and good customer communication,
Then: I find myself wondering exactly what all the fuss is about.

I took the question back to Liz, and she had interesting things to say. I hope sometime she may write them up properly herself, but in the meantime she said it would be fine if I wrote something. So, here's an [edited] transcript of parts of the e-mail conversation we had.

Context

Ant: Yesterday at our local Ruby User Group here in Edinburgh we had a presentation on RSpec and on BDD in general. The talk presented BDD from first principles rather than assuming previous TDD experience. The audience was mostly TDD old-timers, so a lot of the talk seemed like an very familiar tale told in slightly different dialect.

Ant: Our fuzzy consensus was acceptance that BDD is a nicer syntax; its language is a better fit for the good things that test/spec-driven development tries to do and hence the way it should be done from first principles - but, is the change worth it for TDD die-hards who have already completely internalized xUnit? What else do we get?

Ant: We realized none of us had ever come across a talk or article evangelising BDD which particularly spoke to us; for example, by a someone who'd been very dedicated acolyte of TDD and had many wonderful things to report once they'd learned to Behave.

Liz: I can't think of anything off the top of my head that addresses TDD gurus, which is awful, because we should have sorted this out ages ago. Let me see if I can help - please let me know if this is any use!

Ant: I'll devil's advocate some. Personally, I'm a big believer in the importance of language [and] I'm simply not experienced enough to make any claim of having seen it all before [but] I'll proxy for my peers who have been doing/teaching TDD for donkey's years.

Philosophy

Liz: BDD isn't just about reworking the syntax of testing, though it started out that way. It's evolved into something more; a language of examples that's relevant at both a unit and system level, which allows better communication with the business, and less redundant code.

Liz: DDD [Domain-Driven Design] provides the nouns, verbs and adjectives, and BDD provides all the other words which let you make sentences out of them. (Dan North, Eric Evans and I were chatting about this at OOPSLA; I can't remember the exact words they used, but this is close.)

Liz: It's all about the language. You don't really get anything else... but language is pretty powerful! Changing the words we use can change the way we think, and we don't have to change our technology to do it.

Territorial dispute

Liz: TDD is mostly about development - about taking a specification or acceptance criteria, at either a system or unit level, and making the associated tests pass. BDD assumes that it's not easy to get the right acceptance criteria in the first place, so the language - including the famous "should" - is designed to allow the original assumptions about the behaviour of the system to be questioned; to have discussions about that behaviour without having to translate from the technical world, either in speech (which helps us devs talk to BAs and the business) or mentally (which helps us realise when we need to ask a question). Of course, this is very closely aligned with DDD. It's also why I prefer "behaviour" to "spec"; because it's easier to question behaviour than a specification.

Ant: I disagree with this characterization. Particularly at the lower levels where the developer may be doing more of the work than the domain expert, traditional TDD is very much about exploring and evolving the behaviour of a system, one test at a time, asking questions rather than blindly coding up an existing design or specification. I completely buy the idea that the language of BDD helps make the openness to change explicit, though.

Liz: The best TDDers do BDD already, at least at a unit level. It's possible that they don't think of elements of code as stakeholders, though; at least, I haven't found any native TDDers with that mindset. It changes the names of classes, at the very least. I am also aware that some die-hard, expert TDDers concentrate more on the technology when they're teaching than the conversations around tests.

Ant: It helps my understanding if I consider BDD to be taking a concept of TDD that fits particularly well with driving from behaviors, and turning the dials on that concept up to ten. If not plain wrong, this characterization may play well with an old guard XP/TDD target audience.

Frameworks

Liz: First off, let me put to rest something really important: You can do BDD in xUnit.

Ant: would you go so far as to say that if you aren't starting from scratch there may not sufficient advantage to switch from xUnit to a more well-Behaved framework even if one falls for BDD in a big way?

Liz: Don't use JBehave. It's a great teaching tool, but sucks IRL. I tend to use JUnit. JBehave 2.0 should (when it eventually comes out) be easier. It's going to be more like RSpec's plain text story runner, and consist of a lot of JUnit exensions.

Talking to business

Liz: [BDD offers] a language of examples that's relevant at both a unit and system level, which allows better communication with the business We use the word 'scenario' or 'example' because you don't go to the business and say 'Give me an acceptance test'. You say 'Give me an example' or 'Give me a scenario where that happens'. It becomes easier to maintain acceptance tests - and they become human-readable!

Ant: Would you venture further? Non-developers can edit the acceptance test? Non-developer can write the test when paired with developer as translator? Non-developer can even write the test unambiguously without direct developer assistance, given some further translation layer like a plain-text story runner?

Liz: Absolutely. This has worked very well with our QAs.

Ant: The last was a feature of the talk I attended. It drew attention from the old TDD guard, who were all very XP-story-focussed and some of whom had developed such translation layers themselves. It provoked both a "these BDD people sure have the right idea" response as well as a "and it's an idea we had a while back, and is separable from the question of changing terminology at the xUnit level" one.

Liz: Good. "Should" is trivial compared with the given / when / then language.

Ant: Partly putting words in their mouths, and partly exploring something I thought might be interesting: if you do end up with another layer between the non-developer and the runnable test/behavior/spec - and our experience is that perhaps this is worthwhile - then some of the communication advantages of using BDD at the bottom layer are diluted. If you have to use a translation layer like a story runner to pre-process a more plain text description of behavior, then using BDD terminology in the target code doesn't help you so much with respect to communicating with the non-developer, because they don't see that code, they see the source story. At that point the BDD communication argument perhaps reduces to the DDD one (albeit one I buy) that using business language as much as possible in the code itself is worthwhile even when only the developer sees it.

Ant: I guess I was wondering whether you'd challenge the pre-condition - can one make behaviors expressed in the production code language sufficiently like plain English to be writeable by the non-developer? (Perhaps it depends on the language?)

Liz: RSpec's stories now work in plain English. We hope JBehave 2.0's will be the same. It's a nice little given / when / then DSL.

Ant: I guess my point was that some long-time TDDers have so internalized some of the translations from dev-centric terminology to plain English, that they might consider a test very easy to read way before the non-developers do. But purging such language habits would probably ease indigestion nevertheless. And if BDD terminology becomes the standard, there's also a communication benefit amongst developers - the next wave may well see TestCase/test/assert as strange dinosaur language.

Outside-in

Liz: [The BDD spec] leads to the first piece of code which the human stakeholders need, which is the UI - the mechanism through which users (or 3rd party apps, say) interact with our system, and through which behaviour is perceived. So this is how we develop with BDD - from the outside-in.

Ant: This was a part of BDD missing from the talk we heard, and it sounds interesting. I enjoyed your real life example. [Re TDD] with "the YAGNI of outside-in" - whilst YAGNI itself is a TDD cornerstone, I haven't heard the tenet that therefore one should always start with the user interface layer.

Liz: If you don't start with the interface, how do you know that you need it? How do you know that the stakeholder code will be able to use what you're writing in the way that you're writing it? You can pretend that you already have the code, work out how you're going to use it, then code the real thing later - this is exactly what we do with TDD anyway! Now we're taking that pattern into production code too.

Code has needs too!

Liz: We use the same language of examples, to describe behaviour that provides some benefit, at both a unit and a system level. The stakeholder at a unit level becomes the other piece of code which needs that behaviour - so unless we have some code already, we can't start.

Ant: This analogy between human and code clients is not one I'd heard advanced quite so strongly in this context - is it important to BDD?

Liz: Yes!

Ant: [In TDD] this definitely isn't an angle I've seen pushed hard. Trying to make client use of the new code readable, sure, but not the kind of anthropomorphism you're advocating.

Liz: I'm not thinking of the stakeholders as human, as such, though it's a good analogy - code changes just as much as people do! BDD is very need-driven. Why are you coding that? Who's it for? What is going to benefit from that line of code? It changes the names of classes, at the very least. [...] We've started naming things based on the responsibility to its stakeholder, or its job, instead of the design pattern. As an example, we've started calling repositories after the plurals of whatever you get out of them - so, "Puzzles" for my sudoku creator, as opposed to "PuzzleRepository".

Given/Then, When

Ant: Random aside... is it perhaps the case that most mocking layers for xUnits (at least in languages that don't love closures) make given/when/then expression harder? It seems that expressing the expectations of mocks should go in the "then" section, but they usually assume they'll be set up before you execute the code under test within the "when" section.

Liz: Check this new toy out: Mockito My friend Szczepan Faber made it, based on EasyMock. It doesn't make you set up any expectations unless you need something to return (eg: given three fridges in stock). Instead, you get to ask it afterwards whether the interactions needed happened. No finishedRecording(), and it'll automatically return null, 0 etc. if you haven't set up a particular stub.

Conclusion

Well, I hope you found that as interesting as I did.I'd love it if the "TDD old guard" (who I've quite possibly misrepresented) could comment below to take the conversation further...

Credits where credits due

Mon, 04 Feb 2008 23:16:51 GMT

(A piece about TV credits on Mongoose World caused in turn by this piece from the BBC provoked a rant from me on that subject and particularly how it relates to machinima. It seems to have broken my writer's block, so I've copied/pasted/edited it here.)

Full screen credits on plain old broadcast television might still be reasonable, because how else do you get the information? But that medium is increasingly dead. We shall speak of it no more.

Full screen credits on TV as we increasingly watch it (DVR, DVD, IP) seem anomalous. Pack them into an information burst: if I want to read them, I'll pause and advance through individual frames. If I'm on a smart device, put the credits in metadata I can browse with that device. If the device is Net-enabled, put them on a page I can navigate to from the program. Don't make me sit through this stuff. Brand a little if you must, but if I care about obscure details, I'll seek out the information, and be very happy to read much more than you could cram into the credit footage.

Full screen credits in typical machinima (and other YouTube-length pieces) are a sign the medium is far too indebted to its older media parents, and is too slavishly following their inappropriate conventions. See above: viewer is almost certainly watching on a Net-enabled device. And a minute of credits on a five minute piece is a real tax on my time!

Further, there should be two default downloads/streams for episodic machinima:

There should be the new episode for returning viewers (without credits, of course - we already saw them so many times. You can do a "previously" if the plot so demands.)
And there should be everything from the first up to the new episode in a single download/stream for people coming to the story for the first time.

These are your two markets - cater to them. The second is particularly ill-served by having to watch umpteen short previous episodes in order to catch-up, where tedious credits/theme song prevent me getting into your wonderful art.

(Sorry for the explosion of highly subjective opinion. I don't know why I care so much about it. I remember I tried and failed to convince Hugh to do this kind of thing with Bloodspell.)

'Spell is out

Tue, 23 Oct 2007 23:30:18 GMT

Independent film outfit Strange Company have released their feature-length machinima movie, the "punk fantasy" BloodSpell.

My opinions on the film are necessarily messy. I'm an advocate of the medium (machinima is shooting movies within virtual worlds, often using video games), and a friend of the creators - indeed, I worked on one of the tools used to create the film. So I'm unfairly biased. I'm also not a great person to have on your side, because I don't let such things mellow my criticisms or get me hyped - at best I do faint praise, and damn you.

BloodSpell hits a huge range of highs and lows. Almost every aspect - story, performance, cinematography, art, technology/effects, pacing, dialogue - has parts that are truly great, and parts that are pretty poor. (The exception is the audio: sound and soundtrack are good throughout.) A spread of quality is not unusual in a film, but the range is particularly wide here. There are many genius moments and scenes. There are many embarassing ones.

The film is strong at start and end, weak in the middle. This is partly due to the order of shooting: Strange Company learned heaps about their machinima technologies and about movie-making as they created the piece; originally released in episodic chunks, they returned to completely redo the opening scene when editing together this version. It's also partly due to happenstance - the second scene was already strong, whilst the film serves as an archetypal illustration of the cinema textbook tendency to have the story sag in the middle.

This is by a fair way the best machinima feature film I've seen. The dated technology is necessarily distracting, but didn't spoil my enjoyment. Whilst previous machinima movies have excited me about the future potential of the medium, this one is real, now. It simply works as a decent movie in it's own right - I'd watch it through if it came on TV the right midweek evening, probably whilst bestowing it the odd sarcastic aside because, you know, it's lucky to be on the channel I'm indulging.

It's also probably the best budget independent film of its genre (epic action fantasy drama with nods to comedy and contemporary culture - the one word summary is "Buffy".) This is almost by definition: you simply can't shoot real film (or TV) of this flavor for the few thousand pounds it took to make BloodSpell. (Also this genre of film is not blessed with many success stories. This one can only have helped.)

The film was made by volunteers, is licensed under a BY-NC Creative Commons licence, and you can download or stream it for free. I'd even encourage fans of the genre to do so if it didn't contradict my disclaimer at the start!

Digital Privacy Management is inEFFable

Sun, 30 Sep 2007 21:27:20 GMT

It's increasingly convenient to manage personal data up in the cloud. Eben Moglen worries that people entrust important data to organisations without thinking of the downside. Drawing an analogy with open source in software, he encourages people to store data themselves rather than pumping it into proprietary information silos.

I am persuaded by the argument as I've phrased it above. It's timely to consider investing in keeping copies of the personal data that is important to you, because otherwise you're going to lose it when the silo closes. Ideally you would keep the copy rather than a copy, - you would manage the canonical resource and publish it. This is because you have the most interest in and ability to ensure the ongoing correctness of the data. Because DIY is hard, third-party service providers will help you; but they should be neutral hosts rather than aggregators motivated by their own uses for the information.

Actually Moglen seems to be making the stronger argument that you should hold the data because you need to be able to control it, not just publish it. Commercial entities will enter into contracts with you about what use they are allowed to make of this data that you own and are generously granting them access to. His argument is phrased in terms of privacy - your right to control the use of digital data about you. Maybe you'll even sell access and usage rights.

And this is where I differ from him and a whole bunch of organizations with whom usually I find myself in agreement. Trading overzealous stereotyping for a reduced risk of misquoting, I will pick the Electronic Frontier Foundation as a specific example of what I find to be a widespread and questionable position. I find their stance on issues of privacy to be very much at odds with the pragmatic approach they take to, say, DRM.

On the latter, essentially they say: trying to control how digital information is copied and used doesn't work very well. The technologies are too easy to break, and they prevent unintended new uses of the content, restricting freedoms and stifling innovation. Therefore we should avoid purely technical solutions, and retire historic business models that now rely on them. Instead play nice with the world as it is, and try to nudge it in the directions you desire.

That all sounds right to me. So how on Earth will we be able to use any kind of Digital Privacy Management mechanism to control how personal information is copied and used? Bits are bits, and once you publish them you just don't get to control their copying and use. The hazily envisaged DPM is DRM, only without a law-buying cartel behind it. It's not going to work. Move on, and start discussing and tackling the problems that can arise in a space where there are plausible solutions, please.

(You'll also find the open source analogies work better once technical controls are abandonned, I believe.)

Origins of the word "machinima"

Sun, 09 Sep 2007 17:58:52 GMT

Machinima was spelled slightly differently when first-born. The first use of the term, as "machinema", appeared in an e-mail I sent on the 'q2demos' mailing list on 5 Jan 1998. Here is the quote from the mail in question. Conveniently, it happened to go on to explain the need to coin the term.

Anthony:

I think with a new tool-set and a greater awareness of the skills
required to make a good piece of machinema[*] we can expect to see
some truly adventurous stuff being made. Fine and fun as they can be,
slapstick and B-movie action horror are just the start of what *could*
be done with this new medium. If I was a film student with some
technical savvie, I'd be beginning to look at using this stuff as an
alternative to or prototype for an expensive real-world production.

[*]Machinema... yes, sorry, it's a bit of a contrived term... but what
in general *are* we going to call these pieces of cinema that are made
using 3D engines? Not only is "Quake movie" an ugly and confusing
term, it's also fast going to become outdated as other technologies
become relevant. (c: Any ideas?

I chose "machinema" to rhyme with cinema. I used "machine" as a base only because I didn't find an easy pun with "engine" in it. I spelled it with the 'e' because it made it look a bit like cinema. I don't believe it to be the best neologism in the world, and I apologize for that - I didn't know the term was going to get widespread.

The word began to be used fairly regularly on the mailing list and gradually in other related fora, but that was about all. So far as I know, it's first "public" appearance is in a review I wrote for GameSpy Industry's Quake portal site PlanetQuake.com, first published on 02 Apr 1999. (Slightly broken copy of now dead page courtesy of the Internet Archive)

Hugh Hancock of Strange Company wrote up some "machinima" techniques on 10 Jun 1999. We were already in e-mail contact so we discussed the new spelling.

Anthony:

<pedantry>
I'm very happy that you've adopted my silly whimiscal term for cinema
generated by desktop machines, but is there any reason that you have
changed the spelling? The 'e's in both "cinema" and "machine" seem to
suggest the natural spelling for the mangled combination is
"machinema." It's a pun rather than a derivation, so I don't see any
linguistic pressure to follow the path of e.g. "machinist."
</pedantry>

Hugh:

As a linguist I'd prefer to use the "ima", a more common suffix than "ema",
and go with the normal derivation, but to be perfectly honest the main
reason we've changed the spelling is I forgot how you originally spelled it
and "ima" looked more natural! Sorry...

Anthony:

I didn't like "machinima" at first because it seemed to lose the
"cinema" pun, at least at the lexical level. But it's still there
phonically, and now I realise there is a genuine derivation from
"anima" going on as well, so the new spelling actually maybe combines
something plausible and word-play. Anyway, in the interests of
uniformity I'll adopt your spelling.

Hugh tells the machinima etymology story from his side in a far more self-deprecating and entertaining style within the book Machinima for Dummies.

The term crossed the chasm for certain when hub-site machinima.com was launched at the start of 2000.

Blipvert

Sun, 19 Aug 2007 18:07:58 GMT

I'm wondering why there isn't more machinima on blip.tv. There is some, but this video publishing service seems like such a good fit with the needs of machers I would have expected more.

I became aware of blip through this recent interview with co-founder Mike Hudack by Jon Udell. (Good podcast - says some interesting things in general about web service businesses.)

The company seems to really get what a user-gen content aggregator and publisher needs to do to succeed by pleasing their users. You can read their own overview, but items from the interview that impressed me included

a strong distaste for service lock-in (for example, they will automatically mirror your upload to the Internet Archive if you want),
being very open to working with other distributors and publishers (for example, building a system to share tags and download stats),
staying in the background to let creators control branding (for example, they don't embed their logo),
sensible advertising options (you can use their advertising and share the profits, or do your own, or none),
providing a strong free service with professional upgrade.

The whole thing sounds like the right business model: provide a service which offers sustainable long-term value that supports creators.

The disclaimer: I've not used the service. (Anyone who has care to comment?)

Regression Therapy - Contentful Testing

Thu, 19 Jul 2007 00:46:32 GMT

Regression Therapy - Contentful Testing

This article is a fleshing out of a ten-slides-in-ten-minutes talk that I originally gave in July 2007 at work.

Since then I talked on the same subject at the Scotland on Rails 2008 conference. Slides and so forth are available.

1. Introduction

Regression testing is usually seen as the poorer cousin of "proper" domain-abstracted assertion-based testing. Often rightly so!

However, with the right support in place, I have found that this form of testing can work very well in certain contexts.

This article addresses one such context: testing the view content generated by a web app. I discuss the background, then present a concrete example in the form of a plug-in for Rails.

2. Three reasons to test

For the purpose of this discussion, I identify three overlapping but distinguishable uses of tests in software development.

To preserve existing good behaviour

Tests can be used to maintain known good behaviour, preventing accidental bad changes (also known as "regressions".) The tests provide a safety net during refactorings, and enfore a Hippocratic "first, do no harm" oath during other work.

....................................!..!...........................!...........

The test stays green whilst all is well. A red bar tells us that we've messed up and broken something.

To drive intended changes

Tests can be used to change (often, to add) behaviour, providing context to an intended good change. The archetypal example is classic TDD, creating tests first in order to define the problem, and to guide the design and implementation of a solution.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

New tests flash red and are put back to green by introducing production code.

To understand emergent change

A less often discussed use of testing is to explore and validate emergent behaviour.

In a complex system it can be hard to prejudge or predict every knock-on effect of a deliberate change to one part of that system. Tests can provide a context for understanding and controlling such changes and consequences.

.....!!??????...................!!??????............!!??????..............

In this way of working, changes cause flashes of red that are observed and then often approved, updating expectations to recolor things back to green again.

3. Three ways to test

Next I'll categorize three methods of testing, paying particular attention to their use in the context of testing view content.

Live testing

In a scandalously unrepeatable manner that sacrifices reliable coverage, we can test by hand and verify by eye.

Drive the web app through the browser and see if looks right. We all do this. It shouldn't be sniffed at; it is genuinely useful because it is so cheap. However, once the system gets sufficiently complex, it doesn't hold water. I won't consider manual testing further in this article.

To automate testing, we have to be able to provoke the generation of view content that we can then test. Assuming a good web app framework that allows us to do this (no small thing, but not the topic of interest today), there are then two means of testing the content.

Assertion testing

We can explcitly check individual desired properties of the the resulting content.

Regression testing

Or, we can compare the content in its entirety against a previously validated known good copy.

I'll now discuss in greater detail the relative merits of testing view content through assertion and regression.

4. `assert_equals(expected, test_method)`

Assertion tests are prevalent in contemporary developer testing, with good reason. They tend to be very effective, given sufficient investment – we need to express the properties that care about succinctly, and at an appropriate level of abstraction within the domain.

The Rails web framework has evolved a best-of-breed solution for assertion testing view content in the form of assert_select. It uses CSS selectors to locate content, and neatly expresses the conditions it should satisfy. Here's a simple example:

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    assert_select "div#cart tr.total td:last-of-type", "$39.95"
  end
end

This is pretty nice. But it doesn't escape some downsides of assertion testing which happen to be particularly troublesome for view content.

Coverage is selective

We can only capture the behaviors that we think to test for. The importance of many properties of view content are not obvious until we've accidentally changed them.

Exploration is expensive

Assertion tests are generally great for catching regressions and for driving changes. But they are poor for my third type of test use, that of exploring change. The developer has to run around rebalancing the books, fixing up test expectations to agree with the unexpected consequences of a change.

Explicitness is verbose

Many aspects of the view are worth caring about. It takes too much code to express them all explicitly; translation, even into a well-designed and succinct language, still takes time. The most concise summary of all the properties of a piece of content is the content itself…

5. Regression (past life)

…which leads us to our next alternative.

Regression tests work by checking current content against expected content. We test by comparing the entirety of what we generate against some previously approved copy. The expected content might be quoted verbatim in the test; more often it is stored in a file.

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    assert_equals File.new('expected.html').read, @response
  end
end

This is clearly very cheap to express, and capturing existent good behaviour needs no translation: just a copy and paste of the current results.

But like assertion testing, regression testing view content comes with some downsides.

Tests don't come first

Again this form of testing is good for two of the three uses of testing that I identified. It is ideal for exploring changes, and unsurprisingly it catches regressions. But, it is very awkward to drive new behavior.

However, this problem is not serious in the context of view content. Although it would not be plausible to test-drive the generation of content through regression testing, it is not clear that one would ever want to. View content tends to come into existence in a concrete form, designed or composed in toto rather than being built up one abstract essence at a time. Even for those who usually love to test first, test-last of view content does not seem so detestable.

The main problem with regression testing is that it is very expensive if we act naively in response to diffs. There are two common pain points.

Tests are brittle and fragile

Comparing against a complete text copy means that we see every change, even ones we don't actually care about. Every difference needs explicit attention.

Tests are usually noisy

Because there is no abstraction, one change may break many tests. Similar differences need attention in multiple places.

These workflow costs multiply as the system under regression test grows. Investing in the abstraction of assertion tests is usually preferrable. Even in the domain of view content, regression tests will prove too costly if we do not work to make them less so.

6. Regression Contentful testing

So: let's attempt to address the problems, and turn regression testing into contentful testing.

Lessen the noise

Information can be defined as any difference that makes a difference. The diffs that don't are the noise in regression tests that we want to reduce.

Ignore insignificant change

Some textual changes in HTML source (for example, certain whitespace in content; and formatting, quoting, capitalization and attribute order within the mark-up itself) won't affect how the content is displayed.

These can be ignored by allowing a little abstraction back in. We can still use standard text-based diff tools, but to compare unique text representations of DOM-normalized content.

DRY up which content is tested

Rather than see the same diff to the same repeated content over and over, we can focus regression tests on those parts of content that vary across our testcases.

Smooth the workflow

In good automation fashion, we will make the common tasks in the workflow as cheap and as easy as we can.

It has to be simple

to create a new regression test,
to inspect diffs from the expected content when changes occur, and
to accept the changes once they have been checked over and seen to be OK.

And, this needs to work well in batch. We should be able to detect, review and accept changes in many tests at a time when need be.

The Contentful Rails plug-in

I put together a plug-in that tries to do these things within Rails.

% script/plugin install svn://rubyforge.org/var/svn/contentful

The plug-in was evolved in, then extracted from, a small spare-time web application. I've since re-used it successfully in other Rails projects. If you write Rails applications, you can read more and download the plug-in at the Contentful website.

For this discussion, Contentful is simply a single concrete example of a general approach I believe should translate across language boundaries, or from my Rake automation commands into your favorite IDE.

7. Capturing content to create tests

With Contentful plugged-in, we can check the content generated in a test by saying assert contentful.

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    assert_contentful
  end
end

Then when we run the test for the first time, the test passes - and it generates the expected content from the current content, as a side effect. In accord with Rails' culture of convention over configuration, we locate the expected content in a standard place, derived from the name of the test.

% rake test
Started
...................................!...............
Finished in 3.487022 seconds.

  1) Contentful Notification:
test_total(CartExampleTest):
Generated /depot/test/contentful/cart_example/total/expected.html

To avoid duplication, we can focus on a particular subset of the content using a CSS selector. This allows us to ignore components that are common across all our views, such as navigation sidebars and the like.

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    select_contentful "div#cart"
  end
end

% rake test
Started
...................................!...............
Finished in 3.494442 seconds.

  1) Contentful Notification:
test_total(CartExampleTest):
Generated /depot/test/contentful/cart_example/total/div#cart.expected.html

8. Detecting changes and inspecting diffs

Now our tests are up, we should set them running. (Running everything is the norm, but Contentful will build and execute a suite containing only its own tests if asked. I'll show that here.)

Here's what happens when we've broken something:

% rake test:content
Testing expectations in test/contentful
Started
..F.....
Finished in 0.808864 seconds.

  1) Failure:
test_total(CartExampleTest)
diff test/contentful/cart_example/total/*.to_diff
to see the content change

The failing test will have written a temporary changed.html file next to expected.html. Additionally it generates a second pair of files (expected.to_diff and changed.to_diff) which are the same HTML DOMs, but with some added line-breaks and other normalization to make them usefully comparable using a standard text diff.

(Whenever a test passes, the stale three files get cleaned away.)

As well as using the suggested command-line to examine an individual failure, we can inspect all current diffs by running a single command.

% rake test:diff
Finding changes in test/contentful
! Diff in cart_example/total/changed.html
91c91
< <td id="total">
< Total
---
> <td id="total" class="grand">
> Grand Total
! Diff in another_example/something_else/changed.html
[etc...]

(Additionally, if we want to focus on a subset of all the current changes, we can do so by running within a subdirectory of test/contentful.)

9. Accepting change

If a set of changes are good, then often we can tell this by casting an eye over the diffs and observing a plausible pattern.

(Sometimes one can grep the diffs for extra certainty. For bonus points, script anything you find yourself doing by hand three times!)

If all looks good, we want to accept a bunch of changes.

% rake test:accept
Finding changes in test/contentful/cart_example
! Accepting cart_example/total/changed.html
! Accepting another_example/something_else/changed.html

That was a bit better than opening up an editor and copying and pasting the content, right?

(Again we can refine our acceptance to a subset of changes by running within a subdirectory.)

Also, since expected content is generated when absent, we can alternatively use the file system to our advantage, and update expectations by removing particular subdirectories and then re-running tests.

% rm test/contentful/further/specific/*
[etc...]
Generated /depot/test/contentful/further/specific/expected.html

10. Making a vice out of a virtue

Personally, I really like my content tests to be a permanent part of the test suite. Once the workflow is smooth, I find them well worth their maintenence cost. I like the total coverage; they allow serendipidous discovery of changes I didn't consider.

However, even if you won't buy the entire bridge that I'm trying to sell you, perhaps I can interest you in a strut: use the virtues of contentful testing as a vice. (Sorry for the UK spelling. Translate it into the US "vise" if you like, but you'll spoil my pun.)

Suppose we want to perform a pervasive refactoring. For example,

to change the templating framework we're using, or
to factor out a set of view helpers and partials.

Big changes are risky. It would be nice to temporarily pin everything about current content tightly down in place whilst we work. When using Contentful, then we can add a bonus content assertion to every existing functional and integration test by setting a single Rails config flag in environment.rb:

CONTENTFUL_AUTO = true

Running tests once generates expectations and locks down all the current content. Then we can perform our refactoring under their full protection. When done, we unset the config flag and delete the temporary expectations before checking in the safely refactored code.

% wget http://contentful.rubyforge.org/

Thanks for reading! Comments are most welcome.

The views expressed here...

Mon, 04 Jun 2007 18:14:04 GMT

Something I forgot to do on joining my current employer...

The views I express here on my blog do not necessarily reflect the views of Amazon.com.

A Feedsum Engine

Fri, 25 May 2007 21:48:37 GMT

A couple of weeks ago I noticed that I was avoiding subscribing to some informative feeds because they were a bit too informative. I don't want to see as many as a dozen separate news items across the course of a single day, because it makes too much noise in the feed aggregator and gets in the way of rarer, more valuable entries. (In my case, the aggregator takes the form of my LJ Friends page.)

I realized I wanted the equivalent of a digest e-mail for feeds - a daily summary or similar. This sounded like a job for a web service, so after failing to find any existing solution, I registered feedsum.com and coded up a little Rails app that sucks in a source feed, collates items into daily batches, and generates a derived feed in which each item summarizes all the source items for one day.

At first this was simply a stateless service, but I found that source feeds often dried up very quickly - by the end of the day, the earliest items could already have been pushed out. So now I stash source items in a database whenever the source feed is pulled. I also found that syndicators such as LiveJournal were not sufficiently patient to wait for my app to spin up in clunky development mode, starting up a fresh Ruby before pulling the source and generating a summary - they often timed out before the service was done. So finally I have had to learn how to deploy a proper production Rails app using mongrel proxies and all that. There were some fiddly one-time set-up details, but once up and running Capistrano is pure deploy joy.

Currently I'm using the service to source three LJ syndications: slashdot_daily, machinifeedaily and arstechnicdaily. The service can probably handle a further light load yet so others (LJ users or any other feed consumer) are welcome to give it a beta test. It's easy to use: to get a daily summary of some feed

http://example.com/path/index.xml

simply request

http://feedsum.com/daily/example.com/path/index.xml

(You can do other neat stuff like http://feedsum.com/every/8/hours/... and so forth but the daily summaries tend to be the useful ones.)

Do try it out and please tell me if e.g. it completely chokes on any source feeds. (And to avoid a rush of dupes, could anyone who uses it to register further LJ syndication accounts add a link in a comment below.)

Happy Fun Content Surprise

Sat, 12 May 2007 10:48:36 GMT

HBO CTO Bob Zitter claims DRM only has detractors because of the branding - he suggests "Digital Consumer Enablement" instead. Ars technica say: why stop there?

Hacking it for the kids

Sat, 28 Apr 2007 22:29:15 GMT

I remember reading The Little Coder's Predicament some years back. I agreed strongly with the essay: many of my generation were encouraged to code because our home computers et al, bought for games, made it so easy and rewarding to play and experiment. Machines are now much more powerful, and the Net makes the potential rewards far greater, but today's generation lacked such an immediately accessible path into programming.

The author of the article has since become something of a Ruby guru with a track record of turning dreams into code. Recently he's addressed his Predicament and with a little help from his friends brought Hackety Hack into being. It's in an early phase (Windows only, and not quite bug-free) but I think this will be a great playground.

When I was thirteen, my ZX Spectrum let me make beeps, plot pixels and clash colors. Its contemporary equivalent HH lets you work directly with the things that matter today: web pages, feeds, audio, images, video. It's a bit like Pipes without the annoying drag 'n' drop (or Yahoo ID.) The underpinnings are rather too indistinguishable from magic to trust with anything you'd get paid for, but what a great way this would be to write your first real-time networking web app:

chat = Hacker('anthony').channel('test')
Web.popup do
  title('Chat Client')
  box = scrollbox
  editline('Say')
  buttons('Send') { |popup| chat.say(popup['Say']) }
  every(1) { chat.hear.each { |message| add_to_scrollbox message } }
end

Hackety's hackers can share programs and data across the Net very easily - much better than getting the computer to squeal into a tape that you take round to your mate's house! I really hope that this thing will take off and get some people programming.

New angles

Mon, 16 Apr 2007 06:57:30 GMT

Fotowoosh is a service to extract 3D models from 2D images, seeking beta users.

This kind of math is not new, but I've not seen it productized and sold before - seems to be a pretty exciting technology development for machinima-makers. Model asset generation is probably the biggest remaining expense in our medium, expecially films that want to be set outside the virtual world the engine they were made in comes with. (Seen at TechCrunch - the article there has a good pictorial summary)

How to disenchant the pedantic voter

Sun, 15 Apr 2007 21:04:14 GMT

There are local elections here soon. Here's some really dumb propaganda from a party I'm feeling somewhat embarassed to admit I usually vote for.

Three fish in one barrel:

That's sure some excitingly non-linear y-axis you've got going on there.
These are the results from the previous election. Alas, they bear little resemblance to current polling.
"It could not be closer." Yes, of course, because there are no numbers less than five.

Way to push away the constituents with half a brain, local Lib Dems.

Tumbled

Wed, 11 Apr 2007 19:38:24 GMT

I continue to be rather too precious about posting to my main (LiveJournal) blog.

(That's anthonybailey on LJ.)
(Others should bookmark anthonybailey.net/blog, since that should survive any change of blog provider.))

Extravagant essays get drafted, edited and binned before publication. The posts that make it through my filters are often thoughtful, but long and infrequent. And they are quite often ~~dull~~ technical in content.

Most people post all kinds of junk on their blogs. They write quickly and more often. They do quick links and jokes and half-baked ideas and ephemera. They let the reader take care of extracting the signal from the noise.

When its the right person, I like reading these blogs. I don't mind my friends being noisy, and I find the diamonds in the rough.

So I'm trying one of those kind of blogs too...

If you want a noisier, uncut Anthony, you may want to read this instead:
anthonybailey.net/tumblelog

It will get copies of everything I post to the existing blog, so you shouldn't need to read both. Here's the feed. LJ people can friend antbailey_uncut.

(If the terminology confuses: a tumblelog is a deliberately less formal blog. It encourages posting all kinds of junk, fast! Although it doesn't stop said junk from being ~~dull~~ technical in content, or physically hurt me if I don't post; so don't expect too much.)

Quick Link: GeekDad

Wed, 28 Mar 2007 19:27:48 GMT

Sensibly, I have not myself offsprung. But, various of those who read this might be interested in Wired's new GeekDad blog.

Slim Builder - an Untangled Template View

Thu, 08 Feb 2007 01:19:09 GMT

I'd like to draw attention to a pleasant way to implement the view in MVC web apps that is emerging in suitably reflective and dynamic frameworks. It doesn't seem to have a name yet - I'd suggest "slim builder" and/or "untangled template view".

The technique combines the strengths of two traditionally distinct approaches - builders, and server page templates - and thereby solves some of their usual problems. I'll illustrate it through an example drawn from Rails, although you don't need to know the details of this framework or the underlying Ruby language to get the point.

If you do know Rails and its canon then the example will be very familiar: it's the bookstore index page from the "depot" tutorial in Agile Web Development with Rails, as of the end of section B.

(As context for the rest of you, here's how the page renders once its style is wired up. It's simple, but will suffice for our purposes. The approaches described all extend smoothly to more complicated layouts, Ajax updates, and so forth.)

Templates - mark-up and code entangled in a server page

The traditional template view works by embedding executable code fragments into concrete mark-up. The default Rails view system uses rhtml templates processed by ERb.

Here's how we'd implement the store page view. In the tutorial - as in real Rails apps - the page gets split into different modular parts (layouts, partials, etc.) For the sake of this example I'll inline everything into one template:

<?xhtml version="1.0" encoding="UTF-8"?>
<html>
<head>
  <title>Pragprog Books Online Store</title>
  <%= stylesheet_link_tag "depot", :media => "all" %>
</head>
<body id="store">
  <div id="banner">
    <img src="/images/logo.png" alt="Logo"/>
    <%= @page_title || "Pragmatic Bookshelf" %>
  </div>
  <div id="columns">
    <div id="side">
      <a href="http://www....">Home</a><br />
      <a href="http://www..../faq">Questions</a><br />
      <a href="http://www..../news">News</a><br />
      <a href="http://www..../contact">Contact</a><br />
    </div>
    <div id="main">
      <h1>Your Pragmatic Store</h1>
      <% for product in @products -%>
        <div class="entry">
          <img src="<%= product.image_url %>" alt="Cover" />
          <h3><%= h(product.title) %></h3>
          <%= product.description %>
          <span class="price"><%= number_to_currency(product.price) %></span>
          <%= button_to "Add to Cart", :action => :add_to_cart, :id => product %>
        </div>
      <% end %>
    </div>
  </div>
</body>
</html>

To my eyes, this kind of JSP-inspired template is rather ugly - it is the dirtiest plate in the default Rails stack. Angle-bracketed web mark-up is not pretty to start with, and mixing code into it under the protection of even more punctuation does not help matters, forming uglies such as <%, <%=, and -%>. Further, the resulting template is neither fish nor flesh; it isn't validating mark-up, nor is it directly compilable code. Two worlds have been messily ent-angled.

To be fair, if you need to directly mix the two, then having the mark-up around the code is probably a better choice than the converse. Templates rightly ousted sequences of print statements that cobbled together strings of explicit mark-up. Traditional entangled server page templates have a sufficient similarity to straight HTML that tag-aware tools will do a decent job of pretty-printing and coloring them. Similarly tag-aware but non-programming web folk can work with the template directly, especially when helper objects have been used to minimize the size of the scary code fragments and tuck the details safely away.

Still, it's messy. Please do better.

Builders - code that casts a mark-up shadow

When I first ran across the Template View pattern in Martin Fowler's Patterns of Enterprise Application Architecture I imagined it to cover exactly the case above and no more; template views were server pages.

But a refinement of the "cobbling" procedure described above brings us to the builder, which inverts the server page relationship between the tags and the code. By encapsulating all the opening and closing of tags within helper methods, we can hide the actual mark-up whilst persisting its structure in that of the program that knows how to build it, so that the code itself looks something like a template. Fowler's account of the view template shows he considers builders to be a valid variant; the veteran builder from those days was Perl's CGI.pm, which is referenced (albeit briefly) in the pattern.

The CGI.pm builder brews tag soup by recursively nesting expressions each of which evaluate to strings. A set of helper methods that are named after the tags they build take a hash of options followed by further string-returning expressions to concatenate together and enclose. Glossing over a few Perl/Rails gaps, here's how we can get CGI.pm to build up and print an expression that evaluates to the store page.

#!/usr/local/bin/perl
use CGI ':standard';

print(header,
  start_html("Pragprog Books Online Store"),
             # I couldn't figure out how to supply the stylesheet
  div({-id=>'banner'},
    img({-src=>"http://localhost:3000/images/logo.png", -alt=>"Logo"}),
    $page_title || "Pragmatic Bookshelf"
  ),
  div({-id=>'columns'},
    div({-id=>'side'},
      a({-href=>"http://www...."}, "Home"), br,
      a({-href=>"http://www..../faq"}, "Questions"), br,
      a({-href=>"http://www..../news"}, "News"), br,
      a({-href=>"http://www..../contact"}, "Contact"), br
    ),
    div({-id=>'main'},
      h1("Your Pragmatic Store"),
      join '', map {
        $product = $_;
        div({-id=>entry},
          img({-src=>$product->{image_url}, -alt=>"Cover"},
            h3($product->{title}),
            $product->{description},
            span({-id=>'price'}, number_to_currency($product->{price})),
            button({-action=>"add_to_cart/$product->{id}"}, "Add to Cart")
          )
        )
      } (@products)
    )
  ),
  end_html
);

This builder is very Perlish. It achieves high-level brevity at the cost of some low-level heavy punctuation, as well as dragging its helpers into the global scope. We got to use some semi-functional programming cuteness to iterate over the products within the innards of a huge expression. This is charming if, like me, you have an old affection for the language; but even if you find Perl distasteful I hope you'll agree CGI.pm seems to have hit upon a good idea.

The Ruby version most directly descended from CGI.pm is XmlBuilder, and it's the default builder within Rails. One change in approach is that rather than building a page as one big expression that nests method calls and string concatenation, instead we let the building occur as a side effect and hence write a regular sequential program with guards and iterators.

Here's the rxml template to get Rails to render the store page:

xml.instruct! :xhtml, :version=>"1.0", :encoding=>"UTF-8"
xml.html do
xml.head do
  xml.title("Pragprog Books Online Store")
  xml<<(stylesheet_link_tag "depot", :media => "all")
end
xml.body :id=>"store" do
  xml.div :id=>"banner" do
    xml.img :src=>"/images/logo.png", :alt=>"Logo"
    xml.text!(@page_title || "Pragmatic Bookshelf")
  end
  xml.div :id=> "columns" do
    xml.div :id=>"side" do
      xml.a("Home", :href=>"http://www...."); xml.br
      xml.a("Questions", :href=>"http://www..../faq"); xml.br
      xml.a("News", :href=>"http://www..../news"); xml.br
      xml.a("Contact", :href=>"http://www..../contact"); xml.br
    end
    xml.div :id=>"main" do
      xml.h1("Your Pragmatic Store")
      @products.each do |product|
        xml.div :class=>"entry" do
          xml.img :src=>product.image_url, :alt=>"Cover"
          xml.h3(product.title)
          xml<<(product.description)
          xml.span(number_to_currency(product.price), :class=>"price")
          xml<<(button_to "Add to Cart", :action => :add_to_cart, :id => product)
        end
      end
    end
  end
end
end

This is looking OK, but it isn't nice enough to move me from rhtml - it's still a bit hard to read, just from a different angle. We haven't yet reached a DSL for writing pages - web people probably couldn't work with this directly. My DRY eyes are offended by some remaining duplication. What with introducing the xml builder object to scope its helpers, this is actually more verbose as the server page. And, there's too much punctuation (although I confess I've not suppressed everything I could - I'm leaving some tricks for our big finish...)

Still too massive. Let there be light.

Slim builder untangles the template

What happens if we throw further Ruby magic at the problem so as to slim down the builder?

Markaby happens:

xhtml_strict {
head {
  title "Pragprog Books Online Store" 
  stylesheet_link_tag "depot", :media => "all"
}
body.store! {
  div.banner! {
    img :src=>"/images/logo.png", :alt=>"Logo"
    text @page_title || "Pragmatic Bookshelf"
  }
  div.columns! {
    div.side! {
      a "Home", :href=>"http://www...."; br
      a "Questions", :href=>"http://www..../faq"; br
      a "News", :href=>"http://www..../news"; br
      a "Contact", :href=>"http://www..../contact"; br
    }
    div.main! {
      h1 "Your Pragmatic Store"
      @products.each do |product|
        div.entry {
          img :src=>product.image_url, :alt=>"Cover"
          h3 product.title
          text product.description
          span.price number_to_currency(product.price)
          button_to "Add to Cart", :action => :add_to_cart, :id => product
        }
      end
    }
  }
}
}

Markaby is "mark-up as Ruby." It's real pretty.

Punctuation noise is minimal. As a matter of style we've used curlies rather than "do ... end" for the tag blocks, and managed to suppress almost every bracket. This is about as clean as the structure representation could be whilst remaining executable code. (HAML gets briefer still, including Pythonesque semantic indentation, but it only manages this through going back to the being a template that needs pre-processing - using a better rhtml like this is an interesting alternative, though.)

There's much handling of method_missing so that method invocations can be used to build the basic CSS classes and ids without needing hashes. (Like many builders Markaby is also using further reflection to more cheaply create the tag helper methods that these modify.)

The builder is made implicit again by using instance_eval to carry the object's context through to its inner blocks - this avoids scope pollution without incurring a character tax. (XmlBuilder used to turn this trick too, but because it builds general XML the chance of confusion was considered too high and the technique was retired. As an HTML builder Markaby suffers less, and we also get to enforce a restricted builder context so as to control which models can be referenced by the view.)

The code flow has been designed so that natural idioms tend to do the right thing; appropriate encodings are implicitly applied to appropriate string arguments. The inner details can be mysterious, but why would you expect helpful magic to be otherwise?

Talking of doing the right thing by default... Markaby was the only one of the builders I tried that generated completely (i.e. W3C) validating mark-up out the box without any tuning. Further, it also complains if you attempt to build such things as elements with duplicate ids, or tags that aren't allowed to self-close - this validation is a nice benefit of builders. Once armed with a decent Ruby-aware editor working in Markaby can provide plenty of warning of mistakes in either code or tags - the best of both worlds, keeping programmer and web designer happy.

Conclusion

I think this is the right approach for view building. To repeat my proposed terminologies:

It's an untangled template view - all the flexibility of your regular server page view template, but less of the mess. We've disent~~angle~~d the template by collapsing mark-up into the code and outlawed all distressing pointy brackets.
And it's a slim builder - all the power of your regular builder but less of the mass. The result is a DSL for authoring web pages.

What are the slimmest builders in other languages and frameworks?

Burning time in a virtual world

Sat, 27 Jan 2007 16:31:16 GMT

I had three whole weeks of holiday over Christmas/New Year, so I thought I'd indulge myself enough time to play through some computer game. I was intending to catch up with a random well-regarded FPS such as Halo, but I couldn't seem to get into it - Half-Life 2 has made me very demanding, and I'll wait for Episode 2.

Instead I ended up downloading a game through Steam. (This delivery system suits my lazy shop-averse needs, and seems to be working as intended: I downloaded a trailer, then the free demo, and finally bought the complete game.)

Darwinia by Introversion Software

Darwinia is a very fun and involving real-time strategy / god-game combination with a unique vector/retro graphic aesthetic.

The evolving back story concerns a virus-infected AI-inhabited virtual world run on a cluster of unsellable 1980's computers (think Sinclair QLs.) You get to wipe out the virus through various means (usually involving lasers and explosives), to gradually resurrect the world's citizens, and to direct them to work and fight to save their world.

The game is by a small team who clearly cared a lot about it. Apart from the award-winning visual design, there's a good soundtrack and a plethora of start-up sequences. And, on completing the game you unlock its editor...

See, I started writing this review more than a week ago, and went browsing around for background material. (There are lots of extras on the game's official site - the tortuous development history is particularly fascinating.) I found a large list of community mods. I'm still playing through the first I downloaded - Insurrection - and am enjoying it as much as the original game. Involving to the point of life-consuming, then.

Anthony Bailey's blog

Review: Anathem

Review: The Baroque Cycle

Contacts: Let's keep all this between ourselves

Review: Implementation Patterns

Minimal OpenID glue for Rails authentication

Review: Little Brother

Review: Clean Code

Review: Bulletproof Web Design

What does BDD offer TDD die-hards?

Introduction

Context

Philosophy

Territorial dispute

Frameworks

Talking to business

Outside-in

Code has needs too!

Given/Then, When

Conclusion

Credits where credits due

'Spell is out

Digital Privacy Management is inEFFable

Origins of the word "machinima"

Blipvert

Regression Therapy - Contentful Testing

Regression Therapy - Contentful Testing

1. Introduction

2. Three reasons to test

To preserve existing good behaviour

To drive intended changes

To understand emergent change

3. Three ways to test

Live testing

Assertion testing

Regression testing

4. assert_equals(expected, test_method)

Coverage is selective

Exploration is expensive

Explicitness is verbose

5. Regression (past life)

Tests don't come first

Tests are brittle and fragile

Tests are usually noisy

6. Regression Contentful testing

Lessen the noise

Smooth the workflow

The Contentful Rails plug-in

7. Capturing content to create tests

8. Detecting changes and inspecting diffs

9. Accepting change

10. Making a vice out of a virtue

The views expressed here...

A Feedsum Engine

Happy Fun Content Surprise

Hacking it for the kids

New angles

How to disenchant the pedantic voter

Tumbled

Quick Link: GeekDad

Slim Builder - an Untangled Template View

Templates - mark-up and code entangled in a server page

Builders - code that casts a mark-up shadow

Slim builder untangles the template

Conclusion

Burning time in a virtual world

4. `assert_equals(expected, test_method)`