Anthony Bailey's blog

Ask Alexa Dot "Try Amazon Music Unlimited", get "Sorry, I'm having trouble" response? Workaround.

Fri, 06 Jan 2017 20:51:56 GMT

(Context/disclaimer: I do work for Amazon, but not with the Alexa folk: this is me reporting an issue and workaround as a customer rather than as an employee.)

I'm a UK Amazon customer who bought an Echo Dot a few months ago. The deal of £3.99 per month for Amazon Music Unlimited on just that device is a plausible fit for my needs, and a friend had asked me to help sort out some themed music for a New Year's Eve party. I knew lending the Dot to another household was plausible because I'd just taken it on a Christmas visit elsewhere. On that occasion I'd even got my host to log in with their own Amazon Prime account to access their music library.

So on the afternoon of 31st December I logged said previous host out of the Alexa app, signed in myself, then asked Alexa to "try Amazon Music Unlimited". (Henceforth I'll just write that I asked to try AMU.) It reported my free trial had started. A few quick test requests worked really well.

So I took the Dot to the party, told it about Wifi and bluetooth speakers. No luck accessing the 40 million songs there though, and asking again to try AMU got me "Sorry, I'm having trouble." Ah well. Odd spot for the Dot I guessed.

On return home, though, same situation. No access to the music, no evidence of a subscription having started. Asking to try AMU, same report. In the history, I see "SystemFaultMusic.MembershipIntentM..." in the response too. Lastly: not even a record of my brief interaction where I successfully tried AMU and played some songs. Hmmm.

I'm on holiday from work, but I'm still a customer, so I contacted customer services. The Alexa specialist support told me that the subscription process on the device uses Voice Purchasing, and that this simply doesn't work for non-Prime members.

(It remains enabled and selected in my Alexa app, though. And of course I really did successfully subscribe when sure I was signed in as my non-Prime account. As you may have guessed the free trial had been started instead on the previously logged Prime account - my host wisely cancelled it when they got an unexpected welcome email. I will be adding correspondence to a trouble ticket when I return to work I think. But let's just get to the workaround.)

This is the workaround customer service advised: non-Prime amazon.co.uk customers who want to try Amazon Music Unlimited at £3.99 on their Alexa device should instead start a trial of the full service from the website. Once the trial expires you can ask Alexa to downgrade to the £3.99 level.

This may hold for non-UK Amazon too: I don't know. And a proper fix (rather than weird error and support workaround) is in progress, of course.

GoogleCL authentication with accounts that use non-Google email addresses

Sat, 19 Feb 2011 18:09:18 GMT

GoogleCL is a command line tool that uses the Google Data APIs, so that you can interact with your personal wisp of the Google cloud from your shell. I approve: many thanks to the Google devs who run this project in their spare time.

However, there's an authentication issue for Google accounts registered under non-Google email addresses. This blog post describes a workaround that solved the problem for my anthonybailey.net account - but I'll use example.com as the domain name throughout to avoid personalizing the report. (I was clued in to this workaround through a forum comment by someone I can only credit as "sjoub..."..)

So: you have a Google account under an email address in a domain that you own: mail@example.com. You have signed your example.com account up for Google Apps, and installed GoogleCL. When you run it, you'll have to grant it access permissions. This is what you do:

First, sign in to your mail@example.com Google account in your browser.
Run your GoogleCL command, specifying this account, and telling it to ignore any previous failed attempts to authenticate:
```
% google calendar list --user mail@example.com --force-auth
```
GoogleCL will say: Please log in and/or grant access via your browser at https://www.google.com/accounts/OAuthAuthorizeToken?oauth_token=10n9_r4nd0m_Ex4mp13_70k3n_5tr1n9&hd=example.com then hit enter.
(Your browser will probably open up that URL, follow some 302 redirects to a https://www.google.com/a/example.com/ServiceLogin2 page, and report "Sorry, you've reached a login page for a domain that isn't using Google Apps".)
Visit the URL that GoogleCL suggested, but with the "&hd=example.com" part at the end chopped off. This makes Google correctly use the default example.com hosted domain for the logged-in account. You'll get to a https://www.google.com/accounts/b/0/OAuthAuthorizeToken page where you "Grant Access" to the GoogleCL application.
Press enter back in the GoogleCL shell and see the successful results of your command.
Check that GoogleCl is permanently permitted (and has remembered you as the default user) by running again and getting the same results without any need to do special arguments or the browser / cut-and-paste dance:
```
% google calendar list
```

(You'll have to do this for each Google service you access: each one needs to give GoogleCL access permissions. For some services you have to copy a token back from the authentication page to the shell.)

"But", you exclaim, "wasn't this you being dumb? You shouldn't have given mail@example.com as the username - you just needed to give example.com, the domain that Google Apps knows about!"

Alas, no. This doesn't work:

% google calendar list --user example.com

You do indeed get to grant access without any cut-and-paste dance. But Google Apps ends up granting access to the primary name for the account, mail@example.com. And GoogleCL quite reasonably but pointlessly reports:

You specified account example.com but granted access for mail@example.com. Please log out of mail@example.com and grant access with example.com.

I may have missed a trick, but I didn't find any sequence of attempts that worked without the URL-hacking.

"Why didn't you just sign up with a Gmail address in the first place? The system broke because you did something complicated."

Maybe so. But although I love Google services, I like to use my own domain for my identity and my email address. I had the email address before Gmail came along, and I want to be able to carry on using it without having to ask everyone to update their address books, even if I switch away from the Google service.

Review: Homestuck

Tue, 01 Feb 2011 00:16:52 GMT

Homestuck

by Andrew "MSPaint Adventures" Hussie

This epic ongoing webcomic is my favorite new artistic work of the last year.

The core of the story concerns a bunch of children on interwoven hero's quests, exploring something like a video game. It has a few elements in common with Scott Pilgrim but is massively more ambitious and inventive, and the mythos is more fully realized.
The plot mechanics are highly intricate: in particular the time-lines of characters and objects form a spirograph-worthy pattern that makes Primer look like an idle doodle. One character accurately describes the thing as "4 B1G S3LF FULLF1LL1NG CLUST3RFUCK, A HUG3 ORG14ST1C MOB1US DOUBL3 R34CH4ROUND" - and yet it fits together as well as any time-travel story I've seen.
Creator Andrew Hussie tells his tale in a highly singular voice and using a boundary-stretching style. The medium is an unusual evolving experiment: all the MSPaint Adventures are presented in the form of an interactive adventure where readers suggest what the characters should do next. This has become more of a stylistic motif than a core principle as Hussie tells longer stories, and Homestuck ups the ante on this with some of the story's own characters giving instructions in the same manner, and watching each other through the same screens the readership does. The usual meta-fictional elements abound.
Typically several pages of the comic are published every day. Most contain a single GIF image, frequently animated. Often these are accompanied by a short descriptive text, or by character dialog which is always presented in the form of a chat log. Some pages feature Flash content: these range from simple transitions with a little atmospheric audio, through interactive stuff to explore, parodies of boss battles all the way to complete 8-bit style embedded games and full-on soundtracked fast-cut cinematic animations lasting several minutes for some pivotal sections of the story.
OK, powerful many-layered story-telling with cutting avant-garde edges. But the key quality that got me reading this comic, pulled me through the archives over Christmas and now has me impatient for updates every day is that the jokes are consistently great. The humor is quite universal, somewhat geeky, and occasionally nerdy (I confess it was the slapstick sequences based on data structures that hooked me forever.) And the character dialog sparkles: if you pulled the Scooby Gang fifteen years into the future and onto IRC then you might end up with these chats and their hilarious embrace and extend of contemporary kids Net idiom and culture.
Highly recommended, if you can cope.

Review: Red Mars, Green Mars, Blue Mars

Sun, 29 Aug 2010 20:44:32 GMT

The Mars Trilogy (Red Mars, Blue Mars, Green Mars) by Kim Stanley Robinson

I read The Mars Trilogy since it loiters near the top of most of the lists of great sci-fi novels I've encountered. This is mostly a "me too, +1" review.
The three books follow the fortunes of the "First Hundred" Mars colonists. There is plenty of hard structural and terraforming science, and much exploration of the social and political fall-out of founding a separated human world, but it shines as a story about fascinating people and their lengthened lives.
As I read, my simplifying pattern-matching machinery decided "this is Deadwood in space", and it is a strong enough analogy I think that, with enough love and skill, a TV adaptation could work - despite understandable concerns to the contrary.

Review: For The Win

Sat, 17 Apr 2010 16:49:22 GMT

For The Win by Cory Doctorow

Doctorow weaves a highly original story setting from three observations.

Currency and assets in virtual worlds have significant real world value and large economies within which some people work for a living.

The distributed and global nature of the Net is a good fit for mobilizing employees of contemporary corporations.

The nature of wealth in a speculative information-driven economy is open to disruption—to gaming, if you will.

Within this context he presents a fable for young people on the historic and future importance of worker unionization.
No, seriously.
Actually. it works really well: the adventure story and the message play very nicely together. I rate this as a better combination of techno-thriller and political writing than another recent read (Freedom^TM, Daniel Suarez' sequel to Daemon), along both axes: For The Win is more fun despite being better grounded and relevant, and more sophisticated despite being targeted at teenagers.
The ideas are the core but the central characters are more three-dimensional than average in this kind of contemporary fantasy/sci-fi. Even though they are larger than real-life, and sometimes unreasonably heroic / physically capable / aspirational symbols, I cared about them.
Little Brother has grown up good. Recommended.

The hopeful tale of an unspoiled Twilight

Thu, 01 Apr 2010 22:16:28 GMT

Viewers of television shows with long narrative arcs know the Net can be a double-edged sword. It can provide many pleasant words shared with other engaged fans. It can also deliver much pain in just a few unwanted words of spoiler. The key to avoiding the latter, I think, is community and convention.

For many years as a UK viewer, I found myself jealously observing the growth of lively Net discussion of the unfolding stories on US TV. The time difference between when countries aired of the shows meant I couldn't actively participate in those discussions, only read them many months after the fact. Meanwhile even a casual browsing of fan sites for the shows ran quite a risk of being spoiled.

More lately this has been less of a problem for me, since I can download episodes of the few shows I do watch week-by-week shortly after they air in the US. Instead I observe others experiencing an analogous problem at a smaller timescale as they attempt to combine a DVR-based viewing schedule with their eager tracking of the real-time web. "OMG I can't believe X just Y" tweets from those watching live make the trade-off between involved and unspoilt viewing as difficult and as dangerous as ever.

Discussion of the spoilage process itself is even more fraught with difficulty - there's precious little you can assume everyone knows or that no-one will mind hearing.

Given this context, I'm pleased to present a recent personal story of spoilage avoidance.

Although the TV show is over, the official Buffy canon continues in the form of a "Season Eight" series of monthly comics, involving a masked Big Bad known as Twilight. Lately there was a horrendous mess up when the uncensored version of a cover of an upcoming issue which literally unmasked Twilight was unintentionally included in teaser marketing materials, revealing the identity of this villain months ahead of schedule. A significant proportion of the keener fans were spoiled, and there were dozens of discussions and press stories on how this all came about.

Now, I follow Joss Whedon stuff relatively close up on the excellent Whedonesque hub. But this site has a gated and relatively caring community, and a strong convention of omitting spoilers from summaries and clearly marking when they may be present in linked articles, or the site's own discussion boards. I read Whedonesque for many months before the comic in question was finally published, and despite the huge amount of attention "Twilightgate" received I easily remained unspoiled until I reached the intended page. So I want to offer warm fuzzy thanks in the general direction of all those who helped that happen.

(I'll reserve judgement on the overall plot and twist in question until the season is done, mind.)

Review: Sweet Dreams (Pride, and Prejudice, and Zombies)

Sat, 26 Dec 2009 01:38:08 GMT

Sweet Dreams (Philosophical Obstacles to a Science of Consciousness) by Daniel Dennett

When I read the provocatively entitled Consciousness Explained fifteen years ago, I was very impressed by Dennett's "intentional stance" and advocacy of heterophenomenology - applying the standard scientific method to understanding the nature of the consciousness of some third-party, using the subject's own reports about their internal experience together with other evidence. The particular "multiple drafts" model of consciousness he proposed alongside was plausible but very sketchy.
Dennett returns to the topic in Sweet Dreams, and again does a great job clearing the junk from this philosophical space. By his account, one has to fight surprisingly hard for common sense in the study of consciousness - people seem threatened by any model that doesn't make them sufficiently special, and unusually willing to rely on their default intuitions, even if this means entertaining zombies and moving into qualia street.
So, most of the book is a careful and polished philosophical argument that seeks to remove the obstacles that stop us making progress with the "problem" of consciousness. It's Dennett's best presentation yet - but for me, what he builds on the claimed ground remains frustratingly fuzzy. He fits a few new metaphors onto his still plausible model, but I'm still left waiting for others to go and Do Science with the tools and justifications that he has prepared.

Testing with less manual calculation

Mon, 30 Nov 2009 04:52:48 GMT

After seeing some solutions to this Ruby coding kata on generating ASCII representations of LCD calculator numbers I thought writing the test cases could be more pleasant. Because the whole idea is to have the machine deal with the tedious task of generating the fiddly and sizeable output, I really don't like the straightforward approach that means I have to supply lots of example outputs myself, by hand.

I thought either of two alternative testing approaches might help: checking universal properties, or content regression.

Universal coverage

The first approach is that rather than give many specific examples of output, we find some generic conditions that all outputs should satisfy and then check these for many cases. For this to work well the inputs need to be simpler than the outputs, or easy to generate automatically.

A good example from a previous job involved 3D vector geometry. We had code for distances and angles, and for various transformations. An intrinsic feature of this domain is that the distance between any pair of points should be unchanged under translations, rotations and reflections, and multiplied by an appropriate factor under a scaling. Similarly the angle subtended by any three points should be unchanged under any of those four tranformations.

It was easy to check these identities for many combinations of input points and transformations, including both pseudorandom inputs and some specially chosen to provoke edge conditions e.g. zero and near-zero distances, and co-linear, perpendicular and anti-parallel vectors. Much more pleasant than manually calculating the answers for a smaller number of examples — and it gave better coverage too.

Lowest common denominators of LCD

You can find some pretty good constraints that every LCD output should satisfy; certain horizontals and verticals should contain only spaces and hyphens, others only spaces and pipes; it should have certain dimensions; and certain relationships should hold between the outputs for different sizes of the same input, and between outputs for single digits and their combinations.

Actually... once you have pinned down the basic form of each of the digits, it isn't difficult to write a complete specification of the content of each character position in the output expected for a given set of inputs. Now said spec is pretty easy to make executable: just loop over the positions and render the character that should be there. In fact it turns out to be most natural to express this procedurally without explicitly tracking the positions in the final output.

So thinking about this kind of testing informed my choice of implementation instead! An unusual variation on test-driven design.

Implementation rendered obvious

Here is the production code I ended up with.

class Lcd def initialize @cells = <<CELLS.split( '' ) - - - - - - - - | | | | | | | | | | | | | | - - - - - - - | | | | | | | | | | | | | - - - - - - - CELLS end def render( digits, size = 2 ) rows = [ 0, [1] * size, 2, [3] * size, 4 ].flatten rows.map { | row | render_row( row, digits, size ) }.join end def render_row( row, digits, size ) cols = [ 0, [1] * size, 2 ].flatten digits.split( '' ).map do | digit | cols.map { | col | @cells[ row * 40 + digit.to_i * 4 + col ] }.join end.join(' ') + "\n" end end

(I omitted the command-line option handling specified in the original kata — it's two uninteresting lines using the trollop gem.)

I really like the succinctness of this solution, and that the basic digit forms are directly readable in the code. It does have a bit of a primitive obsession, with the source rows and columns and the input digits all represented by plain integers, and the compositions performed by anonymous application of list operators such as join. But in this case I found the indirections and conversions that were introduced if I abstracted these away undermined the readable simplicity of the core ASCII renderer.

(I guess that nostalgia for the idioms of low-level rendering is probably why I kept the "magic numbers" expression that calculates the index to look up in @cells verbatim in the inner loop. Look, just be happy I didn't inline render_row too, OK?)

Contentful, again

Since universal properties drove the production code rather more directly than normal, although I did write some tests of that form, the main approach I used to avoid writing tedious example output myself was to instead capture what production code does, and regress against that once it looks good.

I've blogged and talked about this kind of content testing before, and implemented a Rails plug-in for view content testing. In this case distilled the approach to its unpolished essence.

def regress_render( digits, size ) regress( "#{size}-#{digits}.content", @lcd.render( digits, size ) ) end def regress( name, content ) expected_content = File.open( name + '.expected' ).read rescue $!.message if ( expected_content != content ) File.open( name, 'w' ).write( content ) assert_equal( expected_content, content ) elsif File.exists?( name ) File.delete( name ) end end

So when I first put e.g. regress_render( '88', 3 ) in a test, then my current content was written to 3-88.content and I was told it didn't match "No such file or directory - 3-88.content.expected". Once I'd written enough code that the output was rendered correctly, I captured it (e.g. mv 3-88.content 3-88.content.expected.) This let me build up a solid set of regression tests without ever having to construct their content by hand — I just had to observe results and move them to being expectations once I saw they were valid.

If you found either of my new variations on this old kata diverting, please do comment with your thoughts.

Review: Anathem

Wed, 29 Apr 2009 23:14:48 GMT

Anathem by Neal Stephenson

After the more historic Baroque Cycle, Stephenson returns to speculative sci-fi a la Diamond Age, playing with the parallel evolution of language, culture, and quantum narratives.
I enjoyed this a lot. It oddly reminded me of the Riverworld saga, but with philosophical ideas rather than characters from history making their cameos against a densely detailed unfolding backdrop and story.

Review: The Baroque Cycle

Sat, 21 Feb 2009 20:42:36 GMT

The Baroque Cycle by Neal Stephenson

Size matters, here. The Baroque Cycle is something of an epic, at around one million words long. (Workings: its three books end on page numbers 916 + 815 + 886 = 2617 pages * 45 lines/page * 10 words/line = 1.12Mwords, then round down some whitespace.)

The trilogy is a "behind the scenes" saga spanning fifty years of scientific and economic alchemy, featuring characters from the pages of history but centered on three of their peers whose tales escaped the official record.

In many ways this is the 17th/18th century version of the same author's more contemporary Cryptonomicon, with which it shares the above approach, plenty of adventures colored by cryptography and computation, and more. However, I found its sprawling scope resulted in less value per page: its three books contain only a little more gold in total than Cryptonomicon's one.

Contacts: Let's keep all this between ourselves

Mon, 15 Dec 2008 23:38:46 GMT

I want to describe how I like contact(s) information on the Net to work.

I say "contact(s)" because I have complementary ideas regarding both the contact details for a person, and the details of who their contacts are - you can say "friends" or "social graph" for the latter if you like. To oversimplify: I want the information sources to be distributed, and to delegate to individuals as authorities for their own details. I find that most current technologies don't share this ideology; however, I can still use them to act in tune with it.

An introductory note: I'm going to talk about people "owning" a web page. What I mean is that they have significant control over the main content of the page, which is published under as permanent an URL as possible. They definitely don't have to serve the page from their own machine, or from a host that they rent - a page on a social network or in an ISP's basic web hosting area is fine. The URL can be under someone else's domain, so long as they have confidence it will last a while. (IMHO aside: own the domain.)

My suggestion about contacts assumes some kind of a homepage/identity baseline: that people own some page that is about them - such as a blog, or a presence on a social network. Their homepage URL is then a decent identifier for their identity online. For bonus points, they can prove that they are the person who owns that page through OpenID, or Facebook Connect, or similar. All this is standard for many people online already, and it can easily enough be true for anyone else who cares to make it so.

My stance is that I want to own my contact information - because I believe I'm best placed to be the central authority it. I want to publish it as a resource located on or somewhere that is discoverable from my homepage, in a standard format. (Until the world stabilizes a bit, probably in multiple formats, all views of the same underlying data.) I'd like others to do the same. It's up to them exactly which details they publish. If they care to, they can even restrict access to some information, making it "friends only". (IMHO aside: most privacy is squeamish.)

I also want to own my contacts information - the list of other people that I know. For each of them I want to store and publish a homepage URL, and also a name - given how neither or those identifiers is completely permanent. I also own how I personally would describe my relationship with these people. This perspective need not be reciprocated.

But, I don't want to own my contacts' information. I want to delegate that to them! In as much as my contacts list is an address book, it should be so only through indirection: to find out about a contact, go to the homepage I have linked for them, and discover the contact information that they themselves have published there.

Most current systems do not assume such delegation: I am expected to own other people's contact information, as well as mine. I presume this is a deliberate compromise in order to gain critical mass (of contact details published online in general and on the vendors network in particular.) My own disdain for such practicality comes partly from not finding much need for an address book per se - perhaps if my network(ing) were better this would be otherwise. But, even allowing for that, I don't see the delegation I desire called out as an ideal or an aspiration anywhere in the design or implementation of current systems. Evangelizing this, and exploring how to get there from here, is the motivation for this and my next blog entry.

There are a two main reasons I want things to work this way.

Firstly, there is the golden rule: if I want to own my contact details, I should should grant other people the right to own theirs.
Secondly, I'd rather not have the hassle of complicating my own records with other people's details. Let the person who knows best update their information in a single place, and I'll get their latest details by going there. This also avoids my having to take responsibility for choosing which of their details to publish. Given current preciousness about privacy, this side-steps a potential minefield.

There is a yet more open approach that I have refrained from advocating. In a truly distributed wiki-wiki-world, one could argue that an individual's opinions on information about themself should not be sacrosanct - that we should seek a consensus of everyone's opinions. On most truths, I'm all for this kind of emergence. But when it comes to how to contact an individual I think having them be the single authority is too useful a shortcut, and it may have psychologically benefit, too! Letting the rest of the world speak for those as yet offline, though, could be a decent strategy to cover the gaps between where we are and where I want us to be.

Talking of such strategies: next time I'll write about my current implementation of the plan I advocate using existing contacts technologies - hCard, XFN, FoaF, PoCo, et al.

I have some further opinions that are related to the discussion above but would have distracted from its core, so I've moved them down here.

Own the domain: If you're going to own a page of significance to you, I would advocate controlling the domain name of the URL you publish it under. By all means have another party host the content, just point your own URL at where the page currently is. That way you can change the service/hosting provider without losing the URL by which people find the page.

Most privacy is squeamish: Although many are nervous about publishing it, I think much contact and other "personal" information is most naturally public: going ex-directory is weird. The impact of problems that might arise from publishing e-mail, mailing address or biography is overestimated. You're getting spam anyway. Stalkers are rare. The existence of the concept of "identity theft" occuring through knowledge of simple everyday facts - like a birthday, a hometown, a mother's maiden name - is a symptom of immature authentication procedures, a laughable security through not-even-obscurity. Lastly, contact details are useless without at least some users - and trying to control how others use and copy bits is a mug's game.

Review: Implementation Patterns

Wed, 22 Oct 2008 19:50:10 GMT

Implementation Patterns (Addison-Wesley Signature Series) by Kent Beck

This book concerns the detailed choices we make minute-to-minute when implementing programs.
The first twenty pages express Beck's context and motivation in his usual fashion: some personal history and philosophy, some thoughts on cost, and a prioritized set of values (communication, simplicity and flexibility) and some principles (DRYness, symmetry, preferring declarative to procedural expression, plus three ways to group things that change together) that help in translating those values into specific practices or patterns.
The next hundred pages is a catalog of seventy-seven such patterns.
Two closing chapters offer an overview of collections, and sketch some ideas of how things change if you're coding a framework rather than a specific application.
I think those first twenty pages are excellent. Two of my previous favorite books by Beck outlined a core precept, asked "what if we run with it?", then detailed how to extract far more value than you might initially expect. (To over-simplify: XP Explained follows from keeping the cost-of-change curve as flat as possible, Test-Driven Development follows from a belief that making code testable improves its design.)
The main idea in Implementation Patterns is that making code readable is key in good implementation, and that turning those dials to eleven will provide similar unexpected insight. (For example: given that a good name for a variable describes its role in context, then if all good names seem long, you're probably missing an abstraction of context - i.e. there's a class waiting to be extracted.) Not quite as big an idea as in the other two books, but very plausible, and my appetite was whetted for the patterns where I hoped this idea, and all the values and principles, would be explored and discussed in context.
Alas, no. I was very disappointed. (The above was about the only example I could find.) There was very little exploration and discussion, and the patterns did not often refer back to the values and principles, even implicitly.
OK, since it doesn't build sufficiently on the introduction's abstractions, the meat of the book had better work well as a stand-alone catalog of patterns.
Alas, a second no. One of the useful things explicit lists of patterns do is to give a name to a concept so we can communicate about it. Beck's list does do this useful thing. But, that is about the only useful thing it does. Once named, there is rarely additional insight in the pattern descriptions: they mostly do exactly what they say on the tin. You can read the table of contents to see how simple the concepts under discussion are.
The last two chapters are fine, but unremarkable.
My recommendation: borrow the book to read the material preceding the catalog. Then buy and read Clean Code.

Minimal OpenID glue for Rails authentication

Sun, 19 Oct 2008 21:04:04 GMT

The Rails framework and culture offer a well-developed set of plug-ins and patterns for the authentication of users and the authorization of their actions. These are great for more sophisticated enterprise and social applications - but as a hobbyist, my own apps don't tend to need a database full of users and roles. Typically the only user I care about authenticating is me, for write access to content that everyone else should view read-only without any hassle. I want something very lightweight.

The app behind my ego site is a good example. It presents pages based on a few sets of resources - contacts, and various flavors of permalinks for blog entries which are hosted elsewhere. Everybody should be able to access those pages. Behind the scenes there is some simple scaffolding for managing the resources. Only I should be accessing those.

The simplest solution is security through obscurity - to keep quiet about the management URLs. It's not that bad an approach for a low-stakes app, and I did rely on it for a while. Felt wrong, though.

Having decided to do better, I still wanted to keep things as lightweight as I could. My first preference is to minimize the use of the Rails session: I don't want people who visit my public pages to get lumbered with a pointless cookie, so I switch sessions off by default.

One way to keep authenticated session management out of the way of users and apps is to place it in the HTTP layer. I am a big fan of the http_authentication Rails plug-in. The browser on the client side takes care of the session sign-on UI and state, getting users to log in once on first access to protected pages. In your app, you can use a before_filter with implementation as simple as:

authenticate_or_request_with_http_basic do |ignore_user, password| password == 'your secret' end

But, I have a second preference. I don't want arbitrary secrets in my apps, or in my brain. I want single sign-on, please, and my web app just needs to know that I am who I say I am - let someone else do the verification of that identity. And I am already enthusiastic about a technology for authenticating identity in this way - it's OpenID.

Following a "gem install ruby-openid", I still wanted as little Rails glue as possible. The open_id_authentication plug-in is pretty compact given its power, but it does tend to assume that users will live in the database, and it provides all kinds of store machinery that I don't need.

So I cut non-essentials away from the plug-in source and ended up with this minimal parent for my resource controllers.

require 'openid/consumer' class ResourceProtectingController < ApplicationController @@PROTECTED_ACTIONS = %w(new create edit update destroy) session :disabled => false, :only => @@PROTECTED_ACTIONS before_filter :protect, :only => @@PROTECTED_ACTIONS protected def protect return if session[:authenticated] realm = "#{request.protocol + request.host_with_port}" requested_url = "#{realm + request.relative_url_root + request.path}" identity_url = "http://anthonybailey.net" consumer = OpenID::Consumer.new(session, nil) if params[:open_id_complete].nil? open_id_request = consumer.begin(identity_url) open_id_request.return_to_args['open_id_complete'] = '1' redirect_to(open_id_request.redirect_url(realm, requested_url)) else returned_params = params.reject {|k, v| request.path_parameters[k] } returned_params.delete(:format) open_id_response = consumer.complete(returned_params, requested_url) if open_id_response.status != OpenID::Consumer::SUCCESS render :text => "Forbidden: got #{open_id_response.class} " + "on claiming #{identity_url}", :status => 401 else session[:authenticated] = true end end end end

Some notes on the implementation:

Because OpenID involves a stateful request dance between the relying party and the identity provider, I had to reintroduce the Rails session. However, I can still keep cookies away from the public by only enabling it for the actions needing protection.
Now I have a session, I ensure that only one dance is required, by stashing the authenticated status. This also works around the fact that OpenID can only return using the GET method: in regular use the client will GET a prelude page before it needs to POST, etc. a real change, at which point we've finished dancing with the provider.
Mostly as a conceptual indulgence, I left the read-only actions unprotected. (To keep things GET-first, I moved scaffolding UI to delete resources off index pages and onto edit pages - but I kind of feel that's where it should be anyway.)
Since if you're not me, you're not supposed to be here, I didn't friendly-up the content when access fails.
Finally, as an aside, it amuses me that my identity URL actually is the domain of the URLs I'm protecting. (I don't run my own identity server, mind - I delegate.)

Review: Little Brother

Wed, 01 Oct 2008 23:07:23 GMT

Little Brother by Cory Doctorow

This is a fun coming-of-age adventure telling a very contemporary tale of tech-savvy teenage righteous rebellion against a police state.
The plot is straightforward in the large, but the formula is well-executed, is written in a convincing voice, and features wonderful details. Doctorow regularly writes apparently futuristic science-fiction simply by basing his fiction on today's cutting edge, and the edge cuts keenly in this book - the politics and technologies are all here today or grown tomorrow.
I read the free CC-licensed download, as I did with one of his previous Eastern Standard Tribe - and the business model of this publishing approach (of which Doctorow is one of the earliest adopters and most passionate proponents) proves itself again: I've just ordered a copy of the just released UK paperback to send as geek propaganda to my own younger sibling.

Review: Clean Code

Wed, 03 Sep 2008 19:43:22 GMT

Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin

I wouldn't usually have cared to read yet another book on the generics of good programming practice. But this one was by Bob Martin, "Uncle" of the first family of thoughtful contemporary coding, and whose blog is high on my roll.
The book claims to be unusually hard work due to the substantial amount of code it contains. I disagree - you certainly want to dive deep into the examples, but I've read plenty of texts containing more code and less value. (Although, for a book that encourages reading code, and that argues for the retirement of outdated formatting conventions, I am frankly mystified by the fixed-width, unhighlighted, vanishingly thin font used to present the examples: there might as well be tractor holes down both sides of the print-out.)
Martin makes the pragmatic simplifying choice of presenting everything in Java. That's fine; apart from some concurrency specifics, most of what is promoted will slip easily back and forth to e.g. C++ and Ruby - and if you can make Java pleasant to read, you've earned my respect.
After an introduction featuring great abstract definitions of cleanliness from the near-godly, we get twelve solid chapters of concrete advice. An account of best practice should not be surprising to a decent practitioner - and I found much of Clean Code to be a reassuring mix of the obvious ("oh, I already know that") and the nicely crystallized ("oh, that's what I already know".) TDD, culling comments, careful naming, ultra-DRY, breaking dependencies, small methods, yada yada yada.
There's also sufficient challenge ("oh, you're going to push it that far?") and controversy ("er, wtf? maybe...") to keep it interesting. Uncle Bob is illuminatingly obsessive about the law of Demeter and the single responsibility and open/closed principles. He particularly surprised me with his enthusiasm for keeping plain old data structures around within the object-oriented world.
But the three large case studies that close the book are the real gold here - the summary list of smells drawn from them is no substitute for their glorious, odorous detail. They are very engaging: despite their final shininess, I bet you too will want to polish them a little further.
Recommended.

Review: Bulletproof Web Design

Sun, 20 Jul 2008 16:08:40 GMT

Bulletproof Web Design: Improving flexibility and protecting against worst-case scenarios with XHTML and CSS by Dan Cederholm

This is a slim (270 spacey pages) primer on robust CSS practice. It isn't a reference book, nor really a compendium of recipes; more a set of ransackable case studies through which runs a persuasive essay on how and why to do this stuff right. (The basic precept is a principled separation of concerns: define style for mark-up that is independently semantically sound, rather than hacking div after nested div into your page in order to kludge your target design.)
I bought it a couple of years ago to learn how to do CSS-based layout, and did so by coding through many of the examples. I recently read it through a second time and was again impressed by its clarity and cohesion, and by how its elegant solutions do not sacrifice practicality - hence this review and recommendation. (And apart from a few thankfully obsolete IE5-generation workarounds, the content had not dated at all: though it preaches graceful degradation, the book hasn't yet had to practise it.)

(If it seems strange that I broke blog silence this way - yes, usually I'd put such a brief piece on my tumblelog, but Tumblr eats the hreview mark-up.)

What does BDD offer TDD die-hards?

Mon, 14 Apr 2008 00:06:30 GMT

"As a long-serving test-driven developer, I want to understand how behavior-driven development differs, so that I can decide whether or not to join the revolution."

Introduction

A couple of years ago I was developing medical imaging software for diagnostic radiologists. The field exhibited an archetypal divide between non-coding experts who knew lots about the domain, and developers who did not. In the language of classical XP, we were coding from requirements supplied by a customer. Without a map, we explored many aspects of the customer/testing/developer relationship:

the importance of a ubiquitous language spoken across the whole team
the connection between a spec and an executable test
the automation of customer-written acceptance tests

Very near the end of that time, I attended XP Day 5 in London. Whilst there I talked some with Elizabeth Keogh, who was assisting Dan North with the birth of Behavior Driven Development. It sounded fascinating and very relevant.

But it became less of an immediate concern, because soon after I changed jobs. I moved to a new language and test suite in the context of a (simpler) domain and a culture where the devs were encouraged to internalize the expert/customer role.

Now, over the past couple of years the BDD meme and tools have been surging in popularity. Meanwhile I'm working well in a classical Test-Driven Development place some indeterminate distance away, catching occasional glimpses in my peripheral vision of what looks like a pleasant enough image in a slightly warped fun-house mirror, and trying to work out how the reflection differs.

A couple of months back Tom ten Thij gave a great BDD introduction at our local Ruby Users Group. But he had to handle heckling from many of the audience who met the following spec:

Given: I'm a software developer who loves TDD and communicating well with customers,
When: I hear a talk on BDD that seems to promise exactly what I get from TDD and good customer communication,
Then: I find myself wondering exactly what all the fuss is about.

I took the question back to Liz, and she had interesting things to say. I hope sometime she may write them up properly herself, but in the meantime she said it would be fine if I wrote something. So, here's an [edited] transcript of parts of the e-mail conversation we had.

Context

Ant: Yesterday at our local Ruby User Group here in Edinburgh we had a presentation on RSpec and on BDD in general. The talk presented BDD from first principles rather than assuming previous TDD experience. The audience was mostly TDD old-timers, so a lot of the talk seemed like an very familiar tale told in slightly different dialect.

Ant: Our fuzzy consensus was acceptance that BDD is a nicer syntax; its language is a better fit for the good things that test/spec-driven development tries to do and hence the way it should be done from first principles - but, is the change worth it for TDD die-hards who have already completely internalized xUnit? What else do we get?

Ant: We realized none of us had ever come across a talk or article evangelising BDD which particularly spoke to us; for example, by a someone who'd been very dedicated acolyte of TDD and had many wonderful things to report once they'd learned to Behave.

Liz: I can't think of anything off the top of my head that addresses TDD gurus, which is awful, because we should have sorted this out ages ago. Let me see if I can help - please let me know if this is any use!

Ant: I'll devil's advocate some. Personally, I'm a big believer in the importance of language [and] I'm simply not experienced enough to make any claim of having seen it all before [but] I'll proxy for my peers who have been doing/teaching TDD for donkey's years.

Philosophy

Liz: BDD isn't just about reworking the syntax of testing, though it started out that way. It's evolved into something more; a language of examples that's relevant at both a unit and system level, which allows better communication with the business, and less redundant code.

Liz: DDD [Domain-Driven Design] provides the nouns, verbs and adjectives, and BDD provides all the other words which let you make sentences out of them. (Dan North, Eric Evans and I were chatting about this at OOPSLA; I can't remember the exact words they used, but this is close.)

Liz: It's all about the language. You don't really get anything else... but language is pretty powerful! Changing the words we use can change the way we think, and we don't have to change our technology to do it.

Territorial dispute

Liz: TDD is mostly about development - about taking a specification or acceptance criteria, at either a system or unit level, and making the associated tests pass. BDD assumes that it's not easy to get the right acceptance criteria in the first place, so the language - including the famous "should" - is designed to allow the original assumptions about the behaviour of the system to be questioned; to have discussions about that behaviour without having to translate from the technical world, either in speech (which helps us devs talk to BAs and the business) or mentally (which helps us realise when we need to ask a question). Of course, this is very closely aligned with DDD. It's also why I prefer "behaviour" to "spec"; because it's easier to question behaviour than a specification.

Ant: I disagree with this characterization. Particularly at the lower levels where the developer may be doing more of the work than the domain expert, traditional TDD is very much about exploring and evolving the behaviour of a system, one test at a time, asking questions rather than blindly coding up an existing design or specification. I completely buy the idea that the language of BDD helps make the openness to change explicit, though.

Liz: The best TDDers do BDD already, at least at a unit level. It's possible that they don't think of elements of code as stakeholders, though; at least, I haven't found any native TDDers with that mindset. It changes the names of classes, at the very least. I am also aware that some die-hard, expert TDDers concentrate more on the technology when they're teaching than the conversations around tests.

Ant: It helps my understanding if I consider BDD to be taking a concept of TDD that fits particularly well with driving from behaviors, and turning the dials on that concept up to ten. If not plain wrong, this characterization may play well with an old guard XP/TDD target audience.

Frameworks

Liz: First off, let me put to rest something really important: You can do BDD in xUnit.

Ant: would you go so far as to say that if you aren't starting from scratch there may not sufficient advantage to switch from xUnit to a more well-Behaved framework even if one falls for BDD in a big way?

Liz: Don't use JBehave. It's a great teaching tool, but sucks IRL. I tend to use JUnit. JBehave 2.0 should (when it eventually comes out) be easier. It's going to be more like RSpec's plain text story runner, and consist of a lot of JUnit exensions.

Talking to business

Liz: [BDD offers] a language of examples that's relevant at both a unit and system level, which allows better communication with the business We use the word 'scenario' or 'example' because you don't go to the business and say 'Give me an acceptance test'. You say 'Give me an example' or 'Give me a scenario where that happens'. It becomes easier to maintain acceptance tests - and they become human-readable!

Ant: Would you venture further? Non-developers can edit the acceptance test? Non-developer can write the test when paired with developer as translator? Non-developer can even write the test unambiguously without direct developer assistance, given some further translation layer like a plain-text story runner?

Liz: Absolutely. This has worked very well with our QAs.

Ant: The last was a feature of the talk I attended. It drew attention from the old TDD guard, who were all very XP-story-focussed and some of whom had developed such translation layers themselves. It provoked both a "these BDD people sure have the right idea" response as well as a "and it's an idea we had a while back, and is separable from the question of changing terminology at the xUnit level" one.

Liz: Good. "Should" is trivial compared with the given / when / then language.

Ant: Partly putting words in their mouths, and partly exploring something I thought might be interesting: if you do end up with another layer between the non-developer and the runnable test/behavior/spec - and our experience is that perhaps this is worthwhile - then some of the communication advantages of using BDD at the bottom layer are diluted. If you have to use a translation layer like a story runner to pre-process a more plain text description of behavior, then using BDD terminology in the target code doesn't help you so much with respect to communicating with the non-developer, because they don't see that code, they see the source story. At that point the BDD communication argument perhaps reduces to the DDD one (albeit one I buy) that using business language as much as possible in the code itself is worthwhile even when only the developer sees it.

Ant: I guess I was wondering whether you'd challenge the pre-condition - can one make behaviors expressed in the production code language sufficiently like plain English to be writeable by the non-developer? (Perhaps it depends on the language?)

Liz: RSpec's stories now work in plain English. We hope JBehave 2.0's will be the same. It's a nice little given / when / then DSL.

Ant: I guess my point was that some long-time TDDers have so internalized some of the translations from dev-centric terminology to plain English, that they might consider a test very easy to read way before the non-developers do. But purging such language habits would probably ease indigestion nevertheless. And if BDD terminology becomes the standard, there's also a communication benefit amongst developers - the next wave may well see TestCase/test/assert as strange dinosaur language.

Outside-in

Liz: [The BDD spec] leads to the first piece of code which the human stakeholders need, which is the UI - the mechanism through which users (or 3rd party apps, say) interact with our system, and through which behaviour is perceived. So this is how we develop with BDD - from the outside-in.

Ant: This was a part of BDD missing from the talk we heard, and it sounds interesting. I enjoyed your real life example. [Re TDD] with "the YAGNI of outside-in" - whilst YAGNI itself is a TDD cornerstone, I haven't heard the tenet that therefore one should always start with the user interface layer.

Liz: If you don't start with the interface, how do you know that you need it? How do you know that the stakeholder code will be able to use what you're writing in the way that you're writing it? You can pretend that you already have the code, work out how you're going to use it, then code the real thing later - this is exactly what we do with TDD anyway! Now we're taking that pattern into production code too.

Code has needs too!

Liz: We use the same language of examples, to describe behaviour that provides some benefit, at both a unit and a system level. The stakeholder at a unit level becomes the other piece of code which needs that behaviour - so unless we have some code already, we can't start.

Ant: This analogy between human and code clients is not one I'd heard advanced quite so strongly in this context - is it important to BDD?

Liz: Yes!

Ant: [In TDD] this definitely isn't an angle I've seen pushed hard. Trying to make client use of the new code readable, sure, but not the kind of anthropomorphism you're advocating.

Liz: I'm not thinking of the stakeholders as human, as such, though it's a good analogy - code changes just as much as people do! BDD is very need-driven. Why are you coding that? Who's it for? What is going to benefit from that line of code? It changes the names of classes, at the very least. [...] We've started naming things based on the responsibility to its stakeholder, or its job, instead of the design pattern. As an example, we've started calling repositories after the plurals of whatever you get out of them - so, "Puzzles" for my sudoku creator, as opposed to "PuzzleRepository".

Given/Then, When

Ant: Random aside... is it perhaps the case that most mocking layers for xUnits (at least in languages that don't love closures) make given/when/then expression harder? It seems that expressing the expectations of mocks should go in the "then" section, but they usually assume they'll be set up before you execute the code under test within the "when" section.

Liz: Check this new toy out: Mockito My friend Szczepan Faber made it, based on EasyMock. It doesn't make you set up any expectations unless you need something to return (eg: given three fridges in stock). Instead, you get to ask it afterwards whether the interactions needed happened. No finishedRecording(), and it'll automatically return null, 0 etc. if you haven't set up a particular stub.

Conclusion

Well, I hope you found that as interesting as I did.I'd love it if the "TDD old guard" (who I've quite possibly misrepresented) could comment below to take the conversation further...

Credits where credits due

Mon, 04 Feb 2008 23:16:51 GMT

(A piece about TV credits on Mongoose World caused in turn by this piece from the BBC provoked a rant from me on that subject and particularly how it relates to machinima. It seems to have broken my writer's block, so I've copied/pasted/edited it here.)

Full screen credits on plain old broadcast television might still be reasonable, because how else do you get the information? But that medium is increasingly dead. We shall speak of it no more.

Full screen credits on TV as we increasingly watch it (DVR, DVD, IP) seem anomalous. Pack them into an information burst: if I want to read them, I'll pause and advance through individual frames. If I'm on a smart device, put the credits in metadata I can browse with that device. If the device is Net-enabled, put them on a page I can navigate to from the program. Don't make me sit through this stuff. Brand a little if you must, but if I care about obscure details, I'll seek out the information, and be very happy to read much more than you could cram into the credit footage.

Full screen credits in typical machinima (and other YouTube-length pieces) are a sign the medium is far too indebted to its older media parents, and is too slavishly following their inappropriate conventions. See above: viewer is almost certainly watching on a Net-enabled device. And a minute of credits on a five minute piece is a real tax on my time!

Further, there should be two default downloads/streams for episodic machinima:

There should be the new episode for returning viewers (without credits, of course - we already saw them so many times. You can do a "previously" if the plot so demands.)
And there should be everything from the first up to the new episode in a single download/stream for people coming to the story for the first time.

These are your two markets - cater to them. The second is particularly ill-served by having to watch umpteen short previous episodes in order to catch-up, where tedious credits/theme song prevent me getting into your wonderful art.

(Sorry for the explosion of highly subjective opinion. I don't know why I care so much about it. I remember I tried and failed to convince Hugh to do this kind of thing with Bloodspell.)

'Spell is out

Tue, 23 Oct 2007 23:30:18 GMT

Independent film outfit Strange Company have released their feature-length machinima movie, the "punk fantasy" BloodSpell.

My opinions on the film are necessarily messy. I'm an advocate of the medium (machinima is shooting movies within virtual worlds, often using video games), and a friend of the creators - indeed, I worked on one of the tools used to create the film. So I'm unfairly biased. I'm also not a great person to have on your side, because I don't let such things mellow my criticisms or get me hyped - at best I do faint praise, and damn you.

BloodSpell hits a huge range of highs and lows. Almost every aspect - story, performance, cinematography, art, technology/effects, pacing, dialogue - has parts that are truly great, and parts that are pretty poor. (The exception is the audio: sound and soundtrack are good throughout.) A spread of quality is not unusual in a film, but the range is particularly wide here. There are many genius moments and scenes. There are many embarassing ones.

The film is strong at start and end, weak in the middle. This is partly due to the order of shooting: Strange Company learned heaps about their machinima technologies and about movie-making as they created the piece; originally released in episodic chunks, they returned to completely redo the opening scene when editing together this version. It's also partly due to happenstance - the second scene was already strong, whilst the film serves as an archetypal illustration of the cinema textbook tendency to have the story sag in the middle.

This is by a fair way the best machinima feature film I've seen. The dated technology is necessarily distracting, but didn't spoil my enjoyment. Whilst previous machinima movies have excited me about the future potential of the medium, this one is real, now. It simply works as a decent movie in it's own right - I'd watch it through if it came on TV the right midweek evening, probably whilst bestowing it the odd sarcastic aside because, you know, it's lucky to be on the channel I'm indulging.

It's also probably the best budget independent film of its genre (epic action fantasy drama with nods to comedy and contemporary culture - the one word summary is "Buffy".) This is almost by definition: you simply can't shoot real film (or TV) of this flavor for the few thousand pounds it took to make BloodSpell. (Also this genre of film is not blessed with many success stories. This one can only have helped.)

The film was made by volunteers, is licensed under a BY-NC Creative Commons licence, and you can download or stream it for free. I'd even encourage fans of the genre to do so if it didn't contradict my disclaimer at the start!

Digital Privacy Management is inEFFable

Sun, 30 Sep 2007 21:27:20 GMT

It's increasingly convenient to manage personal data up in the cloud. Eben Moglen worries that people entrust important data to organisations without thinking of the downside. Drawing an analogy with open source in software, he encourages people to store data themselves rather than pumping it into proprietary information silos.

I am persuaded by the argument as I've phrased it above. It's timely to consider investing in keeping copies of the personal data that is important to you, because otherwise you're going to lose it when the silo closes. Ideally you would keep the copy rather than a copy, - you would manage the canonical resource and publish it. This is because you have the most interest in and ability to ensure the ongoing correctness of the data. Because DIY is hard, third-party service providers will help you; but they should be neutral hosts rather than aggregators motivated by their own uses for the information.

Actually Moglen seems to be making the stronger argument that you should hold the data because you need to be able to control it, not just publish it. Commercial entities will enter into contracts with you about what use they are allowed to make of this data that you own and are generously granting them access to. His argument is phrased in terms of privacy - your right to control the use of digital data about you. Maybe you'll even sell access and usage rights.

And this is where I differ from him and a whole bunch of organizations with whom usually I find myself in agreement. Trading overzealous stereotyping for a reduced risk of misquoting, I will pick the Electronic Frontier Foundation as a specific example of what I find to be a widespread and questionable position. I find their stance on issues of privacy to be very much at odds with the pragmatic approach they take to, say, DRM.

On the latter, essentially they say: trying to control how digital information is copied and used doesn't work very well. The technologies are too easy to break, and they prevent unintended new uses of the content, restricting freedoms and stifling innovation. Therefore we should avoid purely technical solutions, and retire historic business models that now rely on them. Instead play nice with the world as it is, and try to nudge it in the directions you desire.

That all sounds right to me. So how on Earth will we be able to use any kind of Digital Privacy Management mechanism to control how personal information is copied and used? Bits are bits, and once you publish them you just don't get to control their copying and use. The hazily envisaged DPM is DRM, only without a law-buying cartel behind it. It's not going to work. Move on, and start discussing and tackling the problems that can arise in a space where there are plausible solutions, please.

(You'll also find the open source analogies work better once technical controls are abandonned, I believe.)

Origins of the word "machinima"

Sun, 09 Sep 2007 17:58:52 GMT

Machinima was spelled slightly differently when first-born. The first use of the term, as "machinema", appeared in an e-mail I sent on the 'q2demos' mailing list on 5 Jan 1998. Here is the quote from the mail in question. Conveniently, it happened to go on to explain the need to coin the term.

Anthony:

I think with a new tool-set and a greater awareness of the skills
required to make a good piece of machinema[*] we can expect to see
some truly adventurous stuff being made. Fine and fun as they can be,
slapstick and B-movie action horror are just the start of what *could*
be done with this new medium. If I was a film student with some
technical savvie, I'd be beginning to look at using this stuff as an
alternative to or prototype for an expensive real-world production.

[*]Machinema... yes, sorry, it's a bit of a contrived term... but what
in general *are* we going to call these pieces of cinema that are made
using 3D engines? Not only is "Quake movie" an ugly and confusing
term, it's also fast going to become outdated as other technologies
become relevant. (c: Any ideas?

I chose "machinema" to rhyme with cinema. I used "machine" as a base only because I didn't find an easy pun with "engine" in it. I spelled it with the 'e' because it made it look a bit like cinema. I don't believe it to be the best neologism in the world, and I apologize for that - I didn't know the term was going to get widespread.

The word began to be used fairly regularly on the mailing list and gradually in other related fora, but that was about all. So far as I know, it's first "public" appearance is in a review I wrote for GameSpy Industry's Quake portal site PlanetQuake.com, first published on 02 Apr 1999. (Slightly broken copy of now dead page courtesy of the Internet Archive)

Hugh Hancock of Strange Company wrote up some "machinima" techniques on 10 Jun 1999. We were already in e-mail contact so we discussed the new spelling.

Anthony:

<pedantry>
I'm very happy that you've adopted my silly whimiscal term for cinema
generated by desktop machines, but is there any reason that you have
changed the spelling? The 'e's in both "cinema" and "machine" seem to
suggest the natural spelling for the mangled combination is
"machinema." It's a pun rather than a derivation, so I don't see any
linguistic pressure to follow the path of e.g. "machinist."
</pedantry>

Hugh:

As a linguist I'd prefer to use the "ima", a more common suffix than "ema",
and go with the normal derivation, but to be perfectly honest the main
reason we've changed the spelling is I forgot how you originally spelled it
and "ima" looked more natural! Sorry...

Anthony:

I didn't like "machinima" at first because it seemed to lose the
"cinema" pun, at least at the lexical level. But it's still there
phonically, and now I realise there is a genuine derivation from
"anima" going on as well, so the new spelling actually maybe combines
something plausible and word-play. Anyway, in the interests of
uniformity I'll adopt your spelling.

Hugh tells the machinima etymology story from his side in a far more self-deprecating and entertaining style within the book Machinima for Dummies.

The term crossed the chasm for certain when hub-site machinima.com was launched at the start of 2000.

Blipvert

Sun, 19 Aug 2007 18:07:58 GMT

I'm wondering why there isn't more machinima on blip.tv. There is some, but this video publishing service seems like such a good fit with the needs of machers I would have expected more.

I became aware of blip through this recent interview with co-founder Mike Hudack by Jon Udell. (Good podcast - says some interesting things in general about web service businesses.)

The company seems to really get what a user-gen content aggregator and publisher needs to do to succeed by pleasing their users. You can read their own overview, but items from the interview that impressed me included

a strong distaste for service lock-in (for example, they will automatically mirror your upload to the Internet Archive if you want),
being very open to working with other distributors and publishers (for example, building a system to share tags and download stats),
staying in the background to let creators control branding (for example, they don't embed their logo),
sensible advertising options (you can use their advertising and share the profits, or do your own, or none),
providing a strong free service with professional upgrade.

The whole thing sounds like the right business model: provide a service which offers sustainable long-term value that supports creators.

The disclaimer: I've not used the service. (Anyone who has care to comment?)

Regression Therapy - Contentful Testing

Thu, 19 Jul 2007 00:46:32 GMT

Regression Therapy - Contentful Testing

This article is a fleshing out of a ten-slides-in-ten-minutes talk that I originally gave in July 2007 at work.

Since then I talked on the same subject at the Scotland on Rails 2008 conference. Slides and so forth are available.

1. Introduction

Regression testing is usually seen as the poorer cousin of "proper" domain-abstracted assertion-based testing. Often rightly so!

However, with the right support in place, I have found that this form of testing can work very well in certain contexts.

This article addresses one such context: testing the view content generated by a web app. I discuss the background, then present a concrete example in the form of a plug-in for Rails.

2. Three reasons to test

For the purpose of this discussion, I identify three overlapping but distinguishable uses of tests in software development.

To preserve existing good behaviour

Tests can be used to maintain known good behaviour, preventing accidental bad changes (also known as "regressions".) The tests provide a safety net during refactorings, and enfore a Hippocratic "first, do no harm" oath during other work.

....................................!..!...........................!...........

The test stays green whilst all is well. A red bar tells us that we've messed up and broken something.

To drive intended changes

Tests can be used to change (often, to add) behaviour, providing context to an intended good change. The archetypal example is classic TDD, creating tests first in order to define the problem, and to guide the design and implementation of a solution.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

New tests flash red and are put back to green by introducing production code.

To understand emergent change

A less often discussed use of testing is to explore and validate emergent behaviour.

In a complex system it can be hard to prejudge or predict every knock-on effect of a deliberate change to one part of that system. Tests can provide a context for understanding and controlling such changes and consequences.

.....!!??????...................!!??????............!!??????..............

In this way of working, changes cause flashes of red that are observed and then often approved, updating expectations to recolor things back to green again.

3. Three ways to test

Next I'll categorize three methods of testing, paying particular attention to their use in the context of testing view content.

Live testing

In a scandalously unrepeatable manner that sacrifices reliable coverage, we can test by hand and verify by eye.

Drive the web app through the browser and see if looks right. We all do this. It shouldn't be sniffed at; it is genuinely useful because it is so cheap. However, once the system gets sufficiently complex, it doesn't hold water. I won't consider manual testing further in this article.

To automate testing, we have to be able to provoke the generation of view content that we can then test. Assuming a good web app framework that allows us to do this (no small thing, but not the topic of interest today), there are then two means of testing the content.

Assertion testing

We can explcitly check individual desired properties of the the resulting content.

Regression testing

Or, we can compare the content in its entirety against a previously validated known good copy.

I'll now discuss in greater detail the relative merits of testing view content through assertion and regression.

4. `assert_equals(expected, test_method)`

Assertion tests are prevalent in contemporary developer testing, with good reason. They tend to be very effective, given sufficient investment – we need to express the properties that care about succinctly, and at an appropriate level of abstraction within the domain.

The Rails web framework has evolved a best-of-breed solution for assertion testing view content in the form of assert_select. It uses CSS selectors to locate content, and neatly expresses the conditions it should satisfy. Here's a simple example:

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    assert_select "div#cart tr.total td:last-of-type", "$39.95"
  end
end

This is pretty nice. But it doesn't escape some downsides of assertion testing which happen to be particularly troublesome for view content.

Coverage is selective

We can only capture the behaviors that we think to test for. The importance of many properties of view content are not obvious until we've accidentally changed them.

Exploration is expensive

Assertion tests are generally great for catching regressions and for driving changes. But they are poor for my third type of test use, that of exploring change. The developer has to run around rebalancing the books, fixing up test expectations to agree with the unexpected consequences of a change.

Explicitness is verbose

Many aspects of the view are worth caring about. It takes too much code to express them all explicitly; translation, even into a well-designed and succinct language, still takes time. The most concise summary of all the properties of a piece of content is the content itself…

5. Regression (past life)

…which leads us to our next alternative.

Regression tests work by checking current content against expected content. We test by comparing the entirety of what we generate against some previously approved copy. The expected content might be quoted verbatim in the test; more often it is stored in a file.

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    assert_equals File.new('expected.html').read, @response
  end
end

This is clearly very cheap to express, and capturing existent good behaviour needs no translation: just a copy and paste of the current results.

But like assertion testing, regression testing view content comes with some downsides.

Tests don't come first

Again this form of testing is good for two of the three uses of testing that I identified. It is ideal for exploring changes, and unsurprisingly it catches regressions. But, it is very awkward to drive new behavior.

However, this problem is not serious in the context of view content. Although it would not be plausible to test-drive the generation of content through regression testing, it is not clear that one would ever want to. View content tends to come into existence in a concrete form, designed or composed in toto rather than being built up one abstract essence at a time. Even for those who usually love to test first, test-last of view content does not seem so detestable.

The main problem with regression testing is that it is very expensive if we act naively in response to diffs. There are two common pain points.

Tests are brittle and fragile

Comparing against a complete text copy means that we see every change, even ones we don't actually care about. Every difference needs explicit attention.

Tests are usually noisy

Because there is no abstraction, one change may break many tests. Similar differences need attention in multiple places.

These workflow costs multiply as the system under regression test grows. Investing in the abstraction of assertion tests is usually preferrable. Even in the domain of view content, regression tests will prove too costly if we do not work to make them less so.

6. Regression Contentful testing

So: let's attempt to address the problems, and turn regression testing into contentful testing.

Lessen the noise

Information can be defined as any difference that makes a difference. The diffs that don't are the noise in regression tests that we want to reduce.

Ignore insignificant change

Some textual changes in HTML source (for example, certain whitespace in content; and formatting, quoting, capitalization and attribute order within the mark-up itself) won't affect how the content is displayed.

These can be ignored by allowing a little abstraction back in. We can still use standard text-based diff tools, but to compare unique text representations of DOM-normalized content.

DRY up which content is tested

Rather than see the same diff to the same repeated content over and over, we can focus regression tests on those parts of content that vary across our testcases.

Smooth the workflow

In good automation fashion, we will make the common tasks in the workflow as cheap and as easy as we can.

It has to be simple

to create a new regression test,
to inspect diffs from the expected content when changes occur, and
to accept the changes once they have been checked over and seen to be OK.

And, this needs to work well in batch. We should be able to detect, review and accept changes in many tests at a time when need be.

The Contentful Rails plug-in

I put together a plug-in that tries to do these things within Rails.

% script/plugin install svn://rubyforge.org/var/svn/contentful

The plug-in was evolved in, then extracted from, a small spare-time web application. I've since re-used it successfully in other Rails projects. If you write Rails applications, you can read more and download the plug-in at the Contentful website.

For this discussion, Contentful is simply a single concrete example of a general approach I believe should translate across language boundaries, or from my Rake automation commands into your favorite IDE.

7. Capturing content to create tests

With Contentful plugged-in, we can check the content generated in a test by saying assert contentful.

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    assert_contentful
  end
end

Then when we run the test for the first time, the test passes - and it generates the expected content from the current content, as a side effect. In accord with Rails' culture of convention over configuration, we locate the expected content in a standard place, derived from the name of the test.

% rake test
Started
...................................!...............
Finished in 3.487022 seconds.

  1) Contentful Notification:
test_total(CartExampleTest):
Generated /depot/test/contentful/cart_example/total/expected.html

To avoid duplication, we can focus on a particular subset of the content using a CSS selector. This allows us to ignore components that are common across all our views, such as navigation sidebars and the like.

class CartExampleTest
  def test_total
    get "/store/add_to_cart", :id => rails_book.id
    select_contentful "div#cart"
  end
end

% rake test
Started
...................................!...............
Finished in 3.494442 seconds.

  1) Contentful Notification:
test_total(CartExampleTest):
Generated /depot/test/contentful/cart_example/total/div#cart.expected.html

8. Detecting changes and inspecting diffs

Now our tests are up, we should set them running. (Running everything is the norm, but Contentful will build and execute a suite containing only its own tests if asked. I'll show that here.)

Here's what happens when we've broken something:

% rake test:content
Testing expectations in test/contentful
Started
..F.....
Finished in 0.808864 seconds.

  1) Failure:
test_total(CartExampleTest)
diff test/contentful/cart_example/total/*.to_diff
to see the content change

The failing test will have written a temporary changed.html file next to expected.html. Additionally it generates a second pair of files (expected.to_diff and changed.to_diff) which are the same HTML DOMs, but with some added line-breaks and other normalization to make them usefully comparable using a standard text diff.

(Whenever a test passes, the stale three files get cleaned away.)

As well as using the suggested command-line to examine an individual failure, we can inspect all current diffs by running a single command.

% rake test:diff
Finding changes in test/contentful
! Diff in cart_example/total/changed.html
91c91
< <td id="total">
< Total
---
> <td id="total" class="grand">
> Grand Total
! Diff in another_example/something_else/changed.html
[etc...]

(Additionally, if we want to focus on a subset of all the current changes, we can do so by running within a subdirectory of test/contentful.)

9. Accepting change

If a set of changes are good, then often we can tell this by casting an eye over the diffs and observing a plausible pattern.

(Sometimes one can grep the diffs for extra certainty. For bonus points, script anything you find yourself doing by hand three times!)

If all looks good, we want to accept a bunch of changes.

% rake test:accept
Finding changes in test/contentful/cart_example
! Accepting cart_example/total/changed.html
! Accepting another_example/something_else/changed.html

That was a bit better than opening up an editor and copying and pasting the content, right?

(Again we can refine our acceptance to a subset of changes by running within a subdirectory.)

Also, since expected content is generated when absent, we can alternatively use the file system to our advantage, and update expectations by removing particular subdirectories and then re-running tests.

% rm test/contentful/further/specific/*
[etc...]
Generated /depot/test/contentful/further/specific/expected.html

10. Making a vice out of a virtue

Personally, I really like my content tests to be a permanent part of the test suite. Once the workflow is smooth, I find them well worth their maintenence cost. I like the total coverage; they allow serendipidous discovery of changes I didn't consider.

However, even if you won't buy the entire bridge that I'm trying to sell you, perhaps I can interest you in a strut: use the virtues of contentful testing as a vice. (Sorry for the UK spelling. Translate it into the US "vise" if you like, but you'll spoil my pun.)

Suppose we want to perform a pervasive refactoring. For example,

to change the templating framework we're using, or
to factor out a set of view helpers and partials.

Big changes are risky. It would be nice to temporarily pin everything about current content tightly down in place whilst we work. When using Contentful, then we can add a bonus content assertion to every existing functional and integration test by setting a single Rails config flag in environment.rb:

CONTENTFUL_AUTO = true

Running tests once generates expectations and locks down all the current content. Then we can perform our refactoring under their full protection. When done, we unset the config flag and delete the temporary expectations before checking in the safely refactored code.

% wget http://contentful.rubyforge.org/

Thanks for reading! Comments are most welcome.

The views expressed here...

Mon, 04 Jun 2007 18:14:04 GMT

Something I forgot to do on joining my current employer...

The views I express here on my blog do not necessarily reflect the views of Amazon.com.

A Feedsum Engine

Fri, 25 May 2007 21:48:37 GMT

A couple of weeks ago I noticed that I was avoiding subscribing to some informative feeds because they were a bit too informative. I don't want to see as many as a dozen separate news items across the course of a single day, because it makes too much noise in the feed aggregator and gets in the way of rarer, more valuable entries. (In my case, the aggregator takes the form of my LJ Friends page.)

I realized I wanted the equivalent of a digest e-mail for feeds - a daily summary or similar. This sounded like a job for a web service, so after failing to find any existing solution, I registered feedsum.com and coded up a little Rails app that sucks in a source feed, collates items into daily batches, and generates a derived feed in which each item summarizes all the source items for one day.

At first this was simply a stateless service, but I found that source feeds often dried up very quickly - by the end of the day, the earliest items could already have been pushed out. So now I stash source items in a database whenever the source feed is pulled. I also found that syndicators such as LiveJournal were not sufficiently patient to wait for my app to spin up in clunky development mode, starting up a fresh Ruby before pulling the source and generating a summary - they often timed out before the service was done. So finally I have had to learn how to deploy a proper production Rails app using mongrel proxies and all that. There were some fiddly one-time set-up details, but once up and running Capistrano is pure deploy joy.

Currently I'm using the service to source three LJ syndications: slashdot_daily, machinifeedaily and arstechnicdaily. The service can probably handle a further light load yet so others (LJ users or any other feed consumer) are welcome to give it a beta test. It's easy to use: to get a daily summary of some feed

http://example.com/path/index.xml

simply request

http://feedsum.com/daily/example.com/path/index.xml

(You can do other neat stuff like http://feedsum.com/every/8/hours/... and so forth but the daily summaries tend to be the useful ones.)

Do try it out and please tell me if e.g. it completely chokes on any source feeds. (And to avoid a rush of dupes, could anyone who uses it to register further LJ syndication accounts add a link in a comment below.)

Anthony Bailey's blog

Ask Alexa Dot "Try Amazon Music Unlimited", get "Sorry, I'm having trouble" response? Workaround.

GoogleCL authentication with accounts that use non-Google email addresses

Review: Homestuck

Homestuck

Review: Red Mars, Green Mars, Blue Mars

Review: For The Win

The hopeful tale of an unspoiled Twilight

Review: Sweet Dreams (Pride, and Prejudice, and Zombies)

Testing with less manual calculation

Universal coverage

Lowest common denominators of LCD

Implementation rendered obvious

Contentful, again

Review: Anathem

Review: The Baroque Cycle

Contacts: Let's keep all this between ourselves

Review: Implementation Patterns

Minimal OpenID glue for Rails authentication

Review: Little Brother

Review: Clean Code

Review: Bulletproof Web Design

What does BDD offer TDD die-hards?

Introduction

Context

Philosophy

Territorial dispute

Frameworks

Talking to business

Outside-in

Code has needs too!

Given/Then, When

Conclusion

Credits where credits due

'Spell is out

Digital Privacy Management is inEFFable

Origins of the word "machinima"

Blipvert

Regression Therapy - Contentful Testing

Regression Therapy - Contentful Testing

1. Introduction

2. Three reasons to test

To preserve existing good behaviour

To drive intended changes

To understand emergent change

3. Three ways to test

Live testing

Assertion testing

Regression testing

4. assert_equals(expected, test_method)

Coverage is selective

Exploration is expensive

Explicitness is verbose

5. Regression (past life)

Tests don't come first

Tests are brittle and fragile

Tests are usually noisy

6. Regression Contentful testing

Lessen the noise

Smooth the workflow

The Contentful Rails plug-in

7. Capturing content to create tests

8. Detecting changes and inspecting diffs

9. Accepting change

10. Making a vice out of a virtue

The views expressed here...

A Feedsum Engine

4. `assert_equals(expected, test_method)`