Judging the Rumble
Whew! In case you didn’t know, last weekend was the Rails Rumble – an annual, 48-hour sprint in which teams of 1-4 people build complete, functioning web applications from scratch. I’d competed in the two prior Rumbles (winning the solo division both years </shameless plug>), but this year I participated in a different way: I was one of the expert panelists tasked with judging the submitted applications prior to public voting. Surprisingly, I found judging to be at least as tiring as competing had been the previous years – the overall quality of submissions this year was noticeably higher than it’s been in the past, which made the evaluations that much more difficult.
As a panelist, I made a commitment to providing constructive comments on each of the applications I judged, as well as rating them on design and interface, completeness, innovativeness, and usefulness. I think I did a pretty good job with the comments; I’ve gotten thanks from several of the contestants for the feedback, which is nice. I wanted to clarify my particular approach to the ratings in this post, however, since it’s the most opaque part of the process of judging. Each of us on the panel has a different background, so my interpretation of, say, usefulness might differ dramatically from some other judge’s.
Design and Interface
When deciding what score to give for design and interface, I looked at several factors. Pleasant-looking sites rated higher for me than sloppier ones, and I gave a fair amount of weight to sites that went beyond a simple themed prototype look. Sites that used the standard error pages lost points, because that’s an easy place to customize and make your site friendlier and more usable. I tried to go easy on minor flaws, since those are bound to occur in such a limited time period – but at the same time, I tried to reward sites that put on those finishing touches.
More important than most of those factors, however, was message: if I couldn’t understand what a site was for, then its design failed. I ran across several sites that lacked the appropriate contextual help to get me to the finish line, which was a problem.
Completeness
This is a hard category to judge, because who’s to say if someone fulfilled their vision or not? Sometimes, it was easier – several teams made the mistake of talking about the features they wanted to implement but couldn’t, or pointing out broken functionality. Here’s a tip to future Rumble contestants: if you don’t finish something, don’t call attention to it. Fix it after the competition, but don’t advertise your failures to the judges.
Other applications had obvious problems – broken authentication and links, that sort of thing. Those were easier to pick out, but you always feel bad when that happens. At the same time, there were applications that gave me everything I could want and more.
Innovation
Here’s another tough category, though it’s not quite as bad as completeness. When evaluating innovativeness, I tried to figure out if the site represented something really new (either wholly original or a surprising revision, like putting multiplayer Asteroids on the web), or if it was Yet Another Twitter Filterer. The Rumble’s interesting because there are usually a few categories of application that have four or five entries – this year, it was stuff-tracking apps (I loaned you X, give it back), secret crush apps, and movie night planning apps. I tend to dock points if you end up in one of those categories, or if you’re a standard entry in an already crowded market.
Even if you move into a popular niche, though, you can still score highly on innovation. The key to doing so is trumpeting your differences – why are you better? Too few apps do a good job of that, even when they have something worth marketing.
Usefulness
My primary metric for evaluating usefulness is myself: can I see myself using this app tomorrow? Next week? Frequently? Rarely? Obviously, this doesn’t fit all sites, so when I’m not part of the target audience I try to project how they might feel about the app – and you could have an app that only needs to be used once to prove its helpfulness. This is the category where I’m most flexible in my ratings – partly because of the odd nature of the category, and partly because I feel it somewhat-unfairly penalizes games and other off-the-beaten path sites.
So, that’s how I did it. Hopefully this sheds some light on what can be an anxiety-producing process, and maybe even helps next year’s competitors (who may have me to contend with again – I was very jealous of all the fun that seemed to be happening over the weekend!)
Update
Jim Minatel has also posted his thoughts on being an expert panelist over on the Wrox blog. It’s nice to see how others were thinking, and I agree with a lot of what he said (though I’m of a different mind on the IE support).
Very cool to see your standards for judging. I’m glad you had some, it makes the experience that much more beneficial for participants. I can only hope you get assigned to ours next year!
Ross Brown
26 Aug 09 at 7:20 pm
what I find interesting is how much the design ticks up the other categories. compare:
http://lowdownapp.com/
http://rakugoka.r09.railsrumble.com/
lowdown, I get the feeling, is going to end up pretty high in the overall rankings in the end. I hadn’t seen nor heard of rakugoka until just today, and even looking at it, I *feel* like it won’t be as useful or complete. even so, their vision is exactly the same, yet I predict lowdown will get hire usefulness and innovation scores.
visnu
26 Aug 09 at 7:32 pm
*higher
visnu
26 Aug 09 at 7:33 pm
That’s a good point. While the category is ‘appearance’, it really includes usability. While Lowdown and Rakugoka might do exactly the same thing, Lowdown’s design does a better job of communicating what it does and how to use it. Usability definitely has an impact on ‘usefulness’ and even ‘completeness’.
Steve Heffernan
26 Aug 09 at 9:00 pm
Lowdown and Rakugoka are similar, but they’re hardly identical in scope or functionality. Rakugoka in particular has an accompanying gem that allows you to import an existing project’s Cucumber features. I’d actually rank it a teensy bit higher on usefulness and innovation as a result. Rakugoka also allows commenting on features and scenarios. Lowdown has a nicer UI, but there are distinct differences apart from that there bear on the other scores.
Ben
26 Aug 09 at 10:05 pm
Hey Ben,
I just wanted to write back and thank you for the constructive criticism on Yield the Floor. There were definitely deficiencies in the site as a whole, but I suppose most Rumble entries will have a few of those. I guess what made it most difficult for us was the lack of a designer on the team… a talent we’ll definitely work harder to recruit for next year. In addition, we have a number of notification mechanisms in the works that will turn the asynchronous nature of the site into a more seamless one.
Anyway, the last thing I want to do is seem like I’m making excuses for our work… that is definitely not the case. I’m proud what we did (especially the BDD nature of the code) and look forward to evolving the concept into a less staccato-like version soon.
Again, thanks for all your feedback!
:brad
brad
26 Aug 09 at 11:33 pm
Hi Ben,
Thanks for this post. You were our first judge for our app, Diversion, and we were blown away by how insightful your comments were and actually in agreement with most of them. Thanks a lot for that; you’ve definitely lived up to your commitment to providing constructive feedback.
We were a bit dissappointed, though, because it seemed we were not able to get across the main strength of the app; you understood it as a ‘versioning system’ (nothing new), but that is just a feature of the app. The real power comes with the ability to work distributedly with documents, allowing other’s to “fork” (we called it divert) your documents and “merge” (we called it grab) other people’s changes into your own document (as you would do with code when working with Git). We wanted to bring to normal human beings the joy that Git brings to us as developers :-)
If you are interested on trying that side of the app, you can merge a couple of changes back into your document here, by clicking “Show” and “Grab” in the “future versions” of your document:
http://diversion.r09.railsrumble.com/users/38-Ben/documents/91/history
Just to be clear, I think we were lucky to have you as our judge and if you didn’t pick up that side of the app, it is a problem with our app!
Again thanks.
Jorge
Jorge Gomez Sancha
27 Aug 09 at 3:44 am
Hi,
I did Rakugoka – I think I made a large mistake by not working on the landing page sufficiently. I was just one guy though, compared to four on the lowdown app team.
Also – I didn’t get the exposure by being on the first teams page or blogging about making the app immediately after the competition like the lowdown guys did – lession learnt: marketing counts :).
Comments have been greatly appreciated though – it is good to get feedback. I’m keen to develop the app further after the rumble – I’m building it to be used by myself, and I’ll be interetested to see what the lowdown guys do. Two apps shows there is probably a general demand for this sort of app.
Roland Swingler
27 Aug 09 at 4:59 am
Hoi Ben!
This is Will from howireallyfeel.org. Thanks for being so positive! :D
We didn’t make it anywhere near the final leaderboard… but hearing “everything I could want and more” from one of the judges is really lovely. Thank you. :)
Will
27 Aug 09 at 9:12 pm
As one of the organizers of the event, I want to send a big thanks to Ben for participating as an expert panel member and also for this writeup. Your ratings were fair and your feedback was everything we were hoping for — constructive, thorough, and above all else, useful. Thanks! I know all the teams that you judged very much appreciated it.
nap
28 Aug 09 at 8:36 am