Sunday, 31 May 2009

Speaking at SkillsMatter Open Source .NET Exchange III

image I’ll be doing a short spot about the Web Platform Installer and msdeploy at the forthcoming SkillsMatter Open Source .NET Exchange evening here in London on July 16th. It’s a kind of “bonsai conference” where we get through six or seven fifteen-minute topics in one evening, giving a sort of whirlwind tour of what’s new and what’s cool in the open source .NET development world.

The Web Platform Installer (below) is a slick GUI installer that’ll download and configure everything you’ll need to get started building and hosting web apps on Microsoft systems – if you’re just getting started with web development, or you want to check out one of the popular open-source frameworks like Subtext, Drupal or DotNetNuke, it’ll set up everything you need in a couple of clicks, including Visual Studio Express, SQL Server express, PHP, and whatever else you’ll need to run your chosen apps and frameworks. And yes, it really is as quick and easy as the screenshot makes it look.


msdeploy is the command-line packaging and deployment engine used internally by the Web Platform Installer, but you can also use msdeploy directly to update, package, publish and manage your own servers. It’ll let you publish updated files, configurations, virtual directories, script handlers – basically your entire web application, set up and ready to run –between servers. We’re using it to publish changes from our staging server to our live servers, and it’s an absolute delight to work with.

We’ve also got Ian Cooper talking about Boo, Scott Cowan talking about the Spark view engine, David Ross on MPI.NET, Gojko talking about acceptance testing with  Concordion.NET, Phil Trelford showing off units of measure in F#, and Seb talking about his rapidly-evolving REST framework, OpenRasta. It should be a good evening with plenty of time for food, beer and chat with like-minded developers. The last one was extremely popular so if you want to come along, please register and we’ll see you there.

Room Service Oriented Architecture

This is a story about breakfasts.Sunrise on the Chobe River channel (c) bjmccray via Flickr

Well, OK, it’s actually a story about hotels and software architecture. But, like anything worth doing, it starts with two breakfasts. (You all know about second breakfast, right?)

The first was the breakfast I ate on my ninth birthday, cooked by my dad on the banks of the Chobe river in Botswana. We were on a camping trip, at a remote spot in the middle of nowhere – no roads, no phones, no shops, no electricity - so to cook this particular breakfast, my folks had brought eggs, bacon, sausages, bread, butter, cooking oil – you name it, they had it, along with a gas-powered fridge to keep it all fresh, and a big tank of water to wash up afterwards, and a gas stove to cook it on, and matches to light the stove, and countless other necessities that were all written down on a big long list taped to the inside of Dad’s camping trailer. This single breakfast, like every meal we’d eat out out there in the bush, involved weeks of planning, preparation, equipment, checklists – did we have enough food? Would it stay fresh long enough? Had we brought enough gas bottles to run the fridge and still have enough left to cook with? Was it safe to wash up with boiled river-water or did we need to use bottled stuff? When it’s just you and your tent in the middle of nowhere, you have to cover every single detail – and if you pack two weeks worth of bacon and then wake up one day and fancy porridge, you’re screwed. It’s bacon or nothing.

They apparently have wildlife in the Big Apple as well. The second breakfast was eaten on the 44th floor of a Manhattan hotel, sometime in 2004. Organising this breakfast involved picking up a phone and saying to Steph in room service "Hey, can you send up some pancakes and coffee?” and then going back to sleep until someone knocked at the door with – you guessed it - pancakes and coffee. Planning involved? Zero. Which was just as well; as I’d discovered the previous night, New York does a pretty good line in cocktails as well, and if I’d needed to plan anything whatsoever that morning, I’d probably have given up and gone hungry.

So, let’s compare the two scenarios. In both cases, someone’s making a request for breakfast, and someone is then responding to that request – ask for breakfast, wait a while, get breakfast. Simple. Thing is, you can clearly see that for my dad, responding to this particular request involves an awful lot of work, whereas Steph probably doesn’t even need to stand up. And the irony is that for all his hard work and planning, the only thing Dad can actually respond with is bacon and eggs, whereas Steph – who probably can’t even cook – can do bacon, eggs, orange juice, pastries, espresso, fresh fruit and even those famous silver-dollar pancakes covered in sugar. And martinis. And probably arrange a shoe-shine, a haircut, a massage and get your tux dry-cleaned while she’s about it.

If you’re only here to hear about breakfast, you can leave now; it’s about to get a bit geeky. Still here? Cool, OK, here we go.

If you’ve built web applications for any length of time, at some point in your career you probably wrote something like an ASP page that connects to a database, reads some files, sends an e-mail or two, updates a database, does some validation, resizes an image, saves it to disk, generates a bunch of HTML and returns it to the client – all in one file. It’s OK, you can admit it; we’re all friends here. For quite a long time, I wrote most web apps this way. At the time it all seemed terribly clever, and if you stuck at it long enough, you normally ended up with something that did the job, paid the bills and kept the client reasonably happy. Problem is, this is normally the client who thinks he has millions of potential customers, but actually the load on his Web 1.0 B2B portal is him, his business partner, and about four guys in Russia who they persuaded to sign up. The wonderful thing about scaling issues is that when you get them in your software, it normally means your business is doing OK. Assuming, of course, you have a business model in the first place…

Anyway. Point is, trying to apply this everything-in-one-page approach in any large-scale app is like using my dad’s catering strategy to make breakfast for 1,200 people. Last time I looked, the only people who set off into the middle of nowhere with fifty tons of bacon and eggs were the Army - and whatever your thoughts on the armed forces, I’m sure you’ll agree that if we’re trying to learn about minimalist efficiency and cost-effectiveness, we’re probably better off looking elsewhere.

So, how does Steph arrange a couple of hundred breakfasts a day using just a phone – and what can we learn from her about designing scalable web apps?

What’s happening is that Steph’s acting as a front controller in a service-oriented architecture. The front controller’s job is to grab the incoming request, call whichever services are necessary to fulfil that request, and tell them what to do with the result.

Steph has a single responsibility – she converts customer requests (imagine Peter from Family Guy ordering breakfast: “yeah, uh, do you have, like… ah… hang on… er… like, pancakes? With syrup? And some tea? No, wait, coffee… no, tea… no coffee. Yeah. Coffee. Did I get pancakes?”) into short, snappy, precise kitchen orders (“Mikey, two pancakes, two coffee for forty-four-oh-seven”). Mikey is the breakfast chef, and he’s just one of the services that Steph can delegate to. Someone orders a martini, she can delegate to Raoul, the bartender service. Someone orders a shoeshine, she can delegate to Chip the shoe-shine guy.

Like Steph, Mikey has a single responsibility. He turns short, snappy kitchen orders into plates full of food, and he does it quickly. He doesn’t buy the ingredients, he doesn’t serve the food – he delegates those responsibilities to other services. Like Tony, the delivery service, who turns up at 4am every day with a truck full of fresh supplies. Or Chet, the bellhop service, who takes the plates, puts them on one of those funny little trolley-tables with some cutlery, and takes them up to 4407 where I’m waiting for my breakfast.

Hotels can teach us a lot about scaling, because they’re good at it – they’ve been in the hotel business a lot longer than we’ve been in the software business, after all.

Make Your Controllers Delegate Heavy Lifting to Services

Think about a typical hotel; there’s probably only four or five points of customer interaction – say the front desk, the concierge, the room service desk, and the bar. There’s probably only one or two people at each desk, but because they’re working as part of a service-oriented architecture, they can fulfil a huge number of incredibly diverse requests, whilst keeping their own responsibilities simple, well-defined and manageable. Think of your controllers like the customer service desks in a hotel – clean, friendly, and completely dependent on supporting services to actually get anything done. That’s how it should be.

Cache Whatever You Can (Within Reason)

For a hotel, this is the early-morning deliveries of food, booze, fresh linen, theater tickets – stuff that you’re probably going to need sometime today, and you don’t want clients sat around waiting whilst you have to run out and fetch it. For web apps, this means setting up local caches of images, persistent data, resource files. But don’t go crazy. Just like hotels don’t order a ten-year supply of lamb chops, you’ll need to consider how much space your cache consumes, and how long it’ll stay fresh before you need to reload it.

Acknowledge Now; Deliver Later

The room-service desk doesn’t stay on the line yakking with you until your pancakes arrive. They tell you it’ll be twenty minutes, and then hang up so they can take the next call – if they didn’t, they’d need fifty room service operators instead of one. You can use asynchronous processing, message queues and Ajax callbacks to achieve the same thing in web applications – when you get a request that’s going to take a while, respond immediately saying it’ll take a while, and let your users get on with something else whilst it’s processing. And be careful with the Refresh button. If I call down to see where my pancakes have got to, it doesn’t mean I want to order more pancakes; I’m just chasing up the first batch.

Tuesday, 12 May 2009

What If SQL Had Abstract Tables And Composite Types?

There are well-documented mismatches between the object-oriented paradigm and the relational database model. Despite the early adopters telling us we should all be using F# running against CouchDB, I’m going to go out on a limb and say that object-oriented languages and relational databases aren’t going away any time soon.

The more I see and learn about ORMs, the more I get this feeling like there’s a real “code vs data” mindset going on. To an object-oriented application developer, the database is a necessarily evil, the DBA is a pain-in-the-ass Luddite who won’t let you write code the way you want to, tables are an inadequate and inferior persistence mechanism for your beautiful object hierarchies, and – well, if someone could wave a magic wand and just make the whole database thing go away, you’d all be very happy. Even in the most wonderfully agile, test-driven greenfield project, there’s still this perception that the more we can abstract the project away from the database, the better off we’ll be. It’s like the Utopian ideal of ORMs is not to embrace the power of the database – they’d rather make the DB completely irrelevant.

I sit on both sides of this particular fence. As a developer, I find persistence as frustrating and time-consuming as everyone else – but as a business stakeholder, I want my data stored in normal, sensibly-named tables that I can get at using SQL-92 queries. Not because that’s what I’m doing now, but because I really have no idea what I’ll be doing with that data in 5-10 years, and I believe the relational model has stood the test of time. 

I’ve spent today at  Ayende’s NHibernate workshops at the Skillsmatter Progressive .NET event, and he’s shown us all sorts of NHibernate magic, including various strategies for modelling inheritance and class hierarchies with polymorphic associations, sparse tables, and the like. There’s been some discussion of stuff like composite keys, searching, indexes, unique constraints – and it’s pretty clear that when it comes to object-relational mapping, it’s the object folks that are doing all the work, and the relational side of things is really not doing anything much to make their lives easier.

There’s two things that stand out as being particular painful when it comes to object-relational mapping – composite keys and inheritance. Inheritance because it has no analogue in the relational model so you have to somehow “fake” it – which makes no difference to the code, but it can leave the DB in a bit of a mess. Composite keys because they have built-in limitations regarding nested queries (WHERE foo IN (SELECT…)) – and complex equality and value semantics, which means most ORMs strongly advise against using composite keys if you can help it.

(From this point forward I’m making stuff up as a sort of thought experiment. Don’t take any of it too literally :))

What if we added ABSTRACT and EXTENDS keywords to SQL? What if you could do this in your database?

  ID int identity(1,1) primary key

  CompanyName varchar(256)

   Forenames varchar(256),
   Surname varchar(256)

Here’s the rules:

  • Tables must be abstract or concrete.
  • You can’t insert, update or delete records from an abstract table
  • Both abstract and concrete tables can participate in foreign key relationships
  • One abstract record must have exactly one concrete record (i.e. you can’t insert a Company whose ID already represents a Person)
  • Concrete tables behave like the union of their own columns and their abstract base table’s columns

SELECT * FROM Customer




Id CompanyName
1 Monkey Butlers Ltd


Id Forenames Surname
2 Eddie Van Halen
3 Jack Sparrow


INSERT INTO Person(Id, Forenames, Surname) VALUES(4, “Ayende”, “Rahien”)

-- will work as expected

INSERT INTO Company(Id, Name) VALUES(4, “Rhino Ltd”)

-- will fail with “Distributed key violation” or some such thing, because ID4 is already taken by a Person and so can’t be used for a Company.

You can, e.g. associate Customer with Address (so every address is ‘owned’ by exactly one customer, regardless of whether the customer in question is a person or a company) – but you can also associate Employee with Company directly, so that your data schema enforces the business requirement that a Person cannot have Employees. As a database concept, it doesn’t really add anything – but with a suitably turbocharged ORM, think of what you could do. A simple Customer.ListAll() could return an array of Customer objects – each of which is ACTUALLY a strongly-typed Person or Company. You could do things like Customer.ListAll(typeof(Company)); you could switch on the type of the objects – and all without the compromise of sparse tables or child table inheritance.

Ok, what about composite keys? Imagine we’re booking flights for an airline. There is a real-world business constraint that the same person cannot be on the same flight twice, and composite keys are tailor-made for modelling this sort of “unique combination” scenario:

CREATE TABLE SeatReservation (
      PassengerId int,
      FlightNumber varchar(5)
   ) primary key,
   SeatRow char,
   SeatNumber int

CREATE TABLE SpecialMeal (

  -- Notice that the “type” of this column is a SeatReservation.Key – i.e. a reference
  -- to the composite type defined as Key in the SeatReservation table.
   SeatReservation.Key SeatReservation primary key,
   IsVegetarian bit,
   IsVegan bit,
   IsHalal bit

SELECT * FROM SeatReservation WHERE Key.PassengerId = 12

SELECT * FROM SeatReservation WHERE Key IN  (SELECT SeatReservationKey FROM SpecialMeal)

Think of this like adding a struct type to SQL, where COMPOSITE defines a collection of columns just as in C# a struct defines a collection of fields. Composites carry field-value semantics for comparison and equality; if all the columns are equal, the struct is equal. Composite literals are defined inline like:

select * from SpecialMeal WHERE SeatReservation.Key = (178189, ‘VS207’)

- using similar syntax to SQL’s familiar INSERT INTO (Column1, Column2) VALUES(Value1, Value2)

This would allow ORMs such as NHibernate to explicitly map a composite key as a struct, benefiting from intrinsic value-object semantics. The resulting database queries could use composite keys in exactly the same way as primary keys – because the database explicitly allows a composite ‘type’ to be used wherever a primitive type is currently supported – and, most importantly, the business meaning of a composite key is explicit in both the database schema and the object model.

Saturday, 9 May 2009

Offsite Backup Recommendations for Really, Really Big Files?

Old media 2 by john_a_ward. via Flickr.

When I’m not coding and being all nerdy, I play guitar and mess around with digital audio recording. Uncompressed digital audio is stored in the same format as old-fashioned audio CDs – 74 mins of raw audio takes up 740Mb or thereabouts. This means that a good afternoon’s recording can easily create 2-3Gb of files – and they’re not like MP3s or movies that are easy enough to replace; these are often the only copy in the world of that one killer solo or that bass riff that I’ve been practising for weeks and just happened to nail it this afternoon.

Stuff like Carbonite or Mozy is all well and good for documents and holiday snaps, but given I have consumer ADSL (512Kb upstream), is there even any point thinking about offsite backup for my audio projects, or am I better off just copying stuff to a USB hard drive and maybe taking a drive into work every couple of months? What do people do with video / post-processing stuff? What’s your backup strategy when the actual work you’re doing is creating gigabytes of files on a daily basis?

[photo from john_a_ward via flickr – thanks for sharing, John.]

Unsubscribe (aka ‘the Internet Way of Saying “It’s Not You, It’s Me.”’)

My personal inbox is full of opt-in marketing… I receive newsletters about hotels, concerts, travel insurance, car insurance, DVDs, cheap flights… seems you can’t buy anything these days without ending up on their list. Spam is one thing, but most of this is stuff I have actually signed up for – albeit unwittingly – over the years. You know the deal - “by purchasing this motor insurance you agree to receive blah blah blah from time to time” and you never get around to opting-out?

So I spent a happy Saturday morning unsubscribing from every piece of legitimate-but-unwanted marketing material in my inbox… and it’s remarkable how diverse something as simple as the unsubscribe experience can get. Here is how unsubscribe should work. I want to click an unsubscribe link. I do not want to log in and update my preferences. I want to click one link. I want to get a page that’s looks like it’s somehow connected to the organisation who are sending me this stuff – even if it’s just a logo at the top of the page. I want a message informing me that my e-mail address has been removed. I see that, I feel all happy, like you care about me, and you care about your brand, and I’ll come back and do business with you again, because I know you’re happy to let me walk away if I want to.


It’s amazing how many unsubscribe links go to a page called unsubscribe.htm – htm! - on unrelated domain (some third-party mailing list provider) with no branding or connection to the company you’re dealing with, and give you some completely impersonal “Your request has been acknowledged” message. Don’t acknowledge my request – take me off the damn list already. “Acknowledging my request” is right up there with my call being important to you, your being an equal opportunities employer, and your product helping weight loss as part of a calorie controlled diet.

Oh – and when I click the same link again, I’d love a message saying “you’re already unsubscribed”. Unsubscribe should not be idempotent. If they really removed you from the list the first time, what exactly are they doing the second time?

PS: Whilst we’re talking usability and logging in and stuff, how many times has this happened to you?

  1. Go to website login form
  2. Enter your e-mail address and password… “sorry, incorrect password”
  3. Click “Forgotten password”…
  4. …and have to enter your e-mail address again?

I just told you my e-mail address… is it too much to ask that you remember it for fifteen seconds?