Tuesday, 28 April 2015

One API, Four Frameworks: The Great .NET ReST Showdown

There’s only two hard problems in software: cache invalidation, naming things, off-by-one errors, null-terminated lists, and choice overload. Back when I started building data-driven websites, classic ASP gave you Request, Response, Application, Session and Server, and you built the rest yourself - and it was uphill both ways! These days, we’re spoiled for choice. Whatever you’re trying to achieve, you can bet good money that somebody’s been there before you. If you’re lucky, they’ve shared some of their code – and if you’re really lucky, they’ve built a framework or a library you can use.

Since the heady days of classic ASP, I’ve built quite a few systems that have to expose data or services to other systems – from standalone HTTP endpoints that just return true or false, to full-featured ReST APIs. I’ve come across several frameworks that are designed to help developers create ReSTful APIs using Microsoft .NET, each of which doubtless has its own strengths and weaknesses – and now I find myself facing the aforementioned choice overload, because if I was starting a new API project right now, I honestly don’t know which framework I’d use. Whilst I’ve got hands-on experience with most of them, I’ve never had the opportunity to try implementing the same API spec in multiple frameworks to compare and contrast their capabilities on a like-for-like basis.

So here goes. Over the next few months, I’m going to be developing the same API in four different frameworks, side-by-side. The APIs have to use the same back-end code – same data access, same business logic – and have to pass the same integration test suite. I’m going to start out really simple = “hello, world!” simple – and gradually introduce features like resource expansion, pagination, OAuth2, content negotiation. The idea is that some of these features will actually break the interface, so I’ll also be looking at how to handle API versioning in each of my chosen frameworks. I’m also going to try and respect the idioms and conventions of each of the frameworks I’m working with – good frameworks tend to be opinionated, and if you don’t agree with their opinions you’re probably not going to find them particularly helpful. 

The Frameworks

Microsoft ASP.NET WebAPI (http://www.asp.net/web-api)

Microsoft’s out-of-the-box solution for building HTTP-driven APIs. Superficially similar to ASP.NET MVC but I suspect there’s much more to it than that. I’ve built a couple of small standalone APIs in WebAPI but not used it for anything too substantial.

ServiceStack (https://servicestack.net/)

For a long while I was completely smitten with ServiceStack. Then it hit v4.0 and went from free-as-in-beer to expensive-as-in-$999-per-developer for any reasonable-sized project – there is a free usage tier, but it’s pretty restrictive. That said, it’s still a really powerful, flexible framework. Version 3 is still on NuGet, is still available under a BSD license, and there’s at least one active project based on a fork of the ServiceStack v3 codebase.  I like ServiceStack’s conventions and idioms; working with it taught me a lot about ReST; it has great support for things like SwaggerUI, and I suspect as I start implementing various API features ServiceStack is going to deliver additional value and capabilities which the other frameworks can’t match. Will it add enough value to justify $999 per developer? We’ll see :)

OpenRasta (http://openrasta.org/)

I’ve played with OpenRasta on-and-off over the years, though I’ve never used it on anything substantial, but I know the folks over at Huddle are using it in a big way and having great results with it, so I’m really curious to see how it stacks up against the others. (I should probably disclose a slight bias here in that Sebastien Lambla, the creator of OpenRasta, is a friend of mine; it was Seb who first got me thinking about ReST via London .NET User Group and prolonged conversations in the pub afterwards.)

NancyFX (http://nancyfx.org/)

This one is completely new to me – until last week, I’d never even looked at it. But so far, it looks really nice – minimalist, elegant and expressive, and I’m looking forward to seeing what it can do.

Other Candidates

It would be interesting to get some F# into the mix – mainly because I’ve never used F# and I’m curious. I’ve heard interesting things about WebSharper and Freya – and, of course, if anyone wants to add an F# implementation and send me a pull request, go for it!

The API

I’m using Apiary.IO to design the API itself – you can check it out at http://docs.restival.apiary.io/

The Code

The code is on GitHub - https://github.com/dylanbeattie/Restival.

If you want to run it, you’ll need to set up IIS applications for each of the four framework implementations – I use a hosts file hack to point restival.local at 127.0.0.1, and I’ve set up http://restival.local/api.webapi, http://restival.local/api.nancy, http://restival.local/api.servicestack and http://restival.local/api.openrasta

The test suite is NUnit, uses RestSharp to make real live HTTP requests, and all the actual test logic is in the abstract base class. There’s four concrete implementations, and the only difference is the URL of the API endpoint under test.

The Backlog

Features I want to implement include, but are probably not restricted to…

  • Pagination. What’s a good way to handle huge result sets without causing timeouts and performance problems?
  • Resource expansion. How do you request a customer, and all their orders, and the invoices linked to those orders, in a single API call?
  • API versioning – using all three different ways of doing it wrong
    • Version numbers in the URL (api.foo.com/v1/)
    • Version numbers in a custom HTTP header (X-Restival-Version: 1.0)
    • Content negotation based on MIME types (Accept: application/vnd.restival.v1.0+json)
  • OAuth2 and bearer token authentication. (You’ll like this one because I’m not using DotNetOpenAuth)
  • API documentation – probably by seeing how easily I can add Swagger support to each implementation

In every case, this is stuff I’ve already implemented in at least one project over the last couple of years, so it’s going to be interesting seeing how readily those implementations translate across to the other frameworks I’m using.

Sound like fun? You bet it does. Tune in and see how it unfolds. Or come to NDC Oslo in June or Progressive.NET in London in July, where you not only get to listen to me talk about ReST, you get several days of talks and workshops from some of the best speakers in the industry.

Friday, 24 April 2015

Spotlight, Dynamics CRM, and the age-old question of “build vs buy”

We’re in the early stages of creating a new membership system built on Microsoft Dynamics CRM 2015. This isn’t a decision we’ve taken lightly – the decision to buy vs. build is always complex where software is concerned, as Mike Hadlow explains in this excellent blog post. In our case, though, there’s two good reasons why we’ve decided to integrate an off-the-shelf solution.

First, we are willing to change our own business process to suit the software. We’ve been printing books since 1927, and our business model is tightly coupled to our publishing process. As part of this project, we’re going to decouple those two areas of activity. Publishing is a differentiator for us, and always will be, but membership is not – whilst it’s important that it works, it’s not hugely important how it works. If CRM can offer us a “pit of success”, we’ll happily do what’s necessary to fall in it.

Second – we really understand what it costs to write our own software. We’ve got a solid, mature agile process, which links in to our time-tracking system. One of the high-level metrics that we track is “cost per point” – how much, in pounds, does it cost us to get a single story point into production? It’s easy for businesses to think of off-the-shelf software as “expensive”, because it has a price tag, and bespoke software as ‘free’ because you’re already paying developers’ salaries, but that’s a pretty naive way to look at it. When you factor in the opportunity cost of all the things we could have been doing instead of reinventing the CRM wheel, the off-the-shelf option starts to look a lot more attractive even when it carries a hefty up-front price tag.

That’s two good reasons why we’re going with CRM2015. Now for two good reasons why CRM projects fail, and what we’re doing to mitigate them.

First – scope creep. CRM vendors will happily sell it CRM as the solution to all your problems, and then they’ll start showing off marketing campaigns and case management and Outlook integration and web portals and everybody’s eyes light up like this is the most amazing thing they’ve ever seen… and next thing you know you’ve got a 300-page “requirements” document and everyone’s got so carried away by what’s possible that they’ve forgotten what they were trying to fix in the first place. I’ve seen this happen first-hand, and it doesn’t work, and the reason it doesn’t work is that the project isn’t being driven by prioritised requirements, it’s being driven by wish-lists.

So… start with a problem. Any successful business probably has dozens of things that could be done better – so list them, analyze them, identify dependencies. Work out which one to solve first, and focus. CRM isn’t fundamentally different to any other software project. Identify your milestones. Be absolutely brutal about the MVP – what is the simplest possible thing that’s better than what we’ve got right now? Build that. Ship that. Get people using it. In our case, CRM’s first outing is going to be as a replacement for GroupMail, our email-merge tool, and that’s it. We’ll integrate just enough data that CRM can send personalised email to a specific group of customers, we’ll ship it, and we’ll use it – and then we’ll iterate based on feedback and lessons learned. We already have a pretty good idea what we’ll do after that, but we’re not going to worry about it until we’ve delivered that first MVP release.

I think the second reason CRM projects fail is over-extension. Dynamics CRM is a really powerful, flexible platform, and with enough consultants effort you can probably get it to do just about anything. But that doesn’t mean it’s the best solution. Sure, there’s going to be cases where it makes sense to customise CRM by adding a new field or some validation rules. Spotlight holds the same core “business” data  as any other company – what’s your name? Where do you live? what’s your email address? Is your account up-to-date? Off-the-shelf CRM is very, very good at managing this sort of information – and once you’ve got this core information in CRM, there’s dozens of off-the-shelf marketing tools available to help you use it more effectively.

But Spotlight stores a lot more than that. We also store all the information that appears on your professional acting CV – height, weight, eye colour, hairstyle, skills, credits. We store details of almost all the productions being cast in the UK, we track tens of thousands of CV submissions every day, and millions of job notification emails each week. We manage terabytes of photography and video clips.  You probably could get CRM to manage all this information. But hey, you can open a beer bottle with a dollar bill – doesn’t mean it’s a good idea, though.

image

The overlap – the green bit – is where CRM solves one of our problems. The blue bit is things CRM does that aren’t really relevant to us – not at the moment, anyway. The red bit is the stuff that we’re going to keep out of CRM. And that dark area on the border… that’s representation, which is a tremendously complicated tangle of operational data, business data and publishing data that we’re going to have to work out as part of our service roadmap. Fun.

Now, different people – and systems! – have different expectations about what “CRM” and “membership” mean.

  • To our customers, good CRM means it’s easy to join Spotlight, it’s easy to manage your account, it’s easy to talk to us and get answers to your questions. You know how you hate phoning the electric company because every time you get through you talk to a different person who has no idea what’s going on, and your “my account” page on their website says one thing and your bill says something else and the person on the phone doesn’t agree with either of them? Yeah. Imagine the complete opposite of that.
  • To our marketing team, good CRM is about accurate data, effective marketing campaigns, happy customers – and, yes, revenue. It’s about helping us work out what we’re doing right and what we’re doing wrong, giving us the intelligence we need to make decisions about new products and initiatives.
  • To our software team,  good CRM means easy access to the data and processes you’ll need to build great products. Responsive systems, logical data structures, simple integration patterns and intuitive API endpoints. In other words, if you want to build an awesome online tool that’s only available to customers with a current, paid-up Spotlight Actors membership, it should be trivial to work out whether the current user can see it or not.

Same system, same data, three radically different use cases. So here’s how we’re proposing to make it work:

image

Astute readers will notice a slim black box marked “abstraction layer”. That’s how we’re going to fool the rest of our stack into thinking that Dynamics CRM 2015 is a ReSTful microservice, so tune in over the next couple of weeks to find out how it works, what sort of patterns and techniques we’re using, and how we’re going to test and monitor it.

(Astute readers may also notice a resemblance between Spotlight’s customer base and the cast of Game of Thrones… well, that’s because Spotlight’s customers are the cast of Game of Thrones. I told you working here was awesome.)

Spotlight, Dynamics CRM, and the age-old question of “build vs buy”

We’re in the early stages of creating a new membership “engine” built on Microsoft Dynamics CRM 2015. This isn’t a decision we’ve taken lightly – the decision to buy vs. build is always complex where software is concerned, as Mike Hadlow explains in this excellent blog post. In our case, though, there’s two good reasons why we’ve decided to integrate an off-the-shelf solution.

First, we are willing to change our own business process to suit the software. We’ve been printing books since 1927, and our business model is tightly coupled to our publishing process. As part of this project, we’re going to decouple those two areas of activity. Publishing is a differentiator for us, and always will be, but membership is not – whilst it’s important that it works, it’s not hugely important how it works. If CRM can offer us a “pit of success”, we’ll happily do what’s necessary to fall in it.

Second – we really understand what it costs to write our own software. We’ve got a solid, mature agile process, which links in to our time-tracking system. One of the high-level metrics that we track is “cost per point” – how much, in pounds, does it cost us to get a single story point into production? It’s easy for businesses to think of off-the-shelf software as “expensive”, because it has a price tag, and bespoke software as ‘free’ because you’re already paying developers’ salaries, but that’s a pretty naive way to look at it. When you factor in the opportunity cost of all the things we could have been doing instead of reinventing the CRM wheel, the off-the-shelf option starts to look a lot more attractive even when it carries a hefty up-front price tag.

That’s two good reasons why we’re going with CRM2015. Now for two good reasons why CRM projects fail, and what we’re doing to mitigate them.

First – scope creep. CRM vendors will happily sell it CRM as the solution to all your problems, and then they’ll start showing off marketing campaigns and case management and Outlook integration and web portals and everybody’s eyes light up like this is the most amazing thing they’ve ever seen… and next thing you know you’ve got a 300-page “requirements” document and everyone’s got so carried away by what’s possible that they’ve forgotten what they were trying to fix in the first place. I’ve seen this happen first-hand, and it doesn’t work, and the reason it doesn’t work is that the project isn’t being driven by prioritised requirements, it’s being driven by wish-lists.

So… start with a problem. Any successful business probably has dozens of things that could be done better – so list them, analyze them, identify dependencies. Work out which one to solve first, and focus. CRM isn’t fundamentally different to any other software project. Identify your milestones. Be absolutely brutal about the MVP – what is the simplest possible thing that’s better than what we’ve got right now? Build that. Ship that. Get people using it. In our case, CRM’s first outing is going to be as a replacement for GroupMail, our email-merge tool, and that’s it. We’ll integrate just enough data that CRM can send personalised email to a specific group of customers, we’ll ship it, and we’ll use it – and then we’ll iterate based on feedback and lessons learned. We already have a pretty good idea what we’ll do after that, but we’re not going to worry about it until we’ve delivered that first MVP release.

I think the second reason CRM projects fail is over-extension. Dynamics CRM is a really powerful, flexible platform, and with enough consultants effort you can probably get it to do just about anything. But that doesn’t mean it’s the best solution. Sure, there’s going to be cases where it makes sense to customise CRM by adding a new field or some validation rules. Spotlight holds the same core “business” data  as any other company – what’s your name? Where do you live? what’s your email address? Is your account up-to-date? Off-the-shelf CRM is very, very good at managing this sort of information – and once you’ve got this core information in CRM, there’s dozens of off-the-shelf marketing tools available to help you use it more effectively.

But Spotlight stores a lot more than that. We also store all the information that appears on your professional acting CV – height, weight, eye colour, hairstyle, skills, credits. We store details of almost all the productions being cast in the UK, we track tens of thousands of CV submissions every day, and millions of job notification emails each week. We manage terabytes of photography and video clips.  You probably could get CRM to manage all this information. But hey, you can open a beer bottle with a dollar bill – doesn’t mean it’s a good idea, though.

image

The overlap – the green bit – is where CRM solves one of our problems. The blue bit is things CRM does that aren’t really relevant to us – not at the moment, anyway. The red bit is the stuff that we’re going to keep out of CRM. And that dark area on the border… that’s representation, which is a tremendously complicated tangle of operational data, business data and publishing data that we’re going to have to work out as part of our service roadmap. Fun.

Now, different people – and systems! – have different expectations about what “CRM” and “membership” mean.

  • To our customers, good CRM means it’s easy to join Spotlight, it’s easy to manage your account, it’s easy to talk to us and get answers to your questions. You know how you hate phoning the electric company because every time you get through you talk to a different person who has no idea what’s going on, and your “my account” page on their website says one thing and your bill says something else and the person on the phone doesn’t agree with either of them? Yeah. Imagine the complete opposite of that.
  • To our marketing team, good CRM is about accurate data, effective marketing campaigns, happy customers – and, yes, revenue. It’s about helping us work out what we’re doing right and what we’re doing wrong, giving us the intelligence we need to make decisions about new products and initiatives.
  • To our software team,  good CRM means easy access to the data and processes you’ll need to build great products. Responsive systems, logical data structures, simple integration patterns and intuitive API endpoints. In other words, if you want to build an awesome online tool that’s only available to customers with a current, paid-up Spotlight Actors membership, it should be trivial to work out whether the current user can see it or not.

Same system, same data, three radically different use cases. So here’s how we’re proposing to make it work:

image

Astute readers will notice a slim black box marked “abstraction layer”. That’s how we’re going to fool the rest of our stack into thinking that Dynamics CRM 2015 is a ReSTful microservice, so tune in over the next couple of weeks to find out how it works, what sort of patterns and techniques we’re using, and how we’re going to test and monitor it.