Monday, 31 July 2017

Deployment Through the Ages

Vanessa Love just posted this intriguing little snippet on Twitter:

And I got halfway through sticking some notes into the Google doc, and then thought actually this might make a fun blog post. So here’s how deployment has evolved over the 14 years since I first took over the hallowed mantle of [email protected].

2003: Beyond Compare (maybe?)

The whole site was classic ASP – no compilation, no build process, all connection credentials and other settings were managed as application variables in the global.asa file. On a good day, I’d get code running on my workstation, test it in our main target browsers, and deploy it using a visual folder comparison tool. It might have been Beyond Compare; it might have been something else. I honestly can’t remember and the whole thing is lost in the mists of time. But that was basically the process – you’d have the production codebase on one half of your screen and your localhost codebase on the other half, and you’d cherry-pick the bits that needed to be copied across.

Of course, when something went wrong in production, I’d end up reversing the process – edit code directly on live (via UNC share), normally with the phone wedged against my shoulder and a user on the other end; fix the bug, verify the user was happy, and then do a file sync in the other direction to get everything from production back onto localhost. Talk about a tight feedback loop – sometimes I’d do half-a-dozen “deployments” in one phone call. It was a simpler time, dear reader. Rollback plan was to hammer Ctrl-Z until it’s working again; disaster recovery was tape backups of the complete source tree and database every night, and the occasional copy’n’paste backup of wwwroot before doing something ambitious.

Incidentally, I still use Beyond Compare almost daily – I have it configured as my merge tool for fixing Git merge conflicts. It’s fantastic.

2005: Subversion

Once we hired a second developer (hey Dan!) the Beyond Compare approach didn’t really work so well any more, so we set up a Subversion server. You’d get stuff running on localhost, test it, maybe share an link (hooray for local wildcard DNS) so other people could see it, and when they were happy, you’d do an svn commit, log into the production web server (yep, the production web server – just the one!) and do an svn update. That would pull down the latest code, update everything in-place. There was still the occasional urgent production bugfix. One of my worst habits was that I’d fix something on production and then forget to svn commit the changes, so the next time someone did a deployment (hey Dan!) they’d inadvertently reintroduce whatever bug had just been fixed and we’d get upset people phoning up asking why it was broken AGAIN.

2006: FinalBuilder

This is where we start doing things with ASP.NET in a big way. I still dream about OnItemDataBound sometimes… and wake up screaming, covered in sweat. Fun times. The code has all long since been deleted but I fear the memories will haunt me to my grave.

Anyway. By this point we already had the Subversion server, so we had a look around for something that would check out and compile .NET code, and went with FinalBuilder. It had a GUI for authoring build pipelines and processes, some very neat features, and could deploy .NET applications to IIS servers. This was pretty sophisticated for 2006. 

2008: test server and msdeploy

After one too many botched FinalBuilder deployments, we decided that a dedicated test environment and a better deployment process might be a good idea. Microsoft had just released a preview of a new deployment tool called MSDeploy, and it was awesome. We set up a ‘staging environment’ – it was a spare Dell PowerEdge server that lived under my desk, and I’d know when somebody accidentally wrote an infinite loop because I’d hear the fans spin up. We’d commit changes to Subversion, FinalBuilder would build and deploy them onto the test server, we’d give everything a bit of a kicking in IE8 and Firefox (no Google Chrome until September 2008, remember!) and then – and this was magic back in 2008 – you’d use msdeploy.exe to replicate the entire test server into production! Compared to the tedious and error-prone checking of IIS settings, application pools and so on, this was brilliant. Plus we’d use msdeploy to replicate the live server onto new developers’ workstations, which was a really fast, easy way to get them a local snapshot of a working live system. For the bits that still ran interpreted code, anyway.

2011: TeamCity All The Things!

By now we had separate dev, staging and production environments, and msdeploy just wasn’t cutting it any more. We needed something that can actually build different deployments for each environments – connection strings, credentials, and so on. And there’s now support in Visual Studio for doing XML configuration transforms, so you create a different config file for every environment, check those into revision control, and get different builds for each environment. I can’t remember exactly why we abandoned FinalBuilder for TeamCity, but it was definitely a good idea – TeamCity has been the backbone of our build process ever since, and it’s a fantastically powerful piece of kit.

2012: Subversion to GitHub

At this point, we’d grown from me, on my own doing webmaster stuff, to a team of about six developers. Even Subversion is starting to creak a bit, especially when you’re trying to merge long-lived branches and getting dozens of merge conflicts, so we start moving stuff across to GitHub. It takes a while – I’m talking months – for the whole team to stop thinking of Git as ‘unnecessarily complicated Subversion’ and really grok the workflow, but we got there in the end.

Our deployment process at this point was to commit to the Git master branch, and wait for TeamCity to build the development version of the package. This would get built and deployed. Once it was tested, you’d use TeamCity to build and deploy the staging version – and if that went OK, you’d build and deploy production. Like very step on this journey, it was better than anything we’d had before, but had some obvious drawbacks. Like the fact we had several hundred separate TeamCity jobs and no consistent way of managing them all.

2013: Octopus Deploy and Klondike

When we started migrating from TeamCity 6 to TeamCity 7, it became rapidly apparent that our “build everything several times” process… well, it sucked. It was high-maintenance, used loads of storage space and unnecessary CPU cycles, and we needed a better system.

Enter Octopus Deploy, whose killer feature for us was the ability to compile a .NET web application or project into a deployment NuGet package (an “octopack”), and then apply configuration settings during deployment. We could build a single package, and then use Octopus to deploy and configure it to dev, staging and live. This was an absolute game-changer for us. We set up TeamCity to do continuous integration, so that every commit to a master branch would trigger a package build… and before long, our biggest problem was that we had so many packages in TeamCity that the built-in NuGet server started creaking.

This started life as an experimental build of themotleyfool/NuGet.Lucene – which we actually deployed onto a server we called “Klondike” (because klondike > gold rush > get nuggets fast!) – and it worked rather nicely. Incidentally, that NuGet.Lucene library is now the engine behind themotleyfool/Klondike, a full-spec NuGet hosting application – and I believe our internal hostname was actually the inspiration for their project name. That was a lot of fun for the 18 months or so that Klondike existed but we were still running the old NuGet.Lucene codebase on a server called ‘klondike’. It’s OK, we’ve now upgraded it and everything’s lovely.

It was also in 2013 that we started exploring the idea of automatic semantic versioning – I wrote a post in October 2013 explaining how we hacked together an early version of this. Here’s another post from January 2017 explaining how it’s evolved. We’re still working on it. Versioning is hard.

And now?

So right now, our build process works something like this.

  1. Grab the ticket you’re working on – we use Pivotal Tracker to manage our backlogs
  2. Create a new GitHub branch, with a name like 12345678_fix_the_microfleems – where 12345678 is the ticket ID number
  3. Fix the microfleems.
  4. Push your changes to your branch, and open a pull request. TeamCity will have picked up the pull request, checked out the merge head and built a deployable pre-release package (on some projects, versioning for this is completely automated)
  5. Use Octopus Deploy to deploy the prerelease package onto the dev environment. This is where you get to tweak and troubleshoot your deployment steps.
  6. Once you’re happy, mark the ticket as ‘finished’. This means it’s ready for code review. One of the other developers will pick it up, read the code, make sure it runs locally and deploys to the dev environment, and then mark it as ‘delivered’.
  7. Once it’s delivered, one of our testers will pick it up, test it on the dev environment, run it past any business stakeholders or users, and make sure we’ve done the right thing and done it right.
  8. Finally, the ticket is accepted. The pull request is merged, the branch is deleted. TeamCity builds a release package. We use Octopus to deploy that to staging, check everything looks good, and then promote it to production.

And what’s on our wishlist?

  • Better production-grade smoke testing. Zero-footprint tests we can run that will validate common user journeys and scenarios as part of every deployment – and which potentially also run as part of routine monitoring, and can even be used as the basis for load testing.
  • Automated release notes. Close the loop, link the Octopus deployments back to the Pivotal tickets, so that when we do a production deployment, we can create release notes based on the ticket titles, we can email the person who requested the ticket saying that it’s now gone live, that kind of thing.
  • Deployments on the dashboards. We want to see every deployment as an event on the same dashboards that monitor network, CPU, memory, user sessions – so if you deploy a change that radically affects system resources, it’s immediately obvious there might be a correlation.
  • Full-on continuous deployment. Merge the PR and let the machines do the rest.

So there you go – fourteen years worth of continuous deployments. Of course, alongside all this, we’ve moved from unpacking Dell PowerEdge servers and installing Windows 2003 on them to running Chef scripts that spin up virtual machines in AWS and shut them down again when nobody’s using them – but hey, that’s another story.

Thursday, 27 July 2017

Securing Blogger with CloudFlare and HTTPS

As you may have read, life is about to get a whole lot harder for websites without HTTPS. Now this site is hosted on Blogger – I used to run my own MovableType server, but I realised I was spending way more time messing around with the software than I was actually writing blog posts, so I shifted the whole thing across to Blogger about a decade ago and never really looked back.

One of the limitations of Blogger is that it doesn’t support HTTPS if you’re using custom domains – there’s no way to install your own certificate or anything. So, since Chrome’s about to crank up the warnings for any websites that don’t use HTTPS, I figured I ought to set something up. Enter CloudFlare, who are really rather splendid.

First, you sign up. (bonus points for them NOT forcing you to choose a password that contains a lowercase letter, an uppercase letter, a number, a special character, the poo emoji and the Mongolian vowel separator).

Second, you tell them which domain you want to protect:


They scan all your DNS records, which takes about a minute – and not only is there a nice real-time progress bar keeping you in the loop, they use this opportunity to play a really short video explaining what's going on. I think this is absolute genius.


Finally, after checking it's picked up all your DNS records properly (it had), you tell your domain registrar to update the nameservers for your domain to CloudFlare's DNS servers, give it up to 24 hours, and you're done. Zero downtime, zero service interruption – the whole thing was smooth, simple, and completely free-as-in-beer.

Yes, I realise this does not encrypt content end-to-end. For what we're doing here, this is absolutely fine. It'll secure your traffic against dodgy hotel wi-fi and unscrupulous internet service providers - and if anyone's genuinely intercepting HTTP traffic between CloudFlare and Google, I'm sure they can think of more exciting things to do with it than mess around with my blog posts.

Having done that, I then had to use the Google Chrome console to track down the resources – photos and the odd bit of script – that were being hosted via HTTP, and update them to be HTTPS. The only thing I couldn't work out how to fix was the search bar that's embedded in Blogger's default page layout – it's injected by JavaScript, it's hosted by Google's CDN (so I can't use any of CloudFlare's clever rewriting tricks to fix it), it's stuck inside an IFRAME, and it points to – see the plain HTTP with no S?


After an hour or so of messing around with CSS, I gave up, posted a question on the ProWebmasters Stack Exchange, and – of course, immediately found the solution; go into Blogger, Layout, find the Navbar gadget, click Edit, and there's an option to switch the nav off entirely.

So there you go. Thanks to CloudFlare, now has a green padlock on it. I don't know about you, but I take comfort in that.


Friday, 21 July 2017

Summer 2017 .NET Community Update

Summer here in the UK is normally pretty quiet, but this year there's so much going on around .NET and the .NET community that I thought this would be a great opportunity to do a bit of a round-up and let you all know about some of the great stuff that's going on.

First, there's the news of two new .NET user groups starting up in southern England. Earlier this week, I was down in Bournemouth speaking at the first-ever meetup of the new .NET Bournemouth group, and thoroughly enjoyed it. Three speakers – Stuart Blackler, Tommy Long and me – with talks on leadership, agile approaches to information security, and an updated version of my "happy code" talk I've done at a few conferences already this year. The venue and A/V setup worked flawlessly, there was a strong turnout, and some really good questions and discussion after each of the talks – I think it's going to turn out to be a really engaging group, so if you're in that part of the world, stop by and check them out. Their next few meetup dates are on already.

Next month, Steve Gordon is starting a new .NET South East group based in Brighton, who will be kicking off with their inaugural meetup on August 22nd with Steve talking about Docker for .NET developers.

Brighton based .NET South East user group logo

There's a great post on Steve's blog explaining what he's doing and what he's hoping to get out of the group, and they're also on (and I have to say, they've done an excellent job of branding the Meetup site – nice work!)

It's an exciting time for .NET – between the cross-platform stuff that's happening around Xamarin and .NET Core, new tooling like JetBrains Rider and Visual Studio Code, and the growing number of cloud providers who are supporting C# and .NET Core for building serverless cloud applications, we've come a long, long way from the days of building Windows Forms and databinding in Visual Studio .NET.

If you're interested in really getting to grips with the future of .NET, join us at the Progressive.NET Tutorials here in London in September. With a great line-up of speakers including Julie Lerman, Jon Skeet, Jon Galloway, Clemens Vasters and Rachel Appel – plus Carl Franklin and Rich Campbell from DotNetRocks, and a few familiar faces you might recognise from the London.NET gang – it promises to be a really excellent event. It goes a lot deeper than most conferences – with one day of talks and two days of hands-on workshops, the idea is that attendees don't just go away with good ideas, they actually leave with running code, on their laptops, that they can refer back to when they take those ideas back to the office or to their own projects. Check out the programme, follow #ProgNET on Twitter, and hopefully see some of you there. 

Then on Saturday 16th September – the day after Progressive.NET – is the fourth DDD East Anglia community conference in Cambridge. Their call for speakers is now closed, but voting is open until July 29th – so sign up, vote on the sessions you want to see – or just vote for mine if you can't make your mind up ;) - and hopefully I'll see some of you in Cambridge.

t_shirt_logo_thumb[23]Finally, just in case any readers of this blog DON'T know about the London .NET User Group… yep, we have .NET User Group! In London! I know, right? We're on, and on Twitter as @LondonDotNet, and we meet every month at SkillsMatter's CodeNode building near Moorgate.

Our next meetup is on August 8th, with Ana Balica talking about the history and future of HTTP and HTTP/2, and Steve Gordon – and on September 12th we've got Rich Campbell joining us for a Progressive.NET special meetup and presenting the History of .NET as you've never heard it before.

New people, new meetups, new platforms and new ideas. Like I said, it's a really exciting time to be part of the .NET community – join us, come to a meetup, follow us online, and let's make good things happen.

Tuesday, 4 July 2017

Use Flatscreens

This started life as a lightning talk for PubConf after NDC in Sydney, back in August 2016… and after quite a lot of tweaking, editing and learning to do all sorts of fun things with Adobe AfterEffects and Premiere, it's finally on YouTube. The inspiration is, of course, "Wear Sunscreen", Baz Luhrmann's 1999 hit song based on an essay written by Mary Schmich. Video footage and stock photography is all credited at the end of the clip, and the music, vocals, video, audio and, well, basically everything else is by me. Happy listening - and don't forget to use flatscreens :)

Ladies and gentlemen of the class of 2017… use flat screens. If I could offer you only one tip for the future, flat screens would be it. The benefits of flat screens have been proved by Hollywood, whereas the rest of my advice has no basis more reliable than my own meandering experience.
I will dispense this advice... now.

Enjoy the confidence and optimism of greenfield projects. Oh, never mind. You will not appreciate the confidence and optimism of greenfield until everything starts going to hell. But trust me, when you finally ship, you'll look back at the code you wrote and recall in a way you can't grasp now how simple everything seemed, and how productive you really were. Your code is not as bad as you imagine.

Don't worry about changing database providers. Or worry, but know that every company who ever used an OR/M in case they needed to switch databases never actually did it. The real problems in your projects are the dependencies you don't control; the leaking air conditioner that floods your data centre at 5pm on the Thursday before Christmas.

Learn one thing every day that scares you.


Don't reformat other people's codebases; don't put up with people who reformat yours.


Don't get obsessed with frameworks. Sometimes they help, sometimes they hurt. It's the user experience that matters, and the user doesn't care how you created it.

Remember the retweets you receive; forget the flame bait. If you succeed in doing this, tell me how.

Keep your old hard drives. Throw away your old network cards.


Don't feel guilty if you don't understand f#. Some of the most productive junior developers I've worked with didn't know F#. Some of the best systems architects I know still don't.

Write plenty of tests.

Be kind to your keys; you'll miss them when they're gone.

Maybe you have a degree; maybe you don't. Maybe you have an open source project; maybe you won't. Maybe you wrote code that flew on the Space Shuttle; maybe you worked on Microsoft SharePoint. Whatever you do, keep improving, and don't worry where your next gig is coming from. There's a big old world out there, and they're always going to need good developers.

Look after your brain. Don't burn out, don't be afraid to take a break. It is the most powerful computer you will ever own.

Launch, even if you have no users but your own QA team.

Have a plan, even if you choose not to follow it.

Do NOT read the comments on YouTube : they will only make you feel angry.

Cache your package dependencies; you never know when they'll be gone for good.

Read your log files. They're your best source of information, and the first place you'll notice if something's starting to go wrong.

Understand that languages come and go, and that it's the underlying patterns that really matter. Work hard to fill the gaps in your knowledge, because the wiser you get,  the more you'll regret the things you didn't know when you were young.

Develop in 86 assembler once, but stop before it makes you smug; develop in Visual Basic once, but stop before it makes you stupid.


Accept certain inalienable truths. Your code has bugs, you will miss your deadlines, and you, too, might end up in management. And when you do, you'll fantasize that back when you were a developer, code was bug-free, deadlines were met, and developers tuned their database indexes.

Tune your database indexes.

Don't deploy your code without testing it. Maybe you have a QA team. Maybe you have integration tests. You never know when either one might miss something.

Don't mess too much with your user interface, or by the time you ship, it will look like a Japanese karaoke booth.

Be grateful for open source code, but be careful whose code you run. Writing good code is hard, and open source is a way of taking bits from your projects folder, slapping a readme on them, and hoping if you put them on GitHub somebody else will come along and fix your problems.

But trust me on the flat screens.