Friday, January 13, 2017

Everyone has to serve somebody, but software has to serve more than one

When people get paid to write software, we very often find some form of friction between the people that build the software and those that pay to have it built.

The company I joined three years ago was no exception. When I joined, they had just launched three months ago, weren't seeing any return on investment and prospects weren't too bright either.

I won't lie, that first year was quite stressful. The business model sucked, the code was even worse. That first year, I wouldn't have been too surprised if the whole team had been let go. In that situation, I could have played it safe, but to be fair, there was something quite exciting about actually having skin in the game and trying to make something out of nothing.

The best thing to come out of this situation, is that each and every engineer acquired a high sense of ownership. When you're a team of only a few people, you have to carry your weight and make an impact. And the only impact that mattered was coming up with changes that would make customers return to us instead of to our competitors. A product without customers, isn't a product at all.

Being extremely customer-focussed, we did set aside many of our own concerns. Both from a personal and technical perspective. We spent too much time at the office, putting in more hours than ideal. We were lucky to have a young team, that didn't have a large day-to-day personal workload though. Since we were also the ones running the operational side of things in the early days, the tooling that was built covered the bare minimum, in a very crude way. On the technical side of things, we played the hand we were dealt. We tried not to make things worse - like treating a slowly healing wound. Improving things when constraints allowed it, still taking the rare short cut when our hand was forced. I still believe today that had we invested in the large technical overhaul the code base deserved - putting ourselves in a position where we would have shipped less features and changes - we would have found ourselves completely pushed out of the market and without a job six months later.

Once things started to turn around for the better, we acquired more and more return customers, but we also acquired a set of customers we didn't anticipate.

When we were still small, we would handle customer tickets ourselves. The owners didn't think the workload warranted another hire. After we threw away the homegrown, not very inviting to use, ticket system, we switched to an off-the-shelf live chat system. This gave us a lot more customer feedback - more than we could manage. Where tickets can be processed in batch, chats are expected to be replied to in real time. Nothing you can juggle in-between programming work, giving us a good reason to hire extra people. For these people to be productive, and actually able to help customers, they also needed access to those crude tools we built earlier. Turns out, those tools didn't really cut it. We needed to listen to their feedback and work towards building tooling that's obvious and easy to use. And there you have it, an extra type of "customer" we needed to satisfy.

But it didn't stop right there. Like a scurry of squirrels after a bird feeder, there was more and more pressure from within the organization to provide more sophisticated tools for the expanding operational needs, like invoicing, content management, customer segmentation, localization and so much more.

So far, we had always put the end customer above all. We had always been very conservative with our technical budgets. We did refactor aggressively, but only changing the inside doesn't always cut it. Larger structural, organizational changes require you to move mountains. Well, more like small hills. We knew the domain and its characteristics well enough to have a pretty good idea where we wanted to go, but the clock had always kept us from making big technical leaps forward.

One could ask: if we have been able to postpone these changes for so long, are they necessary at all? If you would ask non-technical stakeholders, they would probably prioritize the next "one more feature". Hell, they would probably only start caring when the system comes down crashing every few days. Who we should really be asking however is those maintaining the system. If they feel as if they're operating a system that's headed toward needing life support, we should probably pay attention. Eventually something has to give. Eventually you might see your software become a serious liability, or you might see people jump ship even before the tipping point. Engineers are just as much of a customer as your end customer - code is a product too. Engineers also have certain needs you want to satisfy to turn them into happy return customers, who like spending their time, energy and creativity on your product.

It might sound backwards, but even the people you're paying at the end of the month, are very much customers of your product. A monthly pay check might buy you a higher pain barrier, but it's not unbounded either. You might even see some of the most passionate people succumb early on. People on the payroll are as much a vital part of the ecosystem as the end customer. If they don't like being a customer, chances are, your end customer won't either. Highly contagious. While the end customer is still King in this story, to build something truly sustainable for the future, you need to reckon with each and every entity that interacts with the software.

Tuesday, December 27, 2016

Consumed in 2016

I'm keeping the tradition alive, sharing how much I've consumed over the last year highlighting the things that stood out. 18 books, 8 movies and 9 shows. Looks like I consumed more than other years, which probably also explains why I produced less after-hours.


I finished the Dark Tower series after 3 years. Following Roland Deschain and his ka-tet throughout the 8 books has been an epic adventure. Finishing Harry Potter by the time I was 18, I had little hopes to be ever dragged into such a long and captivating tale ever again. The Stand, another epic by Stephen King is also high up on my list. I've seem to have taken a liking to stories that are set in a post-apocalyptic world, in which the antagonist is not necessarily a horde of zombies.
Control the things you can control, maggot. Let everything else take a flying fuck at you, and if you must go down, go down with your guns blazing. - Stephen King, The Dark Tower
Show me a man or a woman alone and I'll show you a saint. Give me two and they'll fall in love. Give me three and they'll invent the charming thing we call 'society'. Give me four and they'll build a pyramid. Give me five and they'll make one an outcast. Give me six and they'll reinvent prejudice. Give me seven and in seven years they'll reinvent warfare. Man may have been made in the image of God, but human society was made in the image of His opposite number, and is always trying to get back home. - Stephen King, The Stand
Culture and Empire and The Psychopath Code by the late Pieter Hintjens were thought provoking, yet easy reads. I wish more authors could break apart complex problems in such an understandable fashion. Pieter's ability to research any given topic and to build a convincing case is a skill I'd like to acquire one day.
For the sake of argument, let's divide society into four roughly equal chunks. We have the bandits, who specialize in taking from others. Then, we have the beggars, who specialize in getting something for nothing. Middle management, perhaps. Then, we have the bureaucrats, who specialize in making rules and keeping things organized. Finally, we have the bakers, who specialize in making things that other people need. - Pieter Hintjens, Culture and Empire
When the cost of secrets held by one person or group outweighs the benefits to society, then it's right that those secrets be leaked. Security does not just trump Liberty, he takes her into a dark back alley, violates her repeatedly, and then beats her senseless with a heavy stick. - Pieter Hintjens, Culture and Empire
As far as technical books go, I've been mostly reading about things that are relevant to my current job. I learned most from Site Reliability Engineering and SQL Server 2012 Internals. They are long, very slow reads, but the information has a depth to it which is hard to get by in other formats. In retrospect, I often wonder if I might have been better of reading them more selectively though.
Covering the same domain, I finally got around to reading the classic Release it! This book was published in 2007, and it shows. However, that doesn't mean it's not worth your time. It's fascinating to see how much the infrastructure landscape has changed over the years. Knowing where we're coming from deploying and running large systems helps me understand why we're doing things in a certain way today.


The Revenant. If you're looking for a rich storyline, you might end up disappointed. However, it's the high quality cinematography and breathtaking scenery that make this film. I'm a sucker for the romance of the American frontier though.

Snowden. Not a documentary, but a Hollywood picture telling the story of the most disputed whistleblower of the century. To be fair, having worked with government tech over the years, I used to be ignorant of the fact that a government agency would be able to acquire the talent needed to build and run a mass surveillance system successfully. Being Belgian, I totally ignored the fact that in other countries there are some really smart people patriotic enough to serve their country no matter what. Although I read up on the subject when it was early days, the massive scale of it all hit me like a ton of bricks halfway through the movie. Generally I'm pretty excited to work in a field that has changed how we communicate, distribute and do work in such a small period of time. However, it has become more and more apparent to me that the centralized nature of the things we build and use, is also a huge threat to our freedom. It's crucial that we learn from this incident, and grow towards a world where pardoning Snowden is the obvious thing to do, and in which the people in power toying with our privacy are the outcasts.
I think the greatest freedom that I have gained, the fact that I don't have to worry about what happens tomorrow, because I'm happy with what I've done today. - Edward Snowden

I got around to watching the first episode of Westworld yesterday, and my expectations are sky-high. Friends have been hyping me up over this show since the first episode.

Earlier in the year, I watched Narcos and Stranger Things. Netflix focussing more on content creation instead of rapidly expanding their portfolio with third party content might pay off big time. Both shows bring high quality, bingeable content.


This American Life has been a favorite of mine for a long time. The Monday morning commute almost becomes enjoyable. Almost.

Software Engineering Daily is the podcast that I use to break out of my technological echo chamber. The pace at which these have been published is unheard-of. Five days a week there's a fresh one hour interview with a high-profile guest involved in building incredibly interesting systems at scale.

Being the accidental DBA at work, the SQL Server Pain Relief podcast has been compensating for the lack of people I know that have first hand experience running large, high available and performance intensive SQL Server production systems.

If there's anything I really missed out on that I should watch or read, let me know!

Sunday, September 18, 2016

Commands and events with JustSaying and AWS

I've been looking into handing a bit of our messaging infrastructure over to a managed alternative. Managing your own messaging infrastructure that should be highly available is not always an investment you want to make in this day and age. Going through the documentation and relying on experiences from some people I trust, I ended up looking at AWS and SNS/SQS.

Making the Github repository rounds, looking for inspiration, I stumbled on JustSaying: a library by the people from JustEat implementing a message bus on top of AWS.

I wanted to find two messaging patterns in this library:
  1. Command queuing. A common pattern in our components is to react to an event by making an HTTP request to an external partner. To improve reliability and throughput, we generally don't make that HTTP request in the projection itself, but rather drop a command onto a queue which will then be processed in parallel using a bounded amount of retries. When things do go wrong, we either retry the messages by moving them from the error queue back to the input queue or we change the reaction and reset the projection checkpoint, sending the commands again.
  2. Pub-sub. Another pattern used when there is a certain level of familiarity between components, is to have a component publish events. Other components can subscribe to these messages and have them delivered to their own queues.
Both these styles are supported by JustSaying.

In this example, I have two commands: BookFlight and CancelBooking, with two related events: FlightWasBooked and BookingWasCancelled.

Since JustSaying requires messages to inherit from a base class, these message definitions live on the outside, far from the domain. This allows to decouple the domain from the outside contracts and to make sure the events go out to the world in the format I want them to be.

To handle these messages, JustSaying requires you to implement the IHandler interface.

Having this out of the way, we need to configure the bus (publishers and subscribers).

First of all, Amazon needs to know who we are and what we're allowed to do.

We should define which region our infrastructure lives in.

Now we can configure our command queue. Commands should be published using an SQS publisher, directly dropping messages into the "Commands" queue. A point-to-point subscriber will directly pull messages from the "Commands" queue and hand them over to the command handlers.

Events are not directly dropped to an SQS queue, but will be created as an SNS topic. We can use SQS to subscribe to these topics and have them delivered to an "Events" queue.

Once the bus has been created, we can start listening and publishing messages.

JustSaying will create two SNS topics and four SQS queues: two input queues and two error queues.

Those topic and queue names are not that descriptive once you introduce multiple components and might cause names to collide. JustSaying allows you to define a custom naming strategy. I've settled on a strategy that is based on the message type and prefixed with the component name. This has the added advantage that each message type now goes into its own queue.

This whole experiment has had a scary low learning curve (maybe a bit too low). While I'm still in the assess-phase, I'm fairly optimistic that running on top of SNS/SQS might take away some of our operational burden. Going over the JustSaying API and code base, it's quite opinionated and there are things I might have approached differently. Some features I'd like to see, like the library providing a message envelope as a first-class citizen (a base message class is something I've regretted in the past) is being worked on, so I'm keeping my eye on those. Since I'm only using command queuing at the moment, I should be pretty safe from future breaking changes to the message format and such.