Sunday, September 18, 2016

Commands and events with JustSaying and AWS

I've been looking into handing a bit of our messaging infrastructure over to a managed alternative. Managing your own messaging infrastructure that should be highly available is not always an investment you want to make in this day and age. Going through the documentation and relying on experiences from some people I trust, I ended up looking at AWS and SNS/SQS.

Making the Github repository rounds, looking for inspiration, I stumbled on JustSaying: a library by the people from JustEat implementing a message bus on top of AWS.

I wanted to find two messaging patterns in this library:
  1. Command queuing. A common pattern in our components is to react to an event by making an HTTP request to an external partner. To improve reliability and throughput, we generally don't make that HTTP request in the projection itself, but rather drop a command onto a queue which will then be processed in parallel using a bounded amount of retries. When things do go wrong, we either retry the messages by moving them from the error queue back to the input queue or we change the reaction and reset the projection checkpoint, sending the commands again.
  2. Pub-sub. Another pattern used when there is a certain level of familiarity between components, is to have a component publish events. Other components can subscribe to these messages and have them delivered to their own queues.
Both these styles are supported by JustSaying.

In this example, I have two commands: BookFlight and CancelBooking, with two related events: FlightWasBooked and BookingWasCancelled.

Since JustSaying requires messages to inherit from a base class, these message definitions live on the outside, far from the domain. This allows to decouple the domain from the outside contracts and to make sure the events go out to the world in the format I want them to be.

To handle these messages, JustSaying requires you to implement the IHandler interface.

Having this out of the way, we need to configure the bus (publishers and subscribers).

First of all, Amazon needs to know who we are and what we're allowed to do.

We should define which region our infrastructure lives in.

Now we can configure our command queue. Commands should be published using an SQS publisher, directly dropping messages into the "Commands" queue. A point-to-point subscriber will directly pull messages from the "Commands" queue and hand them over to the command handlers.

Events are not directly dropped to an SQS queue, but will be created as an SNS topic. We can use SQS to subscribe to these topics and have them delivered to an "Events" queue.

Once the bus has been created, we can start listening and publishing messages.

JustSaying will create two SNS topics and four SQS queues: two input queues and two error queues.

Those topic and queue names are not that descriptive once you introduce multiple components and might cause names to collide. JustSaying allows you to define a custom naming strategy. I've settled on a strategy that is based on the message type and prefixed with the component name. This has the added advantage that each message type now goes into its own queue.

This whole experiment has had a scary low learning curve (maybe a bit too low). While I'm still in the assess-phase, I'm fairly optimistic that running on top of SNS/SQS might take away some of our operational burden. Going over the JustSaying API and code base, it's quite opinionated and there are things I might have approached differently. Some features I'd like to see, like the library providing a message envelope as a first-class citizen (a base message class is something I've regretted in the past) is being worked on, so I'm keeping my eye on those. Since I'm only using command queuing at the moment, I should be pretty safe from future breaking changes to the message format and such.

Sunday, August 21, 2016

My InfoQ interview on DDD, events and legacy

Seems that it's impossible to beat the Gaussian curve of blogging frequency. On the other hand, I spent quite some of my mental blogging budget on an interview with InfoQ.

I'm a bit bummed out that it's such a large wall of text. When submitting the answers, I highlighted some snippets which should make for easier scanning. Too bad the formatting was lost when publishing it. I included some highlights below.

The interview itself can be found here. Let me know what you think!
Extracting components: Starting out, this can be as trivial as trying to model boundaries as namespaces or modules. 
Invariants: Having core properties enforced deep within the model, allows for a better night's sleep.
Sizing aggregates: Make your aggregates as small as they can be, but not any smaller. There's a big difference between an invariant that needs to be strongly held and data that helps the aggregate to make a decision, but which doesn't require strong consistency. 
ORM pitfalls: Being able to navigate through a graph, which basically walks through your whole database, is a great way to lose any sense of transactional boundaries. 
The value of bounded contexts: Now when I switch between bounded contexts, it feels like walking through a door, entering a separate room where you can tackle a problem with an unbiased mindset, allowing you to use the right tool for the job. 
Introducing domain events: When you don't want to or can't afford to invest in the full paradigm shift there's a middle ground. You can try a hybrid approach in which you, next to persisting state, also persist the events that led up to a specific state. This does entail the risk of introducing a bug which causes split-brain in which your events do not add up to your state.  
Designing contracts: If you get the semantics wrong, you will end up with a system that's held together by brittle contracts that break constantly.

Wednesday, April 27, 2016

Pieter Hintjens

Writing doesn't necessarily always come naturally to me. It often takes me days, weeks or even months of toying with an idea, before I think it's mature enough to put it down into writing. I can't afford that luxury this time though, I wouldn't think of myself as much of a friend if Pieter didn't get to read this in time.

I met Pieter the first time in a bar in Vilnius, December 2013, I accidentally ended up sitting next to him during the traditional pre-conf drinks. The first thing that stood out, was what a comfortable warm sweater he was wearing - I still cherish the memory of that grey woolen sweater on cold winter nights. I'm still unsure whether it was the sweater or his undeniable radiant charisma that made its way into my memories. When Pieter talks, people tend to listen, or at least pay attention. That's what I ended up doing that night - listening, sipping in the knowledge, afraid to make a fool out of myself joining the conversation.

That next day Pieter opened the conference with probably the most motivational keynote I ever attended, aptly titled "Building stuff changes everything". Him being a fellow countryman and me having a few Lithuanian beers in me, helped me gather enough courage to properly introduce myself and talk for a bit.

From that moment on, we would only meet a few times a year, traveling to Vilnius with the Belgian delegation or as a guest at the Domain Driven Design Belgium meetup. During that time, we had a few - far from enough - lengthy conversations. Me mostly asking questions, him sharing his point of view, and me trying hard to keep up, taking it all in. Although he was always more than happy to entertain each question you would throw at him, I would always feel a bit selfish keeping him to myself for too long.

The most memorable talk I had with Pieter was during a tête-à-tête in the Vilnius sky bar. We would mingle Dutch and English, whichever language made the next sentence sound best. We shared some personal experiences, he laid out most of the groundwork for what a good year later materialized into "The Psychopath Code", but most importantly he allowed me a peek through his eyes, looking at his playground we like to call life.

You don't need his Mensa membership card, to realize he is a highly gifted individual. He could have pursued anything he wanted and been good at it, but he chose all-out for people, freedom and love - making it his mission to evangelize his core beliefs.

His words - both spoken and written - have inspired me more than he can imagine. And they will likely continue to inspire others for many more years to come. His work has given me a framework to build on for the rest of my life. There's so much to learn from how he is capable of dissecting the world, to document things that are broken and his tireless effort to make it whole again - one protocol, one word, one hug at a time. Where I would often feel overwhelmed by dark hopeless sentiments, he has given me enough tools to overcome those. From his mouth to my heart: "We're not much more than a pile of ants, but together we can achieve tremendous things".

Pieter, I'm not half the writer you are, but I hope these words can serve as a testimony to your children what a great dad they had. If your time comes, know that I'm grateful that I've been able to call you my friend.