Tuesday, May 1, 2018

Amazon Redshift - Fundamentals

Late 2017, we set out to replace and upgrade our existing reporting and analytics infrastructure with something that would be a better fit for our workloads. Keeping costs and required maintenance at a minimum would be a nice plus, making for an easy sell. After a bit of research, it was obvious Amazon Redshift had the potential to tick all the right boxes. While steadily porting the most problematic workloads away from our existing infrastructure, I started writing an investigative article on the fundamental concepts of Amazon Redshift. I learned a lot studying each individual building block, allowing me to make some small, but impactful changes to our own setup along the way.

The outcome is a 10.000 word document (1 hour reading time), covering 7 topics:
  • Storage
  • Distribution
  • Importing data
  • Table maintenance
  • Exporting data
  • Query processing
  • Workload management

The text is available in three formats:

The project is open source and available on Github.

Thanks to everyone who proof-read earlier iterations and provided me with indispensable feedback.

I hope this work can teach you as much as it thought me. I'm looking forward to your feedback.

Wednesday, January 3, 2018

Consumed in 2017

Another year, another 17 books, 6 shows and 3 movies consumed. Here's this year's highlights.


1. The Zen and Art of Motor Cycle Maintenance

The author is a tormented soul on a quest to define quality. You're his passenger, driving shot gun on a CB77 Super Hawk, in for an exhausting intellectual journey through the high mountains of reasoning. You will often fear getting lost and feel slightly anxious that the driver might drive of a cliff any moment, but he won't. Once you see the top of the mountain for the first time, you'll be happy he doesn't make it too easy on you, and you'll be more appreciative of the road that took you there.
Throughout the process of fixing the machine things always come up, low-quality things, from a dusted knuckle to an accidentally ruined “irreplaceable” assembly. These drain off gumption, destroy enthusiasm and leave you so discouraged you want to forget the whole business. I call these things “gumption traps.” 
Peace of mind isn’t at all superficial to technical work. It’s the whole thing. That which produces it is good work and that which destroys it is bad work. 
It’s the style that gets you; technological ugliness syruped over with romantic phoniness in an effort to produce beauty and profit by people who, though stylish, don’t know where to start because no one has ever told them there’s such a thing as Quality in this world and it’s real, not style.

2. The Soul of a New Machine

The type of writing I wish there was more of. It's the closest I'll ever get to experience building a mini-computer. It makes one appreciate how much we're standing on the shoulders of giants. So much has changed in 40 years and even more hasn't changed at all. People will be people.
Adopting a remote, managerial point of view, you could say that the Eagle project was a case where a local system of management worked as it should: competition for resources creating within a team inside a company an entrepreneurial spirit, which was channeled in the right direction by constraints sent down from the top. But it seems more accurate to say that a group of engineers got excited about building a computer. 
In the sixties there was proposed a “National Data Bank,” which would, theoretically, improve the government’s efficiency by allowing agencies to share information. The fact that such a system could be abused did not mean it would be, proponents said; it could be constructed in such a way as to guarantee benign use. Nonsense, said opponents, who managed to block the proposal; no matter what the intent or the safeguards, the existence of such a system would inevitably lead toward the creation of a police state.

3. Winter is Coming: Why Vladimir Putin and the Enemies of the Free World Must Be Stopped

I've been trying to get better educated on the history of world politics and long-lasting international conflicts. Growing up in Western Europe with very little direct conflict, you're never really taught why other parts of the world seem to be so messed up and why they're so angry at us.

Although you might find Kasparov to be a bit too convincing, he has good reasons to hold strong opinions and is in a unique position to shed light on what's been happening under the Putin regime. It's like listening to someone who just escaped an abusive relationship. I was remembered of Hintjens' The Psychopath Code more than once.
Like a weed, evil can be cut back but never entirely uprooted. It waits for its chance to spread through the cracks in our vigilance. It can take root in the fertile soil of our complacency, or even the rocky rubble of the fallen Berlin Wall. 
If the road to hell is paved with good intentions, compromises on principles are the streetlights. 
He and his junta have turned the country into a petro-state, and exporting natural resources to an insatiable global market doesn’t require entrepreneurs or programmers, let alone writers and professors. 
Putin and his defenders abroad bragged about Russia’s rising GDP, but it was like taking the average temperature of all the patients in a hospital. 
The hypocrisy of condemning weak dictatorships while embracing strong ones destroys American and European credibility and undermines any attempt at global leadership; in fact, it seems to encourage smaller autocracies to aspire to greater ambitions.


1. Westworld

In a not so distant future, the rich will be spending their free time visiting amusement parks inhabited by lifelike robots. Once you pay your entry ticket, you can be the protagonist of any story you want. Maybe you want good value for money and go on an epic quest chasing a bad guy to the edge of the park, or maybe you just want to drink, gamble and maybe kill a few randoms for fun in the saloon just feet away from the drop-off point? Like playing Red Dead Redemption post-virtual reality.

The setting, cast and especially the story line are out of this world. Reading up on fan theories after the show is half the fun. It's amazing how many plausible theories were put out and how a small community is able to dissect every little scene looking for hints to figure out the park's mysterious past.

This show made me question if I understood what it is that makes us human. Aren't we all just the result of our pre-programmed genetics and the events we experience throughout our lives?


2. Stranger Things 2

I just finished watching this one last night. It's like reading a really good Stephen King novel, but in color. I noticed yesterday's season finale was 61 minutes long; good things do happen when you're not constrained to TV time.



Commutes have gotten earlier and longer this year. I've cut back on the technology podcasts in favor of a more broad range of topics.

1. The Joe Rogan Experience

It's a bit like sitting in on a conversation between two people at a bar. One person is a talkative and enjoyable guy, not afraid to ask questions and the other is happy to drop knowledge on a specific (and often fringe) topic. Topics range from diet and fitness, to economy and politics. Some episodes that stood out for me recently are the ones with Colin Moriarty and Nina Teicholz.

2. Conversations with Tyler

The same concept, being a one-on-one conversation covering a wide range of topics. However, this one is more formal and often a bit (too?) academic. This podcast on Marcroeconomics, Mentorship and Avoiding Complacency might give you a good idea on what to expect.

3. Dan Carlin's History X

Extremely thoroughly researched lectures on important periods of our history. The Destroyer of Worlds is a 6 hour long, but captivating piece on the history of nuclear warfare and filled in a lot of gaps for me.

Not sure what to watch next. Any recommendations?

Friday, December 15, 2017

Passing the AWS Certified Solutions Architect exam

Before last week, the only certification exam I ever passed was the Microsoft .NET Framework Application Development Foundation certification. This was almost eight years ago. My manager back then thought getting certified was the best way for me to get a raise. It would be a win-win. I for one would learn something along the way, and the company would have less trouble keeping its Microsoft Gold partnership. As far as I remember, I spent a good six months reading, studying and memorizing this 794 pages thick book. Although the book did teach me a fair amount of solid .NET framework internals, most time was spent force feeding myself the ins and outs of framework API's you only need once in a blue moon and should just Google for when needed.

This time around though, it was my own decision to get certified. Mid 2016, our components were getting more and more structured in a way that allowed us to deploy them away from our on-premise data center. Components that didn't own data bound to a specific territory by regulatory requirements and that would allow for some down-time were the obvious candidates.

Moving some of our infrastructure to the cloud, we had a few goals in mind:
  • Take advantage of managed cloud services to reduce operation cost significantly.
  • More freedom to scale up or down. The structure of  the contracts with our data center (and regulations that require us to own our own racks) generally forces us to over provision our infrastructure. Making changes halfway the contract takes time and is costly.
  • Ease into learning how to run software on the cloud for when we move to other markets or when we build services that have less strict territoriality constraints.

Getting started with AWS is easy enough. Starting an EC2 instance, attaching a disk, using a managed database, configuring a load balancer is child's play. But when it came to networking, security, fault-tolerance and properties guaranteed by AWS, I had a lot of questions. I hoped to find answers going over the AWS Certified Solutions Architect material. Why not set myself an artificial goal and get certified while I was at it?

After three months of studying, I passed the exam with a score of 95%. Here's a list of the resources I used, including how much money and time were spent.

Exam Blueprint

You should go over the exam blueprint to understand what they expect you to know to pass the exam. It's a good idea to go over the document while studying and to tick off domains you feel comfortable with.

Money spent: €0, time spent: 30 minutes

A Cloud Guru

A good collection of short videos and mini-exams covering all the topics needed to pass the exam. Details that require extra attention to pass the exam are highlighted throughout the course.

You can get the videos on acloud.guru for €99. I got it through Udemy for only €10. Worth every cent.

Money spent: €10, time spent: 26 hours

FAQs and Whitepapers

AWS advises you to read a specific set of whitepapers and FAQs. The material can be a bit dry, but it's extremely useful. Not just to pass the exam, but to avoid nasty surprises in production.

Money spent: €0, time spent: 6 hours

AWS Open Guide

An open-source effort to document real world experiences running environments on AWS.

Money spent: €0, time spent: 1 hour


Somewhere around 500 practice questions that tease out the topics you don't completely master. When I got a question wrong, I would read up on the topic and play around in the AWS console until I felt like I got it.

Although there were some questions that were very similar, you can't pass the exam by just studying these questions.

Money spent: €20, time spent: 12 hours

Exam Guru

Mobile app affiliated with A Cloud Guru containing more practice questions. These are less scenario based and less in-depth. The Whizzlabs questions are much more in the direction of what to expect on the actual exam. Disappointing to be fair.

Money spent: €20, time spent: 2 hours

Test exam

A small set of questions provided by AWS in the style of the actual exam. This was very much a waste of time and money. Whizzlabs had copied all of these questions word for word.

Money spent: €20, time spent: 20 minutes

Test day

The least enjoyable part of the experience.  55 multiple choice questions need to be answered in 80 minutes. Half of the questions are quite straight forward. The other half are more involved. For the longer questions, I first read the answers and wrote down the options. For then to read the question and strike through the options that definitely were not a part of the answer. This helped me to focus on the important bits of the question and to gain momentum plowing through the questions at a steady pace. I finished with 25 minutes left.

Money spent: €135, time spent: 90 minutes. Extra money spent on parking in the city center of Brussels: €10, searching for a spot: 60 minutes

In short... Go over the material, practice, take notes, practice some more, review your notes until you get sick of them.