How We Use Infrastructure as Code in HarperDB Cloud

Aron Johnson — Tue, 20 Oct 2020 18:13:42 +0000

Why Infrastructure as Code?

Earlier this year we launched HarperDB Cloud, a fully-managed, cloud-hosted version of HarperDB. We wanted HarperDB Cloud to be a fast and low-friction way to get HarperDB. Sign up, answer a couple of questions, and a few minutes later you have a working instance of HarperDB. To make this happen, we understood that all aspects of deploying the infrastructure to host customer workloads must be automated. In other words, we needed to leverage Infrastructure as Code (IaC.)

What is Infrastructure as Code?

Traditionally, the deployment of IT infrastructure (creating virtual machines and installing applications, provisioning storage and networking, etc.) was largely a manual effort. Meaning, IT administrators would use a series of CLI and GUI tools, maybe some scripts, to build and configure each piece of the infrastructure. This way of building infrastructure is very inefficient and prone to error. IaC is the response to these challenges.

IaC is a method of defining a set of infrastructure components in one or more configuration files (usually in the form of YAML, JSON, or a similar format), which are then passed to an IaC software tool (such as HashiCorp Terraform or AWS CloudFormation) that will provision those components on a cloud platform. IaC allows infrastructure to be built in a fast, consistent, and repeatable fashion!

How and Why We Use CloudFormation

HarperDB Cloud consists of:

HarperDB Studio, our hosted application for managing your HarperDB instances.
AWS Lambda functions that build AWS resources for running customer workloads (the functions also do various other things, such as update our payment and CRM platforms.)
The AWS resources that run customer workloads, such as VPCs, EC2 Instances, EBS Volumes, Elastic Load Balancers, etc.

The Lambda functions use CloudFormation to build AWS resources. Sure, we could call the individual AWS APIs for VPC, EC2, Route53, etc. to build each of these resources, but we would have to create the logic to deal with resource dependencies (wait to build this until that is built) and rollback when a resource fails to build (that EC2 instance doesn't do any good if there is not a related ELB rule.) An IaC tool like CloudFormation takes care of all that, and more, for us! 😁

What About Terraform?

There are lots of great IaC tools besides CloudFormation out there. We currently only place HarperDB Cloud customer workloads on AWS, and we already use AWS services for the other backend parts of HarperDB Cloud (David has an excellent article for more about that), so using the AWS-native IaC tool made sense. Don't worry all you HashiCorp fans out there, we use and love HashiCorp tools (we use Packer to automate the build of all our Amazon Machine Images), so of course we considered using Terraform! CloudFormation and Terraform each have different strengths, but either IaC tool would work.

With a Little Help From Systems Manager Automation

While CloudFormation is awesome at provisioning and terminating AWS resources, the management of existing infrastructure, particularly the guest OS and applications of an instance, often requires another tool. Need to patch Linux, then upgrade HarperDB, and make a configuration change, to hundreds of EC2 instances, in just a few minutes? AWS Systems Manager (SSM) Automation can make that happen! Yup, there are other great tools for doing this kind of work (Cough! Ansible. Cough!), but SSM Automation works great for us.

What's Next?

This technology stack works great for us, but what's next? Lots! You probably noticed that I said a few icky words like "EC2 instance" and "patch," but I didn't mention "containers" or "immutable infrastructure." Well, don't worry, we are making plans. Also, it now seems likely we will have customer demand to host HarperDB Cloud workloads on cloud platforms other than AWS. Hmm, about that Terraform thing... 😅

If you are interested, Margo has a great article about what makes HarperDB different and special. Also, you can try out HarperDB, for free!