Forem: Tony Morris

The Vision for CloverleafTrack.com

Tony Morris — Thu, 26 Sep 2024 19:41:15 +0000

So, here’s the deal—I’m both a track and field coach and a bit of a tech geek. I’ve always been into the numbers side of things when it comes to sports. Track and field, especially, is all about numbers: personal bests, meet results, season records. You name it. But keeping track of all those stats? Not as easy as you’d think. Spreadsheets are a pain to deal with, and don’t even get me started on paper records—they always seem to go missing. And the tools out there? They just don’t quite fit what a high school track program really needs.

That’s why I decided to build CloverleafTrack.com. My goal here is simple: I want to create a custom platform that tracks every performance from every athlete on Cloverleaf High School’s track and field team. But it’s not just about storing data. I want this platform to actually help coaches, athletes, parents, and even the fans interact with all that data in ways that are actually useful. You know, things like spotting trends, seeing how someone’s improving, or comparing results between meets and over different seasons. I want CloverleafTrack.com to be the go-to place for everything track and field related at Cloverleaf.

Why Start from Scratch?

Now, you might be wondering why I didn’t just use some existing software. I thought about it, but honestly, there are a few reasons I wanted to build this thing from the ground up:

Customization The pre-made solutions out there? They’re fine, I guess, but they don’t let me do exactly what I want. CloverleafTrack.com is going to be tailored specifically for our team’s needs. I’m talking full control over everything—performance tracking, team management, visualizing the data. I don’t want to deal with the limitations of some off-the-shelf tool that can’t handle the unique quirks of track and field events.
Scalability As more athletes join the team and we gather more data, this platform needs to keep up without slowing down or getting bogged down in old data. By building it from scratch, I can make sure the system will scale up as we grow—and even bring in historical data from past seasons without a hitch.
Ownership and Flexibility By hosting the platform ourselves, we’re not relying on some third-party service that could change its terms, raise prices, or go offline whenever they feel like it. We’re in full control. That means we can adapt and evolve the platform as our needs change without worrying about anyone else’s decisions.
A Passion Project Honestly? This is kind of a labor of love for me. Track and field has always been a big part of my life, and combining that with my love of tech just feels right. I’m not just building something functional; I’m building something I can continuously tweak and improve, making it more useful for everyone involved.

Where I’m Starting

For the first version of CloverleafTrack.com, I’m focusing on these four areas:

Athlete Profiles Every athlete will have their own profile that shows all their performance history, personal bests, and how they’ve improved over time. Whether they’re in individual events or relays, their profile will give a full picture of what they’ve accomplished.
Meet Management This is where I’ll be organizing all the data from meets, showing results across all events and athletes. It’ll make comparing performances between different meets and seasons much easier.
Performance Tracking and Scoring I’m also building a detailed scoring system that will give coaches more context around the numbers. It won’t just show raw results—it’ll show how those results stack up against different scoring systems, whether for local meets or even potential benchmarks for college-level competition.
The Leaderboard This feature is one of the more exciting parts. There will be two types of leaderboards:
1. School Record Leaderboard: This will track the top performances for every event throughout the school’s history. It’ll give everyone a clear look at who holds the record for each event and when it was set.
2. Event Leaderboard: This one will track performances for a specific event over time, showing how results have improved and where current athletes rank against historical data.

Looking Ahead

There’s still a lot more I want to build, and this is just the beginning. Down the road, I plan to add:

Data Visualization I’m looking to create charts and graphs that’ll make it easier for coaches and athletes to spot trends and see where things are headed over time.
Mobile-Friendly Interfaces Everyone’s on their phone these days, so I’ll be making sure the platform works seamlessly on phones and tablets. That way, anyone can check results or track performances while on the go.
Meet and Season Scoring Metrics In the future, I’m going to build out scoring metrics that can show how the entire team performed across a meet or even over a full season. This will let coaches compare results from one meet to the next, and across seasons, making it easier to see team strengths and where improvements need to happen.

CloverleafTrack.com is more than just a place to track numbers. I’m hoping it becomes an interactive, useful tool that makes the data more meaningful for everyone involved—from coaches and athletes to parents and fans. By building it from scratch, I can make sure it’s built exactly for the unique needs of our track and field program, and I’ll keep improving it as we go.

Self-Hosted Audiobooks

Tony Morris — Fri, 10 Mar 2023 02:11:00 +0000

I typically spend around two to five hours per week in a car, driving back and forth to work. While some weeks I work from home more regularly, I still end up in the car a few hours per week.

During these longer commutes, I love listening to audiobooks. However, being the self-hosted advocate that I am, relying on a SaaS provider, like Audible, for my audiobooks runs counter to my beliefs. This post describes my self-hosted setup that lets me access my library anywhere.

Audiobook Source

I exclusively use Audible to get my audiobook files. While Amazon is generally the devil, Audible has amazing quality and range of audiobooks. Also, they let you download the .m4b without any issues.

I’ve got an Audible Premium Plus subscription, which gives me a free audiobook every month for $14.95/month. This price is typically cheaper than purchasing audiobooks at retail, so it works out for me. I also don’t typically go through more than one audiobook per month, so the cadence works great.

Audiobook Downloader

To automate the downloading of the audiobooks from Audible, I use a tool called OpenAudible. I pay $18.95/year for the license, and it’s tied to the version that you purchase. In other words, if, after a year, a new version is out, I have to pay another $18.95 in order to upgrade. However, I can choose to not upgrade and stay on the version I purchased permanently. It’s a pretty standard perpetual license model.

OpenAudible is configured to connect to my Audible account and download all my audiobooks to a well-known network share. I run it on my personal MacBook Air.

Download Location

Without going into too much detail, I have a QNAP TS-431 on the network that houses all my network shares. In this case, I have a network share called Audiobooks that I mount on my Air. I then configure OpenAudible to export all downloaded files to that network share.

Media Server

I run Plex on a server that is connected to my network as my media server. Adding the library and configuring it is a bit of an adventure in metadata, but generally it’s straightforward.

Before adding any libraries, I added the Audnexus library agent to Plex following the instructions on its GitHub README.

I added an Audiobooks library that uses the Music type. From there, I add the auto-generated books directory that OpenAudible creates in the Audiobooks mount point.

Within the configuration of the library, I chose the Plex Music Scanner for the scanner and the newly-added Audnexus Agent for the agent. The Audnexus GitHub README includes the recommended configuration for the library, as well.

Listening to the Audiobooks

Now that these audiobooks exist on a self-hosted media server, it’s time to listen to them! While it’s true that I could use the native Plex app on my devices, I’ve found that Prologue is an upgrade in every way. In order to get offline playback, you have to spend $5 one time, which is well worth the price.

From here, I just plug my phone into my car and listen to the audiobooks via CarPlay.

Wrapping It Up

All-in-all, my self-hosted audiobook configuration is pretty straightforward, and it requires almost no maintenance. Every month or so, I open my MacBook Air, fire up OpenAudible, sync my library, and everything just works. I typically listen to books after downloading them locally in Prologue. It all works really well!

README

Tony Morris — Tue, 23 Aug 2022 23:32:23 +0000

Inspired by Manager Readme, I recently decided to write up my own version. It's important to explicitly call out that this exercise was not to replace any formal onboarding that a new team member goes through, but instead to supplement it. There are quite a few detractors to Manager Readmes out there, but I'm working really hard to make it as introspective as possible. Anyway, enjoy!

Hi! Welcome to the team! I’m Tony. My official title is Director of Cloud Enablement, but to you, I’m just your manager (or peer, or manager’s manager, or whatever our working relationship is).

This document is intended to enhance, not replace, our working relationship. Indeed, the intention of this document is to understand how I think, act, react, and work. This document is intended as a user guide for me and how I work. It captures what you can expect out of the average weekly working with me.

While I may have some opinions and nuance about how I work, this document should not, in any way, influence the way you work.

My Leadership Style

I’ve been working professional on technology since 2006. My career has taken me from software engineer, team/tech lead, software architect, software manager, enterprise architect, and now Director of Cloud Enablement. Throughout that journey, I’ve refined my leadership approach, but it’s still built on the same foundational elements.

You are an adult. This is, by far, the most important value you need to hear. I trust adults implicitly. Adults are self-sufficient. Adults know when to escalate bad situations. Adults know how to praise one another. Adults tell one another when they are overloaded and/or stressed out. I will always treat you as an adult first, until you are no longer being an adult. At that point, I will treat you like a child. Children need to earn trust. Children need their hands held crossing the street. Children throw tantrums. Children are unable to prepare for the future. Don’t be a child.

Do less. Delivery of value will always take precedent over being busy. The easiest way to deliver value is to have less balls in the air. In fact, I will regularly challenge you to take items off your plate, either by delegation or by backlogging the work for later. If you aren’t sure of the priority of items in the backlog, just ask!

Bias towards action. This is an Amazon leadership principle, and I am shamelessly stealing it. I do not want to be in hours of contemplative, theoretical meetings, and I do not want you to be either. Actions are very rarely irreversible. Try something small, detail your experiment, and learn from it. Keep experimenting.

Be honest. We don’t lie to each other. This includes both positive and constructive feedback. This is part of the social contract between you and me. I expect this to be difficult for you sometimes, and that’s okay! We will get better at it together.

My Schedule

I’m technically an in-office employee. This means that I’ll be in Westlake around 3 days per week. On those days, expect me to be in the office from 9:30am US Eastern until about 5:00pm US Eastern. For the days that I’m remote, the hours typically shift an hour earlier.

I’m very particular about my calendar in Outlook--it’s always up-to-date, especially if I am unavailable. Additionally, except in very specific cases, my entire calendar is available to everyone in the organization.

Interruptions

My job is to be interrupted, especially by you. I am, first and foremost, a servant leader to you. Do not hesitate to ask for time with me! I’ll be very honest about my availability, and I will typically make the time for you.

Focus Time

Upon inspection of my calendar, you may notice that I aggressively block many hours of the week as Focus Time (thanks Microsoft Viva!). This Focus Time is to block meeting scheduling from people outside of my organization. It does not apply to you! Book right over the top of it.

About The Spring

In the Spring (from February to the first weekend in June), I coach Track and Field at our local High School. Practice runs from about 3:00pm US Eastern until 5:30pm US Eastern. During those times, I have my cell phone on me, but I’m mostly unavailable. To accommodate this schedule interruption, I am typically picking up extra time in the evening, outside of most people’s normal working hours. If something is urgent, I’m available.

You may notice that I have explicit Commute Time on my calendar during these months. I can be considered Available, But Working Remotely during those specific times.

If you feel like you can’t reach me during these times, do not hesitate to reach out! I can’t fix a problem if I don’t know about it.

Communications

Nowadays, there are about three hundred different ways to communicate with me. In an effort to set some expectations, here are the different ways that I regularly receive communications.

Face-to-Face (Physical or Virtual). This is for both scheduled meetings and impromptu conversations. Need to discuss something that doesn’t fit a textual model? Message me on Slack to confirm availability, then set it up. If it’s not urgent, schedule a meeting.

Phone Call. This is a very urgent mode of communication. If you get a unprompted phone call from me, expect that the contents of the phone call need immediate attention. I will not call just to talk about my day.

Text Message. I rarely will text you. If you get a text from me, it’s typically because I’m unable to call at the moment. You can safely assume it is significantly less urgent than a phone call.

Instant Message (Slack / Teams). This is our normal mode of conversation. I consider IMs to be asynchronous communication. You should have no expectation of immediate responses from me. I try to respond quickly, and I will be upfront if I’m unable to respond with substance.

Email. The purest form of asynchronous communication. There’s almost no expectation of me responding to an email in less than 24 hours.

1:1 Meetings

1:1 meetings are for you. When you start, I will send you a survey about 1:1’s to determine your ideal cadence, day, time, and topics that you would like to discuss. From there, I will schedule our regular 50-minute 1:1 at your defined cadence, day, and time.

The Agenda

You own the agenda for our 1:1. When I schedule our 1:1, I will be explicitly clear on when you should expect to receive the collaborative agenda, and I try to make it at least one full business day prior to the 1:1. This agenda will live in my personal space in Confluence in an area that only you and I can see and contribute.

I will very likely have items in every agenda, but this is, first and foremost, your opportunity to tell me how you’re doing, what you need, what you wish could be different, how you feel about our team and your teammates, what your career goals are, etc. These are for the conversations you might not necessarily have with me when we’re sitting at our desks amongst coworkers. If you’d like to give me a brief status update on things you’re working on or that you’re stuck on, that is fine with me, but those are generally better-suited to a quick chat while I’m at my desk, a Slack message, or a separate meeting.

The Slack Channel

I create a private Slack channel for our 1:1s. These are explicitly to be used as a sort of scratchpad for items that you or I want to discuss in an upcoming 1:1 whose agenda is not yet available. These are removed from Direct Messages (DMs) on purpose, as I reserve DMs for normal work chatter, and 1:1 topics will typically get lost in the shuffle.

Skip-Level 1:1 Meetings

I am trying very hard to have 1:1’s with everyone in the department. I am bad at this, and it is something I am working to improve. If you and I haven’t met, anticipate that I will be correcting that.

Feedback

I’m going to be asking you for your feedback on how the team and department are running. I’m going to do this a lot. I’m going to do this in a multitude of ways. Sometimes, I will put you on the spot in a 1:1, hoping for a top-of-the-mind answer. Other times, I will send out an anonymous survey, hoping to get more detailed, challenging feedback.

These are not the only times to provide feedback! Feedback should be provided and received as soon as possible. Instead of waiting for our next 1:1, or the next State of the Team survey, or the next watercooler conversation, grab me and let’s talk it out.

Quarterly Check-Ins

At the conclusion of every quarter, you will be expected to complete a Quarterly Check-In. This process is for you to provide self-reflection feedback to me. From there, you and I will have a discussion about that feedback. During this, I will be supplying my own feedback. This is currently the only HR-mandated feedback cycle at Hyland, so I will be very keen for you to complete it on time.

Leadership Reviews

Once a year, I will be sending out Leadership Reviews, in which you will provide me with feedback on my leadership competencies. I ask that everyone in the department fill this out for me. Following the review cycle, I will be sharing the results of the feedback with the department.

The Tony Experience

I’ve been using this term to describe what it’s like to work with me the first time. The alternative title was “My Nuances,” but I figured this would land more elegantly. Below is a list of interesting personality and professional quirks that I’m aware of.

I am an extrovert. Working with people, especially groups of people, is empowering for my mind and soul. I am most effective when I am working collaboratively with people. This isn’t a quirk, but it’s something to understand about me.

I’m very open , and I will talk about anything. I want to know about every non-protected detail in your life, be it unique or mundane. This isn’t some management consulting exercise to remember you--I genuinely care about you and your life. However, I recognize there are lines that are not crossed, and I will endeavor to never cross them. You should feel no obligation to share anything with me, either. I understand there’s a weird power imbalance with these last two statements, and I am working on defining these boundaries more explicitly.

Strong opinions, held weakly. When a decision has to be made, but there is no one making it, I will make a decision, even if it’s mostly wrong. In fact, I will typically come to the table with a decision in my head, and I’ll explain it to everyone with the caveat that I am likely wrong, but it’s a good start. You should challenge me on my assertions and assumptions regularly.

I work in visible bursts. My job is mainly around strategic initiatives and large shifts in organizational and operational structures. This type of work can be difficult for me to crowdsource, so I tend to wander off into the desert for a few weeks at a time before coming back with a completed work product. Yes, this runs directly counter to my previous statement about collaborating with people, I know. This isn’t the best way to work, and I’m actively working on improving this.

When I ask you to do something nebulous , it’s likely because I haven’t thought about it all the way through. You should ask me to clarify and prioritize the work if you are uncertain. Sometimes, this will make the nebulous thing go away completely.

I’m great at starting new things , and poor at finishing them when I know the end result. This is especially true when the end result is weeks or months away. If I tend to give you work that feels started-but-unfinished, it’s because I know you are an exceptional operator of work that can take it across the finish line. As my career has moved further away from individual contribution, this quirk has diminished.

I’m regularly hyperbolic. Things are rarely as good or as bad as my inner salesman believes. Most of this can be chalked up to me being excited about the topic, in one way or another. Call me out on something in which you don’t believe the hype--I won’t be offended.

Static Website Playground: Terraform on AWS

Tony Morris — Tue, 31 May 2022 17:30:28 +0000

I need a side project, so here I am.

I make a lot of strong statements about my opinions of public cloud providers and infrastructure-as-code tooling. I figure I may as well back them up with real actual projects.

This is the first post in a series that I'm calling Static Website Playground. Each post will have the same premise: build a static website and deploy it to a specific public cloud, utilizing some type of infrastructure-as-code tooling.

This post is about using Terraform to deploy to AWS.

The important piece first:

URL: https://aws-terraform-static.morriscloud.com/

I was going to draw a diagram to represent the architectural layout here, but... it's super simple. So I'll just discuss the interest bits below.

GitHub

URL: https://github.com/morriscloud/static-website-playground/tree/main/terraform/aws

Each of these static websites is stored in the same repository and organized by IaC provider and then by cloud provider.

Cloudflare

Every post in this series will be using Cloudflare as the public DNS provider. I've got a Zone that I created previously (morriscloud.com), so this part is super simple. I look up the Zone using a data resource, then I create a new CNAME Record in the Zone that points to the S3 bucket's website endpoint. More on that part later.

Note that I recently had to update this to use the new S3 resources in the Terraform AWS provider, so I'm now using aws_s3_bucket_website_configuration.this.website_endpoint as the CNAME target, rather than the previous aws_s3_bucket.this.website_endpoint.

AWS

This deployment is as bare-bones as possible in AWS. It creates an S3 bucket, uploads an HTML file index.html, and sets some properties on the bucket to enable it to be viewed as a public website. There's a small bucket policy that's also attached to the bucket, allowing GetObject access to the objects in the specified bucket from anywhere in the world.

Terraform Cloud

To deploy the resources, I use the free version of Terraform Cloud. I connect a workspace to my GitHub repository, then I use the working directory to point to the correct subdirectory in the repository.

Using this project as an example, I have a workspace named static-website-playground-aws-terraform, pointed to the GitHub repository linked above, and then set to terraform/aws as its working directory.

Using Terraform Cloud gives me a nice hook directly into Pull Requests (if you enable it), plus I can see the changes that I'm making in a feature branch easily.

It's important to note that my favorite feature of using Terraform Cloud is the ability to run terraform plan from my local console pointed to the remote workspace. This runs the plan in a Terraform Cloud runner with all the variables defined there, rather than somewhere on my laptop that I could accidentally commit to source.

What's Next?

Next up will be using Pulumi to deploy to AWS. Obviously it won't have Terraform Cloud, but I imagine most of the rest of it will be exactly the same. Stay tuned!

Regular Feedback From My Team

Tony Morris — Sun, 08 May 2022 01:13:14 +0000

As a brand new-ish leader of people at Hyland, I am regularly in search of ways to solicit feedback from my team.

Hyland has a quarterly review process that is completely optional, so it can be tough to get the organizational support that implementing such a process would entail. Additionally, this quarterly review process is not anonymous, so it can be tough for people to provide "true" feedback.

Yearly, Hyland has a Leadership Evaluation Process which does provide that level of anonymity that I require, plus the organizational support around it. Unfortunately, this only reviews the people in leadership, and not necessarily the health of the team. Back to the drawing board.

My wife solves the problem

My wife mentioned to me offhand that she her organization's weekly "Voice of the Team" survey is moving to a biweekly cadence. This sparked my interest, mainly because I had no idea what she was talking about.

She explained it as such:

The survey is completely anonymous.
The first question is a 1-10 rating: "How satisfied are you with the team?"
The second question is a Yes/No, and it's a different question every week. This is typically statements about the leadership or the team in general.
The third question is an open response: "Do you have any other feedback?"

That's it. Three simple questions, sent weekly, to gauge the health of the team and to provide the team a manner in which to provide feedback.

My solution

Using Hyland's Microsoft 365 subscription, I created a simple Microsoft Form for the given week, as seen in the screenshot below.

Note that we are not recording names! Totally anonymous.

Then I copied that form multiple times, one for each subsequent week. I gave each of those forms a different second question:

AWS Hosting Team Leadership cares about me as an individual.
AWS Hosting Team Leadership holds the team accountable for its actions.

For now, those are the only three leadership-focused questions I've come up with. Future questions will likely include state-of-the-team type questions, in addition to some product roadmap ones.

Every Friday morning at 10am Eastern, I send out the following email:

Team,

In an effort to become more a more transparent and feedback-driven organization, I will be sending out a very short survey every week on Friday.

This survey is completely anonymous. While it requires you to be logged into your Hyland Office 365 account, names and emails are not recorded.

I strongly encourage you to respond honestly! The responses to these surveys will drive a portion of every department meeting.

https://forms.office.com/r/---

Thanks,
Tony

How it's going

My team spans three very distinct geographic regions: US, Poland, and India. Because I send this out Friday morning Eastern time, most of the non-US team members are not likely to respond on the same day.

I am not planning on closing the form until the next one comes out, so it gives the team a chance to provide feedback when they see fit.

I would love to get insight into other survey-style questions that anyone else might think fit in this type of feedback gathering! Let's hear it!

Don't Manage Terraform Enterprise With Terraform

Tony Morris — Sat, 07 Aug 2021 18:03:27 +0000

First, some facts.

Terraform Enterprise is a wonderful product. It's the self-hosted distribution of Terraform Cloud for organizations that want the privacy and scale of an enterprise-grade installation.
There exists a Terraform Cloud/Enterprise Provider that can easily template and manage how an organization creates Workspaces (and other TFC/TFE resources).

Given the generally painful user experience of entering dozens of Workspace Variables into the TFE, it makes sense that nearly everyone I've worked with has stated a desire to use the tfe provider to manage workspaces. I'm here to tell you why this ends up being a rougher idea than you hoped for.

Pricing

Terraform Enterprise is priced by the number of Workspaces you have.

They're not cheap.

If you start dedicated Workspaces to creating and managing other Workspaces, you're effectively shorting yourself out of your own licenses.

On the other hand, if you're in Terraform Cloud, you're not paying per-Workspace, so feel free to use this method if the following hiccups don't pertain to you.

For what it's worth, if HashiCorp had a concept of "Configuration Workspaces" that didn't hit against your Workspace count, then this would obviously not be an issue.

Multi-Step Updates

Let's say you can get over the pricing issue. Talking through how the Configuration Workspaces would be architected and deployed brings up some other potential issues.

First, a quick overview of how we had tried this out.

The Setup

Workspace Repository

The most important piece of functionality here is the Terraform module that uses the tfe provider to create the Workload Workspaces.

Everything about the Workload Workspace is contained within this Workspace Repository, including the Workspace Variables.

Another name for this repository could be a "Configuration Repository," as it configures the implementation of the Workload Repository.

Workload Repository

The Workload Repository defines the Terraform resources to create the workloads that you are configuring. For us, this is a bunch of resources from the aws provider, but it could be anything.

Configuration Workspace

The Configuration Workspace in Terraform Enterprise is pointed to the Workspace Repository. When it executes a Run, it generates Terraform Enterprise resources, such as the Workload Workspaces and the requisite Workspace Variables in each one.

Workload Workspace

The Workload Workspace is created by the Configuration Workspace and pointed to the Workload Repository. When it executes a Run, it generates Workload-specific resources. In our cases, this is generally AWS resources such as EC2 instances, EBS volumes, etc.

The Problem

The biggest problem with this setup is the back-and-forth you have to do in order to make changes.

Let's say you want to add a resource to the Workload Repository. This is straightforward, and it doesn't cause many issues. You would just commit your changes to the Workload Repository, and the Workload Workspace would pick those up.

What if, however, that introduces a new variable to the repository? The changes you would have to make in order to get it through the system look something like this:

Add the Workspace Variable resource to the Workspace Repository.
Push the Run through the Configuration Workspace to add the TFE Workspace Variable to the Workload Workspace.
Add the variable to the Workload Repository.
You can finally push the Run through the Workload Workspace.

Four steps for a simple variable addition seems like a tough solution to roll out to any team.

What Should You Do Instead?

Let's be honest. I'm certain there are some really creative ways to work around these limitations managing Terraform at-scale. Feel free to use them and tell me what they are! Hit me up on Twitter if you find a really neat solution!

For our use cases, we are building out a Control Plane that abstracts that business-level functions from TFE itself. So, instead of thinking about "I need to create this TFE Workspaces with these Workspace Variables," we are now thinking "This team needs to create this application cluster."

This abstraction is not unique. I've talked to many people in the community that do a similar thing.

This solution works really well for us because we have a number of backend systems that we need to integrate together during an "application cluster spin-up." These include SaaS tools, such as PagerDuty, Splunk, New Relic, and many others. Given that Terraform Enterprise has a robust API, it makes our Control Plane much more straightforward to implement.

Finding S3 Batch Operations Failures in CloudTrail

Tony Morris — Sun, 01 Aug 2021 14:29:34 +0000

At work, I recently got the distinct opportunity to copy millions of objects from one S3 bucket to another.

There are roughly a dozen separate ways to do this (as with everything in AWS), but the "right" way is to use an S3 Batch Operation to copy everything from an S3 Inventory Report.

The only problem with an S3 Batch Operation is that it fails in surprising and hidden ways, especially if there's a misconfigured IAM permission. For example, our most recent job was failing due to a missing KMS permission. To determine what the missing permission was, we would typically head to CloudTrail and hunt down the failed requests.

As S3 Batch Operations run as an assumed role, hunting these logs can be slightly more difficult, but we finally found the right way to accomplish it.

The first, most important, piece is to hunt down the S3 Batch Operation's Job ID. You'll find this on the details screen clear at the top.

S3 Batch Operation Job Details Screen

Next, shoot on over to CloudTrail and filter by User Name. The value you'll want to use is s3-batch-operations_{Job ID}, where {Job ID} is your S3 Batch Operation's Job ID retrieved in the previous step.

CloudTrail Filtering on Job ID User

Heads Up! If you've got KMS enabled for the Job, then you're going to get a whole heck of a lot of logs. I tend to download the logs to use Excel to look through it, but when you're moving more than 19 million records, you're going to have a bad time. If you need to dive into the reasons even more, I recommend using an Athena table.

In case anyone is curious, the missing permission for the role was kms:GenerateDataKey*. KMS is super fun.

Auto-Generating Terraform State Storage in Azure

Tony Morris — Sun, 15 Sep 2019 13:21:26 +0000

In my latest Azure/Terraform post, I touched on how I solved the “Chicken and Egg” problem with Terraform: how you need cloud resources in order to store Terraform state, but you can’t use Terraform to generate those cloud resources. This post details the solution to that problem.

The Problem

The "Chicken and Egg" problem with Terraform, for me, can be succinctly defined with four points:

I want to write Terraform to generate resources in a cloud provider.
I don't want to run Terraform locally.
I want to store my Terraform state in the same cloud provider.
How do I generate the cloud provider's resources that will store my Terraform state?

The Solution

Instead of relying on Terraform to generate the cloud provider's resources that will store your Terraform state, you instead use the cloud provider's native tools to generate and manage those resources. As I am currently working in Azure, I will use the Azure's az CLI tool. This type of solution is also relevant to AWS's aws CLI and GCP's gcloud CLI, if you want to use those instead.

The Script

In each of my repositories that house Terraform definitions (which is every repository at the end of the day), I have an extra script: create-storage.sh.

This is a Bash script because I develop using .NET Core, and my build/deploy machines are all Linux-based in Azure DevOps. This could just as easily be a PowerShell script if it needed to be executed in a Windows environment.

Heads Up!: If you want the most recent version of the script, please go here instead of trusting this static website.

#!/bin/sh

# Heads up! You need to define the following environment variables:
# RESOURCE_GROUP_NAME for the resource group that will contain the Azure Storage Account that will house your Terraform state files
# STORAGE_ACCOUNT_NAME for the name of the Azure Storage Account
# KEYVAULT_NAME to store the Storage Account's access key, so you don't have to manually keep track of it
# LOCATION for the location of the Azure resources
# KEYVAULT_SECRET_NAME for the name of the secret in Key VAult of the Storage Account's access key
# CONTAINER_NAME for the Azure Blob Storage's container that will hold the Terraform state file(s)

RESOURCE_GROUP_NAME=mm-terraform-rg
STORAGE_ACCOUNT_NAME=mmterraform
KEYVAULT_NAME=mm-terraform-kv

# Create resource group
az group create --name ${RESOURCE_GROUP_NAME} --location ${LOCATION}

# Create storage account
az storage account create --resource-group ${RESOURCE_GROUP_NAME} --name ${STORAGE_ACCOUNT_NAME} --sku Standard_LRS --encryption-services blob

# Get storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group ${RESOURCE_GROUP_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --query [0].value -o tsv)

# Create Key Vault
az keyvault create --name ${KEYVAULT_NAME} --resource-group ${RESOURCE_GROUP_NAME} --location ${LOCATION}

# Store account key in secret
az keyvault secret set --name ${KEYVAULT_SECRET_NAME} --vault-name ${KEYVAULT_NAME} --value $ACCOUNT_KEY

# Create blob container
az storage container create --name ${CONTAINER_NAME} --account-name ${STORAGE_ACCOUNT_NAME} --account-key $ACCOUNT_KEY

Environment Variables

There are a few environment variables that are required with the script above.

Variable	Purpose
`RESOURCE_GROUP_NAME`	The name of the resource group that will house all of the resources generated for the Terraform state storage
`LOCATION`	The location of the resource group (and subsequent resources located within the resource group)
`STORAGE_ACCOUNT_NAME`	The name of the Azure Storage Account that we will be creating blob storage within
`CONTAINER_NAME`	The name of the Azure Storage Container in the Azure Blob Storage. This will actually hold the Terraform state files
`KEYVAULT_NAME`	The name of the Azure Key Vault to create to store the Azure Storage Account key. This allows us to automate the running of Terraform files without having to store the key ourselves
`KEYVAULT_SECRET_NAME`	The name of the secret that will store the Azure Storage Account key

Pay special note to the fact that we are generating an Azure Key Vault and Azure Key Vault Secret to manage the storage of the Azure Storage Account's key. While this isn't technically necessary, and we could just query the Azure Storage Account itself for the key anytime we needed it (as seen in the Get storage account key section of the script), Azure DevOps has tight integration with Azure Key Vault, and this step simplifies our future deployment of Terraform resources.

The Process

The functionality in the file itself is pretty similar. The file itself isn't special, as it's mostly the same as what Azure provides in their docs. The interesting bits that are special are the exporting of the Storage Account's key into Key Vault for later use in an Azure Pipeline during deploy-time.

The Pipeline

Having this file exist is useful, but it doesn't truly tie together delivery of the resources and the "auto-generating" part in a real-world scenario. To accomplish this, I use Azure Pipelines in the Azure DevOps offering.

Build Pipeline

As Azure Pipelines can use YAML to define the build pipelines, I've got an azure-pipelines.yml file in the root of each repository. Within that YAML file, I've got the following snippet:

variables:
  AZURE_SUBSCRIPTION: 'xxx'
  APPLICATION: 'xxx'
  CONTAINER_NAME: $(ENVIRONMENT_PREFIX)terraform
  KEYVAULT_NAME: $(APPLICATION)-terraform-kv
  KEYVAULT_SECRET_NAME: $(ENVIRONMENT_PREFIX)-storage-account-key
  LOCATION: 'eastus'
  RESOURCE_GROUP_NAME: $(APPLICATION)-terraform-rg
  STORAGE_ACCOUNT_NAME: $(APPLICATION)terraform
  TF_IN_AUTOMATION: 'true'

stages:
- stage: Setup
  jobs:
  - job: SetupDevelopmentStorage
    variables:
      ENVIRONMENT_PREFIX: 'd'
      ENVIRONMENT_NAME: 'development'
    displayName: 'Setup Development Storage'
    steps:
    - task: AzureCLI@1
      displayName: 'Run Setup Script'
      inputs:
        azureSubscription: $(AZURE_SUBSCRIPTION)
        scriptPath: './create-storage.sh'
  - job: SetupStagingStorage
    variables:
      ENVIRONMENT_PREFIX: 's'
      ENVIRONMENT_NAME: 'staging'
    displayName: 'Setup Staging Storage'
    steps:
    - task: AzureCLI@1
      displayName: 'Run Setup Script'
      inputs:
        azureSubscription: $(AZURE_SUBSCRIPTION)
        scriptPath: './create-storage.sh'

  - job: SetupProductionStorage
    variables:
      ENVIRONMENT_PREFIX: 'p'
      ENVIRONMENT_NAME: 'production'
    displayName: 'Setup Production Storage'
    steps:
    - task: AzureCLI@1
      displayName: 'Run Setup Script'
      inputs:
        azureSubscription: $(AZURE_SUBSCRIPTION)
        scriptPath: './create-storage.sh'

Nothing too special in this build pipeline, to be clear. It is just executing the create-storage.sh script with different environment-specific parameters being passed in. You can see that I use the environment-specific parameters in the definition of the other environment variables (e.g., CONTAINER_NAME has $(ENVIRONMENT_PREFIX) at the start of its definition). This lets me have separate containers in the same Azure Storage Account that house environment-specific Terraform state files. Alternatively, I could just use differently-named Terraform state files, but I like consistency for the file names themselves.

Deploy Pipeline

This is where the magic happens. As of the time of this writing, I don't currently use multi-stage pipelines, so this is all done within the Classic Release Pipelines web UI.

The first step in the Release Pipeline is to retrieve the Key Vault secret that was stored from the create-storage.sh script via the AzureKeyVault step. This is then stored within an Azure Pipelines variable named after the secret name itself. For example, if the secret name is d-storage-account-key, the Azure Pipeline's variable will also be d-storage-account-key. We will see this in use in a future step.

For the Terraform-specific steps, I use the very wonderful Terraform Build and Release Tasks by Charles Zipp found on Azure Marketplace.

The first Terraform step I use is merely to install Terraform to the agent, using the TerraformInstaller step. As of this writing, I am using Terraform v0.12.3, but I'm sure that'll change soon.

The second Terraform step is to run terraform init by using the TerraformCLI step. I pass in the following Command Options: -backend-config="access_key=$(d-storage-account-key)" -backend-config="storage_account_name=$(APPLICATION)terraform" -backend-config="container_name=$(ENVIRONMENT_PREFIX)terraform" -backend-config="key=$(APPLICATION).tfstate".

As I mentioned previously, you'll see that I'm using the $(d-storage-account-key) Azure Pipeline variable to retrieve the access key. Straightforward, but something to keep in mind if you're building it out.

The final Terraform step is to run terraform apply by using the TerraformCLI step again. The only Command Options I pass in are the environment-specific Terraform variables: -var-file="./environments/$(ENVIRONMENT_NAME)/terraform.tfvars". The usage of these variables can be seen in the previous Azure/Terraform post.

tl;dr

Use native cloud provider CLI tools to generate and store access keys for use later in build and deploy pipelines. Pretty straightforward, yeah?

Azure and Terraform, Round Two

Tony Morris — Sun, 08 Sep 2019 20:37:47 +0000

I recently blogged about using Terraform to manage resources in Azure. To be honest, my implementation was okay, but it could definitely improve. This post is an update on how I’ve updated the structure and usage of Terraform within projects.

Project Structure

On any given project that has Terraform resources, my folder structure looks like this:

project
│   .gitignore
│   azure-pipelines.yml
│   create-storage.sh  
│
└───terraform
    │   data.tf
    │   locals.tf
    │   main.tf
    │   provider.tf
    │   variables.tf
    │   versions.tf
    │
    └───environments
        └───development
            │   terraform.tfvars
        └───staging
            │   terraform.tfvars
        └───production
            │   terraform.tfvars

.gitignore

Pretty standard .gitignore file here. I use JetBrains IDEs, so I pull in the IntelliJ-standard entries, plus a few more. Go here for the exact .gitignore I use.

azure-pipelines.yml

As my resources are in Azure, it makes sense to use Azure DevOps for build and deploy pipelines. The build pipeline is explicitly defined with Azure Pipeline's YAML schema. The release pipeline, unfortunately, is currently only defined within the web UI of Azure Pipelines (it's really just a terraform apply at the end of the day, anyway).

Generally speaking, the Terraform bits in my azure-pipelines.yml is the same from project to project. Note that I truncated the file to only include the development environment, but the other environments are basically the same but with updated variables.

variables:
  AZURE_SUBSCRIPTION: 'xxx'
  BASE_ENVIRONMENT_PATH: 'terraform/environments/$(ENVIRONMENT_NAME)'
  CONTAINER_NAME: $(ENVIRONMENT_PREFIX)terraform
  KEYVAULT_NAME: 'product-terraform-kv'
  KEYVAULT_SECRET_NAME: $(ENVIRONMENT_PREFIX)-storage-account-key
  LOCATION: 'eastus'
  STORAGE_ACCOUNT_NAME: 'productterraform'
  TERRAFORM_PATH: 'terraform'
  TERRAFORM_STATE: 'product_infrastructure.tfstate'
  TERRAFORM_VERSION: '0.12.3'
  TF_IN_AUTOMATION: 'true'

stages:
- stage: Setup
  jobs:
  - job: SetupDevelopmentStorage
    variables:
      ENVIRONMENT_PREFIX: 'd'
      ENVIRONMENT_NAME: 'development'
    displayName: 'Setup Development Storage'
    steps:
    - task: AzureCLI@1
      displayName: 'Run Setup Script'
      inputs:
        azureSubscription: $(AZURE_SUBSCRIPTION)
        scriptPath: './create-storage.sh'

- stage: Test
  dependsOn: Setup
  jobs:
  - job: TestDevelopmentTerraform
    variables:
      ENVIRONMENT_PREFIX: 'd'
      ENVIRONMENT_NAME: 'development'
    displayName: 'Test Development Terraform'
    steps:
    - task: AzureKeyVault@1
      displayName: 'Azure Key Vault: $(KEYVAULT_NAME)'
      inputs:
        azureSubscription: '$(AZURE_SUBSCRIPTION)'
        KeyVaultName: '$(KEYVAULT_NAME)'
        SecretsFilter: '$(KEYVAULT_SECRET_NAME)'

    - task: charleszipp.azure-pipelines-tasks-terraform.azure-pipelines-tasks-terraform-installer.TerraformInstaller@0
      displayName: 'Use Terraform $(TERRAFORM_VERSION)'
      inputs:
        terraformVersion: $(TERRAFORM_VERSION)

    - task: charleszipp.azure-pipelines-tasks-terraform.azure-pipelines-tasks-terraform-cli.TerraformCLI@0
      displayName: 'terraform init'
      inputs:
        command: init
        workingDirectory: '$(BASE_ENVIRONMENT_PATH)'
        commandOptions: '-backend-config="access_key=$(d-storage-account-key)" -backend-config="storage_account_name=$(STORAGE_ACCOUNT_NAME)" -backend-config="container_name=$(ENVIRONMENT_PREFIX)terraform" -backend-config="key=$(TERRAFORM_STATE)"'

    - task: charleszipp.azure-pipelines-tasks-terraform.azure-pipelines-tasks-terraform-cli.TerraformCLI@0
      displayName: 'terraform validate'
      inputs:
        command: validate
        workingDirectory: '$(BASE_ENVIRONMENT_PATH)'
        commandOptions: '-var-file=".\environments\$(ENVIRONMENT_NAME)\terraform.tfvars"'

- stage: Package
  dependsOn: Test
  jobs:
  - job: PackageTerraform
    displayName: 'Package Terraform'
    steps:
    - task: PublishBuildArtifacts@1
      displayName: 'Publish Terraform Artifacts'
      inputs:
        pathToPublish: '$(TERRAFORM_PATH)'
        artifactName: tf

That's a lot of configuration, but I'll attempt to condense it down. The pipeline is broken up into three separate Stages: Setup, Test, and Package.

Setup Stage

The Setup stage solves what I call "The Chicken and Egg Problem." It boils down to requiring Azure resources to store Terraform state, but we cannot create those Azure resources via Terraform because it doesn't know where store it yet. Instead of relying on Terraform to create those resources, I call a separate script. It sets some environment variables, and then it calls out to a shell script located in source: create-storage.sh. The contents of this script are below.

#!/bin/sh

RESOURCE_GROUP_NAME=product-terraform-rg
STORAGE_ACCOUNT_NAME=productterraform
KEYVAULT_NAME=product-terraform-kv

# Create resource group
az group create --name $RESOURCE_GROUP_NAME --location ${LOCATION}

# Create storage account
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob

# Get storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query [0].value -o tsv)

# Create Key Vault
az keyvault create --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP_NAME --location ${LOCATION}

# Store account key in secret
az keyvault secret set --name ${KEYVAULT_SECRET_NAME} --vault-name $KEYVAULT_NAME --value $ACCOUNT_KEY

# Create blob container
az storage container create --name ${CONTAINER_NAME} --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY

The script itself is pretty straightforward. It ensures a standard resource group for each given product exists. Within that resource group, it creates a storage account, key vault, key vault secret, and a blob container. The script pulls the storage account's key from the Azure CLI and stores it within the key vault secret. This key will be used to in future terraform init calls. The blob container will hold the Terraform state files created later in the process.

In case the application being deployed to Azure requires a database, I have a slightly altered version of the script that will generate a random database password and store it within the same key vault, but in a separate secret. That version can be seen below.

#!/bin/sh

RESOURCE_GROUP_NAME=product-terraform-rg
STORAGE_ACCOUNT_NAME=productterraform
KEYVAULT_NAME=product-terraform-kv

# Create resource group
az group create --name $RESOURCE_GROUP_NAME --location $LOCATION > /dev/null

# Create storage account
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob > /dev/null

# Get storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query [0].value -o tsv)

# Create Key Vault
az keyvault create --name $KEYVAULT_NAME --resource-group $RESOURCE_GROUP_NAME --location $LOCATION > /dev/null

# Store account key in secret
az keyvault secret set --name $KEYVAULT_SECRET_NAME --vault-name $KEYVAULT_NAME --value $ACCOUNT_KEY > /dev/null

# Check if database password exists
DB_SECRET_INFO=$((az keyvault secret show --name $DB_PASSWORD_SECRET_NAME --vault-name $KEYVAULT_NAME) 2>&1)

# Create the database password if it doesn't exist
if [[ $DB_SECRET_INFO =~ "(SecretNotFound)" ]]; then
  NEW_UUID=$(openssl rand -base64 24)
  az keyvault secret set --name $DB_PASSWORD_SECRET_NAME --vault-name $KEYVAULT_NAME --value $NEW_UUID > /dev/null
fi

# Create blob container
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY > /dev/null

As you can see, this is mostly the same script, but with a small UUID generator if the database password has not already been generated. There are a number of ways to generate a random string, but the openssl rand -base64 24 was the most straightforward (and it worked on the Azure Linux worker machines).

Test Stage

The Test Stage installs a specific version of Terraform, runs a terraform init with assistance from the values retrieved from the previously-created key vault, and then runs a terraform validate.

You'll notice that the terraform-init uses the $(d-storage-account-key) variable. The Azure Key Vault step prior to that will pull out the value from the key vault secret into that variable. Unfortunately, I haven't discovered a way to double-reference a variable, so I have to keep it as a hard-coded reference. For reference, I would much rather have something like $($(KEYVAULT_SECRET_NAME)), but that doesn't seem to be possible currently.

The terraform validate step's details are important: it points directly to the environment-specific terraform.tfvars. This is how I accomplish multi-environment releases with a single codebase.

Package Stage

The Package Stage is the simplest of the pipeline: it just runs an out-of-the-box PublishBuildArtifacts task, pointed to the terraform directory and dropping it into the tf artifact. This will be used later in the release pipeline.

Terraform Artifacts

I've broken down the Terraform artifacts into a number of files for ease of use.

data.tf

For infrastructure-only repositories, this file is very straightforward:

data "azurerm_subscription" "this" {
}

However, if the given repository is building off another repository (e.g., an application-specific repository building on top of an infrastructure-specific repository), there will obviously be other data blocks here. A sample one can be seen below.

data "azurerm_resource_group" "this" {
  name = local.resource_group_name
}

data "azurerm_app_service_plan" "this" {
  name                = local.app_service_plan_name
  resource_group_name = data.azurerm_resource_group.this.name
}

data "azurerm_storage_account" "this" {
  name                = local.storage_account_name
  resource_group_name = data.azurerm_resource_group.this.name
}

data "azurerm_sql_server" "this" {
  name                = local.sql_server_name
  resource_group_name = data.azurerm_resource_group.this.name
}

locals.tf

I typically use the locals.tf file to define aggregated resource names that I'm going to be using in a number of places.

locals {
  resource_group_name   = "${var.environment_prefix}-${var.application_name}-rg"
  app_service_plan_name = "${var.environment_prefix}-${var.application_name}-plan"
  scope                 = "/subscriptions/${var.subscription_id}/resourceGroups/${azurerm_resource_group.this.name}"
}

main.tf

My main.tf is where I create the Azure resources themselves. There's very little interesting or unique about this file, except that I'm generally not creating my own modules to group items. I simply haven't had a good reason to at this point.

It is likely useful to point out that each repository only has one main.tf defined. This is important, as it alludes to the fact that each environment has the same types of Azure resources. While everything is variable-driven, so the resources themselves can be configured differently, each different environment will have the same resources in total.

provider.tf

Nothing crazy here.

terraform {
  backend "azurerm" {
  }
}

provider "azurerm" {
  version = "~>1.30.1"
}

I try to make it a point to upgrade my provider and Terraform versions as much as possible, but I'm typically working across 10-15 repositories at a time, so once I get all the repositories on a single version, I'll stick to that version for awhile.

variables.tf

Again, nothing special here. Fancy new Terraform v0.12 usage in the role_assignments variable below!

variable "environment_prefix" {
}

variable "application_name" {
}

variable "location" {
}

variable "subscription_id" {
}

variable "app_service_plan_sku_tier" {
}

variable "app_service_plan_sku_size" {
}

variable "tags" {
  type = map(string)
}

variable "role_assignments" {
  type = list(object({ username = string, object_id = string, role_definition = string }))
}

versions.tf

I like to explicitly define what version of Terraform to support for a given repository. This is where that's done.

terraform {
  required_version = ">= 0.12"
}

terraform.tfvars

Each environment has its own terraform.tfvars file. This is where the values for the given variables (defined in variables.tf above) are passed in if they are free to be exposed publicly. If there are secret values that need to be passed in, they are stored within a key vault and pulled in during the release pipeline, similar to the storage account key above.

environment_prefix        = "d"
application_name          = "product"
location                  = "eastus"
subscription_id           = "xxx"
app_service_plan_sku_tier = "Shared"
app_service_plan_sku_size = "D1"

tags = {
  "terraform"   = "true",
  "environment" = "Development",
  "application" = "Product"
}

role_assignments = [
  {
    username        = "xxx"
    object_id       = "xxx",
    role_definition = "Owner"
  }
]

Release Pipeline

As stated previously, Azure DevOps has a limitation in that it only allows Release Pipelines to be edited with the in-browser UI. This sucks, but I've come to live with it.

The Release Pipeline for any given project generally looks the same:

Pull secrets from Azure Key Vault
Install Terraform
Run terraform init
Run terraform apply

Then, if the pipeline requires it, and there's an application to deploy:

Set Terraform outputs to Azure Pipeline variables
Deploy application to Azure App Services
Set values from pipeline variables as necessary

This section is intentionally light on details, as there's not really much to talk about it.

tl;dr

All-in-all, my approach to Terraform on Azure has changed pretty heavily in the past 7ish months. Instead of defining resources for each environment, I've now consolidated resource creation into a single file, and I'm setting the variables in each environment directory instead. Again, this is explicitly because I don't have a use case which requires different resources per environment.

In addition to the project structure changes, the "Chicken and Egg Problem" has been solved within the Azure Pipeline itself. Instead of having to manually create resources before running Terraform the first time, I can now rely on the pipeline itself to manage the backing data storage. This has been my biggest improvement to how I run pipelines in Azure DevOps.

As always, if there's something you want to chat about more directly, hit me up on Twitter, as that's where I'm most active.

Don't You Dare Update That Copyright Manually

Tony Morris — Fri, 18 Jan 2019 13:44:34 +0000

A quick reminder for developers: now that we're in a new year, if you update your copyright to manually be "2019" there's almost certainly a spot for you in hell. Don't kick this down the road--solve the problem forever with something like DateTime.UtcNow.Year (or whatever your language of choice does for current year).

I just approved a ticket that changes 2016-2018 to 2016-{DateTime.UtcNow.Year} because there were three separate pull requests in the past that didn't address this. Now, barring any systemic changes to how .NET calculates the current UTC DateTime, we will never have to update this again.

July 12, 2016

February 16, 2017

July 13, 2018

January 17, 2019

Azure and Terraform

Tony Morris — Tue, 15 Jan 2019 00:00:00 +0000

I have always believed that Delivery is one of the most important aspects of software development. I blogged about it previously (My Core Values). Software delivery isn’t just putting the bits into the final resting location; it must also include the infrastructure provisioning to explicitly define where the bits will actually land. Terraform helps bridge that gap, especially given a public cloud offering like Azure.

The Project

I was recently contracted to implement a deployment pipeline for a financial services startup. The client had a special need to have the application environments built out in a reliable, scalable manner. There were two applications within the client's solution: a single-page application and a web service tier.

From a non-functional perspective, there were a number of requirements from the technology team:

The development team must be able to deploy updated code assets to the environment in a reliable, well-known manner, without manual intervention
The QA team must be able to deploy a feature branch to a specified environment for feature-specific testing
System administrators must be able to quickly commission and decommission all aspects of environments, including the entire environment itself

The Proposal

The non-functional requirements detailed above pointed me in an obvious location: host the application in Microsoft Azure and to utilize as many platform services as possible. Additionally, utilize Azure DevOps products to enable tight integration between the application code and the application environments.

Microsoft Azure

Within Azure, App Services would be used as application hosts and Azure Database for PostgreSQL would be used for the database management. In addition to the core platform services, there will also be a few other products involved:

Key Vault: Used to manage application secrets (e.g., connection strings, encryption keys, etc.)
Application Insights Used to provide Application Performance Monitoring for both the single-page application and the web service
Storage: Used as a blob storage location for the applications

Azure DevOps

Azure DevOps (previously known as Visual Studio Team Services, previously known as Team Foundation Server) was chosen as the set of tools to manage source control and the build and release pipelines. Azure Repos is the remote source control repository and Azure Pipelines is the build and release pipeline tool.

In a future iteration, it is possible that the work item tracking will migrate to Azure Boards from Jira and to Azure Test Plans from Zephyr, but this was not on the initial set of work.

Terraform

Terraform is a product from HashiCorp to implement Infrastructure-as-Code.

That sounds like a lot of jargon, so let's boil it down: Infrastructure-as-Code (IaC) allows you to prescriptively define your infrastructure implementation in source control. Terraform is then a tool that executes these definitions to ensure the implemented infrastructure is equivalent to the specification. It moves the best-of-breed software development practices (e.g., source control, code reviews, deployment pipelines, etc.) into the infrastructure management realm.

Additionally, Terraform was chosen as the IaC tool rather than Azure Resource Manager Templates (ARM Templates) due to the extensive Terraform community and my personal expertise. I've worked with ARM Templates previously, but Terraform offered the same output with less initial startup work. While I hate YAML with an undying passion, it is much less verbose than the JSON that is used with ARM Templates.

The Solution

Implementing the solutions for each of the applications was generally the same. The only main difference was the Terraform implementation and the steps within the Azure Build Pipeline itself, but the concepts are similar.

Azure Pipelines

Azure Pipelines has two concepts: Build and Release Pipelines. While I generally believe this should be one pipeline in a perfect world, I can understand why they're separated. The Build Pipeline is used to generate artifacts, and the Release Pipeline is used to move those artifacts to separate stages (i.e., environments).

Recently, Azure Pipelines added source control-defined Build Pipelines to its offering. As a firm believer in source controlling everything, an azure-pipeline.yml file now exists in both the single-page application's repository and the web service's repository. These are detailed in the sections below.

Single-Page Application Build Pipeline

The single-page application's pipeline can be seen in its entirety below. At a high level, it's a straightforward build:

Run npm install to get the application's dependencies
Lint the application to ensure it conforms to the Angular's specifications
Build the application
Publish the built application
Publish the Terraform artifacts

Both the built application and the Terraform artifacts are used in the related Release pipeline.

It is important to note that this build pipeline is used for all branches. This is to ensure the entire repository is deployable at any given commit, assuming it passes the build pipeline successfully.

In the future, the application team plans to implement automated tests in this pipeline to ensure the application is stable enough to push through to a release pipeline.

# Single Page Application

pool:
  vmImage: 'Ubuntu-16.04'

variables:
  publishPath: 'dist/single-page-app'
  terraformPath: 'terraform'

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '8.x'
  displayName: 'Install Node.js'

- script: |
    npm install -g @angular/cli
    npm install
  displayName: 'npm install'

- script: |
    ng lint
  displayName: 'ng lint'

- script: |
    ng build
  displayName: 'ng build'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Website Artifacts'
  inputs:
    pathtoPublish: '$(publishPath)'
    artifactName: drop

- task: PublishBuildArtifacts@1
  displayName: 'Publish Terraform'
  inputs:
    pathtoPublish: '$(terraformPath)'
    artifactName: terraform

Web Services Build Pipeline

While the web services pipeline has many more steps, it accomplishes roughly the same concepts as the single-page application:

Restore the solution's dependencies with nuget restore
Build the background data processor
Build the web service
Build the database migrator application (used to run SQL scripts needed for the environment (e.g., schema changes, data changes, etc.))
Build the database seeder application (used to setup first-time data for the SQL database (i.e., default database users and their roles))
Publish each of the built applications and Terraform

As before, the built applications and Terraform are used in the future release pipeline. Additionally, this build is run for all branches for the same reason as before: having a deployable codebase is vitally important.

You'll notice that dotnet publish is used for both the database migrator and database seeder projects; this is on purpose, as the application is planned to move toward .NET Core in the future.

The absence of any automated testing (e.g., unit tests) is especially glaring in this build pipeline. The application development team is planning on adding those in the future, as with the single-page application.

# Web Service

pool:
  vmImage: 'VS2017-Win2016'

variables:
  solution: '**/*.sln'
  buildConfiguration: 'Release'
  processorProject: 'WebService.Processor.csproj'
  webServiceProject: 'WebService.csproj'
  migratorProject: 'WebService.DbMigrator.csproj'
  seederProject: 'WebService.DbSeeder.csproj'
  webServicePublishPath: 'WebService/obj/Release/Package'
  migratorPublishPath: 'WebService.DbMigrator/bin/Debug/netcoreapp2.1/publish'
  seederPublishPath: 'WebService.DbSeeder/bin/Debug/netcoreapp2.1/publish'

steps:
- task: NuGetToolInstaller@0
  displayName: 'Install NuGet'

- task: NuGetCommand@2
  displayName: 'Solution NuGet Restore'
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  displayName: 'Build Processor'
  inputs:
    solution: '$(processorProject)'
    configuration: '$(buildConfiguration)'

- task: VSBuild@1
  displayName: 'Build Web Service'
  inputs:
    solution: '$(webServiceProject)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true'
    configuration: '$(buildConfiguration)'

- script: dotnet publish $(migratorProject)
  displayName: 'Build and Package Database Migrator'

- script: dotnet publish $(seederProject)
  displayName: 'Build and Package Database Seeder'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Web Service Artifacts'
  inputs:
    pathtoPublish: '$(webServicePublishPath)'
    artifactName: drop

- task: PublishBuildArtifacts@1
  displayName: 'Publish Terraform Artifacts'
  inputs:
    pathtoPublish: 'terraform'
    artifactName: terraform

- task: PublishBuildArtifacts@1
  displayName: 'Publish Database Migrator Artifacts'
  inputs:
    pathtoPublish: '$(migratorPublishPath)'
    artifactName: migrator

- task: PublishBuildArtifacts@1
  displayName: 'Publish Database Seeder Artifacts'
  inputs:
    pathtoPublish: '$(seederPublishPath)'
    artifactName: seeder

Terraform Configuration

For each of the applications, I define the infrastructure and platform services alongside the application's source code, generally within a terraform folder. Each terraform folder is organized as such:

terraform
  env-dev
    destroy.bat
    main.tf
    setup.bat
    variables.tf
  env-production
    destroy.bat
    main.tf
    setup.bat
    variables.tf
  env-staging
    destroy.bat
    main.tf
    setup.bat
    variables.tf
  env-test-1
    destroy.bat
    main.tf
    setup.bat
    variables.tf
  env-test-2
    destroy.bat
    main.tf
    setup.bat
    variables.tf
  env-test-3
    destroy.bat
    main.tf
    setup.bat
    variables.tf
  modules
    ... some number of modules ...

Modules

I use Terraform modules to encapsulate specific sections of the infrastructure. The modules that I'm using across the two applications are detailed below.

Name	Short Name	Description	Azure Resources
API	api	Defines the Web Services hosting services.	App Service Plan Application Insights App Service
Database	db	Defines the PostgreSQL server and database.	PostgreSQL Server PostgreSQL Firewall Rules PostgreSQL Database
Key Vault	kv	Defines the secret store and the default secrets.	Key Vault Key Vault Access Policies Key Vault Secrets
Resource Group	rg	Defines the resource group for the given environment that contains all the Azure resources.	Resource Group
Storage	storage	Defines the storage account used by the application.	Storage Account
Web	web	Defines the Single-Page Application hosting services.	App Service Plan Application Insights App Service

To be a bit more explicit, below are the exact files that I am using for each of these modules.

API

Folder Structure

modules
  api
    main.tf
    variables.tf

Files

# main.tf

resource "azurerm_app_service_plan" "default" {
  name                = "${var.environment_prefix}-ws-plan"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  tags                = "${var.tags}"

  sku {
    tier = "${var.app_service_plan_sku_tier}"
    size = "${var.app_service_plan_sku_size}"
  }
}

resource "azurerm_application_insights" "default" {
  name                = "${var.environment_prefix}-ws-ai"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  application_type    = "Web"
  tags                = "${var.tags}"
}

resource "azurerm_app_service" "api" {
  name                = "${var.environment_prefix}-ws"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  app_service_plan_id = "${azurerm_app_service_plan.default.id}"
  tags                = "${var.tags}"

  app_settings {
    "APPINSIGHTS_INSTRUMENTATIONKEY"                  = "${azurerm_application_insights.default.instrumentation_key}"
    "APPINSIGHTS_PROFILERFEATURE_VERSION"             = "1.0.0"
    "APPINSIGHTS_SNAPSHOTFEATURE_VERSION"             = "1.0.0"
    "ApplicationInsightsAgent_EXTENSION_VERSION"      = "~2"
    "DiagnosticServices_EXTENSION_VERSION"            = "~3"
    "InstrumentationEngine_EXTENSION_VERSION"         = "~1"
    "SnapshotDebugger_EXTENSION_VERSION"              = "~1"
    "XDT_MicrosoftApplicationInsights_BaseExtensions" = "~1"
    "XDT_MicrosoftApplicationInsights_Mode"           = "recommended"
  }
}

# variables.tf

variable "environment_prefix" {}
variable "location" {}

variable "tags" {
  type = "map"
}

variable "resource_group_name" {}
variable "app_service_plan_sku_tier" {}
variable "app_service_plan_sku_size" {}

Database

Folder Structure

modules
  db
    main.tf
    variables.tf

Files

# main.tf

resource "azurerm_postgresql_server" "default" {
  name                = "${var.environment_prefix}-app-db"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  tags                = "${var.tags}"

  sku {
    name     = "${var.db_sku}"
    capacity = "${var.db_capacity}"
    tier     = "${var.db_tier}"
    family   = "${var.db_family}"
  }

  storage_profile {
    storage_mb            = "${var.db_storage_mb}"
    backup_retention_days = "${var.db_backup_retention_days}"
    geo_redundant_backup  = "${var.db_geo_redundant_backup}"
  }

  administrator_login          = "${var.db_administrator_login}"
  administrator_login_password = "${var.db_administrator_login_password}"
  version                      = "${var.db_version}"
  ssl_enforcement              = "${var.db_ssl_enforcement}"
}

resource "azurerm_postgresql_firewall_rule" "azure" {
  name                = "azure"
  resource_group_name = "${var.resource_group_name}"
  server_name         = "${azurerm_postgresql_server.default.name}"
  start_ip_address    = "0.0.0.0"
  end_ip_address      = "0.0.0.0"
}

resource "azurerm_postgresql_firewall_rule" "userhome" {
  name                = "user-home"
  resource_group_name = "${var.resource_group_name}"
  server_name         = "${azurerm_postgresql_server.default.name}"
  start_ip_address    = "xx.xx.xxx.xxx"
  end_ip_address      = "xx.xx.xxx.xxx"
}

resource "azurerm_postgresql_database" "app" {
  name                = "app"
  resource_group_name = "${var.resource_group_name}"
  server_name         = "${azurerm_postgresql_server.default.name}"
  charset             = "UTF8"
  collation           = "English_United States.1252"
}

# variables.tf

variable "environment_prefix" {}
variable "location" {}

variable "tags" {
  type = "map"
}

variable "resource_group_name" {}
variable "db_sku" {}
variable "db_capacity" {}
variable "db_tier" {}
variable "db_family" {}
variable "db_storage_mb" {}
variable "db_backup_retention_days" {}
variable "db_geo_redundant_backup" {}
variable "db_administrator_login" {}
variable "db_administrator_login_password" {}
variable "db_version" {}
variable "db_ssl_enforcement" {}

Key Vault

Folder Structure

modules
  kv
    main.tf
    variables.tf

Files

# main.tf

data "azurerm_client_config" "current" {}

resource "azurerm_key_vault" "default" {
  name                            = "${var.environment_prefix}-app-kv"
  location                        = "${var.location}"
  resource_group_name             = "${var.resource_group_name}"
  tags                            = "${var.tags}"
  enabled_for_template_deployment = true
  tenant_id                       = "${var.tenant_id}"

  sku {
    name = "standard"
  }
}

resource "azurerm_key_vault_access_policy" "azuredevops" {
  vault_name          = "${azurerm_key_vault.default.name}"
  resource_group_name = "${var.resource_group_name}"
  tenant_id           = "${var.tenant_id}"
  object_id           = "${var.azuredevops_object_id}"

  secret_permissions = [
    "get",
    "list",
    "set",
    "delete",
  ]
}

resource "azurerm_key_vault_access_policy" "current" {
  vault_name          = "${azurerm_key_vault.default.name}"
  resource_group_name = "${var.resource_group_name}"
  tenant_id           = "${data.azurerm_client_config.current.tenant_id}"
  object_id           = "${data.azurerm_client_config.current.service_principal_object_id}"

  secret_permissions = [
    "get",
    "list",
    "set",
    "delete",
  ]
}

resource "azurerm_key_vault_access_policy" "adminuser" {
  vault_name          = "${azurerm_key_vault.default.name}"
  resource_group_name = "${var.resource_group_name}"
  tenant_id           = "${var.tenant_id}"
  object_id           = "${var.adminuser_object_id}"

  secret_permissions = [
    "get",
    "list",
    "set",
    "delete",
    "recover",
    "backup",
    "restore",
  ]

  certificate_permissions = [
    "create",
    "delete",
    "deleteissuers",
    "get",
    "getissuers",
    "import",
    "list",
    "listissuers",
    "managecontacts",
    "manageissuers",
    "purge",
    "recover",
    "setissuers",
    "update",
    "backup",
    "restore",
  ]

  key_permissions = [
    "get",
    "list",
    "update",
    "create",
    "import",
    "delete",
    "recover",
    "backup",
    "restore",
  ]
}

resource "azurerm_key_vault_secret" "secret_1" {
  name      = "secret_1"
  value     = "${var.key_vault_secret_1}"
  vault_uri = "${azurerm_key_vault.default.vault_uri}"
  tags      = "${var.tags}"
}

resource "azurerm_key_vault_secret" "secret_2" {
  name      = "secret_2"
  value     = "${var.key_vault_secret_2}"
  vault_uri = "${azurerm_key_vault.default.vault_uri}"
  tags      = "${var.tags}"
}

# variables.tf

variable "environment_prefix" {}
variable "location" {}

variable "tags" {
  type = "map"
}

variable "resource_group_name" {}

variable "tenant_id" {
  type        = "string"
  description = "The Azure Active Directory tenant ID that should be used for authenticating requests to the key vault."
  default     = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

variable "azuredevops_object_id" {
  default     = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

variable "adminuser_object_id" {
  default     = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

variable "key_vault_secret_1" {}
variable "key_vault_secret_2" {}

Resource Group

Folder Structure

modules
  rg
    main.tf
    output.tf
    variables.tf

Files

# main.tf

resource "azurerm_resource_group" "default" {
  name     = "${var.environment_prefix}-app-rg"
  location = "${var.location}"
  tags     = "${var.tags}"
}

# output.tf

output "name" {
  value = "${azurerm_resource_group.default.name}"
}

output "location" {
  value = "${azurerm_resource_group.default.location}"
}

# variables.tf

variable "environment_prefix" {}
variable "location" {}

variable "tags" {
  type = "map"
}

Storage

Folder Structure

modules
  storage
    main.tf
    variables.tf

Files

# main.tf

resource "azurerm_storage_account" "default" {
  name                     = "${var.environment_prefix}appstorage"
  location                 = "${var.location}"
  resource_group_name      = "${var.resource_group_name}"
  tags                     = "${var.tags}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

# variables.tf

variable "environment_prefix" {}
variable "location" {}

variable "tags" {
  type = "map"
}

variable "resource_group_name" {}

Web

Folder Structure

modules
  web
    main.tf
    variables.tf

Files

# main.tf

resource "azurerm_app_service_plan" "default" {
  name                = "${var.environment_prefix}-web-plan"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"

  sku {
    tier = "${var.app_service_plan_sku_tier}"
    size = "${var.app_service_plan_sku_size}"
  }
}

resource "azurerm_application_insights" "default" {
  name                = "${var.environment_prefix}-web-ai"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  application_type    = "Web"
}

resource "azurerm_app_service" "web" {
  name                = "${var.environment_prefix}-web"
  location            = "${var.location}"
  resource_group_name = "${var.resource_group_name}"
  app_service_plan_id = "${azurerm_app_service_plan.default.id}"

  app_settings {
    "APPINSIGHTS_INSTRUMENTATIONKEY"                  = "${azurerm_application_insights.default.instrumentation_key}"
    "APPINSIGHTS_PROFILERFEATURE_VERSION"             = "1.0.0"
    "APPINSIGHTS_SNAPSHOTFEATURE_VERSION"             = "1.0.0"
    "ApplicationInsightsAgent_EXTENSION_VERSION"      = "~2"
    "DiagnosticServices_EXTENSION_VERSION"            = "~3"
    "InstrumentationEngine_EXTENSION_VERSION"         = "~1"
    "SnapshotDebugger_EXTENSION_VERSION"              = "~1"
    "XDT_MicrosoftApplicationInsights_BaseExtensions" = "~1"
    "XDT_MicrosoftApplicationInsights_Mode"           = "recommended"
  }
}

# variables.tf

variable "environment_prefix" {}
variable "location" {}

variable "tags" {
  type = "map"
}

variable "resource_group_name" {}
variable "app_service_plan_sku_tier" {}
variable "app_service_plan_sku_size" {}

Environments

Generally, each of the environments is the same look and feel. The point of having each of these separate environment folders (e.g., env-dev, env-production, etc.) is to allow Terraform to easily run its normal scripts without any more configuration in the release pipelines. For example, if we are deploying the application to the development environment, we change the current working directory to env-dev and execute the same scripts that we would elsewhere.

A more detailed look at the files can be seen below.

Single-Page Application Environments

The single-page application contains only a few Terraform modules, as its implementation is much simpler.

# main.tf

terraform {
  backend "azurerm" {
    container_name = "terraform"
    key            = "tfbackend-web.tfstate"
  }
}

provider "azurerm" {
  version = "=1.20.0"
}

module "rg" {
  source = "../modules/rg"

  environment_prefix = "${var.environment_prefix}"
  location           = "${var.location}"
  tags               = "${var.tags}"
}

module "web" {
  source = "../modules/web"

  environment_prefix        = "${var.environment_prefix}"
  resource_group_name       = "${module.rg.name}"
  location                  = "${module.rg.location}"
  tags                      = "${var.tags}"
  app_service_plan_sku_tier = "${var.app_service_plan_sku_tier}"
  app_service_plan_sku_size = "${var.app_service_plan_sku_size}"
}

# variables.tf

variable "environment_prefix" {
  description = "The prefix for the environment."
  type        = "string"
  default     = "d"
}

variable "location" {
  description = "Specifies the supported Azure location where the resource exists. Changing this forces a new resource to be created."
  type        = "string"
  default     = "eastus"
}

variable "tags" {
  description = "The tags to associate with the resources."
  type        = "map"

  default = {
    "terraform" = "true"
  }
}

variable "app_service_plan_sku_tier" {
  type        = "string"
  description = "Specifies the plan's pricing tier."
  default     = "Shared"                             # Shared | Basic | Standard | ...
}

variable "app_service_plan_sku_size" {
  type        = "string"
  description = "Specifies the plan's instance size."
  default     = "D1"                                  # D1 | B1 | S1 | ...
}

Web Services Environments

The Web Services contain a significant amount more infrastructure than the single-page application does. This is mainly to accommodate the required database for the application and some configuration secrets.

# main.tf

terraform {
  backend "azurerm" {
    container_name = "terraform"
    key            = "tfbackend"
  }
}

provider "azurerm" {
  version = "=1.20.0"
}

module "rg" {
  source = "../modules/rg"

  environment_prefix = "${var.environment_prefix}"
  location           = "${var.location}"
  tags               = "${var.tags}"
}

module "db" {
  source = "../modules/db"

  environment_prefix              = "${var.environment_prefix}"
  location                        = "${module.rg.location}"
  tags                            = "${var.tags}"
  resource_group_name             = "${module.rg.name}"
  db_sku                          = "${var.db_sku}"
  db_capacity                     = "${var.db_capacity}"
  db_tier                         = "${var.db_tier}"
  db_family                       = "${var.db_family}"
  db_storage_mb                   = "${var.db_storage_mb}"
  db_backup_retention_days        = "${var.db_backup_retention_days}"
  db_geo_redundant_backup         = "${var.db_geo_redundant_backup}"
  db_administrator_login          = "${var.db_administrator_login}"
  db_administrator_login_password = "${var.db_administrator_login_password}"
  db_version                      = "${var.db_version}"
  db_ssl_enforcement              = "${var.db_ssl_enforcement}"
}

module "api" {
  source = "../modules/api"

  environment_prefix        = "${var.environment_prefix}"
  resource_group_name       = "${module.rg.name}"
  location                  = "${module.rg.location}"
  tags                      = "${var.tags}"
  app_service_plan_sku_tier = "${var.app_service_plan_sku_tier}"
  app_service_plan_sku_size = "${var.app_service_plan_sku_size}"
}

module "kv" {
  source = "../modules/kv"

  environment_prefix               = "${var.environment_prefix}"
  location                         = "${var.location}"
  tags                             = "${var.tags}"
  resource_group_name              = "${module.rg.name}"
  tenant_id                        = "${var.tenant_id}"
  azuredevops_object_id            = "${var.azuredevops_object_id}"
  key_vault_secret_1               = "${var.key_vault_secret_1}"
  key_vault_secret_2               = "${var.key_vault_secret_2}"
}

module "storage" {
  source = "../modules/storage"

  environment_prefix  = "${var.environment_prefix}"
  location            = "${var.location}"
  tags                = "${var.tags}"
  resource_group_name = "${module.rg.name}"
}

# variables.tf

variable "environment_prefix" {
  description = "The prefix for the environment."
  type        = "string"
  default     = "d"          # this is different for each environment
}

variable "location" {
  description = "Specifies the supported Azure location where the resource exists. Changing this forces a new resource to be created."
  type        = "string"
  default     = "eastus"
}

variable "tags" {
  description = "The tags to associate with the resources."
  type        = "map"

  default = {
    "terraform" = "true"
  }
}

variable "tenant_id" {
  type        = "string"
  description = "The Azure Active Directory tenant ID that should be used for authenticating requests to the key vault."
  default     = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

variable "azuredevops_object_id" {
  default     = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

variable "adminuser_object_id" {
  default     = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

variable "app_service_plan_sku_tier" {
  type        = "string"
  description = "Specifies the plan's pricing tier."
  default     = "Shared"                             # Shared | Basic | Standard | ...
}

variable "app_service_plan_sku_size" {
  type        = "string"
  description = "Specifies the plan's instance size."
  default     = "D1"                                  # D1 | B1 | S1 | ...
}

variable "db_sku" {
  type        = "string"
  description = "Specifies the SKU Name for this PostgreSQL Server. The name of the SKU, follows the tier + family + cores pattern (e.g. B_Gen4_1, GP_Gen5_8)."
  default     = "B_Gen5_1"
}

variable "db_capacity" {
  type        = "string"
  description = "The scale up/out capacity, representing server's compute units."
  default     = 1
}

variable "db_tier" {
  type        = "string"
  description = "The tier of the particular SKU. Possible values are Basic, GeneralPurpose, and MemoryOptimized. "
  default     = "Basic"
}

variable "db_family" {
  type        = "string"
  description = "The family of hardware Gen4 or Gen5, before selecting your family check the product documentation for availability in your region."
  default     = "Gen5"
}

variable "db_storage_mb" {
  type        = "string"
  description = "Max storage allowed for a server. Possible values are between 5120 MB(5GB) and 1048576 MB(1TB) for the Basic SKU and between 5120 MB(5GB) and 4194304 MB(4TB) for General Purpose/Memory Optimized SKUs."
  default     = 51200
}

variable "db_backup_retention_days" {
  type        = "string"
  description = "Backup retention days for the server, supported values are between 7 and 35 days."
  default     = 7
}

variable "db_geo_redundant_backup" {
  type        = "string"
  description = "Enable Geo-redundant or not for server backup. Valid values for this property are Enabled or Disabled, not supported for the basic tier."
  default     = "Disabled"
}

variable "db_administrator_login" {
  type        = "string"
  description = "The Administrator Login for the PostgreSQL Server. Changing this forces a new resource to be created."
  default     = "adminuser"
}

variable "db_administrator_login_password" {
  type        = "string"
  description = "The Administrator Login's password for the PostgreSQL Server. Required to be passed in."
}

variable "db_version" {
  type        = "string"
  description = "Specifies the version of PostgreSQL to use. Valid values are 9.5, 9.6, and 10.0. Changing this forces a new resource to be created."
  default     = "10.0"
}

variable "db_ssl_enforcement" {
  type        = "string"
  description = "Specifies if SSL should be enforced on connections. Possible values are Enabled and Disabled."
  default     = "Enabled"
}

variable "key_vault_secret_1" {
  type        = "string"
  description = "The first secret"
}

variable "key_vault_secret_2" {
  type        = "string"
  description = "The second secret"
}

Setup and Destroy Scripts

For those of you who were paying attention, you'll notice that I have two scripts in each of the environment folders: setup.bat and destroy.bat. Each of these files are ignored in the .gitignore file, as they contain Azure secrets, but they're useful for while I was developing the solutions. The contents can be seen below.

rem setup.bat

@ECHO OFF

set ARM_ACCESS_KEY=xxxxx

terraform init -backend-config="storage_account_name=dterraformstorage"
terraform fmt
terraform validate

rem destroy.bat

@ECHO OFF

set ARM_ACCESS_KEY=xxxxx

terraform init -backend-config="storage_account_name=dterraformstorage"
terraform destroy

Azure Release Pipeline

I detailed the source-controlled Build Pipelines above, but I explicitly left out the Release Pipelines in the conversation. Azure Pipelines currently has no support for configuration-as-code for Release Pipelines yet. While this is a detriment to the product offering as a whole, the web UI of the Release Pipeline is generally a good one.

Single-Page Application Release Pipeline

There exists a single Release Pipeline for the entire single-page application, each with a number of stages defined. Within each of these stages, a number of tasks are run.

SPA Release Pipeline Stages

A given stage can be seen as the functional equivalent to an environment.

The image defines a few paths that a given set of artifacts from the Build Pipeline can take:

Automatically deployed to the Development stage if it originates from the develop branch
Manually deployed to any of the three Testing environments
Automatically deployed to the Staging stage if it originates from the master branch. It can then get manually elevated to the Production stage if team members approve the release.

SPA Release Pipeline Stage Tasks

You'll notice that each of the stages have four tasks: each of these tasks is generally the same, minus a few environment variables.

These tasks are relatively straightforward. Note that the Terraform tasks are added from the Marketplace.

Download the Terraform binaries

Initialize the Terraform backend (using azurerm in this case)

Apply the Terraform configuration to the specified environment

Deploy the dist directory (the built application) into the previously-created App Service

Web Services Release Pipeline

The web services release pipeline is very similar to the single-page application's release pipeline in its concepts. There just happen to be a few more steps, specifically related to the database automation.

WS Release Pipeline Stages

Each of the stages below, just like the previous release pipeline, is analogous to a given environment for the application.

The paths an artifact can take are the same as the single-page application:

Automatically deployed to the Development stage if it originates from the develop branch
Manually deployed to any of the three Testing environments
Automatically deployed to the Staging stage if it originates from the master branch. It can then get manually elevated to the Production stage if team members approve the release.

WS Release Pipeline Stage Tasks

Each of the stages for this Release Pipeline has six tasks. Each of the stages are the same six steps, just with some differences in the environment variables.

The tasks for this application's release pipeline can be seen below.

Download the Terraform binaries

Initialize the Terraform backend (using azurerm in this case)

Apply the Terraform configuration to the specified environment

Run the database migrator application

Run the database seeder application

Deploy the packaged web application into the previously-created App Service

Next Steps

Sorry, this got really long.

As stated in the previous sections, there's still a bit of work to do from the application development teams with respect to automated testing. The pipelines detailed above allow for easy adaptation to utilize any of the testing frameworks needed.

There are a number of small improvements or refactorings that can be done across the application tiers, but those will be pushed to later in the project's lifecycle.

ASP.NET Static Site Generation

Tony Morris — Tue, 21 Aug 2018 00:00:00 +0000

As hinted at in a previous post, I have transitioned from hosting dynamic web applications in a cloud hosting service (like AWS Elastic Beanstalk or Azure App Service) to hosting static websites in my document storage location of choice (Amazon S3 in this case). This post discusses my development process for a project like this.

Existing Projects

I have a number of projects that currently utilize this deployment paradigm. They are all some flavor of ASP.NET developed locally with a local database, which are then exported and uploaded to S3.

MorrisPhotos.com

My photography website, MorrisPhotos.com, is a static website hosted in S3 and pushed to the edge via Amazon CloudFront. I've got CloudFlare handling DNS and threat detection, and it all just works out really nicely.

LodiCornFest5k.com

A local race website, LodiCornFest5k.com, is also a static website hosted in S3. I don't have CloudFront in front of this guy, as it's less important to aggressively cache the data on the website.

OhioTrackStats.com

My pride and joy, OhioTrackStats.com, is a static website hosted in S3. There will be later blog posts about this one at another date.

The Process

Local Development

I'm a .NET developer. This means I am most comfortable slinging some ASP.NET code, especially for local development. As .NET Core 2.1 is the de facto standard, all my new projects are moving toward that.

A simple example is my MorrisPhotos.com source. Maybe one day I'll go into further detail on how I code, but this is not the purpose of the post today.

My local setup is pretty straightforward: I've got Visual Studio 2017 Community running on my home machine with ReSharper plugged in. Nothing special here with the setup. You could do all of this with VS Code instead, but I like the fully-featured IDE.

With respect to local development, I'm adding new features, fixing any issues, or just generally running the application locally on my home machine to see how it's working. Nothing too special here, but this is the last place that the application is truly dynamic.

Local Database

I've got SQL Server 2017 Express running on my home machine. It's nothing fancy, but it's got the databases that drive each of the projects listed above. To access the databases in the code, I typically use either Entity Framework Core or ServiceStack ORMLite. I don't really have any speed or performance requirements with the database access layer, so I'm basically writing bad code to access the data.

S3 Setup

In order to publish a website using S3, I follow this guide. It's very self-explanatory, and I recommend you go through each step in order to get it correct.

For example's sake, I've got both a www.ohiotrackstats.com bucket and a ohiotrackstats.com bucket in S3 currently for the OhioTrackStats.com website. The www bucket redirects directly to the non-www bucket. This is the pattern that I've got for each of my projects.

Domain Setup

As mentioned previously, I've got DNS in CloudFlare's infrastructure. For MorrisPhotos.com, this looks like the following:

Domain registered through Google Domains
Name Servers set to CloudFlare's name servers
A couple CNAME records in CloudFlare
- morrisphotos.com -> non-www S3 bucket hostname
- www.morrisphotos.com -> www S3 bucket hostname

Static Site Generation

This is where the real magic happens. Given a dynamic web application, I want to create a static HTML website that can be hosted directly through S3.

I personally use HTTrack Website Copier to crawl the locally-running ASP.NET web application and generate the HTML, CSS, and JavaScript files to a folder structure. The beauty of this tool is that it will update all the links on a website to be relative paths, allowing for the website to be run from anywhere that a web server exists.

I make sure to exclude all external CSS and JS files (files hosted on external CDNs, like the Bootstrap files, for example), as I don't want to be responsible for hosting their content in my website. Beyond that, the setup is very straightforward, and once I've got a website setup once, I never have to update it.

In order to properly execute this step, I need to make sure I'm actively running the web application locally, and then I point to the tool the port that IIS Express is running on. I try to make sure to run the application in Release mode in order to get the built-in benefits of bundling and minification, as well.

Deployment

Once I've got the static site generated, all I've got to do is copy the output to the non-www S3 bucket. I can do this via the AWS Console, but I'd rather not upload the entire web site every time I make a small update.

To combat full uploads, I am utilizing the AWS CLI. Specifically, I am using a batch file that runs the static site generator, then runs the S3 sync CLI command to push only the changed items. A sample batch script can be seen below. I'll describe after.

"C:\Program Files\WinHTTrack\httrack.exe" "http://localhost:59141" -O "C:\My Web Sites\LodiCornFest5k" --update
aws s3 sync "C:\My Web Sites\LodiCornFest5k\localhost_59141" s3://lodicornfest5k.com --acl="public-read"

The first line in the batch script is to run the HTTrack software from the command line, pointing to http://localhost:59141, which is the IIS Express website that I set up for my LodiCornFest5k project in Visual Studio. The -O switch points the application to the mirror location, which is something I already set up in the GUI with the options I wanted. Finally, we use the -update switch to ensure that it only writes files that have changed since the last time we ran the tool.

The second line in the batch script is the upload line. It runs the aws s3 sync command, which has already been configured on my machine to use an IAM user that has write access to all my S3 buckets. This isn't the best security, as I should probably limit that IAM user to just the buckets that I need, but it'll do for now. I then point it to the location of the files on the host machine (C:\My Web Sites\LodiCornFest5k\localhost_59141), and then the bucket location to push the files to (s3://lodicornfest5k.com in this case). In addition, I add the --acl="public-read" flag to the uploaded files to ensure that they can be viewed over internet without authenticating, as my website is publicly available.

Again, since I'm generating the static site within this batch file, I need to make sure that I'm running the website locally. I've kicked this batch file off a number of times without doing that, and the whole thing borks pretty hard.

Future Enhancements

There is one major enhancement that I want to get to, but I haven't tried it yet. Instead of manually running the batch file locally when I make changes, I would love for a CI server to be able to do it instead. It would require me to package up the httrack.exe binary with my application. Then, upon commit to the master branch, I would have something like a Continuous Integration tool like AppVeyor run a post-build command that is similar to the batch file itself. This would ensure that the website in S3 directly matches what's in source control, which is the holy grail.

I'll probably get around to that one in the next year or so, so stay tuned!

Review

In all, the "do work" stage of building my custom web applications hasn't really drastically changed with this new hosting model. I'm still running and testing locally whenever I make changes. Instead of pushing the built bits out to a web application server, though, I am just pushing out the generated files instead. It's a much more elegant and speedy solution for both me and the end user!

Cost Saving

Previously, I was using a hosted application platform for something like MorrisPhotos.com (Elastic Beanstalk, specifically). This meant that I was incurring charges on an EC2 instance, an RDS instance, plus some networking and hardware costs. This cost upwards of $100/month just to host and run the web application.

Now, with just the S3 buckets hosting the static website (and the thousands of photos in a separate S3 bucket), plus CloudFront as the CDN layer in front of the S3 bucket, I am paying less than $10/month. I don't have the exact numbers at the moment, but we're looking at roughly a 90% savings per month. I'd say this was absolutely worth it.

Time Saving

I don't have hard numbers on the actual time savings for the end users, but I do have a general understanding that, instead of a web application server (IIS in this case) receiving a request, handing off to the web application to process, handing off to the database to retrieve data, then pushing it back up the pipeline to the end user, I am simply giving the user a typically-cached HTML/JS/CSS/image file instead. The speed that the end user is seeing is significantly faster, on both a time to first byte aspect (due to the edge locations of CloudFront in MorrisPhotos.com's case, generally) and a total download time aspect.

All in all, this new deployment and hosting paradigm has greatly improved everyone's experiences with my "dynamic" web applications, and it's certainly something I'll be doing from now on for web applications that need to be database-driven but only updated somewhat regularly.