Forem: Adam Rozanski

What Breaking My Ankle Taught Me About DevOps

Adam Rozanski — Fri, 20 Mar 2020 21:09:04 +0000

Yikes. Unfortunately it’s true. I broke my ankle doing something I took for granted; walking. As I sit here and write this I am still in recovery, but it’s given me a few interesting perspectives on DevOps. PSA - please do not go breaking your ankles, or anyone elses. I am not advocating for this. Please.

A broken process needs time to heal

When you get extremely well-practiced at a task, things become as routine as walking down the hall, until you trip holding a ping pong paddle while batting at stressballs. No? Just me?... Let’s talk about CI/CD as an example of this. Last year the entire routine for the developers and DevOps got flipped on its head. Devs went from pressing all the buttons to none of the buttons. What used to be routine suddenly became verboten as I and the rest of DevOps took on roles and tried to implement procedures with Dev management to make up for the void that had been created by the walls that were constructed to delineate the distinction of each team’s priorities.

In our case we knew what exactly we had broken: the process. This was very straight-forward to us, from coding, to QA, to Releases, everything was broken. We limped along like Batman post-Bane because of our indomitable wills. The builds moved in mysterious ways, and thus releases were fragmented. You couldn’t easily know what was in production without having to login to BitBucket and review the commit SHA and then review the commits (a major undertaking for any dev who was trying to release anything). Tickets were QA’d individually so it wasn’t always clear where or how they interacted with each other until later, introducing bugs and frequently resulting in hot deploys. It had been like this since I joined the company two and a half years ago, and we hadn't had a reason to change. Why would we? Everything worked, didn't it?

For the sake of brevity I won’t go further, but you get the idea. Like me, this whole CI/CD pipeline needed surgery. Although we have surgically performed the painful part and that is behind us, recovery will take weeks. Sometimes though, it takes longer. Just like with an injury, if you try to do too much at once with it before it’s completely healed you injure yourself again and hurt yourself worse, thus you can drag a 3 week recovery into 2 months, or in our case a 1 business quarter task into 2.

Of course, we had had a DevOps department for a couple of years but in my opinion we never really did DevOps as a company until recently. Since the foundation of the department in 2017, we had been in a constant state of firefighting mode that made it impossible to get ahead of the curve. It took 3.5 months to straighten out the CI/CD pipeline to a point that you could trust the setup across the board. I remember the times where I'd have to rebuild a single branch 3 times to get it between QA, UAT(formerly Staging), and finally Production. Worse yet, the environment and configuration settings weren't identical so every environment was setup differently. It took another 6 months to migrate all of the critical systems' deploy pipelines into Octopus Deploy so that the overhead of managing it would be lessened for all parties involved. We did all of this while accepting infrastructure requests and performing migrations to newer hardware in an effort to keep up with the evolving technologies at DealerOn. As I look back I realize that if we had attempted this DevOps transformation back then, that we would never have completed as much of the work that we have in the timeframes that I listed above. We simply were not ready then, but that is not true today.

The moral of my story is that DevOps isn't magic. Although there are times that I feel like an arch-wizard with a mastery of the arcane, don't be fooled. DevOps won't instantly transform your business, and it certainly isn't a switch where it is suddenly ON and everything works better. DevOps is a mindset and a set of practices, and they take time to implement and reiterate as you come up with cleaner and better ways to do the tasks. Anyone who tells you otherwise is selling something. Though I am fairly new to the DevOps scene, I had never truly considered the scale of how a DevOps transformation would affect the department and company. After all, it was not until we had broken our metaphorical ankles that we had really started to heal and improve...

At the time of publishing this article, I'm pleased to announce that my leg is almost completely healed, and I'll be walking normally again very soon.

Mercurial Matters

Adam Rozanski — Fri, 07 Jun 2019 19:31:13 +0000

So, you want to hear another story, eh? One filled with DevOps and DevOps-ey things? Ha! I've got a story for you. This is a tale of key vaults, automation, and getting things done.

The Vault

It starts with an Azure Key Vault. This is a native service that stores application secrets. Traditionally the application.config or web.config, for a service or website respectively, contains connection strings that your code requires in order to run. This is fine, however doing this is equivalent to writing your passwords on a post-it note stuck somewhere underneath your keyboard (I'm looking at you, Jerry!). If anyone gets to your desk they'll know where to look for it, and once they grab your connection strings it's game over.

To combat this, folks will often encrypt it. The drawback to this though is that the encryption key is often on the machine, which isn't much better. If the Nuget package or whatever mechanism you are using to decrypt it at runtime isn't kept updated, your application is likely to break because it can't read the config file, and it leaves you with what looks like Matrix code in your files (this has happened to me, and it's terrifying). The best place to go then is Azure Key Vaults (AKV) to store them in a specially encrypted external database which allows for not only credential-based authentication, but use of a unique-to-the-individual SSL certificate for authentication to avoid having to write your password somewhere and remember it.

Empowering Mercury

One of our development teams(Mercury) had recently switched to using AKV for something new that they were creating. They needed to add a lot of secrets to their new vaults and it's quite honestly very tedious. The portal generally allows you to upload 1 at a time, and it's a lot of clicking around. Since you're likely copy-pasting all the values in, it's easy to copy the wrong one in. So I gave them a workaround I had used on previous projects:

The PowerShell script that allows upload of multiple secrets.

Since it's possible to use special characters in keys and secrets, and secrets can be very long, such as a JSON object, it's not a great idea to have them be paired in a 2D array, or to try and split a string based on a unique characters.

The general guidance that I give my coworkers is to make sure you verify the secrets beforehand for validity to ensure that every key has a secret, as secrets are not allowed to be blank (rule of Azure). If there was a mistake they could easily fix the value and re-run this, and if there was a scenario where a secret was given the wrong value, it would just be given a new one and only the new version would be active. They now had one of my favorite scripts, and things were fine for a time.

Bad Moon On the Rise

Happily armed with their new tool, the Devs got to work. Over the weeks that followed I received requests during the day to upload new secrets to production, as our Devs are not allowed direct change access to those resources. Once I even came in at night after hours to help with a deploy. It was about this time that all of us began to notice the issue: Me.

It wasn't that I was a problem, or intentionally holding them back, but rather I was inadvertently holding them back from deploying more often than not because I would usually be in a meeting or otherwise predisposed with something else. Sometimes we realized that the key vault values needed to be changed because they weren't what we were expecting a few hours later, and I'd need to step in. Every minute I wasn't available to them was a minute they had to wait to fix the issue.

Since this was a brand-new process, we didn't have a lot of existing guidelines in place because we didn't know how frequently Devs would ask us to make the updates.

The entire point of DevOps is to be empowering and hands-free, and this was turning into something akin to what large companies have with CAB meetings, and the like. None of us expected this simple process to turn out to be this complicated. It had become clear to everyone involved that we needed to implement a better workflow for this, pronto.

Enter the Octopus

The suggestion came up that we automate this to avoid direct DevOps intercession. Agreeing, I did what I should have done at the start - added the script from above as an executable step within Octopus Deploy to allow updating the AKV. It fit directly into their other deploy procedures, and now Dev is able to deploy key vault updates as they need without worrying about who on DevOps is there to assist. This ended a cycle of what was becoming a 3x-a-week-request-with-the-occasional-wait pain point and turned it into a no waiting, self serve request submission.

What are the morals of this story?

Automation does not solve all of your problems.

It sometimes is not enough to script out the task to make it less tedious. Just because we can does not mean that we should.

There are times when a process still needs to have automation driving it forward so that it doesn't hold up production, and there are times we are better off not automating things. This is a nuanced process that requires a formula with non-static variables to get close to right: (saved_cost_of_task - time_spent_troubleshooting) > cost_of_task. The amount of time saved by having the automated task combined with the time spent troubleshooting should be greater than the cost of not automating the task itself. If a task saves you a lot of time but you waste more time troubleshooting it than you do having it run, because it's a complicated process that requires human thought (and assuming you did this in an optimal manner) you're likely wasting more time than you'd be saving by automating it.

As an example, I could automate Team City and Octopus to deploy every single SQL script to run on all databases after it successfully goes to production. While this saves time, it wastes a lot of effort troubleshooting: There's a lot of permutations of scenarios I would have to capture with a deploy process for SQL where the order that I deploy one more more apps in a specific order. The amount of knowledge of the unique scenarios places an unfair burden on Dev, and becomes a detrimental necessity for DevOps to be available to assist. I could write an entirely separate post about this, but I think you get my point.

Be prepared to redefine the process.

We can't always foresee where or when we will need automation. We need to be willing to periodically take a step back and re-assess our processes and re-align our thinking so that we can make the right decisions.

As time progresses and you grow as a team, the willingness to change needs to exist. Before we started this journey down Key Vault Lane, updates to existing keyvaults were rare and infrequent. As we built more of the application, it became apparent that we had outgrown the process, and like an old pair of pants that no longer fits, we had to go get a new one that fits our needs and still make us look good.

The Tragedy of the Jigsaw Bot

Adam Rozanski — Fri, 24 May 2019 13:01:01 +0000

Our story begins on the 25th of march. I was working on a ticket that had been in front of me for a week or two regarding our build server (Team City) not sending any emails to folks. The build server software’s official support team was taking quite a while to get back to me, and I had tried everything I could think of. Not wanting to leave this untouched for a while, I decided to come up with a plan.

“What if instead of an email, we get the notification of a failed build sent via slack directly to the user in a private chat? Most of my teammates don’t like email anyway, and this would put me a step or two further on my goals for integrating the entire CI/CD pipeline with the chat client.”

Over the course of the week in my free time between major tasks, I experimented with a few plugins for the build server. Nothing seemed to work correctly and the server logs didn’t overtly indicate that I had a problem or that it was attempting to use the shiny new tool that I had configured. However, with some help from support I was able to learn how to enable the logging correctly. Apparently, I was missing several scopes for the bot to talk to me in my slackbot channel.

I had been writing PowerShellon the side to send myself simple messages so that I could test the permissions of the bot. As it turns out, there’s a lot of documentation for Slackbots, but I had not seen a unified guide to setting them up with all the permission scopes you’d want in one place in the manner I was seeking, namely something that explained them in a method that made it so I could understand. I knew that you could send bot chat to a channel, but private users was another thing, and that didn’t have (to me) a straightforward logic to doing so.

Typically, a public/private channel only requires an incoming-webhook for a scope, and an incoming webhook URL in order to post to a channel. However, a person’s private slackbot channel is unique to them. The scopes I ended up granting the bot are: incoming-webhook, channels:write, chat:write:bot, groups:write, bot, users:read:users:read:email.

**$stuff** = “_I have the shiniest meat bicycle_!”

**Invoke-RestMethod** -Method POST -Uri “https://slack.com/api/chat.postMessage?token=xoxb-SUPER\_LONG\_TOKEN&channel=SlackUserID&text= **$stuff**" -ContentType “application/json”

“I have the shiniest meat bicycle!” — _was the very first thing I saw in my slackbot channel from my own bot (test messages in my opinion should be fun, so I use Borderlands 2 quotes). Soon after, I was sending URLS and semi-complex text structures in urlencoding, followed by JSON objects. _Success!

However this was only the beginning. It would be a major task, and a huge spoiler of fun to ask my pals in development to go through a semi complicated procedure to get the slack token I needed and then to record it somewhere in a kind of database or file. The next logical step then was to tell my script my email address, have it figure out the rest from there, and then send a message that way. The good news for me is that Team City has a built-in variable for that, and I had been asking everyone who registered on the server to use their work email so they could actually get emails when builds failed.

**$who** = “%teamcity.build.triggeredBy.username%”

**$targetUser** = **Invoke-RestMethod** -Method POST -Uri “https://slack.com/api/users.lookupByEmail?email= **$who@emaildomain.com** &token=xoxb-super\_long\_token" -ContentType ‘application/x-www-form-urlencoded’ | **ConvertTo-Json**

( **$targetUser** -split “`r`n”) | **ForEach-Object** {
**$userID** = **Select-string** -InputObject $\_ -Pattern ‘“id”: ‘
 if( **$userID** ){

**Write-Output** “FOUND”
**$cleanID** = **$userID** -Replace ‘“id”:’, “” -replace ‘“‘, “” -replace ‘,’, “”

**$cleanID** = **$cleanID**.trim()
 }
}

Above I’m doing some cleanup. The response from the API is a JSON object, and in order to lookup my string ID that comes from being in Slack I need to parse it. Once I grab a line that has “id”: “MY_ID”, I just remove the extra stuff I don’t need and that leaves me with just my target ID as a string.

Within an hour I had it set up so that if i gave it my email, it would look up my userID in Slack, clean up the token it got out of the JSON that it returned, and then send it back to slack with the api call to send the private user a message. Easy, right?

It’s now Friday and I have about half an afternoon left in my work week when the realization hits me —

April 1st is a Monday this year. I could prank the dev team by challenging them to a game of Russian Roulette at random on each build triggered. If I took the current time, took the last digit of the minute off and used that in a switch, I’d generate a sensibly fair roulette. An example of this is: Imagine it’s 1:25pm, we grab the last digit in the time, 5, and generate a result based on that. Using the logic I present below, a build triggering at 1:25 would gain the user nothing, but a build triggered at 1:29 would earn them 200 points, and a build triggered at 1:23 would cost them 100 points.

**$a** = Get-Date
**$array** = [int[]](($a.Minute -split ‘’) -ne ‘’)
**$yei** = 0
switch( **$array** [-1]){
 0 { **$yei** =0}
 1 { **$yei** =-25}
 2 { **$yei** =50}
 3 { **$yei** =-100}
 4 { **$yei** =100}
 5 { **$yei** =0}
 6 { **$yei** =25}
 7 { **$yei** =-50}
 8 { **$yei** =0}
 9 { **$yei** =200}
default { **$yei** =”ID:10-T Error”}
}

It would be an ideal way to test out the build notifier as a concept and turn it into a real thing if it works out. I already had it well figured out how to look up the user in the system and verify their authority to run specific builds/execute scripts against certain servers from a previous project.
All the pieces to make this work are here, it just needs someone to assemble the jigsaw puzzle. If I doubled-down , I could give it a Jigsaw flavor and feel but not overtly pressure them into playing my game if they are not interested or do not have the time. I get to have my love of pranks and to be useful at the same time.

By COB that day I had Jigsaw bot tested out and working. The whole process works like this:

A user triggers a build, and at random they’re challenged to a game.
I look up the user on the whitelist and blacklist. If the user is not whitelisted or blacklisted when they’re chosen for a challenge, then they are issued a challenge by the Team City build server directly in Slack. They’d be instructed to (if they wanted) play the game by using the SAW build at the bottom of the page, in a section reserved for Utilities.
Team City Updates the whitelist by allowing that user to play.
Team City sends me a message, so I have a record that a game is coming and should be prepared to send them points.
The user plays the game. Based on another time-based roulette like the one shown above, they’re given or penalized an minuscule, arbitrary amount of points. Any user caught trying to play the game who wasn’t invited via whitelist would be blacklisted and mocked in Slack in front of everyone on their team. Once the user completes the game they’re also blacklisted from playing again in order to keep it simple and prevent everyone from winning too much. If the RNG engine was not being generous with the developers and there were not a lot of players by midweek, my plan was to clear the blacklist and let them try again, for the fun of it.

The challenge messages here were put on a slight delay timer, so you wouldn’t get them all at once, to attempt to make the bot feel life-like. This is a sample of what the user would get in their message inbox.

This is what I expected to see in my personal Slackbot channel.

This has a couple of test examples of winning and losing the game. The user gets a message saying what they win or have to do, and I get a message knowing to expect payment or have to make a payment.

Nothing could go wrong with this plan, right?

Riiiiiiiiiight…

Poor Jigsaw, he has no idea who he’s talking to. %teamcity.build.triggeredby% will only be a name he will get angry at and be triggered by. For the record though, I did track down the user who beat my game and gave him 100 points just for helping me fix the bugs.

Come Monday, hell had pretty much broken loose in my slackbot channel. It turns out that unless the user clicked the button [RUN] or […] next to it to initiate a build, it would set the default service account name as the %teamcity.build.triggeredby.username% system variable value when it went to do a lookup. Slack would then try to lookup this invalid user and then send nothing to me. I was 100% in the dark on what was going on. It wasn’t until Tuesday of that week before I had an actual user get labeled as the trigger-er of builds. By then I already had about 100 IMs in chat saying that %teamcity.build.triggeredby.username% has been challenged to a game.Very helpful. Veeeeeery helpful.

Remember the whitelist/blacklist function that I mentioned earlier? Its only purpose was to be a build artifact that was a composite of all the other users who had been challenged and add their names to it as the week progressed. As a user gets challenged, they’re put on the whitelist, never to be bothered again, until they play the game. The whitelist function wasn’t working for anyone, so Matt came to me by midweek saying that he was getting challenged to a game every 5 minutes while running a series of database changes on multiple databases. Although he wasn’t upset (he was actually mildly amused), he did ask if I could turn it off while I looked for a way to make Jigsawbot not play favorites with him.

Oh, and did I mention that almost everyone found this irritating and deemed it to be some social engineering scam? Pretty much everyone I spoke to either believed that it was a social engineering scam, or just me goofing around, and no one reported it to me or my team either way, save for the 4 users who actively contacted me because they were just amused enough to ask. Out of that subset of developers, half of them were even remotely interested in playing along and did so (to which I rewarded as per rules of the game). At least our IT department’s anti-phishing training was not lost on my developer friends.

Everything was going so smoothly.

So I worked on it, and after some hair-pulling moments realized where my errors were, and got it to leave poor Matt alone.

Pseudo-code for what I was doing:

If (we should challenge the user) {
 challenge the user;
 if (not on whitelist and not on blacklist) {whitelist them;}
}

What I instead should have been doing:

if (we should challenge the user){
 if (not on whitelist and not on blacklist){
 challenge the user;
 whitelist them;
 }
}

This never came up in my tests because I’d play the game directly after being challenged. I didn’t have a test case scenario that would adequately cover the instance of “I need to run 5 different builds at once that would trigger this before playing” and the gap in my code’s behavior wasn’t readily apparent until it was out for the “world” to see.

With the correct logic in place, I was able to fix what my problems were, and Matt finally got some peace and quiet. I, however, wasn’t satisfied. If I wanted to make this into a usable bot script for build errors, I needed to understand how the build software gets the VCS changes from Git to learn who the user was.

With my week of sad and failed pranks nearly over, I began efforts to grab the username from Git so I could extrapolate it into my script and the circle would be complete. With a bit of git commands piped to a file, and a sprinkle of regex later, I had cooked up something that would reliably get emails for usernames, or for users who wanted their first and last name to be signed on the commit, and can accurately report it back to the script so I can do all the fancy things that I have described above without being concerned for getting a bad username passed in because the system’s default is not valid for the entire script’s usage.

The icing of this cake is that I got the plugin working while I was doing this(you know, the one that started this whole damn journey). Ironically, this script isn’t usable for me in it’s current form without me completely redoing all of our builds and sending them back to the dark ages.

However, should I ever need to migrate to another build software where there are less plugins and the UI is less friendly and more command line driven, I have a script that will send private messages. Here it is:

Cover image by Jose Francisco Morales on Unsplash