Forem: Gavin Campbell

Configuration as Code using Azure DevOps variable templates

Gavin Campbell — Sat, 31 Oct 2020 00:00:00 +0000

One of the principles of the twelve-factor app methodology is strict separation between code and config, where config means everything that is likely to vary between deployments, and code is everything that doesn’t. Historically, was not a good fit for .NET Framework applications, which relied on tools such as web.config transformation and Slow Cheetah to apply build-time transformations to application configuration files. These transformations are based on environment-specific config files stored alongside the application code in the source repo. In the past, this led some commentators to conclude that certain aspects of 12-factor were just not applicable to .NET applications.¹ The world has moved on since then, as has the .NET framework, and separation of code and config is much easier to accomplish. This article will describe one technique for managing version-controlled configuration data alongside - but separate from - our application code using Azure DevOps variable templates.

The example

The example project I am using is the sample ASP.NET Core MVC application for Cosmos DB, which is described in a tutorial on docs.microsoft.com as well as being available to download from GitHub. In brief, this consists of a simple website that provides CRUD² operations for a “to-do list” stored in Azure Cosmos DB. I chose this particular example as it requires a handful of configuration items, of which one - the Cosmos DB API key - is a secret. Most real life projects, at least the ones I’ve worked on, require a much larger number of variables to be set.

By default, ASP.NET Core web apps are provided with a default configuration builder that reads configuration from a number of sources in a predetermined order. The important thing to know here is that environment variables take precedence over values in appsettings.json. This means that appsettings.json and its cousins appsettings.dev.json, appsettings.test.json and so on can be relegated to providing configuration values on the developer’s desktop, with all other environments being configured via environment variables. As it happens, there is an open-source nuget package DotNetEnv which makes it easier to dispense with appsettings.json altogether, but I won’t discuss this further here since the example project doesn’t use it. The disadvantage of appsettings.json is of course that it’s too easy to end up doing this kind of thing.

The config repo

The project contains two repos; sample-app-code and sample-app-config.

sample-app-code is just a fork of the public repo from GitHub, with the addition of an azure-pipelines.yml file to build and release the application. sample-app-config contains one .yml file per environment, which I’ve named after the environment.

One template file per environment

These .yml files are known as pipeline templates. When we refer to a pipeline template in our pipeline, the entire contents of the template file is inserted at that point. Pipeline templates can be used to define and re-use a number of pipline objects, such as stages, jobs, steps, and, in our case, variables.

In general you can’t mix different types of objects, such as variables and steps for example, in a single template - they would end up being inserted at the “wrong” place in the pipeline.

Each template file contains a variable for everything that is likely to change between deployments, or in other words the configuration. These variables will be available to our pipeline, as well as being available as environment variables to our scripts³.

# This is a comment. Take that, appsettings.json!
variables:
#deployment config
serviceConnectionName: myNonProdServiceConnection
resourceGroupName: rg-todo-staging
resourceGroupLocation: uksouth
#Cosmos DB Config
cosmosAccountName: cosmos-ac-todo-staging
keyVaultName: kv-todo-staging-fdkjiwh
#Web app config
appServicePlanName: asp-todo-staging
servicePlanSku: S1
# need single quotes around double quotes to pass the pipe and the double quotes!
runtimeVersion: '"DOTNETCORE|3.1"'
webAppName: todo-staging-vdorpsbzaoiku

If you do this in “real life”, it’s up to you whether you create one config repo per code repo, or share a single config repo between a number of code repos. In the latter case you’d need to come up with some sort of folder structure and/or naming convention, e.g.

.
├── app1
│ ├── production.yml
│ └── staging.yml
└── app2
├── otherenv.yml
├── production.yml
└── staging.yml

.
├── app1-production.yml
├── app1-staging.yml
├── app2-otherenv.yml
├── app2-production.yml
└── app2-staging.yml

The Pipeline

The pipeline file is pretty short, so I have included it in its entirety here. During the build stage, we just do dotnet publish and publish the resulting package as a pipeline artifact for later consumption. The deployment stages have been encapsulated into a job template that takes two parameters. The first is the name of the environment we are deploying to, and the second is the name of a variable group containing some additional secret variables - since we don’t want to store secret variables in our config repo.

Enviroments in Azure Pipelines

I created an approval on the production environment using the UI.

Observant readers will have noticed that the value we pass in the stageName parameter matches the name of the yml file in the config repo as well as the name of the enviroment as defined in Azure Pipelines; this matters as we are going to use this parameter to figure out which variable template to use as well as which enviroment to deploy to. Of course, if you wanted these names to be different, you could accomodate this with some additional mapping logic. Unlike the variable templates, the job template pipelineTemplates/doTheDevOps.yml is stored in the same repo as the code, since it doesn’t vary between environments.

The effect of this is that every stage in our pipeline is deployed identically. We use the same template for deployment, and we tell the template where to retrieve its configuration. This should greatly increase our confidence in deploying to production, as since we use the same code everywhere we have lots of chances to “practice” before we do the production deployment.

trigger:
- master
pool:
vmImage: 'ubuntu-latest'
resources:
repositories:
- repository: config
type: git
name: sample-app-config
stages:
- stage: buildAndPublish
displayName: Build project and publish artifact
jobs:
- job: buildAndPublish
displayName: Build and Publish
steps:
- task: DotNetCoreCLI@2
displayName: dotnet publish
inputs:
command: 'publish'
publishWebProjects: true
arguments: '-c Release -o $(Build.ArtifactStagingDirectory)/todo'
modifyOutputPath: false
- task: PublishPipelineArtifact@1
displayName: Publish todo.zip as Pipeline Artifact
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)/todo/todo.zip'
artifact: 'todozip'
publishLocation: 'pipeline'
- stage: deployStaging
displayName: Deploy Staging Environment
jobs:
- template: pipelineTemplates/doTheDevOps.yml
parameters:
stageName : sample_app_staging
variableGroupForSecrets : SecretVarsForStaging
- stage: deployProduction
displayName: Deploy Production Environment
jobs:
- template: pipelineTemplates/doTheDevOps.yml
parameters:
stageName : sample_app_production
variableGroupForSecrets : SecretVarsForProduction

The deployment job

As the name suggests, this file is where the action takes place. In summary, we create our supporting infrastructure⁴ - the CosmosDB account, a key vault to hold the CosmosDB Key, the App Service plan, and then deploy our web app with appropriate configuration.

In order to demonstrate the use of variable groups to hold secret variables for YAML pipelines, I’m also adding an additional sensitive setting that is never used by the app. Other than for holding secrets that need to be masked in the pipeline log output, I think using variable templates removes the need for variable groups. Once again the template file is short enough to reproduce in full here:

parameters:
- name: stageName
type: string
- name: variableGroupForSecrets
type: string
jobs:
- deployment: deploy${{ parameters.stageName }}
displayName: Deploy ${{ parameters.stageName }} Environment
environment: ${{ parameters.stageName }}
variables:
- template: ${{ parameters.stageName }}.yml@config
- group: ${{parameters.variableGroupForSecrets}}
strategy:
runOnce:
deploy:
steps:
- task: AzureCLI@2
inputs:
azureSubscription: ${{ variables.serviceConnectionName }}
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
 az group create --name $(resourceGroupName) --location $(resourceGroupLocation)
az appservice plan create --name $(appServicePlanName) --resource-group $(resourceGroupName) \
--sku $(servicePlanSku) --is-linux
az webapp create --resource-group $(resourceGroupName) --plan $(appServicePlanName) --name $(webAppName) \
--runtime $(runtimeVersion)
az webapp identity assign --name $(webAppName) --resource-group $(resourceGroupName)
# Grant the web app identity permission to read secrets from the vault
az keyvault create --resource-group $(resourceGroupName) --location $(resourceGroupLocation) --name $(keyVaultName)
principal=$(az webapp identity show --resource-group $(resourceGroupName) --name $(webAppName) | jq -r .principalId)
az keyvault set-policy --name $(keyVaultName) --secret-permissions get --object-id $principal
# Add our secret var to the key vault
az keyvault secret set --vault-name $(keyVaultName) --name PointlessSecret --value $(PointlessSecret)
# Create cosmosdb account and retrieve uri and key for app settings
az cosmosdb create --name $(cosmosAccountName) --resource-group $(resourceGroupName)
cosmosUri=$(az cosmosdb show --name $(cosmosAccountName) --resource-group $(resourceGroupName) | jq -r .documentEndpoint)
cosmosAccountKey=$(az cosmosdb keys list --name $(cosmosAccountName) --resource-group $(resourceGroupName) | jq -r .primaryMasterKey)
# Add cosmos key to key vault 
az keyvault secret set --vault-name $(keyVaultName) --name cosmosAccountKey --value $cosmosAccountKey
# Get Secret Uris to add to app settings
cosmosKeyKvUri=$(az keyvault secret list-versions --name cosmosAccountKey --vault-name $(keyVaultName) --maxresults 1 | jq -r .[0].id)
pointlessSecretUri=$(az keyvault secret list-versions --name PointlessSecret --vault-name $(keyVaultName) --maxresults 1 | jq -r .[0].id)
# Set up app settings
az webapp config appsettings set --resource-group $(resourceGroupName) --name $(webAppName) \
--settings Logging __LogLevel__ Default=Warning \
AllowedHosts="*" \
CosmosDb__Account=$cosmosUri \
CosmosDb__Key="@Microsoft.KeyVault(SecretUri=$cosmosKeyKvUri)" \
CosmosDb__DatabaseName=Tasks \
CosmosDb__ContainerName=Item \
PointlessSecret="@Microsoft.KeyVault(SecretUri=$pointlessSecretUri)"
# set web app to mount zip file read-only https://docs.microsoft.com/en-us/azure/app-service/deploy-run-package
az webapp config appsettings set --resource-group $(resourceGroupName) --name $(webAppName) --settings WEBSITE_RUN_FROM_PACKAGE="1"
az webapp deployment source config-zip --resource-group $(resourceGroupName) --name $(webAppName) --src $(Pipeline.Workspace)/todozip/todo.zip

There isn’t really much going on here, we create an app service plan and web app, along with a key vault to hold the Cosmos Key, as well as the secret we are reading from the variable group. The web app is created with a managed identity, which we retrieve in order to grant access to the key vault. Next, we create the Cosmos DB Account, and store the account key in the key vault. Finally, we set up all of the app settings, including the key vault references, and deploy the web app. Since this application doesn’t write anything to local paths, we are taking advantage of the facility to mount our zipped deployment package directly as a read-only filesystem.

When we run the pipeline, we build the app and deploy it to each of our stages in succession:

The completed pipeline run

If we inspect the logs, we can see that the steps from our template have been included just as if they were defined in the main azure-pipelines.yml file:

The pipeline logs

The payoff

Whilst this approach requires a bit of work to set up, it delivers a number of benefits.

There is no need to have any logic in our source repo based on what environment we are deploying into.
Changes to our application config don’t mean we have to rebuild our application code, we can redploy the same version with the new config.
Since the config repo can be versioned, it is possible for different code repos to pin to different versions of the config.
Config repos often contain cost-sensitive items, such as the number or size of VMs being deployed to a particular environment. Having this stored in Azure Repos enables us to use the usual source-code management and audit techniques to control changes to these variables
We can add a new environment to our release pipeline just by creating a new config file in the config repo and a stanza of the following form in the release pipeline.

 - stage: deployWhatever
displayName: Deploy Whatever Environment
jobs:
- template: pipelineTemplates/doTheDevOps.yml
parameters:
stageName : sample_app_whatever
variableGroupForSecrets : SecretVarsForWhatever

This stackoverflow thread is from 2012. It’s included here not as a criticism of the participants, but as a criticism of the ideas that were current at the time. The post that states that the twelve-factor authors “clearly don’t understand the use of configuration files in .NET” has been awarded a “bounty” of 50 points! ↩︎
https://en.wikipedia.org/wiki/Create,_read,_update_and_delete ↩︎
A quirk of Azure DevOps is that variable names are upper-cased when being exposed as environment variables. ↩︎
In the interest of brevity, this example uses the Azure CLI to create all the infrastructure. On a real project I might use a different tool to do this, but the variable templates would be the same, as would the other parts of the pipeline. ↩︎

Human aspects of Dev / Ops Convergence

Gavin Campbell — Fri, 24 Jan 2020 08:57:33 +0000

Over the last decade or so, much has been made of the need to "bridge the chasm"¹ between software development teams and IT operations teams. This promises a number of technological and organisational benefits, such as faster delivery cycles, fewer defects, reduced time to market, and greater profitability.

For those companies "born in the cloud", who have never deployed anywhere other than Firebase and think that Helm is how you steer the yacht you bought with the Series C funding, what follows will be largely unfamiliar. These companies have, in effect, NoOps² and no chasm to bridge.

There is a much larger number of organisations that are somewhere on the spectrum between having siloed dev, test, and operations teams, and the NoOps unicorns.

For this second group, there are a number of challenges, many of which have technological solutions, but the most intractable problems, as ever, are the human ones.

The DevOps Centre of Excellence

A common first step on the DevOps Journey is to reorganise the existing menagerie of Security Specialists, Network Architects, and Database Administrators into a DevOps Centre of Excellence, ready and able to respond to the needs of the developer tribe in the blink of a Service Request, within documented and agreed SLAs, subject to approval by the CAB which meets every second Tuesday.

The purchase of a foosball table and a kegerator can be considered optional at this stage on the DevOps journey.

It should be uncontroversial by now that this isn't an improvement on the previous arrangement of Dev and Ops silos ³, as it preserves almost all the downside with little upside other than the kegerator:

The motivations of the "DevOps Team" and the "Dev Team(s)" remain misaligned.
- Failure to deliver a feature on time rarely has any negative consequences for the DevOps team.
- Delivering features in a manner which is complicated or expensive to deploy and patch rarely has any negative consequences for the developers.
The traditional antipathy between Dev and Ops remains present.
- The DevOps team continue to regard the developers as rogue elements with no concern for the security or stability of the system.
- The developers continue to regard the DevOps team as dinosaurs whose only purpose in life is to slow down development.
- In many organisations, both of these characterisations are accurate.

I have touched on these issues in a previous article on this site:

Throwing code over a different fence

Gavin Campbell ・ Dec 11 '17 ・ 3 min read

#culture #devops

These downsides remain regardless of the number of tools-based initiatives undertaken by the DevOps team.

As long as the CloudFormation and ARM templates, the Ansible playbooks and the Helm charts remain the responsibility of the central DevOps team the misalignment of incentives will persist.

This obtains even though the tools and techniques - source control, CI/CD servers, Agile methods, might be identical across the two teams. None of these tools can break down the communication barriers between the two teams.

The Distributed DevOps Team

The next step along the journey often involves dismantling the central DevOps team and "embedding" one or more DevOps engineers within each development team. This is often inspired by the notion that "Facebook does it that way ⁴".

However:

Your company isn't Facebook.
Facebook's Production Engineers probably write more code than your developers

For most companies other than Facebook, the main benefit of embedding DevOps engineers within development teams is the reduction of communication overhead between developers and DevOps Engineers.

Since the management structures associated with the software teams are "unable to provide career progression for Ops people", and the software leaders are "unable to manage Ops people", the embedded ops people are often managed separately to the developers with whom they work, and may even be rotated between otherwise stable development teams.

The child who is not embraced by the village will burn it down to feel its warmth

Anon., the Internet

The consequence of this is that whilst communication overhead may be reduced, incentives are still not aligned between the feature developers and the embedded Ops people. Rather than "throwing code over the fence" into the Ops field, we throw code into the Ops "cage" embedded within every team.

Furthermore, tasks remain strictly categorised - and often "tagged" - as "Dev tasks" and "Ops tasks". This means that the developers continue to complain about the DevOps people, the DevOps people are not motivated to help out the developers.

Another effect is the daily standup where two or three groups of people take turns discussing their otherwise unrelated activities, supervised by the "scrummaster slash project manager". Such meetings are a well-rehearsed variation on the theme of "standup as status meeting"⁵.

The changing nature of IT Operations

There's a case to be made that not much has changed in the world of feature development over the fifty years or so of the software industry. After all, we still type code into editors, the code gets compiled or otherwise assembled into something useful, we might do a bit of testing, and we ship the end product to the users.

The same cannot be said of IT Operations. Whilst automation tools have existed since the very beginning - before interactive editors, in fact⁶ - much of the history of IT Operations has involved defining procedures and documenting them in a way that they can be executed by humans.

For many years now, most application software has been installed in a more or less scripted way, and many companies have found automated ways to ensure that their operating system environments are consistently configured.

More recently, we have seen the rise of Infrastructure as a Service and Platform as a Service offerings, with many modern applications reliant on both. This has brought about changes in tooling to the extent that almost any operating environment can be defined by a set of text files to be processed by some tool or other. This is what is meant by "Infrastructure as Code".

The real DevOps story isn't collaboration, it's convergence

Infrastructure as Code turned out to be so convenient that developers were able to provision their own environments without reference to the Operations team. Naturally, this was accompanied by much grumbling about "rogue developers", "shadow IT", and "out-of-control spending". Meanwhile, the operations teams were busy learning to use these new tools to provision more infrastructure, and faster, than ever before.

So, if application development and infrastructure development involve much the same tools and techniques, is there any value in having separate people to do each of these tasks? And given the relatively high degree of coupling between applications and the operating environments they consume, does it make sense to manage the code that defines them separately?

Given that the lines between front-end code, back-end code, database code, and infrastructure code become fainter with every year that passes, isn't it time we just referred to it as "code"? After all, aren't we all supposed to be "full-stack developers" these days?

Should we not be aiming to develop teams of "T-Shaped" people able to maintain any part of our product, from the infrastructure to the CSS?

Convergence within the team

The hardest part of this convergence is how to manage the integration of the people already working in the organisation, in particular the "Ops professionals" with little to no coding background.

In the modern software workplace, many tasks that were previously done by the Ops teams now need to be done with code. This means that it is increasingly unacceptable for a Database Administrator, a Security Analyst, or a Network Specialist to claim to be "not a coder", just as it is unacceptable for a Full-Stack Developer to "only know jQuery".

A more subtle problem is that the Ops teams may be proficient in writing code, in the form of scripts, templates, or playbooks, but less experienced in the other practices that go into software development such as continuous integration, automated testing, and branching strategies.

Our aim is to capture the expertise of the Ops people, as well as the skills of the developers, since both are important to the quality of the end product.

The developers don't want to do operations and the operations people don't want to do development

Overheard in the elevator

I can't think of any technological solutions, not even Pulumi and friends, that are capable, by themselves, of achieving this convergence.

Tools are important, of course, and it has been my experience that the more developer-friendly the tools can be made, the more the developers are going to entertain the idea of "doing operations". In the first decade of the cloud, a lot of infrastructure code was written using vendor-specific DSLs such as CloudFormation and ARM templates. These satisfy the definition of "Infrastructure as Code", but I think the reason for the popularity of Terraform is that it has features that make it more "code like", including easier composability and reduced verbosity.

The history of database development is a similar story; for many years there were people specialised in creating database objects and PL/SQL procedures, yet in most modern applications these have been replaced by abstractions such as Hibernate, Entity Framework, and Dapper.

The promise of Pulumi is that it will offer a similar abstraction for our infrastructure, and I think it will be interesting to see how this turns out.

Rather, the solutions are all working practices, most of which should be very familiar.

Pair Programming

Pair programming has been around for a very long time, but is still a challenge for many teams⁷. In the context of Dev/Ops convergence, it has many benefits.

Pairing promotes collective code ownership, including infrastructure code, which should mean the end of "application issues" and "infrastructure issues".

If the team members come from a range of different backgrounds, all the better. The developers learn to "do infrastructure" whilst the Operations people learn to "do development". In time, it is hard to tell who used to be a developer and who used to be a Senior Infrastructure Analyst, as both are equally comfortable maintaining the codebase.

This is a pattern that is already emerging in software testing; Microsoft, among others, claim to have abandoned the distinction between developers and testers⁸, and I suspect that in the future we will be saying the same thing about infrastructure engineers.

One enabler for the knowledge-sharing benefits of pairing is the practice of pair rotation, in which we encourage each team member to pair with a range of other team members. This can be reinforced by tools such as the pair programming matrix⁹.

In the situation where we are integrating people with non-development backgrounds, this is even more valuable, as it prevents any one developer from being the one who always has to pair with "the guy from ops". It also allows the people with infrastructure backgrounds to become familiar with many different areas of the codebase, and to share their expertise with many other team members.

Pair programming and pair rotation are not without challenges, and required dedicated effort to ensure success.

It is worth mentioning the practice of "mob programming¹⁰", in which the whole team works together to deliver a feature. This can also be an effective way to promote knowledge sharing, remove barriers, and develop the skills of the whole team.

Automated Testing

If a user story is independently valuable, it stands to reason that it must be independently testable. The convergence of applications and infrastructure means that we can test both at the same time.

To do this we need to write tests not only for the functionality of our application, but also for the security, availability, and observability of our entire environment. This by itself is a huge benefit of the converged approach.

Infrastructure testing is integration testing with real dependencies, and this is an area where the contributions of people with operations experience can be especially valuable.

Thin vertical slices

It is a well-known agile practice to attempt to define user stories as "vertical slices" of functionality¹¹. This means that rather than attempt to deliver a complete database, followed by a complete business tier, followed by an Enterprise Service Bus, followed by a front end, we deliver just enough of each tier to be able to show something of value to the end user.

As well as these tiers, we can also deliver just enough infrastructure to support the feature we want to deliver. This practice might even result in reduced infrastructure costs.

Adding infrastructure and associated tests to every story may make these stories larger and more complex to deliver. The answer, of course, is to make the slices thinner, thin enough that they are deliverable whilst still adding value.

Defining sufficiently thin vertical slices is not always easy. However, investing time in this will make the other practices easier - thin slices are easier to test and easier for a pair of developers from diverse backgrounds to deliver.

Conclusion

Infrastructure as code in't the future, it's the present. The transition to everything-as-code is likely to be harder for the infrastructure professionals than it is for the developers.

There is a disconcerting remark in the Microsoft article about testing convergence linked above⁸

It was a painful transition for the company. We worked hard to move [test engineers] to other roles, and some did, but a good number of them didn’t.

You don't want this to be your company and you don't want these to be your infrastructure people. These are difficult problems, but the practices outlined above, along with others designed to detoxify the workplace, can go some way towards mitigating them.

There may be some infrastructure professionals who resist the transition to everything-as-code. For the time being there may be work for them to do in the "infrastructure that supports the infrastructure", meaning the subscriptions, contracts, invoices, and service agreements. These tasks, which are more administrative than technical, represent the future of the "non-coding IT professional".

For the others, convergence creates an opportunity to develop new skills, and to have a more visible impact on the finished product. Across the team, the aim is not an unrealistic homogenisation, where every team member is a "resource", but to benefit from the diversity of experience of the team members.

A google search for devops bridge chasm returns around 15,000 results at the time of writing. ↩
I think https://go.forrester.com/blogs/11-02-07-i_dont_want_devops_i_want_noops/ is the earliest published usage of this term. ↩
https://web.devopstopologies.com/#anti-type-b ↩
https://engineering.fb.com/category/production-engineering/ ↩
https://www.scrum.org/resources/blog/scrum-myths-daily-scrum-not-status-meeting ↩
https://en.wikipedia.org/wiki/Support_programs_for_OS/360_and_successors#IEBUPDTE Yes, Wikipedia. IBM documentation from the 1960s isn't easy to find online. ↩
https://martinfowler.com/articles/on-pair-programming.html ↩
https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/evolving-test-practices-microsoft ↩
http://alaverdyan.com/readme/2010/12/pair-programming-matrix-board/ ↩
https://www.agilealliance.org/resources/experience-reports/mob-programming-agile2014/ ↩
https://xp123.com/articles/invest-in-good-stories-and-smart-tasks/ , see also https://www.thoughtworks.com/insights/blog/slicing-your-development-work-multi-layer-cake ↩

The rise of the non-coding Scrum Master

Gavin Campbell — Wed, 18 Apr 2018 07:29:34 +0000

Or rather, not so much "non-coding" as "never-coded".

I came across this phenomenon during a recent brush with "Enterprise Agile".

In particular, the notion of "Agile", or more specifically "Scrum" as a skill distinct from software development, was an entirely new one to me.

This notion has given rise to individuals, and indeed teams of individuals, who are entirely conversant - and expensively trained, by "boutique" consultancies - in the terminology and rituals of Scrum and its "Enterprise" cousins - stand-ups, grooming, planning poker, retrospectives, release trains, the all-important "velocity", the list goes on and on.

However, these same individuals are not at all familiar with the daily grind of merge conflicts, broken dependencies, technical debt, and off-by-one errors that make up the life of a modern software developer, as their training and experience is purely in "Agile" and "Scrum", to the exclusion of any considerations relating to the actual writing of code.

In many ways, the "Agile Capability" has come to resemble a tight-trousered, plaid-shirted version of the venerable Project Management Office, or "PMO", with its colourful "information radiators", issue-tracking spreadsheets, and remarkably granular organisational structure - "Junior Product Owners", "Product Owners", and "Senior Product Owners", anyone?

In the interests of disclosure, I should perhaps state that I myself am a Certified ScrumMaster® , a by-product of having attended Jeff Sutherland's course on Scrum a few years ago.

One of the many interesting things he pointed out on that course was that when they were codifying what we now call "Scrum" back in the 1990s, they never envisaged that "Scrum Master" would become a job title. Now, in 2018, it seems to have transcended the job title and become an entire career path.

Is it just me who is a little sceptical about all this, or is this the way of the future?

Throwing code over a different fence

Gavin Campbell — Mon, 11 Dec 2017 11:10:14 +0000

We're often told that the existence of a DevOps Team is something of an antipattern, or indeed "considered harmful", but it wasn't until I saw this in action that some of the reasons for this advice became really clear in my mind, and I thought I'd note some of them down here.

In the "bad old days", the developers used to write code and "throw it over the fence" to the operations team, who were the custodians of the organisation's infrastructure and responsible for all software deployment and maintenance. This procedure had an associated cost, of course, in paying the operations team to come to work at night or at the weekend in order to "babysit the deployment". In order to reduce this cost, the "routine" deployment and maintenance procedures could even be outsourced to lower-paid workers elsewhere in the world.

This approach also created cultural rifts, of course, in that the developers weren't incentivised to "own" the reliability of the product, and the operations teams had little interest in accelerating the shipping of features, preferring to devote their attention to maintaining the integrity of the infrastructure, particularly when this was the sole deliverable associated with their outsourcing agreement.

Added to this was the increase in cycle time due to the need for "hand-offs" between the various teams involved in getting a feature from MacBook to production, which was exacerbated further when the teams involved in the hand-offs had different goals and priorities, and even different employers.

The DevOps Revolution promised to solve all of these problems by aligning the interests of the developers and the operations teams in shipping features more quickly as well as more reliably.

Naturally, since shipping more features more reliably sounded like an excellent idea, many organisations decided to get a headstart on their DevOps Journey by recruiting a dedicated team to establish their DevOps Capability. The DevOps Team arrived armed with a vast arsenal of tools, and set to the task of establishing carefully crafted pipelines, dashboards with a myriad of metrics, and deployments in blue, green, and all the colours of the rainbow.

The problem, of course, is that the interests of the DevOps Team were no better aligned with those of the feature developers than those of the Ops team had been previously. So, whilst the DevOps Team were busy high-fiving, having just migrated their deployment orchestrations to the latest tool of the day, and created a brand new dashboard with every metric imaginable regarding code quality, the feature teams were still very unclear about how to deploy and monitor the features they were creating, and were reduced to throwing code over the fence to the DevOps Team.

The way to break down the fences, of course, is that the feature teams need to own the delivery and monitoring processes themselves. It's already uncontroversial to suggest that development teams should seek to cultivate "T-shaped people", with deeper functional expertise in one area, such as databases, testing, or front end coding, as well as a base level of understanding of all the tasks required to successfully deliver a feature. In the modern world, of course, one of these competencies is the ability to create test and deployment automation, as well as infrastructure and monitoring, or in other words the tasks that are being done by the newly established DevOps Team on the other side of the fence.

The business case for shipping more often

Gavin Campbell — Thu, 09 Nov 2017 23:22:31 +0000

A large number of words have already been written about why shipping software in smaller increments more quickly is a good thing to do; by deploying more frequently we become more practiced and our automation becomes better and less error-prone, we can "fail faster" by discovering we haven't shipped the features our customers want, and by extension succeed faster by shipping the features our customers do want, faster and more reliably than our competitors across the street.

There's an additional benefit to shipping more often that is touched on less often in the literature, namely a reduction in our inventory.

Imagine for a moment that rather than being in the business of shipping software, we are living the dream at our very own coffee'n'bagels cart with a suitably droll pun in the name.

Now, we could start out on this venture by spending a million dollars on the finest estate-grown coffee and artisan-rolled bagels, and wait for the customers to come, enticed by our viral marketing copy.

In all likelihood, this would lead to a couple of adverse consequences.

We would run out of money before we could sell sufficient coffee'n'bagels and be unable to meet the other costs of running our cart. This is because we had too much of our starting capital tied up in inventory, leaving us with insufficient funds to keep the business alive. Failure to manage cashflow is responsible for nine out of ten small business failures, according to the Internet Department of Made-Up Statistics.
Assuming we did survive long enough to at least shift some of our coffee'n'bagels mountain, our customers would soon start to complain that the coffee'n'bagels were stale at the point of delivery. By keeping excess inventory, we are no longer able to deliver the products that are relevant to our customers' needs, viz. fresh coffee'n'bagels.

So how does all this relate to the business of producing software?

Well, the inventory is all the software we have paid to develop - in developer salaries, Aeron chairs, and indeed coffee'n'bagels, but have not yet delivered to our customers. Put another way, this is "value" that we have potentially generated, but have been unable to realise due to our failure to deliver it to our customers.

These customers need not be paying customers of course, they could just as easily be the internal users who have been wondering whether those guys and girls over there in "IT" actually do anything other than consume coffee'n'bagels whilst sitting in their fancy chairs, and whether the all-singing, all-dancing, relationship-managing intranet will ever actually be delivered.

By keeping all this inventory of undelivered software on hand, we risk suffering the same fate as the coffee'n'bagels cart, namely exhausting our supplies of cash - or the patience of our users - before we can put the software in their hands so it can create the value they have been waiting for.

Likewise, by keeping our software "on the shelf" for weeks and months, we run an ever increasing risk that it will become stale, and no longer relevant to our customers' needs.

So, how do we go about reducing this inventory? By shipping more often, of course, in smaller increments to reduce risk, as well as reducing the quantity of undelivered value "on the shelf".