Forem: James Cook

Azure Functions Deployment using GitHub Actions

James Cook — Wed, 09 Mar 2022 13:13:53 +0000

Have you ever been in a situation where you have been asked to set up CI/CD for your new Azure Function App? Have the requirements been to use a single repository and workflow in GitHub to distribute the application to deployment slots?

In this post, you will follow step by step instructions to set up deployment slots within your Azure Function resource, implement a branching strategy in your repository, identify and create environments in GitHub, and assemble a single workflow to run in GitHub Actions as your CI/CD.

What you need

You would have already created the Azure Function resource in advance of following the below. You will also need access to this resource to configure it and download publication profiles.

Deployment Slots

My workflow for this project will be targeting three deployment slots:

Development
Test
Staging

Deployment Slots allow me to run multiple environments within a single Azure Function App. This will provide a unique public endpoint for each of them, allowing me to develop and test without impacting the staging and live slots. I haven't listed a production slot, and this is because a default slot is available after creating an Azure Function for production use.

If you want to create a Deployment Slot:

Open Azure Function App
Select Deployment Slots from the side menu
Select Add Slot
Provide a name for your slot. The Function App name will automatically appear at the start, so it's unique across the .azurewebsites.net domain. You can then click the Add button when you are ready.

Branch Strategy

I will be using three git branches to represent the slots I've created above. They are:

main (will deploy to Staging slot)

test (will deploy to Test slot)

develop (will deploy to the Development slot)

I will not require a branch for production as I will either run a manual Swap of the Staging slot with Production or incorporate a release approval gate that will trigger the Swap with Staging and Production in the workflow.

Publish Profile

To publish our code to a deployment slot, we require the Publish Profile for each slot (excluding production). To do this, follow each step below and repeat for each slot you have:

Open Azure Function App
Select Deployment Slots from the side menu
Select a Slot (one with the blue hyperlink)
At the top of the window, select Get Publish Profile. This will download a copy of the publish profile as a file.

Repeat these steps per deployment slot.

Environments

I will create an environment in GitHub for every deployment slot I created, allowing me to associate each publish profile to its environment using secrets. To create an environment:

Open your GitHub repository in the Web UI
Select Settings
From the side menu, select Environments
Click the New environment button
Give your environment a name and click Configure environment

Repeating the above steps, I have the following environments:

Development
Test
Production (where we will publish to the Staging slot)

I have enabled Required reviewers in each environment and added myself. When we run a job within a workflow against any of these environments, I do not want it to proceed until my approval. If you don't have this selected, our workflow will automatically deploy to the environment once the Pull Request merges.

Secrets

Within each environment, we will be creating a new secret that will contain the contents of the publish profile. First, select Add Secret under the heading Environments secrets in the environment you are configuring.

In the Name field, give your secret an appropriate name. The name you provide will be used when referring to the secret in the workflow, so make it identifiable. In the Value field, copy and paste the content from the Publish Profile file. This includes all the information required for authenticating and deploying your Azure Function App to the correct slot. Select Save when you are ready.

For this post, my secret will be named AZURE_FUNCTIONAPP_PUBLISH_PROFILE.

Workflow

I will break down each part of the workflow we will use to deploy our Azure Function.

name: Function App Deployment
on:
  push:
    branches: 
      - main
      - test
      - develop
  workflow_dispatch:

Above is the first part of our workflow. I have defined the workflow's name as Function App Deployment and specified this workflow to run when push (updates) happen on one of the three branches. I have also included workflow_dispatch, so I can run the workflow manually.

jobs:

  cd-build-deploy-dev:
    name: Continuous Deployment - Development
    runs-on: ubuntu-latest
    environment: Development
    if: github.ref == 'refs/heads/develop'
    env:
      AZURE_FUNCTIONAPP_NAME: MyHTTPTrigger
      AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
      DOTNET_VERSION: '3.0.13'

Next, we define the jobs we are going to run. We will have in total three jobs, one for each environment. First, we will define the development environment job. We will set the job id to cd-build-deploy-dev and the name to Continuous Deployment - Development. We will be using ubuntu-latestfor all the runs, but we will be setting environments to each environment we create in GitHub.

We use an if statement here to say, "if push happens on the develop branch, run the job". We do this for all three of our jobs to run based on the branch push. If the if statement is not met, the job will skip.

And we define environment variables for the job. These will be:

AZURE_FUNCTIONAPP_NAME - the name of the function app

AZURE_FUNCTIONAPP_PACKAGE_PATH - the path to the code (mine is at the root of the repo so I specified ".")

DOTNET_VERSION - the version of .NET I am using.

    steps:

      - name: Checkout
        uses: actions/checkout@main

      - name: Setup DotNet 3.1.x Environment
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.x

      - name: Resolve Project Dependencies Using Dotnet
        shell: bash
        run: |
          pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
          dotnet build --configuration Release --output ./output
          popd
      - name: Run Azure Functions Action
        uses: Azure/functions-action@v1
        id: fa
        with:
          app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
          package: '${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}/output'
          publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}

Once the job is configured, we want to configure each step we require to get the deployment going.

The first step is Checkout; this will clone the branch to the local runner.

The second step will set up .NET on the runner. Here I have specified to install the latest version of 3.1. We are using the GitHub action project setup-dotnet to complete the installation on the runner.

The third step will trigger a build ready for the function app to deploy.

The final step is for deployment to take place to our Function App. We specify environment variables and GitHub secrets to complete the app name, package and publish profile parameters. This uses a supplied GitHub action from Microsoft called functions-action.

Now you have to complete this for the other jobs (listed below) and copy the steps below each job (the steps should be identical).

  cd-build-deploy-test:
    name: Continuous Deployment - Test
    runs-on: ubuntu-latest
    environment: Test
    if: github.ref == 'refs/heads/test'
    env:
      AZURE_FUNCTIONAPP_NAME: MyHTTPTrigger
      AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
      DOTNET_VERSION: '3.0.13'

  cd-build-deploy-prod:
    name: Continuous Deployment - Production
    runs-on: ubuntu-latest
    environment: Production
    if: github.ref == 'refs/heads/main'
    env:
      AZURE_FUNCTIONAPP_NAME: MyHTTPTrigger
      AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
      DOTNET_VERSION: '3.0.13'

The final workflow should look similar to this:

name: Function App Deployment
on:
  push:
    branches: 
      - main
      - test
      - develop
  workflow_dispatch:

jobs:
  cd-build-deploy-dev:
    name: Continuous Deployment - Development
    runs-on: ubuntu-latest
    environment: Development
    if: github.ref == 'refs/heads/develop'
    env:
      AZURE_FUNCTIONAPP_NAME: MyHTTPTrigger
      AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
      DOTNET_VERSION: '3.0.13'

    steps:

      - name: Checkout
        uses: actions/checkout@main

      - name: Setup DotNet 3.1.x Environment
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.x

      - name: Resolve Project Dependencies Using Dotnet
        shell: bash
        run: |
          pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
          dotnet build --configuration Release --output ./output
          popd
      - name: Run Azure Functions Action
        uses: Azure/functions-action@v1
        id: fa
        with:
          app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
          package: '${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}/output'
          publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}

  cd-build-deploy-test:
    name: Continuous Deployment - Test
    runs-on: ubuntu-latest
    environment: Test
    if: github.ref == 'refs/heads/test'
    env:
      AZURE_FUNCTIONAPP_NAME: MyHTTPTrigger
      AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
      DOTNET_VERSION: '3.0.13'

    steps:

      - name: Checkout
        uses: actions/checkout@main

      - name: Setup DotNet 3.1.x Environment
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.x

      - name: Resolve Project Dependencies Using Dotnet
        shell: bash
        run: |
          pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
          dotnet build --configuration Release --output ./output
          popd
      - name: Run Azure Functions Action
        uses: Azure/functions-action@v1
        id: fa
        with:
          app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
          package: '${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}/output'
          publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}

  cd-build-deploy-prod:
    name: Continuous Deployment - Production
    runs-on: ubuntu-latest
    environment: Production
    if: github.ref == 'refs/heads/main'
    env:
      AZURE_FUNCTIONAPP_NAME: MyHTTPTrigger
      AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
      DOTNET_VERSION: '3.0.13'

    steps:

      - name: Checkout
        uses: actions/checkout@main

      - name: Setup DotNet 3.1.x Environment
        uses: actions/setup-dotnet@v1
        with:
          dotnet-version: 3.1.x

      - name: Resolve Project Dependencies Using Dotnet
        shell: bash
        run: |
          pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
          dotnet build --configuration Release --output ./output
          popd
      - name: Run Azure Functions Action
        uses: Azure/functions-action@v1
        id: fa
        with:
          app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
          package: '${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}/output'
          publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}

Commit the workflow once completed and update your branches, so they are up-to-date.

Deployment

Now we have our workflow in place, we can trigger the workflow either by pushing to a particular branch or running manually.

For this post, I am going to trigger the deployment manually. To do this, I select the Actions tab and select my workflow from the left side menu.

From there, I click on the Run workflow button and select the develop branch to deploy the function app to the development slot.

If you've set up reviewers on your environments, the deployment will not start until a reviewer approves the job.

Once the deployment completes and I can confirm it's working as intended, I can repeat the above to deploy to the testing slot and then the stagging slot.

Once I am happy for the app to go into the production slot, I will use the Swap button in the Deployment blade to swap Stagging with Production (within the Azure Function App resource).

Azure Key Vault Secrets in GitHub Actions

James Cook — Mon, 15 Nov 2021 17:02:39 +0000

The fundamental rule to a secret is to not share a secret. Once shared it's more likely going to be shared again and in an unsecure format, but how do we keep a secret a secret?

When it comes to Cloud technology we can use resources that store our sensitive information in a secure environment. For example, Azure Key Vault allows us to store secrets, certificates and keys where we can set access control using authentication methods like Azure AD.

But when we add secrets into a secure resource like Key Vault, how do we access them when running deployments?

In this blog post I will be covering how we get the secrets from an Azure Key Vault for a deployment in GitHub Actions.

GitHub Workflow

We will need login to Azure using the Azure CLI. The first workflow step will be the following:

      - name: Azure CLI Login
        uses: Azure/login@v1.1
        with:
          creds: '{"clientId":"${{ secrets.AZ_CLIENT_ID }}","clientSecret":"${{ secrets.AZ_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZ_SUBID }}","tenantId":"${{ secrets.AZ_TENANT_ID }}"}'

The following are GitHub Secret values that need to exists before running the workflow:

AZ_CLIENT_ID - Service Principal Client ID

AZ_CLIENT_SECRET - Service Principal Client Secret

AZ_SUBID - The Subscription ID you are connecting to as part of this workflow

AZ_TENANT_ID - The Tenant ID where the Service Principal exists

Once logged via the Azure CLI, we will utilise the Get Key Vault Secrets GitHub Action where we will specify the Key Vault name and the Secrets we want:

      - name: Azure Key Vault Secrets
        id: azurekeyvault
        uses: Azure/get-keyvault-secrets@v1
        with:
          keyvault: "MyVaultName"
          secrets: 'MyFirstSecret, MySecondSecret, MyThirdSecret'

You would replace the following values with your own:

MyVaultName - You would replace this with the name of your Key Vault

MyFirstSecret, MySecondSecret, My ThirdSecret - Replace these with the name of the secrets in your Key Vault (not the values).

Now when you want to use these secrets in the workflow, you just need to use the following format:

steps.azurekeyvault.outputs.MyFirstSecret

Replace the following for your configuration:

azurekeyvault - This would be the id of the Key Vault action

MyFirstSecret - Replace this with one of the secret names you listed to get

Service Principal Access

The above workflow uses a Service Principal to connect to Azure. It would be used to access the Azure Key Vault and will require access permissions to access the secrets. You can do this within the Key Vault itself, either by using RBAC or Access Control (depending on what authentication method you set for the Key Vault).

The GitHub Action only gets the secret from Azure Key Vault, meaning you only need to set permissions with the minimum to be able to get the specified secret you want.

Example Usage

Below are some examples of using the above Azure Key Vault action to use secrets within other actions.

Terraform

      - name: Install Terraform
        uses: hashicorp/setup-terraform@main
        with:
          terraform_version: latest

      - name: Terraform Init
        id: init
        run: terraform init

      - name: Terraform Plan
        id: plan
        run: terraform plan
        continue-on-error: true
        env:
          TF_VAR_az_tenant_id: ${{ secrets.AZ_TENANT_ID }}
          TF_VAR_MyFirstSecret: ${{ steps.azurekeyvault.outputs.MyFirstSecret }}
          TF_VAR_MySecondSecret: ${{ steps.azurekeyvault.outputs.MySecondSecret }}

Docker

    - name: Docker Login
      uses: azure/docker-login@v1
      with:
        login-server: myregistry.azurecr.io
        username: ${{ steps.azurekeyvault.outputs.MySecondSecret }}
        password: ${{ steps.azurekeyvault.outputs.MyThirdSecret }}

Terraform: Remove Resource from a Remote State in Azure Storage Account

James Cook — Mon, 30 Aug 2021 06:19:50 +0000

Have you been in the situation where cleaning up your Infrastructure as Code (powered by HashiCorp Terraform) to delete deprecated resources resulted in the Terraform apply taking longer than expected? Maybe this is what you are seeing:

azurerm_backup_protected_vm.rs_name: Still destroying... [id=/subscriptions/***/***], 1h19m50s elapsed]

Once the deployment timed out I found that the resource was already deleted via the Azure portal. The Terraform state file still believes it exists and it will continue to fail the deployment, how do I resolve the issue?

What you need

Based on a Windows client, you will need:

You will also need a Azure account that has permissions to access the Azure Storage Container which stores the Terraform state file.

Steps to resolve the problem

First you should clone your repository so you can locally validate the actions you take have worked (you can complete these steps without cloning but you won't be able to follow steps to validate if the actions worked without running the pipeline again).

Create a override.tf in the location where you stored your Terraform configuration files. Within the file set the resource group name, the storage account and container name and key where the remote state file is stored.

terraform {
  backend "azurerm" {
    resource_group_name  = "resource_group_name"
    storage_account_name = "storage_account_name"
    container_name       = "container_name"
    key                  = "stafe_file_location/terraform.tfstate"
  }
}

Once you've done this and saved the file, run az login (in a terminal of your choice) to authenticate with an account that has access to the Storage Account Container you specified above.

🚀❯ az login

Now you need to set the subscription you are working with. This should be the subscription that your state file manages.

🚀❯ az account set --subscription "SUBSCRIPTION NAME"

The Azure CLI has now been utilised to complete authentication. You will now need to change the local directory your terminal is using to the location where you have cloned your respoistory. On Windows, changing a directory usually is:

🚀❯ cd "C:\Users\CloudJames\***\***"

Once you are in the correct directory, run the terraform init to initialise the configuration so it downloads providers, modules, etc...

🚀❯ terraform init

Once completed, you can run terraform state list to list the resources that are in your remote state file.

🚀❯ terraform state list

The results should appear like this:

🦄❯ terraform state list
***
***
***
***
azurerm_backup_protected_vm.rs_name
***
***

Find the resource that no longer exists in the Azure environment and take note of the name in full (format is resourcetype.resourcename).

We are now ready to remove the resource from the state file. We will use terraform state rm to achieve this. Here is an example:

🚀❯ terraform state rm azurerm_backup_protected_vm.rs_name

When ran, you should get an output like the below:

🦄❯ terraform state rm azurerm_backup_protected_vm.rs_name
Removed azurerm_backup_protected_vm.rs_name
Successfully removed 1 resource instance(s).

To validate this has worked (if you cloned the repo as described at the beginning), you just need to run a terraform plan.

🚀❯ terraform plan

You should not see the resource listed at all for destruction. This will mean you can run your pipeline again for it to continue as normal.

Azure DevOps: Terraform variables with Azure Key Vault

James Cook — Mon, 09 Aug 2021 07:23:36 +0000

We use variables when creating Terraform configuration files to be able to change and adapt our code to be reusable. When configuring variables, the best method for me was selecting a single location for both sensitive (secrets) and non-sensitive (resource names, etc..) information, allowing me to manage variables in one place. In this post I will cover how to use variables in Terraform, how to store variables in Azure Key Vault and how to use these variables in Azure DevOps as part of a deployment.

How to configure variables using Terraform

First you need to declare a variable in the Terraform code you are writing. For example:

variable "VMPASS"{
    type = string
}

I would usually put all my variables in a separate file but in the same directory to make it easier for myself to locate and manage them.

Once configured, you can use the variable in the code by replacing the string you would enter with the variable, for example:

password = var.VMPASS

Make sure you commit your code to a repository in Azure DevOps.

Storing variables in Azure Key Vault

Very simple, create your Azure Key Vault if you haven't done so already. From within the Key Vault resource you will need to create a secret by selecting Secrets from the side menu.

Now select Generate/Import.

And here create the name of the secret for the variable and the value you require for your code. Please note Azure Key Vault does not support curtain characters, for example underscore which is something we require when using external sources for variables. Click here for more information but we will cover in the next section how we remap these.

Link Azure DevOps to Key Vault

We now need to link our Azure DevOps to Azure Key Vault. Open your project within Azure DevOps and from the side menu select Pipelines then Library. Here select Variable group.

Give your Variable Group a name and enable the Link secrets from an Azure key vault as variables toggle. From here you want to select the Azure Subscription and Key Vault you created your Terraform variables in (if you haven't linked your Azure subscription to Azure DevOps, use the Manage link to create a Service Principal).

You might be asked to Authorize the access to the Key Vault but once this is done, you can select the Add option to add secrets to your variable library.

Now click Save to complete the library creation.

How to use variables in your pipeline

Finally we need to either edit an existing pipeline or create a new one. We need to include the variable library we created that connects to Key Vault, to do this you need to select the Variables tab and then select Variable groups.

Here select the Link variable group and select the newly created variable group.

Now we need to add a Bash Script task to run remap commands so the Azure Key Vault variables are in the supported format. Terraform expects from an external source the format to be TF_VAR_NAME (underscores not supported in Azure Key Vault and why we have to remap), where name is the variable name. Below is an example of the command we need to add to the bash script task, repeated for each variable that needs to be remapped for Terraform.

echo "##vso[task.setvariable variable=TF_VAR_VMPASS;]$tf-var-vmpass"

Once this is done, you can add the Terraform tasks to install, initialize and plan/deploy. Once the pipeline runs, the script will map the Azure Key Vault variables to new names that can be identified by Terraform.

Azure Update Management

James Cook — Mon, 07 Jun 2021 06:40:59 +0000

An Azure Automation Account has a feature called Update Management that can manage your Windows and Linux Operating System (OS) updates for Azure, On-Premise and other third party Cloud environments. In this post I will explain what Update Management is, how to switch it on and how to add your servers to it.

What is Update Management

Update Management is a toggle on feature of Azure Automation Account. Within Update Management, you can add servers from multiple environments and manage both Windows and Linux updates. Microsoft does have a supported Operating Systems list here but I will outline in brief what is supported:

Windows 2012, 2012 R2, 2016 (excl. core), 2019
Centos 6, 7, 8 (excl. 7.5)
Red Hat Enterprise 6, 7, 8
SUSE Enterprise 12, 15, 15,1
Ubuntu 14.04, 16.04, 18.04

From within Update Management you can assess servers for missing updates, configure a schedule for updates to run within a maintenance window and configure what to include/exclude from the update cycle.

How do I enable Update Management

To enable Update Management you need to create a Automation Account and Log Analytics Workspace. Once you have both of these created:

Open the Automation Account you created
Select Update Management from the left side menu
Here you are presented with a configure window. Select the Log Analytics Workspace you created and click Enable to complete.

Update Management is now enabled and will monitor any servers that have been configured to use the Log Analytics Workspace.

How do I add Azure VMs

To add an Azure VM, all you need to do is either:

From within the Update Management blade within the Automation Account resource, select Add Azure VM from the top menu bar
Select a VM from the list provided (note that you may have to move Log Analytics Workspaces if it reports the VM using an alternative).

From within the Virtual Machine* select the **Guest + host updates blade from the left side menu
Now select the Go to Update management option and then select Log Analytics Workspace and the Automation Account to be used for updates.

How do I add On-Premise Servers

First you need to install the Log Analytics agent onto either Windows or Linux. When running the installer you will be asked to select a Log Analytics Workspace, select the one you created earlier.

Once the installer completes, allow 30 minutes for it to appear in Azure. To add the server to Update Management:

From within the Update Management blade within the Automation Account resource, select Add non-Azure VM from the top menu bar
Select a server from the list provided.

What type of update can I manage

On Windows you can manage:

Critical Updates
Security Updates
Update Rollups
Feature Packs
Service Packs
Definition Updates

On Linux:

Critical and Security Updates
Other Updates (those not specified as Critical or Security)

How do you schedule updates

From within the Update Management blade within the Automation Account, select Schedule update deployment from the top menu. Here you can select:

The type of OS you want this deployment to apply to (cannot be both Windows and Linux)
What Virtual Machines or groups containing VMs you want to apply this to
What update categories you want to apply as part of this schedule
Any updates to include (leaving this blank means it will do them all) or exclude
Maintenance Window
Date and Time to start
Recurrence (if you want it to repeat)
Reboot options

Select Create once configured. Use the Deployment tab to monitor the progress or history of a scheduled deployment.

Static Code Analyses - Terrascan, Terraform and Azure DevOps

James Cook — Mon, 10 May 2021 06:27:17 +0000

In my previous post I looked at Static Code Analyses with two of the three tools I am going to use in this post. We are now going to look at Terrascan as our analyses tool and have it running from CI/CD platform Azure DevOps which will also host the Terraform code we want to review.

To follow with the post in configuring this setup, you will need the above mentioned tools with permissions on Azure DevOps to be able to create a Pipeline, add extensions from a marketplace and commit to a repository.

The Code

As per my previous post, I will be reusing the example Terraform configuration file which contains bad practices like password in plain text. This will allow me to test the tool as it should flag some of these bad practices.

This is the code example:

In my example, this is stored in a Azure DevOps repository but you can use a third party repository like GitHub as an alternative.

Pipeline Configuration

We will create a separate pipeline within Azure DevOps rather than use the same one we used in the previous post (you can combine them but I will cover this later). This will be used to run Terrascan to analyse the code. You will want to open your project within Azure DevOps and go into Pipelines.

Now you want to create a pipeline for this. Select the new pipeline option and within the new window select Use the classic editor.

Here you want to select the repository where the configuration file is stored. I have stored it in an Azure DevOps repository so will select this as my location.

Once selected, you will then need to select Empty job as the template option for this pipeline.

The first fields will appear asking you to give the pipeline a name and select the agent pools you want to use. For this demo, I have selected to use Hosted Agents where I will run Terrascan on an Ubuntu OS. Below are the configurations I set.

After all fields are filled, you want to select the Run on agent option and configure the agent job name. I opted to calling the agent Terrascan Analyses as it seemed appropriate for what it is doing.

Now we are going to select the plus icon on the run on agent field to add a job. You will be asked to select something from your currently installed extensions or from the marketplace. We will initially need to install Terraform as this is a prerequisite of Terrascan, so we will need to use the Terraform extension from the marketplace (you may have this already so skip this step).

Once acquired from the marketplace you can then select to install Terraform.

Within the Terraform configuration window of the extension, select the version of Terraform you want to run on the Hosted Agent (as of writing this, v0.15.0 of Terraform has a bug that stops the initialisation, this may cause Terrascan not to function so use an earlier version).

Select the plus icon on the Run on agent and select the Bash extension.

Here you want to install Terrascan using the inline function. Here is what I used to install the software.

Repeat the process of adding another Bash extension to the pipeline and this time we are configuring the inline so Terrascan can run the analyses and output the results into an xml file. Make sure to also tick under Control Options heading the Continue on error option or it will fail the pipeline run.

Again, select the plus icon on Run on agent and select the Publish Test Results extension.

Now we are importing the xml output from Terrascan into the test results feature in Azure DevOps. Here is the configurations I used to import.

Once all configured, select Save on the Pipeline.

Pipeline Run

You are now ready to run the pipeline. All you need to do is select the Run option under the three dotted icon next to the pipeline name. The pipeline will report a failure if Terrascan flags something in its analyses, if nothing is flagged the pipeline will succeed.

In my code, I have been flagged by Terrascan which has set the status of the pipeline build as failed.

Code Analyses Report

Now we have the pipeline running and the report being published into the Azure DevOps test reports, we can review these reports in two location. The first is within the pipeline build, select the pipeline job and open the tab Tests. Here you will see the tests than was ran by Terrascan, what passed and failed and reasons for this.

Clicking on the flagged test failure, you will see more details as to why it failed.

Alternatively, you can view the test reports via the side menu under Test Plans and Runs

You can do more with Terrascan but this will not be covered in this post but future posts on the topic. In the meantime, checkout the Terrascan GitHub page for more information.

Diving into Azure Management Groups

James Cook — Mon, 03 May 2021 05:49:54 +0000

When I first heard of Management Groups I thought it was just a way to group subscriptions in Azure. After in depth research on the feature, I found there was more you can do with them so in this post I will cover what are Management Groups and what can you do with them.

What are Management Groups

Management Groups is a feature of Azure used to control RBAC (Role Based Access Control), apply governance via policies and implement cost management to subscriptions that are organised within these groups. You might be familiar with these features already within subscriptions but being able to duplicate configurations from one subscription to another can be a headache to manage. What Management Groups allows us to do is add these subscriptions to one group and then apply these configurations to the group which then populates to the subscriptions and its resources. You can also add a Management Group within a Management Group which will also inherit the configurations set.

Where and how to create

To create a Management Group is straight forward, first we need to locate where to find this feature. Within the Azure Portal search for Management Groups and select the result as per the below image.

On the page you will notice there is already a Management Group called Tenant Root Group. This will contain all your subscriptions. When you create a group it will appear under the Tenant Root. To create a Management Group, select Add from the top menu.

Here you enter the ID and display name for the group you want to create.

Once fields are completed, select Submit. Once created, you can select it to start configuring.

If you create multiple Management Groups and want to move them inside of each other, select the Move option while in one of these groups and select the location.

Subscriptions

Within the group you created, select Subscriptions from the side menu. Here you can select the Add option from the top menu to add the subscriptions you want within this group.

A subscription can only be in one management group at one time.

IAM

Select IAM from the side menu within the Management Group. Here you can configure RBAC in the same way as you would do within a subscription. This will populate down to other Management Groups under this one, subscriptions and their resource groups.

Click here for more information assigning roles in IAM.

Security

Within the Security option, you can review all subscriptions and resource groups security recommendations. You will also see an overall security score rating for the Management Group with a summary of the lowest rated subscriptions.

Click here for more information on using the Security blade to enable Security Center.

Policy

The Policy side menu option allows you to apply governance policies, either pre-built or custom. This will populate down to your subscriptions.

Click here for more information on implementing policies.

Cost Analysis

To analyse what the costs of resources within a Management Group, select Cost Analysis within the side menu. Here you will get an overview of resource costs and cost breakdown based on each subscription within the group.

Click here for more information on using Cost Analysis.

Budgets

You can set budgets to the top level of a Management Group to monitor and control costs. Select Budgets from the side menu, here you can create a budget for the group.

Click here for more information on how to create budgets.

Static Code Analyses - Checkov, Terraform and Azure DevOps

James Cook — Mon, 26 Apr 2021 06:32:02 +0000

Static Code Analyses is a method of reviewing code against policies before deploying it, identifying weaknesses before they are live vulnerabilities in your environment. This is not new, tools for this purpose have been around for a while for development teams but we are talking about Infrastructure as Code (IaC), a practice that hasn't been around as long so the tools available are very few in comparison.

In this post I am covering Static Code Analyses using three tools, these are:

Azure DevOps - a CI/CD platform provided by Microsoft for developers. We will use this to store the IaC and run the code analyses.

Terraform - this is the IaC tool we will use to write our code for Azure infrastructure.

Checkov - we will use this tool by bridgecrew (their open source version) to analyse the code we write in Terraform.

In future posts I will look at other Static Code Analyses tools and will draw comparisons between them.

The Code

For this, I have created an example Terraform configuration file which contains bad practices like password in plain text. This will allow me to test the tool as it should flag some of these bad practices.

This is the code example:

This will be stored in an Azure DevOps repository but you can also use a third party repository like GitHub.

Pipeline Configuration

First we need to create the pipeline within Azure DevOps. This will be used to run Checkov to analyse the code. You will want to open your project within Azure DevOps and go into Pipelines.

Now you want to create a pipeline for this. Select the new pipeline option and within the new window select Use the classic editor.

Here you want to select the repository where the configuration file is stored. I have stored it in an Azure DevOps repository so will select this as my location.

Once selected, you will then need to select Empty job as the template option for this pipeline.

The first fields will appear asking you to give the pipeline a name and select the agent pools you want to use. For this demo, I have selected to use Hosted Agents where I will run Checkov on an Ubuntu OS. Below are the configurations I set.

After all fields are filled, you want to select the Run on agent option and configure the agent job name. I opted to calling the agent Checkov Analyses as it seemed appropriate for what it is doing.

Now we are going to select the plus icon on the run on agent field to add a job. You will be asked to select something from your currently installed extensions or from the marketplace. We will initially need to install Terraform as this is a prerequisite of Checkov, so we will need to use the Terraform extension from the marketplace (you may have this already so skip this step).

Once acquired from the marketplace you can then select to install Terraform.

Within the Terraform configuration window of the extension, select the version of Terraform you want to run on the Hosted Agent (as of writing this, v0.15.0 of Terraform has a bug that stops the initialisation, this may cause Checkov not to function so use an earlier version).

Select the plus icon on the Run on agent and select the Bash extension.

Here you want to install Checkov with further prerequisites using the inline function. Here is what I used to install the prerequisites and Checkov.

Repeat the process of adding another Bash extension to the pipeline and this time we are configuring the inline so Checkov can run the analyses and output the results into an xml file. Make sure to also tick under Control Options heading the Continue on error option or it will fail the pipeline run.

Again, select the plus icon on Run on agent and select the Publish Test Results extension.

Now we are importing the xml output from Checkov into the test results feature in Azure DevOps. Here is the configurations I used to import.

Once all configured, select Save on the Pipeline.

Pipeline Run

You are now ready to run the pipeline. All you need to do is select the Run option under the three dotted icon next to the pipeline name. The pipeline will report a failure if Checkov flags something in its analyses, if nothing is flagged the pipeline will succeed.

In my code, I have been flagged by Checkov which has set the status of the pipeline build as failed.

Code Analyses Report

Now we have the pipeline running and the report being published into the Azure DevOps test reports, we can review these reports in two location. The first is within the pipeline build, select the pipeline job and open the tab Tests. Here you will see the tests than was ran by Checkov, what passed and failed and reasons for this.

Clicking on the flagged test failure, you will see more details as to why it failed.

Alternatively, you can view the test reports via the side menu under Test Plans and Runs

You can do more with Checkov but this will not be covered in this post but further posts on the topic. In the meantime, checkout the Checkov GitHub page for more information.

Azure Disk Encryption for Data Disk on Linux

James Cook — Mon, 19 Apr 2021 06:46:00 +0000

When configuring a new Linux Virtual Machine (VM) you may think your data is stored on Azure hardware which means is encrypted so no further encryption method is needed. It's true Microsoft encrypts its data but this is at rest, meaning it is protected at the hardware level when there is a physical attack. Other than this, there is no logical encryption of the data disk, leaving you vulnerable if someone was able to download the disk. What I will cover in this post is what type of encryption does Azure uses for Linux Data Disks and how to enable this and attach to a VM.

What encryption method is used

The Azure platform uses DM-Crypt to encrypt Linux VMs data. This is the only method available from Azure when encrypting Linux data.

How does the encryption key generate and where is it stored

As encryption is a supported method offered by Microsoft, the Azure platform integrated data disk encryption with Azure Key Vault. As part of the encryption process, you will be asked to select a Key Vault (or create a new one) and select or create the key that will be used for the encryption.

Are there prerequisites

There are a couple requirements you must meet to be able to configure encryption on a data disk:

The Linux virtual machine must have at least 2GB of RAM (8GB if you are doing both OS and Data Disk)
The OS must be one of the supported operating systems Microsoft have outlined in their documentation
You will need to mount the data disk in advance of encryption so the virtual machine can mount after encryption is enabled. You can follow Microsoft documented approach to this here.
The Azure Key Vault you use must have Enable Access to Azure Disk Encryption for volume encryption policy enabled. If not, the Key Vault used for the procedure below will not work.

How to enable encryption using the Azure Portal

Make sure you first mount the data disk to the virtual machine and turn it off ready for encryption to start. When ready, follow these steps:

Open the Virtual Machine resource
Select Disks from the left side menu
Select Additional Settings from the top of the window
From the Disk to encrypt drop down, select Data disks
Select the Click to select a key option that appeared after the above step completed
Here complete the fields, either by select an existing Key Vault or creating a new. Once done, click the Select* button
You will return to the configuration window to finish by selecting Save

How to enable encryption using Azure CLI

Run the following command

az vm encryption enable -g "ResourceGroupName" --name "LinuxVMName" --disk-encryption-keyvault "NameOfKeyVault" --volume-type DATA

Replace the following values with your own:

ResourceGroupName - The name of the resource group that the Linux VM is.
LinuxVMName - Name of the Linux VM where the data disk will be encrypted
NameOfKeyVault - Key Vault name you are using for storing the encryption key

Best Practice: Terraform State in Azure Blob Container

James Cook — Mon, 12 Apr 2021 06:46:08 +0000

Locating your Terraform state file remotely in an Azure Blob Storage shouldn't be as easy as creating a container and configuring the backend, you should consider some best practices. In this post I will outline practices I've used when securing and implementing redundancy to a Storage Account containing Terraform state files.

Note: Some settings may increase the cost of your Storage Account so please refer to the Microsoft's pricing page. You should also consider each practice as a recommendation and evaluate based on your setup/needs.

Subscription and Resource Permissions

When storing Terraform state files in a Storage Account, you need to review the permissions of the Subscription, Resource Group and Resource. Terraform state files contain sensitive information and should itself be considered sensitive. Check IAM (Identity and Access Management) to make sure those who need access to the resource have permissions and those who shouldn't are removed. Consider SAS (Shared Access Signature) as another means of authentication (mentioned further down in this post).

You can locate IAM in the left sidebar with the name Access Control (IAM) from any subscription, resource group or resource.

Selected Networks

You can secure access to your Blob container by allowing access through selected networks. Specifying networks reduce outside threats to the data because only users who are on the specified networks are granted access after authentication.

To configure this, go into your Storage Account and select from the left side menu Networking. From here you can then change the radio button from All Networks to Selected Networks where you can then configure your network settings.

Azure Defender

Azure Defender by default is disabled but can be enabled on the entire Storage Account. Azure Defender detects access attempts on containers that are deemed to be harmful. If you are only using the Storage Account to contain state files in a Blob container, then the cost based on per 10,000 transactions is very low and should be considered as a setting to be enabled.

To enable, select Security from the left side menu within the Storage Account.

From here you can then select the Enable Azure Defender for Storage (you will also be given a current transaction amount this storage account has processed to help with estimating costs).

Geo Replication

There are three forms of replications you can configure, LRS (Locally Redundant Storage), GRS (Geo Redundant Storage) and RA-GRS (Read Access Geo Redundant Storage). Depending on the availability of the data, I would consider either GRS or RA-GRS replication on an environment where Terraform is used to manage many resources. GRS will perform the same synchronization of data that of LRS (three copies to the local region) but will then copy the data to a secondary region. RA-GRS will do the same as GRS but the data will only be read only. Choosing one of these replication types depends on if you want to have a read-only copy of the data in another region or if you want to actively use that copy of data when the source region is down.

Geo Replication can be configured during setup of the Storage Account and reconfigured after its created. To reconfigure an existing Storage Account's replication, select Configuration from the left side menu.

Within the window you can select the from Replication dropdown the type of replication you want for the Storage Account.

Soft Delete

As a precaution, make sure soft delete is enabled. The more days you add onto the soft delete, the more you pay as the data is stored somewhere so set something sensible like 30 days. After the specified soft delete time has been reached, the data is permanently deleted.

To check soft delete is configured, select Data Protection from the left side menu.

Here you can enable and configure Turn on soft delete for blobs.

Versioning

Versioning is an important setting when it comes to file recovery. You can select the Terraform state file and look back at every change that's been uploaded for that file, allowing you to recover from a previous version by making it the current version. Turn this on if not already enabled.

To enable versioning, select Data Protection from the left side menu.

Then check the box next to Turn on versioning for blobs

Change Feed

For observability I recommend enabling Blob Change Feed so changes made on the Blob container are audited for security and investigation purposes. You can either read the log files within the container or use alternative methods to ingest this data to monitor/read.

Enabling change feed, select Data Protection from the left side menu.

From here you can then check the box next to Turn on blob change feed.

Public Access

By default, Blob public access is enabled. To prevent any misconfiguration, disabling public access is the sensible option so only authenticated methods can access resources. Refer to the Microsoft documentation on preventing anonymous read access to Blob storage.

To disable public access, select Configuration from the left side menu.

Select Disabled under the heading Allow Blob public access.

Access Level

There are three access level tiers to a Blob Container, Private, Blob and Container. Private is the only container that prevents anonymous access to the data inside and should be the one to choose. If you've already disabled Public Access, then Private would be applied to all containers and the other two options would not be available.

Select Container from the side menu.

Select a Blob container and select Change access level from the top of the window to then be able to change access level tier.

Delete Lock

Prevent your Storage Account from being deleted by configuring a Lock so deletion cannot happen without the lock being removed.

To set a delete lock, select Locks from the side menu within Storage Account.

From here you can then create a new lock and set the type as Delete.

SAS Tokens

Consider SAS tokens if you are granting access to individuals/applications for a specific period. SAS tokens can then be used to authenticate to the backend using the specified configuration HashiCorp lists on their site. This reduces the need of creating Service Principals or Managed Service Identities (MSI).

Snapshots

I would only recommend taking a snapshot if you are going to manipulate the state file. This would give you an additional recovery point if the state file breaks in some way (I have failed manipulating a recovery file and no recovery point to revert back to). Versioning is an option to recover from but as a precautionary measure, I feel a snapshot is good to have and then delete once you confirm the file is working as intended.

Ctrl+Shift+A in Windows Terminal to launch Azure CLI authentication

James Cook — Mon, 29 Mar 2021 07:56:11 +0000

I found logging into Azure via CLI a repetitive task but not a long one. I did however utilised Windows Terminal Actions to create key bindings to type in the login command.

Configuring settings.json

Launch Windows Terminal;
Select the down arrow next to the add tab option and select Settings;
The settings.json file will open up with your default code editor. Within the file you need to add the below line of code:

    "actions": [
        { "command": { "action": "sendInput", "input": "az login\return" }, "keys": "ctrl+shift+a" }
    ],

Add this code in a place appropriate, it should look something like this:

Once the code has been included in the file, save it and relaunch Windows Terminal (any errors, review and correct).

How to launch

Now your ready to sign in to Azure CLI using the key bindings, launch Windows Terminal and hold down the following keys:
Ctrl + Shift + A

Once you do this, the below will display on your terminal:

You will also have a web browser window appear asking you to login to Azure to authenticate.

What else can you do

You can amend the command line field so you can include some more information to include as part of the authentication. Find all sign in methods from Microsoft Azure CLI documentation. There is other ways to improve on this or to do something different but remember to consider security before implementing these, you do not want to reduce security to improve convenience.

My Journey to Microsoft Certified Azure DevOps Engineer Expert

James Cook — Mon, 22 Mar 2021 07:27:23 +0000

Before I tell you my journey, I want to let you know this is not my first Microsoft certification. The last Microsoft certification I took was in 2014 and since then changed technology specialism between Apple and Google where I became certified. My experience with exams isn't new to me, I’ve done many, so my journey may come off differently to others who explain their experience.

It started in December 2019 where the interest to skill up took me to the Microsoft Certification Poster. I’ve been using Azure in drips and drabs over the year and wanted to learn more about the service range and felt it was best to start with Azure Fundamentals.

Microsoft Certified Azure Fundamentals (AZ 900) 🏁

I first went to the certification page to look at the learning content Microsoft recommends. The page listed all the learning modules from the Microsoft Learn site that is related to the exam topics. To make it convenient for myself, I created a Collection (select the plus symbol next to the module) within my MS Learn profile so I can collate them in one place to access later. Once I done this, I started working through the material which includes scenario driven examples, video tutorials and hands on labs. I allowed 2 hours a day to work through the material (with the casual day off), using the Collection progress bar to monitor how far I was from completing all the modules.

Once I completed all the modules, I went back to the certification page to review the exam skills outline document. I used this to cross off anything I felt confident I understood and covered as part of the MS Learn material. By doing this, I was happy that the content I covered from Microsoft Learn prepared me for the exam which I then booked and passed. This preparation took a month but felt both my existing knowledge and the MS Learn content helped with the fast turnaround.

Microsoft Certified Azure Administrator Associate (AZ 103 / 104) 📚

After passing the fundamentals exam (in January), I set a target for myself to be prepared for the Azure Administrator exam in five months (June). This target was based on a slow paced learning, dedicated to reading material, hands on labs and deploying my own resources. This target was derailed after I found a new interest in learning how to manage Azure resources using HashiCorp Terraform. This put an extra two months (August) onto my planned target, but I did come out learning an IaC (Infrastructure as Code) product and certified in it.

Over the course of seven months I referred to the Microsoft Learn content listed on the Certification page. While working through this content I was also deploying resources in my own subscription by using the trial service Microsoft offers. Even though Microsoft provides hands on labs in their learning material, using Azure outside of the learning content was necessary in my opinion to familiarise with the service settings and to get a feel of the services working with each other. When reviewing the exam skills outline document, my confidence on some subject areas was not great as I felt I was missing some topics that need clarification. After some thought I decided to go ahead and search for an instructor led video course that included video tutorials and the information I was missing. There is many courses out there, and there is a lot of decent instructors who create them so it was hard to choose. I came across Scott Duffy’s course and felt his way of teaching suited me, and yes everything I watched helped with explaining what I felt was missing from my learning and made other knowledge I had stronger. Once completing the course, I went ahead and scheduled the exam and passed in August.

Designing and Implementing Microsoft DevOps Solutions (AZ 400) 🏆

I now have Fundamentals and Administrator Associate exams passed on first attempts within eight months. I wasn’t planning on taking any other exams in 2020, I was going to focus more on the Azure Architect content but from attending a Microsoft Ignite event in January I had a free exam voucher that was expiring at the end of the year. As stated in the last section I started learning IaC where I was using services like GitHub and Azure DevOps. I felt taking the DevOps exam might be beneficial both to my current understanding of DevOps practices and to the work I was doing with IaC. I did some research on the exam and found even passing the Designing and Implementing Microsoft DevOps Solutions exam, there was a prerequisite of passing one of two exam before earning Microsoft Certified Azure DevOps Engineer Expert (AZ-400). The two exams are Azure Developer Associate (AZ-204) and Azure Administrator Associate (AZ-103/104), one of which I've passed as part of my journey, making me eligible to earn DevOps Engineer Expert status if I pass the AZ-400 exam.

Again, I started with the Microsoft Learn content but felt from reading the exam skills outline document that there was huge gaps in my knowledge around development tools (possibly AZ-204 could of helped me). At the time Pluralsight was partnered with Microsoft to provide free content on certifications (this ended in January 2021), the material included information around development skills needed for the exam. I used this to fill in the missing knowledge I had, implementing what I learnt in some of my own DevOps practices, leaving me feeling prepared for the exam. In December I took the exam and passed, completing my twelve month journey to skill up.

Takeaways and What’s Next ⏭️

I learnt during those twelve months that even though the Microsoft Learn material is great in detail and content, there is additional work on your behalf to build on the information they provide. Don’t just read, actually complete the hands on labs and tutorials that’s provided. I also learnt that time commitment is necessary and must be regular so you retain the information.

Now does this mean I can relax with three Microsoft certifications under my belt? Am I skilled up to where I want to be? The answer to both is no, Cloud and DevOps are both constantly changing. I must stay in the loop with advancements in technology and practices as well as renewing my certification yearly so they stay valid. I have started writing blog posts to share my experience and knowledge as well as trying to contribute to both Cloud and DevOps communities, in return I am receiving advice, support and information that is helping me to grow.