Forem: Jonny Evason

Getting into Azure with Terraform

Jonny Evason — Sat, 17 Oct 2020 17:12:28 +0000

Terraform for beginers with Azure

I originally posted this on my personal blog site where I have more posts explain how I got the site up and running.

Working with Terraform

This post will discuss some of the basic concepts around using terraform and how it has been used to build this blog site.

What is Terraform

Terraform is an open source tool used to create, configure and maintain a wide variety of system infrastructure.

In most cases it's used to manage systems built on the cloud, whether that's Azure, AWS or Google Cloud Platform (or a mixture!) but it can also be used to manage other things like Datadog amongst many more.

This is typically done via "providers" which can be thought of extensions to the base terraform tool with specific functionality to create resources.

In this example we will be using the Azure RM (resource manager) provider.

Prerequisites

Your gonna need an Azure account (if you don't have one already).
You will also need the terraform tool

How does it work

I won't profess to known the inner workings of Terraform, but I will go over what I know.

It's all about state

State is how Terraform knows what you've currently got managed via the tool. As we will see later as the terraform tool is used it creates a .tfstate file which is basically a big block of JSON containing all of the knowledge about what you are intending to deploy or have deployed. It's also used by Terraform to work out what you haven't currently got managed under it's system.

State plays a very important role and if the state file is deleted or becomes corrupted it can be a bit of a head ache!

State can be configured to either be a local file or stored externally, in the case of this site I had chosen to upload the state file to blob storage. It's all automated (probably a post for another day) so I don't have to overly think about losing the state.

Command the helm - init, plan and apply

Terraform has a wide variety of commands to manage resources, but in my experience so far the main 3 I've been working with are init, plan and apply.

init initialises the directory ready for use with Terraform, it scans through the .tf files to do a sort of "pre assesment", here it determines what prodivders will be required and goes and fetches them.

It also determines the backend configuration i.e if it will keep the state file local or should use on refernced elsewhere.

plan Is the second main command I use, as the name implies it generates a plan that will be executed by the apply command. Running the plan displays a list of changes that the Terraform has calculated, be it new resources or configuration denoted by +, changes to existing resources denoted with ~ or the "destruction" of resources denoted with the -. This is really useful for giving a visual check before you up and remove say a database.

apply Happy with the plan? Hit apply. With this final command Terraform simply executes and creates, changes or destroys the resources. There really isn't anything more to it.

Bonus command

destroy is probably worth mentioning, as it's useful when you're done with the resources and want to completely remove them.(Note it will only destroy the resources currently managed by Terraform).

Creating the Resources

Now I'm going to go through some steps to spin up a serverless static website.

Getting set up

So you've got the azure account and downloaded Terraform, then you're ready to go.

Open up your favourite IDE and let's begin.

Firstly if you haven't already grab the resources on my github then open a terminal in the src directory (I use VS Code).

I'll breifly explain the files.

There is the main file, which specifies the version of the azurerm provider (terraform recommends sticking to a specific version, mainly for continuity purposes) as well as a resource group.

    provider "azurerm" {
    version = "=2.23.0"
    features {}
    }

    resource "azurerm_resource_group" "resource-group" {
    name     = "simple-terraform-app"
    location = "ukwest"
    }

The other file static_site contains the definitions for the static site

    resource "azurerm_storage_account" "static-site" {
    name                     = "simpleterraformsite" #This will need changing to a unique value
    resource_group_name      = azurerm_resource_group.resource-group.name
    location                 = azurerm_resource_group.resource-group.location
    account_tier             = "Standard"
    account_replication_type = "LRS"
    account_kind             = "StorageV2"

    enable_https_traffic_only = false

    static_website {
        index_document = "index.html"
        error_404_document = "404.html"
    }
    }

From here it's a case of running terraform init which should give results as below:

Next step is to run the terraform plan again the out put will bt similar to the below

Once that's done and dusted it's time to run terraform apply here terraform should display the plan but this time prompt you to confirm the plan, simply type yes.

You will then see more console output from terraform detailing the current completion state.

Checking the resources

Now head over to Azure to confirm that the resource group has been created as expected.

In this resource group you should find a single storage account resource

If you then inspect the Static Website tab in the left hand panel of the resource you should be able to see details on the sites configuration

From here the next steps to get a working site are to upload the html files found in the repo.
Navigate to the container in the storage account and upload the files in the $web container.

Once complete when you navigate to the web address of the blob container you should get the index page.

A static site hosted in blob is a low cost (practicaly free) way of creating a website in Azure.

That said it's always a good idea to remove resources you don't need, so it you don't plan on keeping the site run terraform destroy and confirm to remove the resources

And that's it, hope you enjoyed.

Where to find me

Best place would be twitter

Azure Functions - Creating a serverless API with NodeJS

Jonny Evason — Fri, 22 May 2020 16:29:03 +0000

Recently I have been working with the team here at Footasylum on building a serverless API using Azure Functions.

The API needed to be simple and lightweight, we were already using Azure and had familiarity with Functions (we write a lot of them) as well as using Cosmos DB as a dynamic store of data.

We wanted to store primary product information from the ERP and make it accessible to other parts of the business via a web API, thus reducing the load on the EPR and creating a way to the other parts of the business to embellish the primary information, so that it’s fit for customer use.

As we didn’t want to impose too many restrictions on what we received, Javascript felt like the right choice of language for the function.
The system needed a way to create, read and update any records and can be seen in the diagram below:

Azure API Management is a great tool for managing access and control over the consumers of our APIs and we use it as the entry point for all the microservices we are developing, more on this another time.

Functions and their Magic

With the system overview out of the way lets discuss the implementation details of the functions. I’ll focus on the Create Product Function first.
To begin we initialised a function app via the command line tooling func init and then created the function within that using func new. The selected runtime of the app was Node and the type of function here was a HTTP trigger.

When the commands complete you are left with some initial boilerplate app structure like below:

where the boilerplate code looks something like this:

    module.exports = async function (context, req) {
        context.log('JavaScript HTTP trigger function processed a request.');

        if (req.query.name || (req.body && req.body.name)) {
            context.res = {
                // status: 200, /* Defaults to 200 */
                body: "Hello " + (req.query.name || req.body.name)
            };
        }
        else {
            context.res = {
                status: 400,
                body: "Please pass a name on the query string or in the request body"
            };
        }
    };

The func new function creates an index.js file, which is the entry point into the function as well as a function.json file, the contents of which looks like this:

    {
    "bindings": [
        {
        "authLevel": "function",
        "type": "httpTrigger",
        "direction": "in",
        "name": "req",
        "methods": [
            "get",
            "post"
        ]
        },
        {
        "type": "http",
        "direction": "out",
        "name": "res"
        }
    ]
    }

For the CreateProduct function we needed to take the request body and save it into the cosmos DB.

Traditionally as developers we would want to create or use some sort of client library to mediate and control the persistence of data objects into some sort of database, however the “magic” here is in the function.json file.
In my (somewhat limited) understanding of the concept, the bindings represent actions that the function can complete in the given context. When the func new command runs, it creates a binding that’s related to the type of function we are creating, but the rules aren’t limited to just one binding or type, as well will see.

When we selected the type “httpTrigger” this created some boilerplate configuration in the function.json (see picture above) we can see a type “httpTrigger” as well as a direction of “in” and a name of “req”, along with httpTrigger specific parts like “methods”. Below that we can see another binding of type “http”, name “res” and direction “out”.
The bindings here determine the input and the output of this function, and bind various bits to objects that are matched to the name property.

For example in the code above, we can see a req parameter that gets passed into the function. This represents the httpTrigger request object that carries all sort of information related to the http request it has received, such as the query string or the body of the request.
On the reverse we can see the context.res creates the response, that is sent back out of the function to the caller.
The behaviour here is determined by those binding properties.

In the case of the Create Product Function we simply needed to define a binding to handle saving data to the cosmos:

    {
      "name": "newProduct",
      "type": "cosmosDB",
      "leaseCollectionName": "leases",
      "connectionStringSetting": "Products_DOCUMENTDB",
      "databaseName": "Products",
      "createIfNotExists": "true",
      "collectionName": "Products",
      "createLeaseCollectionIfNotExists": "true",
      "direction": "out"
    }

As shown above, we can see that we define a type of “cosmosDB” and define its direction as “out”, along with a connection string (this is actually the key in key value pair stored in the functions configuration), a database name, a collection name and whether or not to create it if it doesn’t exist.

Ready for the fun part? Saving the incoming payload now distills down to a single line of code. Below is the full function code, with some validation.

    module.exports = async function (context, req) {
        if (!req || !req.body ) {
            context.res = {
                status: 400,
                body: {
                        Success: false, 
                        Message: "Received an empty payload"
                    }
            };
        }
        else {
            context.bindings.newProduct = req.body;
            context.res = {
                status: 200,
                body: {
                    Success: true
                }
            };
        }
    };

Hopefully you spotted the line of code for saving the payload to the cosmos but in case you didn’t it’s this line here:
context.bindings.newProduct = req.body;
As we can see the context object that has been passed in contains the bindings object, with our cosmosDB binding which we named newProduct and that’s all that is required to save the payload to the cosmosDB.

Reading Cosmos Data

Now that we were saving stuff, we needed a way to retrieve it. First step was to create a new ReadProduct function, again this would be a http trigger type, configured to respond to a GET with some route parameters.

    {
      "authLevel": "anonymous",
      "name": "req",
      "type": "httpTrigger",
      "direction": "in",
      "methods": [
        "get"
      ],
      "route":"Product/{productId}"
    },
    {
      "name": "res",
      "type": "http",
      "direction": "out"
    }

In the above we have added in some extra binding details in the form of the route attribute, this simply appends the definition on the funcitons URL. The ProductId here is the internally generated ProductId coming in from the ERP system.

Hopefully you’ve guessed the next step, adding in a cosmosDB binding with a direction of “in”

    {
      "type": "cosmosDB",
      "name": "readProduct",
      "databaseName": "Products",
      "collectionName": "Products",
      "connectionStringSetting": "Products_DOCUMENTDB",
      "direction": "in",
      "sqlQuery": "SELECT * from c where c.ProductId = {productId}"
    }

The only other noteworthy part in the above binding is the sqlQuery, here we defined a SELECT to get us all the data in a document of a given productId.

Function code

    module.exports = async function (context, req, readProduct) {

        if (!readProduct || readProduct.length === 0) {
            context.res = {
                status: 404,
                body: {
                        Message: "Product not found"
                    }
            };
        }
        else {
            context.res = {
                status: 200,
                body: readProduct
            };
        }
    };

In the above, we have defined a parameter readProduct, this is the same as the name of the cosmosDB binding we defined in the function.json.

When this function is called the productId we pass in the URL route parameter is taken by the function and injected into the SQL query defined on the cosmos binding, if a record is found then it is assigned to readProduct and subsequently returned on the context.res (the http out binding in the function.json)

Cool, now we are saving and reading records out of the cosmos DB, but what about updating records?

Update Product Function

Ready for the coolest part in all of this?

So an update would require, an incoming payload and the original record and that would be and in and an out on the cosmos DB. Again we created another HTTP function, similar to the read, however we also then combined the Create stuff to create the following function.json

    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": [
        "put"
      ],
      "route":"Product/{productId}"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    },
    {
      "type": "cosmosDB",
      "name": "readProduct",
      "databaseName": "Products",
      "collectionName": "Products",
      "connectionStringSetting": "Products_DOCUMENTDB",
      "direction": "in",
      "sqlQuery": "SELECT * from c where c.ProductId = {productId}"
    },
    {
      "name": "updateProduct",
      "type": "cosmosDB",
      "leaseCollectionName": "leases",
      "connectionStringSetting": "Products_DOCUMENTDB",
      "databaseName": "Products",
      "collectionName": "Products",
      "direction": "out"
    }

Here you can see that we have 2 entries for the cosmos DB a readProduct and an updateProduct which are in and out respectively.
The code for the function can be see below:

    module.exports = async function (context, req, readProduct) {
        context.log('JavaScript HTTP trigger function processed a request.');

        if (!req || !req.body) {
            context.res = {
                status: 400,
                body: {
                        Success: false, 
                        Message: "Received an empty payload"
                    }
            };
        }
        else if (!readProduct || readProduct.length === 0) {
            context.res = {
                status: 404,
                body: {
                        Message: "Product not found"
                    }
            };
        }
        else {

            var cosmosId = readProduct[0].id;
            var updateProduct = req.body
            updateProduct.id = cosmosId;

            context.bindings.updateProduct = updateProduct;
            context.res = {
                status: 200,
                body:  {
                    Success: true
                }
            };
        }
    };

It works in the same way as the previous two functions, the productId is passed in via a route parameter into the SQL of the cosmosDB “in” binding and any object that is found is assigned to the read product, in the else clause we then create the updateProduct object, assign the value of the request body and then append the cosmosId from the readProduct result.

When the line

`context.bindings.updateProduct = updateProduct;`

is called this overwrites the existing document with the new document passed in on the body of the update request.

And that is it. Thanks for sticking with me on this, hopefully you can see how powerful (and easy) it is to get a web API up and running with Node functions and cosmos DB.

Hello 2020!

Jonny Evason — Tue, 31 Dec 2019 20:48:35 +0000

Hello! Thanks for stopping by, I do hope you will stay to the end.

TL; DR

Hi first post.
2020 is going to be a great year and the start of another great decade.
Follow along with my journey and hopefully we can grow together in the world of DevOps!

Full blown Story

At the time of writing it's not quiet 2020 but it's only a couple of hours away.

I am writing this in order to introduce myself to the dev world and hopefully gain some followers.

I'd like to firstly reflect on the past decade, which for me as been both challenging and interesting.

I will briefly skip over the first few years, mainly because it's not all too interesting and secondly because it was 10 years ago!

2010 started with Uni and 2014 saw some life experiences that have resulted in a side step / career change leading me up to this very point.

I studied Acoustics in Uni (not guitars, but engineering and science around sound waves) and this naturally progressed into a career in that industry, firstly as a junior consultant then lastly as an acoustic engineer.

In 2014 I became a father, and this was about the same time I picked up a Raspberry Pi and started to tinker with some personal IT projects at home.

A couple of months in and I knew at this point that I wanted to work in IT.

Using the little bit of C# knowledge form uni I manage to land my first IT job in 2015, starting as a junior in both IT Support and Developement for a small 3 person team. I quickly moved into a permanent Software Role taking on some interesting projects, whilst constantly absorbing as much knowledge as possible.

Which brings me to the past 6 months or so.

Current Role

Enter Footasylum a leading retailer of foot wear (what gave it away right?) here in the U.K, who are currently undergoing a "digital journey".

My previous IT role was sort of eCommerce and so I felt at home when I joined Footasylum. The problems somewhat familiar, the nature of the business and business goals not too dissimilar.

I work as part of the platform team here developing microservices that aim to replace a legacy system, that has become some what of a pain point for users and the wider business.

But the team is so much more than that.

Yes we are developing microservices to replace business functionality, however we are also seeding and growing the culture, the attitude, the way in which we work building and delivering software in a concerted effort to help bring DevOps into the business.

Enter 2020 & My DevOps future

DevOps (at least in my understanding) is about delivering quality software safely so that the business can exceed it's goals, for a more structured definition I particularly like the CALMS framework for achieving DevOps.

A culture of working alongside the business; empathising with its cause and sympathising with its customers. Enabling the various departments to be innovative, market distributors because we servie them, we are their enablers.

Automation so we can deliver faster, safer stronger and with 100% confidence every time.

Looking at our systems as a whole and watching their every move, so that we can fail fast see those errors, spot a user trend and react.

Sharing knowledge (because sharing is caring) with team mates, lifting and elevating them to new heights.

This to me is a fascinating concept and one that I truly believe in.

So...

In 2020 I will be taking a change of role from a developer into a DevOps engineer, which will entail aiding the business in getting to its DevOps goal.

In some ways this will be new territory for me, so I am therefore cordially inviting you my reader(s) to join me on this journey as I discuss and dissect various parts along the way, highlighting various pitfalls and traps as well as triumphs and "wins" that may occur.

Stick with me as I (and hopefully you) learn all about taking a business through the DevOps mill.