Forem: Dylan Morley

Azure Service Bus - Replay Messages CLI

Dylan Morley — Fri, 05 Aug 2022 14:57:00 +0000

When working with an Asynchronous messaging product such as Azure Service Bus (ASB), you're going to be working with queues and publish/subscribe scenarios. In either case, you'll have times when message processing isn't successful - perhaps required data isn't available, perhaps some dependent system is offline.

When this happens, we need to handle the failures, and what we need to do depends on your implementation pattern. Generally, after a certain amount of attempts by your compute processing to handle the message, the transport will dead letter it. You may be using the dead letter sub-queue section of the ASB entity as your strategy for failed attempts, and messages will be located here until you do something about them

You could also be sending failures on to a centralised errors entity - but whatever pattern has been chosen, you'll have some outstanding messages you need to process. At small message volumes, this is achievable with tools such as Service Bus Explorer and the inbuilt portal tooling, but this soon becomes problematic if you have many 1000's of messages to deal with.

While your systems should be self-healing as much as possible, there are times when you need to intervene and would like to have an automated way to deal with scenarios.

For example, simply always replaying messages that dead letter immediately isn't a good strategy - if there's a genuine system problem you're just going to create a request storm of errors. You need to replay when you're happy that processing is going to be successful, and you want an automated way to do this

CLI support

There are numerous ways you can replay messages - you could write a function app, a logic app, or provision any other type of compute that allows you to execute some code.

However, replaying messages felt like a very common problem where we could offer a reusable solution. Creating CLI support would make this easy for anyone to consume - as a CLI tool, it can be installed easily and it becomes a cross-platform solution which allows people to work in the way they want.

We therefore created Asos.ServiceBus.MessageSiphon, which allows you to define a configuration file that represents the message work you want to perform.

In this example, we're connecting to a source namespace using a SAS key, peeking messages from a topic-subscription and cloning them into another namespace, this time connecting using RBAC.

{
    "Logging": {
        "LogLevel": {
        }
    },    
    "ReplayMessagesJob": {
        "JobType": "SourceToTarget",
        "JobName": "Clone-Message-To-Other-Namespace",
        "NumberOfConcurrentProcesses": 5,
        "ServiceBusDetails": [
            {
                "Name": "Source",
                "ConnectionMode": "ConnectionString",
                "ConnectionString": "Endpoint=sb://namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=access-key"
            },
            {
                "Name": "Target",
                "ConnectionMode": "Rbac",
                "FullyQualifiedNamespace": "namespace1.servicebus.windows.net"
            }
        ],
        "SiphonWork": [
            {
                "SiphonMode": "Clone",
                "SourceConnectionName": "Source",
                "SourceEntity": "topic-entity",
                "SourceSubscriptions": [ "test-subscription" ], 
                "SourceBatchReceiveSize" : 40,
                "TargetConnectionName": "Target",
                "TargetEntity": "copy-of-topic"
            }
        ]
    }
}

There are various examples in the README we've published with the package, but usage is always the same. Define a configuration file, then execute it via the CLI after installing the tool

siphon-asb-messages -n D:\temp\file-with-config.json

Wrapping up

By using a configuration file based CLI tool, we put the power in the hands of the user and allow them to define various configurations - supporting a variety of common scenarios and ways to filter the messages, such as by age or message header.

The tool can be installed on build agent pipelines and executed on a schedule, or can be installed by any engineer in their development environment.

This allows you to handle requirements such as network and RBAC restricted namespaces. Instead of allowing engineers to connect to namespaces and manipulate data, you can define it as a job that's executed from a build pipeline in a controlled way. An example Azure Devops Pipeline is included in the README

Source code will be available on Github at https://github.com/ASOS/asos-servicebus-message-siphon very shortly

Azure Logic Apps Standard - Part 2 Build Pipelines and Provisioning

Dylan Morley — Wed, 25 May 2022 12:23:50 +0000

In Part 1, we looked at the requirements and the solution design - now let's think about how to build a hello world application to prove out some deployment Pipelines using Azure DevOps.

I believe making sure your foundations are in a good shape first then building upon them really pays dividends in a new software project - you'll go a bit slower at first whilst you identify and fix issues, but you'll move so much faster as the delivery progresses.

You achieve this through building the necessary automation, testing your build and deploy process until you're happy with a few different scenarios. Invest time and get this right at the beginning. Do this by not focussing on the application itself, which we can just use a walking skeleton / hello-world type application, and instead focus on the deployment process.

What we'll aim to have working as part of this is

A simple, HTTP triggered logic application workflow that returns 'Hello World'
A build pipeline that produce an artifact, ready for deployment
A provisioning pipeline that will use the artifact, create infrastructure and deploy the logic application into

Workflow Extension Model

Logic App Standard is built as an extension on top of the functions runtime. This means the logic app runtime and all of the built-in connectors binaries need to be made available to your application, and you have a couple of choices here how to distribute them.

Bundled runtime

In this mode, you provide values for application settings AzureFunctionsJobHost__extensionBundle__id and AzureFunctionsJobHost__extensionBundle__version. All the required binaries are acquired by the functions host using these settings and this is essentially a managed process for you. This will result in a smaller build time artifacts, but you cannot use custom connectors - only the built in connectors that are in the version of the nuget package are available.

In this mode, you could have a simple deployment story - it's just the JSON files that make up the workflows that would need to be deployed

Package reference mode

In this mode, we don't provide the application settings and instead treat the logic app runtime as just another nuget package. This is available as package Microsoft.Azure.Workflows.WebJobs.Extension, which contains the logic app capabilities and the built-in connectors. We provide a package reference for the version of the runtime we want to consume, and any other packages we want in our solution. This is a more traditional model - we package up everything needed to run the application and deploy it.

If you want to use custom connectors, you must use package reference mode - and as that's a requirement for us, this choice is made for us.

Built in and Custom Connectors

Why built in connectors?

Connectors are either built-in or managed, but what's the difference? When you choose a managed connector, you're going out of process and are making a call into a Microsoft provided connector. This will run in it's own infrastructure, and may come with certain limitations such as maximum size or number of calls per minute. When working in the consumption model, managed API connections can be a good choice and enable powerful integrations with little effort, but the limitations may impact your ability to perform particularly large workloads or have the performance profile you're aiming for,

With single tenant logic apps, we can provision our own plans to run our applications and built-in connectors run in-process on that infrastructure. Therefore, we don't have any rate limits enforced and the only limit is the infra itself. Keeping the call in-process will also provide performance benefits not previously available to us.

While there are a number of built-in connectors (and more on the way) when there's something not currently available, logic apps has an extensibility model that allows you to design your own functionality and make available via a nuget package. ASOS open sourced the Asos Cosmos Connector, which shows how you might go about building a connector.

The output of that is a nuget package, which we'll need to include in our deployments - that's easy enough, as it's just a csproj package reference. However, in order to get a local design time experience, the package also needs to be installed locally in your logic app runtime directory.

Custom Connectors - local experience

When working with a custom connector locally, you'll be using the VS Code IDE to view the workflows at design time, so any custom connectors you create need to be available to the IDE in order to understand how to display them to you, and display the relevant UI for particular actions based on the connector specification. The current version of the extension bundle is installed on your local machine in your user profile, in a sub-directory azure-functions-core-tools - this contains all the built-in connectors code, which are distributed as DLLS alongside the extension code.

Therefore, if we want our custom connectors to also be available, we need to register the connector so it's picked up by the extension. IF you've packaged up your connector as a nuget package, then you can use this to make it available on the local file system so that VS Code can display the relevant information.

Here, we demonstrate how we can follow a convention based pattern to distribute and register our custom connectors using a powershell script. The script we've used for this always expects the libraries to follow a certain pattern and to be available in a nuget feed. This then allows us to extract the package locally, and install the contents in the extensions directory. It's available as a gist to demonstrate - https://gist.github.com/dylan-asos/309f7dd3a59c43dab60fd645293aad3e

At runtime, the application also needs access to the package details in order execute and display the workflow details in the portal - this is a simple process, achieved by packaging all binaries and deploying via a zip process.

Other local considerations

When working as part of a team, the barrier for entry should be as low as possible when someone goes to work on a repository. It should be simple to get up and running - everything needed to get the application running should be understood. A new engineer should be able to clone, build, test and run with minimal effort - how simple this is can be a key key indicator of the health of the solution.

This goes for any local settings files as well - it should be easy getting local settings populated, and this might include sensitive values such as connection strings. Including a script with your project that an engineer can run that will fully populate their local setting file will save you time and make for a more secure solution, so we'll demonstrate how you can do this.

Build Pipelines

Since we're building a project that's consuming nuget packages, we just need to produce a zip file that we'll deploy to our logic app that we'll provision. The order is

1) Get Sources
2) Perform versioning
3) Build projects
4) Run all non-integration tests
5) Publish test results
6) Perform Packaging
7) Publish Artifact

We want a reusable set of pipelines that will let us build any component we built, and follow a 'build once, deploy multiple times' approach.

We now have an artifact, that contains everything what we need to deploy and test our logic app.

What to provision?

Based on the architecture requirements from Part 1, I see the provisioning requirements for this in 3 parts

1) The network - vnet, subnets, public ips and nat. Both the logic application and bastion will depend on this, and this should be provisioned independently from both of them
2) The application and dependencies - This is the main deployment and would contain everything needed to have a working application. Can be created and destroyed independently from the network
3) The Bastion and VM for support purposes

In this way, we can think about the provisioning in smaller chunks - this also allows us to destroy parts of the infrastructure independently from each other - e.g. I may choose to delete my infrastructure at night or at weekends in my dev environment.

Both the application and the bastion VMs depend on the network, but we can disconnect and remove them from the vnet independently, whilst leaving the network in place. This can be helpful, as we may incurr costs from our application deployment if it depends on compute and other PaaS services, but the network is generally low cost and dependent on the traffic generated.

Provisioning logic apps using Terraform

There are a number of tools you could choose from, but the PaaS nature of the architecture fits nicely for Terraform. We'll follow a modules and blueprints approach for our solution, which will mean our integration applications can easily reuse what we build.

At the initial time of design, Terraform didn't support logic apps standard, so an ARM template approach was the only viable solution. Since then, I've created a resource for this in Terraform so you can follow the native approach - documentation is at azurerm_logic_app_standard

resource "azurerm_logic_app_standard" "example" {
  name                       = "test-azure-logic-app-standard"
  location                   = azurerm_resource_group.example.location
  resource_group_name        = azurerm_resource_group.example.name
  app_service_plan_id        = azurerm_app_service_plan.example.id
  storage_account_name       = azurerm_storage_account.example.name
  storage_account_access_key = azurerm_storage_account.example.primary_access_key

  app_settings = {
    "CONFIG_SETTING" = "example"
  }
}

Our provisioning pipeline should

Take the build artifact for the workflow
Run the Terraform stage to create the infrastructure
Zip Deploy the application into the newly provisioned infrastructure

The Terraform is just about provisioning the empty shell of the logic app - it doesn't contain any of the workflows themselves, which we'll provide by deploying into the newly provisioned application.

Provisioning considerations

As we want to secure our applications using VNET and service endpoints, there are a few settings and other considerations to ensure the application is in a working state and correctly associated with the network subnets.

Deploying

We've provisioned our application and now we're ready to deploy - this is the simple bit as this is no different from an App Service or Azure Functions type deployment, so we simply take the zip artifact we produced and use a zip deploy task to upload to Azure.

Demo project

A demo project that shows how this will all fit together will be available in github soon.

net6 API Prometheus Metrics

Dylan Morley — Fri, 20 May 2022 10:11:20 +0000

If you're creating a dotnet API to run in Kubernetes, chances are you'll be wanting to use Prometheus to gather metrics about your application.

A library that makes this really easy for us is prometheus-net - we can create a minimal API that exposes metrics quickly and demonstrate a few different ways of creating metric data

The source for this is in github at https://github.com/dylanmorley/dotnet-api-prometheus-metrics-reference

Metrics Configuration

The majority of the work for this is done in the Program startup class, let's look at a few of the important points

EventCounterAdapter - listens for any dotnet event counters and converts them into Prometheus format at the exporter endpoint

MeterAdapter - listens for dotnet metrics created via System.Diagnostics.Metrics Meter

app.UseHttpMetrics - we want the application to expose metrics about each one of our endpoints, such as number of requests, request durations

app.UseEndpoints -> MapMetrics - we'll expose the /metrics endpoint as part of the application

That's pretty much it, not many things to configure and our app is up and running

Controller usage

In the weather forecast controller there are a couple of examples of how metrics can now be created.

If you execute the WeatherForecast/Get endpoint, you'll be incrementing a metric created by a System.Diagnostics.Metrics Meter instance, and if you hit WeatherForecast/PredictStorm you'll be using a Histogram from the prometheus-net library.

After executing either of these methods, you can check the /metrics endpoint and will see various event counters, custom counters & http controller endpoint metrics, all converted into Prometheus format and ready for scraping.

A docker file is included in the repo, to demonstrate creating an image that could then be run alongside a Prometheus instance on a kubernetes cluster.

That's it - a super simple reference solution for getting metrics up and running

Azure Devops - Secure Reset of Terraform State

Dylan Morley — Wed, 13 Oct 2021 10:00:08 +0000

If you're using Terraform for Azure Infrastructure provisioning, you're likely using the Azure Storage Backend type for your state file.

When running a plan and apply, Terraform acquires a lock on the the state file to control concurrency (i.e. so that multiple deployments don't interfere with each other), and sometimes if a pipeline terminates abruptly you're left with a lock on the state file.

Next time you run the pipeline, you'll get something like

Error locking state: Error acquiring the state lock: 
state blob is already locked

We've network restricted our storage accounts and are using a VM Scale Set associated with a subnet for our Azure Devops build pools. The subnet is allow listed on the storage account network restrictions, and the containers holding the state file are also RBAC restricted to the SPN associated with the Devops Service Connection. This means, only our Devops pipelines can interact with our Terraform state and therefore we need an azure pipeline to perform the unlock for us - so that we can authenticate and execute under the correct security context.

This is pretty simple, the only requirement here is to break the lease on the state file blob, which can be achieved with an AzureCLI task. By using this task type, authentication is performed via the specified Service Connection, which performs token management and allows access to the RBAC restricted container.

parameters:
- name : TerraformStateFile
  displayName: Terraform State File (e.g. whatever.statefile.tfstate)
  type: string

- name: ServiceConnectionName
  default: YourServiceConnectionDefault
  displayName: The name of the service connection that gives access to the storage account for the state file
  type: string

- name: StorageAccountName
  default: YourStorageDefault
  displayName: The name of the storage account that holds the Terraform state files
  type: string

trigger: none

steps:
  - task: AzureCLI@2
    displayName: "Break lease on terraform state"
    name: BreakLease
    inputs:
      azureSubscription: ${{ parameters.ServiceConnectionName }}
      scriptType: "pscore"
      scriptLocation: "inlineScript"
      inlineScript: "az storage blob lease break --container-name 'terraform' --blob-name '${{ parameters.TerraformStateFile }}' --account-name '${{ parameters.StorageAccountName }}'"

NB: The container name is hardcoded to 'terraform' in this example, you could parameterise or set to whatever default works for you

Now, any team members can run the pipeline to reset state, without requiring any direct access to the storage account.

Azure Devops - Managed Identity for Automation Tests

Dylan Morley — Wed, 06 Oct 2021 17:19:36 +0000

You're writing some integration tests and as part of doing so you need to check on some Azure Resources - perhaps look in a database, check service bus messages - you need access to some Azure infrastructure to assert that the expected things have happened. When running from Azure Devops, you can take advantage of a form of Managed Identity and avoid any connection strings in your test code

Azure Devops

If you're writing Devops Pipelines, you can setup Service Connections in your project which give you access to Azure subscriptions. A service connection to Azure from Devops is associated with a Service Principal Name (SPN).

Once you have a connection and SPN, your YAML pipelines can use this to authenticate with Azure when running certain tasks. A sneaky way of achieving identity management is possible that's really useful for integration testing.

Testing task

For this to work, your test project needs to be a dotnet project and the tests should be able to execute by using dotnet test.

Within the test project, I'm going to assume you want to do a few things.

Have different configurations, depending on the environment you're testing & connect to different Azure resources, depending on the environment.
Inherit the Security context of the Service Connection, so no need to have any connection strings in your test pack.

We can execute dotnet test and wrap it up with an AzureCLI task, passing in the service connection and environment details, like so

By doing this, we get some goodness for free. The AzureCLI task will authenticate with the service connection and obtain a token before running your script, and will perform tidy up afterwards - we get a form of token management for us.

By passing in ASPNETCORE_ENVIRONMENT we can take decisions in code to load up local settings files and build our configuration values based on a various sources. You can then have appsettings files for the different environments, allowing you to specify different resources to interact with



private IConfigurationRoot GetConfiguration(string outputPath)
{
    // Read the ASPNETCORE_ENVIRONMENT variable we passed in 
    var envName = EnvironmentData.AspNetCoreEnvironment; 

    return new ConfigurationBuilder()
        .SetBasePath(outputPath)
        .AddJsonFile("appsettings.json", true)
        .AddJsonFile($"appsettings.{envName}.json", true)
        .AddEnvironmentVariables()
        .Build();
}

OK - so you've authenticated with Azure by using the AzureCLI task - but how can you use the token in your code?

Test Pack

This is the nice part & Microsoft have made this really easy for us with the DefaultAzureCredential class from the Azure.Identity package. As described, the class will check a number of places in order to try and obtain security context, and because we've authenticated by the AzureCLI task we'll get the token for our Service Connection. Nice!

In code, will look like so.



private async Task<AccessToken> GetAccessToken()
{
    var dac = new DefaultAzureCredential(
        new DefaultAzureCredentialOptions { ExcludeSharedTokenCacheCredential = true });

    return await dac.GetTokenAsync(
        new Azure.Core.TokenRequestContext(new[] { $"https://management.azure.com/.default" }));
}

NB: I found I had to ExcludeSharedTokenCacheCredential for consistent results.

Once you're at this point, you'll find many of the Azure SDKs will have entry points that allow you to use the class directly. You can pass an instance of DefaultAzureCredential in to class - see here for some examples.

It's now just a case of making sure the SPN you're using has permissions to the things you want to integration test with. You might need to perform some role assignments to set these permissions up.

Wrapping it up

By setting up an SPN and service connection, we can execute our tests using the AzureCLI task, which will authenticate with azure and make a token available that we can then pick up from a c# test pack by using DefaultAzureCredential.

This is really powerful - as it means that when running the tests locally, you can either have the tests run under your security context as your identity, or you can provide environment variables and acquire the context of an SPN. No change of code, and your test pack is completely portable and will happily run in Windows or Linux environments.

This means we don't have any connection strings in our test solutions & we're using Access tokens and RBAC in Azure, which keeps configuration management simple and takes advantage of a consistent Azure security architecture.

This is simple to get going and I found it really made our test interactions with Azure easy to manage.

AZ CLI - Deleting Terraform Test Resource Groups

Dylan Morley — Sat, 02 Oct 2021 10:41:29 +0000

This is a series of quick posts, tips and tricks when working with the Azure CLI.

Deleting Multiple Resource Groups

When working with the Terraform Provider for Azure, you may be adding a new feature that requires you to run the test automation packs.

In cases where there are errors in the Provider as you're developing the feature, you might end up with errors like so

=== CONT  TestAccEventGridEventSubscription_deliveryMappings
testcase.go:88: Step 1/2 error: Error running pre-apply refresh: exit status 1

        Error: Argument or block definition required

          on terraform_plugin_test.tf line 47, in resource "azurerm_eventgrid_event_subscription" "test":
          47:     [

        An argument or block definition is required here.

    testing_new.go:63: Error retrieving state, there may be dangling resources: exit status 1

--- FAIL: TestAccEventGridEventSubscription_deliveryMappings (10.18s)
FAIL

Note part of the error description - there may be dangling resources:

Because the test didn't complete, the destroy phase of the test might not have completed, which means you're in a state where there are resources in Azure that have been created by the test code that need tidying up

Luckily, we can do this with a bit of Powershell + AZ CLI.

$resource_groups=(az group list --query "[? contains(name,'acctest')][].{name:name}" -o tsv)

foreach ($group in $resource_groups) {
    Write-Output "Deleting resource group $group"

    az group delete -n $group --yes 
}

Assumes you're logged in and in the subscription you want to work with

Terraform tests create resource groups following an acctest naming convention, so we find all of those that match, then delete them one by one.

AZ CLI - Assign User Role

Dylan Morley — Wed, 08 Sep 2021 11:03:45 +0000

This is a series of quick posts, tips and tricks when working with the Azure CLI.

Assigning Roles for RBAC

To take advantage of the built in roles and fine grained RBAC support many of the resources in Azure support, you should assign Roles to Security Principals.

To do so, the principal that will be performing the assignment must have the relevant permissions. You'll have this if you are an Administrator or Owner of an Azure subscription, but if not you'll explicitly need the permissions to read & write by granting the roleAssignments, which looks like so

"permissions": [
    {
        "actions": [
            "Microsoft.Authorization/*/read",
            "Microsoft.Authorization/roleAssignments/read",
            "Microsoft.Authorization/roleAssignments/write",
            "Microsoft.Authorization/roleAssignments/delete"
        ],
        "notActions": [],
        "dataActions": [],
        "notDataActions": []
    }
]

OK, so you've authenticated as a principal that has the correct permissions and you want to assign a role to another principal - documentation for this is available at the az cli role page

However, I noticed a little quirk when trying to assign a role to a user principal, where the assignee here is the object id of a user principal from AAD.

az role assignment create 
--assignee 00000000-0000-0000-0000-000000000000 
--role "Storage Account Key Operator Service Role" 
--scope $id

ForbiddenError: Operation failed with status: 'Forbidden'. Details: 403 Client Error: Forbidden for url: https://graph.windows.net/{guid}/getObjectsByObjectIds?api-version=1.6

What's going on here? I have permissions to assign roles, this command is working for me for other principal types, but for user principals I'm receiving this error.

You can always pass the --debug flag to your az cli to see what's going on in a bit more depth, for this command we can see

msrest.http_logger : {"odata.error":{"code":"Authorization_RequestDenied","message":{"lang":"en","value":"Insufficient privileges to complete the operation."},"requestId":"{guid}","date":"2021-09-08T08:50:40"}}
msrest.exceptions : Operation failed with status: 'Forbidden'. Details: 403 Client Error: Forbidden for url: https://graph.windows.net/{guid}/getObjectsByObjectIds?api-version=1.6

What's happening is the command is trying to perform a user account lookup which means it requires additional privileges to do so, specifically 'Read directory data' permission with Azure AD Graph API.

You could grant the account the permission and that would solve, but I'd rather keep to least privilege and not do that if possible. Luckily, the command provides another way to assign the role, by passing parameters in a slightly different form

az role assignment create 
--assignee-object-id "guid" 
--assignee-principal-type "User" 
.--role "The Role Name" 
--scope "the/full/resource/id"

By using the command in this format, you won't perform a call to the graph API and therefore don't need 'Read directory data' permission - nice!

Azure Logic Apps Standard - Part 1 Solution Design

Dylan Morley — Fri, 27 Aug 2021 12:19:21 +0000

Integration - in a Microservices world, it's one of the hot topics for a business. In order to scale, we've decomposed monoliths into smaller, discrete applications that manage their own data. We're creating event streams and the data we're producing isn't in a single, central database anymore - it's in many different systems, in many different forms. We're also mixing our custom software with SaaS and other applications that give us capabilities without requiring us to build, own and operate all the software we require.

We therefore need technology that allows us to perform integration, giving us the capability to ingest and move data between boundaries, flowing data from one system to another, changing shape and enriching it. We need to turn disparate and distributed data into something we can understand, bringing information back into Data Lakes where we can analyse and produce joined up reports that can give us a single, coherent view over multiple systems.

To do this, there are some very common patterns and steps you'll need to perform - get data from a source system, transform it to a new shape, split it into chunks, some conditional statements depending on state - generate payload and send it to a target system.

Logic Applications Standard are part of the Azure Integration Services suite and allow us to perform complex integrations in Azure. Logic applications are triggered by an event, and will then perform a sequence of actions through to completion - this is considered our workflow.

As the team building the workflow, it's up to you to chain together the actions that perform the work needed - you can build a sequence of decisions and other branches that perform conditional work based on various criteria. You can then orchestrate workflows, having multiple workflows calling each other - a useful approach that allows you to decompose a large business sequence into smaller, easier to understand chunks.

Logic apps standard introduces some important changes, and the runtime is now built as an extension point on top of Azure Functions. This really opens up the hosting possibilities - previously, they were ARM definitions and Azure Only deployments. In Standard, you now have a VS Code extension for designing the flows, can run and test them locally, and can package and deploy them as you would any other piece of software you create. If you containerise, you can host and run wherever you like.

We're going to look at what might be a typical real world use-case for a business.

We want to integrate with a third-party API and perform some data retrieval - We need to import a minimum of 100,000 rows of data. This is a once daily batch operation.
We're going to read API data, transform from XML to JSON, persist some state and raise service bus events.
The third party has a requirement that it must allow-list requests from a static IP.
All PaaS resources must be VNET restricted. We're going to interact with Storage, Service Bus, Key Vault and Cosmos.
We need to do this in approximately 20 minutes, so will set ourselves a NFR of 100/rps throughput. In order to fan out, we need to consider splitting data into chunks and performing updates and notifications in smaller batches than what we initially receive.

Let's look at what these requirements might translate to as a Physical Infrastructure diagram.

Network Requirements

To produce a static IP, we need to provision a Public IP address and a NAT Gateway. We then need to route all outbound traffic from our logic application through the VNET that is associated with the NAT gateway. A good article that goes into more detail on this is at note to self - azure functions with a static outbound ip - it's the same process for Logic Apps.

Since resources are VNET restricted, any support access to the PaaS components needs to be considered - in this example we solve the problem with Azure Bastion and an Azure VM in the same VNET. Engineers would need to connect to the VM via Bastion, before accessing whatever resources they require. Within the VNET, we'll therefore create a number of subnets, for the logic workflow and for management operations. Finally, we can take advantage of Service Endpoints for each of the PaaS resources, which fits nicely for this serverless and 100% Azure implementation.

The requirement for VNET configuration forces us to take premium versions of certain Skus, the app service plan to run the logic app and service bus are prime examples of this, where VNET support is only available on the premium offering.

Building the Applications

Logic App Standard does allow for a low-code & build approach, but we're going to put some discipline behind this and build a full CI/CD Azure DevOps Pipeline. This allows us to produce a build artifact from a CI job that contains everything we need to provision, deploy and test the application, then use the artifact in a multi-stage pipeline.

To do so, we'll need to generate a zip file with all the required components needed to deploy to azure.

This gives us a fully automated means to provision the physical infrastructure, deploy the workflows and all required dependencies, and test that the workflows are in a good state and doing what we want them to.

Custom Connectors

One of the advantages of Logic Apps Standard is they allow you to build your own integration connectors and use them within your logic flows. This allows you to produce an nuget package that contains the connector code, and reference that from your applications.

Logic Apps have 'hosted connectors' that you can make use of which will run out-of-process from your logic application. Whilst these are good for a number of use cases, they do come with limitations such as number of requests allowed per minute, or maximum payload sizes. If you have a high throughput application and you're running it on your own infrastructure, it makes sense to use the 'Built In' connector approach which allows you to run completely on your own application plan without any imposed limits (other than that of the underlying infrastructure)

One of the requirements for the demo application for this article was to persist some state in Cosmos Database, and at the time of writing this there was no in-built connector for Cosmos DB. We therefore wrote our own connector, which we've made available as ASOS open source. As well as being a useful connector, it's should demonstrate how to build and test any custom connector you could want to design.

Creating The Infrastructure

We need to build the infrastructure where we'll deploy and run the logic workflows - if you're working in Azure then you'll have a number of choices for representing physical infrastructure as source code. ARM templates, Azure Bicep, Ansible and Terraform are all options, for this example we're going to use Terraform.

One issue you may encounter with Terraform is that is uses the Azure SDK for Go - it often takes some time for an Azure API to become available in the Go SDK before being implemented as a Terraform module (Indeed, Day 0 support is one of the strengths of Bicep). This means that certain bleeding edge features might not be available, and at the time of writing there is no Terraform resource for Logic Applications Standard.

To solve this problem, we'll wrap up an ARM template the creates the logic app in a Terraform module, allowing us to provision the application in a state ready to be deployed to. Mixing ARM with Terraform is an acceptable approach until a native module is made available. I find the AzureRm provider for Terraform pretty easy to understand and have made a few contributions when a feature hasn't been immediately available to me - it's open source, dive in and build!

All our infrastructure provisioning process should be entirely handled by Terraform code - we'll provision the infrastructure, then zip deploy our application into what we've created. This should be a modular, easily repeatable process that we could get good reuse out of, for any other integrations we care to build.

Test Approach

An important consideration before we begin - how are we going to test the workflows? One of the advantages of Logic Apps is they promote a declarative approach that doesn't require writing code, however - that means we're not going to be using unit testing to help design the application. We need to think about what our test boundaries are within the overall logic app. Let's look at some more traditional test boundaries in a Model -> View -> Controller application

Our requirements for the application are to retrieve data from a third party, transform data, persist into cosmos and transmit events to service bus. We could treat the entire integration as a black box and just test at the boundaries - trigger the processing to start and assert that messages appear on Service Bus (an end to end test). While these have value, they will slow down your feedback loop, make assertions more difficult and are more prone to errors. With data storage and service bus messaging in the mix, you'll need to consider how to isolate data so that concurrent test executions don't interfere with each other.

Ideally, since the workflows are run as an extension on top of Azure functions, I'd create an instance of the process during the build pipeline, mock out all the dependencies and run the whole thing in memory, black box testing in the build pipeline before a deployment takes place. I want to test my application business logic at build time, not necessarily interactions with the underlying transports.

Unfortunately, it isn't possible at the time of writing - improved test support is something that is being addressed by Microsoft as the offering matures. Until that time, we should consider :

What parts of the workflows can be unit tested? If I'm performing transformations using liquid templates, can I unit test just the transformations work OK?
If I'm using node inline code - how can I test the Javascript?
If my overall logic application is composed of multiple workflows, can I test individual workflows in isolation without requiring the whole end to end piece? Does that offer value?
Can I test in the build pipeline, or should we deploy then test? Should we do both?!

In fact, testing the workflows is worth an article by itself, which you can see in Part 3 'Testing Logic Application Workflows'

Summary

That's it for this article, which is just an introduction to how we might go about designing the solution and some of the moving parts we need to consider that will allow us to build, deploy and test the applications.

We've turned business requirements into Infrastructure, and made some technology choices to allow us to build and deploy, and we're ready to start the implementation - which we'll look at next. Check back for the follow up articles, which will be on these topics.

Part 2 - Build Pipelines and Provisioning
Part 3 - Testing Logic Application Workflows
Part 4 - Designing for scale out and throughput
Part 5 - Operating and Observability

Finally, we'll produce the demo application and make that open source, so you can deploy to your own Azure subscriptions.