Forem: Leonard Soetedjo

Customising Mermaid diagram font and colors

Leonard Soetedjo — Tue, 13 Feb 2024 02:35:21 +0000

In the past, I've been using Microsoft Powerpoint to draw diagrams. However, recently I've been intrigued to try out diagram-as-code as it fits well to my mode of working: I can organise my code, docs (in markdown) and diagrams together in a single repository, and be able to update them easily when I'm on Linux. I find updating Powerpoint on Linux or on mobile is difficult even with Powerpoint web version.

I've been reading up the book Creating Software with Modern Diagramming Techniques by Ashley Peacock which covers mermaid diagramming and I've grown to love the benefit that diagram-as-code provides. However, not all diagrams can be drawn easily / clearly with mermaid. Sometimes I still need to use Powerpoint. The only issue is how do I get both mermaid diagram to be consistent with my Powerpoint diagram (especially fonts as it's not mentioned in mermaid docs).

After looking at Mermaid flowchart CSS documentation and digging into the html page containing the diagram, below are the configuration that I use inside my markdown:

<style>
.nodeLabel, .flowchartTitleText
    { font-family: 'firago', 'trebuchet ms' !important; }

.GreenShape > rect, .GreenShape > g *, .GreenShape > path, .GreenShape > circle, .GreenShape > polygon
    { fill: #009245 !important; color: #fff !important; stroke: #000 !important; stroke-width: 2px !important; }

.YellowShape > rect, .YellowShape > g *, .YellowShape > path, .YellowShape > circle, .YellowShape > polygon
    { fill: #f7931e !important; color: #000 !important; stroke: #000 !important; stroke-width: 2px !important; }

.RedShape > rect, .RedShape > g *, .RedShape > path, .RedShape > circle, .RedShape > polygon
    { fill: #cb4b4f !important; color: #fff !important; stroke: #000 !important; stroke-width: 2px !important; }

.BlueShape > rect, .BlueShape > g *, .BlueShape > path, .BlueShape > circle, .BlueShape > polygon
    { fill: #0071bc !important; color: #fff !important; stroke: #000 !important; stroke-width: 2px !important; }

.PurpleShape > rect, .PurpleShape > g *, .PurpleShape > path, .PurpleShape > circle, .PurpleShape > polygon
    { fill: #9200c2 !important; color: #fff !important; stroke: #000 !important; stroke-width: 2px !important; }

.GreyShape > rect, .GreyShape > g *, .GreyShape > path, .GreyShape > circle, .GreyShape > polygon
    { fill: #eee !important; color: #000 !important; stroke: #000 !important; stroke-width: 2px !important; }

.Boundary > rect
    { fill: #fff !important; color: #000 !important; stroke: #aaa !important; stroke-width: 2px !important; stroke-dasharray: 3 3 !important; stroke-linecap: round !important; }
</style>

With the above, I can set my mermaid diagram using the classDef directive as per mermaid docs.

So far I've tried (& specified in the css):

<rect> for node with round edges
<polygon> for rhombus
<path> for cylindrical and
<circle> for circular shapes.

Another item is the directive .GreenShape > g *. It's required to set the shape's text color because the structure of each shape is as follows:

<g class="GreenShape">
    <rect ...></rect>
    <g class="label">
        <foreignObject...>
           <div ...>
               <span...>Shape label</span>
           </div>
        </foreignObject>
    </g>
</g>

Liquid template in Logic App

Leonard Soetedjo — Fri, 06 May 2022 04:37:41 +0000

I was testing out Shopify liquid template to perform XML to XML/JSON data transformation. Liquid template is one of the templating tool mentioned in Microsoft's website to perform data transformation in Logic App (the other tool being the venerable XSLT).

I've not used XSLT before, but comparing to liquid template, the latter seems more friendly. As such, I embarked on using liquid template for my research.

Below are the 2 observations I found, and the workaround I need to do to fulfill the task on hand.

Syntax differences between liquid and dotliquid (used by Microsoft):
a. Capitalisation of first letter, i.e. Append instead of append.
b. Different naming, i.e. DividedBy instead of divided_by

The problem for syntax differences is that there's no error. The script just run successfully, but the filter functions are not executed 🙄.
Unable to differentiate single child node vs. list of child nodes. To give an example, below are 2 valid snippets that can be interpreted differently.

Snippet 1
```
<order>
    <orderItem>
        <itemName>Item 1</itemName>
    </orderItem>
</order>
```
Snippet 2
```
<order>
    <orderItem>
        <itemName>Item 1</itemName>
    </orderItem>
    <orderItem>
        <itemName>Item 2</itemName>
    </orderItem>
</order>
```
The first snippet interprets order as an object containing orderItem. On the other hand, the second snippet interprets order as an _array _of `orderItem 😅

Fortunately, Microsoft has JSONArrayFor. Using JSONArrayFor instead of For loop, we're able to force order to be treated as an array regardless of the number of orderItem insider order.

Parameterising Logic App (Standard) connections.json with bicep - Part 3

Leonard Soetedjo — Tue, 03 May 2022 13:11:50 +0000

In my previous 2 posts here and here, I briefly showed how we can create the bicep snippet to create API connection resources and subsequently use that to parameterise our Logic App's connections.json. However, I've come to realise that I've missed out one configuration: API access policy. 😨

The API access policy dictates the object (in my case it's the Logic App) that can access the API connection. In this post, I'm covering this additional configuration as well as an observation on the behaviour of bicep.

Creating the API access policy is simple. We've to create Microsoft.Web/connections/accessPolicies, and add the bind our Logic App to this new access policy.

The snippet below creates the API access policy resource given that we've retrieved the particular Logic App resource that we want to allow access to.

resource apiAccessPolicy 'Microsoft.Web/connections/accessPolicies@2016-06-01' = {
  name: '${logicApp.name}-${guid(resourceGroup().name)}'
  location: location
  parent: serviceBusConnApiConnection
  properties: {
    principal: {
      type: 'ActiveDirectory'
      identity: {
        tenantId: subscription().tenantId
        objectId: reference(logicApp.id, logicApp.apiVersion, 'Full').identity.principalId
      }
    }
  }
}

And that's it. The only peculiar behaviour that I've seen is the naming of this resource. For a service bus API access policy, we need to name it with the prefix of the service bus connection name as in the above snippet. However, that is not the case for azure blob.

For azure blob, using the naming structure above will give us an error ~~Expected resource name to not contain any "/" characters. Child resources with a parent resource reference (via the parent property or via nesting) must not contain a fully-qualified name.bicep(BCP170). ~~Turns out we can just name it without the prefix of the API connection's name 🙄, i.e. it's simply

name: '${logicApp.name}-${guid(resourceGroup().name)}'

~~I'm still trying to understand the reason for this, but well, it is what it is 😄~~

UPDATE: Turns out I missed out the parent keyword which indicates what's the connection resource for the access policy 😄. Once the parent is specified, we can name the access policy without the prefix.

Parameterising Logic App (Standard) connections.json with bicep - Part 2

Leonard Soetedjo — Sun, 01 May 2022 03:18:41 +0000

Continuing from where we left off, the next step is to update the appsettings, and set connections.json to read from appsettings.

This is where it gets interesting. After lots of searching thru' the internet and testing, there are a couple of points to highlight:

We can't replace the managedApiConnections information as a whole string, i.e. the below setting won't work

...

"managedApiConnections": {
  "azureblob_1": {
    "api": {
      "id": "@appsetting('azureBlobApiId')"
    },
    "authentication": {
      "type": "ManagedServiceIdentity"
    },
    "connection": {
      "id": "@appsetting('azureBlobConnectionId')"
    },
    "connectionRuntimeUrl": "@appsetting('azureBlobConnectionRuntimeUrl')"
  }
}
...

With the above, our connections seems to be working. Looking at the API connections' status, it reports "Connected". All is well, it seems...until we try to save a workflow. When we try to save the workflow, we're shown a puzzling error:

Workflow saved successfully. Workflow validation failed for the workflow 'xxxxxxxxxx'. The file 'connections.json' contains invalid expressions. The allowed expression functions are 'xxxxxxxxxx'.

To workaround the above issue, we can find the hint from this doc Create cross-environment parameters for workflow inputs in Azure Logic Apps. In essence, rather than the above, we should instead create the following (the below code will still cause an error when saving a workflow. see step 3 for the last observation and workaround)

...

"managedApiConnections": {
  "azureblob_1": {
    "api": {
      "id": "/subscriptions/@{appsetting('WORKFLOW_SUBSCRIPTION_ID')}/providers/Microsoft.Web/locations/@{appsetting('WORKFLOW_LOCATION')}/managedApis/azureblob"
    },
    "authentication": {
      "type": "ManagedServiceIdentity"
    },
    "connection": {
      "id": "/subscriptions/@{appsetting('WORKFLOW_SUBSCRIPTION_ID')}/resourceGroups/@{appsetting('WORKFLOW_RG_NAME')}/providers/Microsoft.Web/connections/azureblob"
    },
    "connectionRuntimeUrl": "@appsetting('azureBlobConnectionRuntimeUrl')"
  }
}
...

Note: only connectionRuntimeUrl can successfully use appsettings. apiId and connectionId can somehow be partially parameterised.

Last observation, the above Microsoft doc recommendation for interpolated format by enclosing expression using curly brackets {} doesn't seem to work. In the end, we have to remove the curly brackets so that the final connections.json will look like this:

...

"managedApiConnections": {
  "azureblob_1": {
    "api": {
      "id": "/subscriptions/@appsetting('WORKFLOW_SUBSCRIPTION_ID')/providers/Microsoft.Web/locations/@appsetting('WORKFLOW_LOCATION')/managedApis/azureblob"
    },
    "authentication": {
      "type": "ManagedServiceIdentity"
    },
    "connection": {
      "id": "/subscriptions/@appsetting('WORKFLOW_SUBSCRIPTION_ID')/resourceGroups/@appsetting('WORKFLOW_RG_NAME')/providers/Microsoft.Web/connections/azureblob"
    },
    "connectionRuntimeUrl": "@appsetting('azureBlobConnectionRuntimeUrl')"
  }
}
...

Once the above settings are done, we're now able to make use of the API connections and save workflows successfully.

Parameterising Logic App (Standard) connections.json with bicep - Part 1

Leonard Soetedjo — Sun, 01 May 2022 02:45:35 +0000

Yet another issue I found when dealing with Logic App in Azure. This time, I want to parameterise the connections.json with the values from appsettings. Specifically, it's the managedApiConnections section of connections.json as this requires a particularly new item: connectionRuntimeUrl.

...
"managedApiConnections": {
  "azureblob_1": {
    "api": {
      "id": "/subscriptions/xxxxxxxxxx/providers/Microsoft.Web/locations/xxxxxxxxxx/managedApis/azureblob"
    },
      "authentication": {
        "type": "ManagedServiceIdentity"
      },
      "connection": {
        "id": "/subscriptions/xxxxxxxxxx/resourceGroups/xxxxxxxxxx/providers/Microsoft.Web/connections/azureblob"
      },
      "connectionRuntimeUrl": "https://xxxxxxxxxx.azure-apihub.net/apim/azureblob/xxxxxxxxxx"
    }
}
...

Looking around Microsoft's documentation on Web Connections, there don't "seem" much useful info on the list of parameters that it requires to build the connections in bicep. However, if we go down further, there're quickstart templates that we can refer to for storage or service bus queue.

As we'll be building in bicep, we start with decompiling (az bicep decompile --file azuredeploy.json) the ARM template into bicep and try to analyse what's required.

resource serviceBusConnectionName_resource 'Microsoft.Web/connections@2018-07-01-preview' = {
  location: location
  name: serviceBusConnectionName
  properties: {
    api: {
      id: subscriptionResourceId('Microsoft.Web/locations/managedApis', location, 'servicebus')
    }
    displayName: 'servicebus'
    parameterValues: {
      connectionString: serviceBusConnectionString
    }
  }
}

Of interest is the item under parameterValues as it's not documented. With this, we can retrieve the service bus connection string and put it into the bicep script. Do note that the above bicep is only for service bus. We can use the other quickstart template for other resources such as azure blob which will show us a different set of required parameterValues to build the bicep script.

Once we've the above, the next step is then to update the Logic App (Standard) connections.json to retrieve the necessary information from e.g. appsettings. This is another journey into the uncharted which I will cover in the next post.

Automating Azure APIM configuration based on Logic App (standard) workflows' callback URLs

Leonard Soetedjo — Wed, 20 Apr 2022 14:33:31 +0000

Continuing from my research to retrieve Logic App (standard) workflow callback URL, my next step was to automatically configure the Azure API Management in bicep.

My plan is to retrieve list of workflows for a particular Logic App (standard), and then iteratively configure APIM to add the backend endpoints. I had thought that it's an easy case, but it's not.

The resulting json from listcallbackUrl() is as follows (sample from Microsoft website):

{
  "value": "https://prod-03.westus.logic.azure.com/workflows/d4690301f3b340de9330256bb2e83974/triggers/requestTrigger/paths/invoke?api-version=2016-06-01&se=2018-04-19T16%3A00%3A00.0000000Z&sp=%2Ftriggers%2FrequestTrigger%2Frun&sv=1.0&sig=JXbHjs3qzLPDyk-IM_VzsN-WL_mNql3v_uWbBbKgtVk",
  "method": "POST",
  "basePath": "https://prod-03.westus.logic.azure.com/workflows/d4690301f3b340de9330256bb2e83974",
  "queries": {
    "api-version": "2016-06-01",
    "se": "2018-04-19T16:00:00.0000000Z",
    "sp": "//*",
    "sv": "1.0",
    "sig": "JXbHjs3qzLPDyk-IM_VzsN-WL_mNql3v_uWbBbKgtVk"
  }
}

My requirements need me to populate APIM's set-backend-service and rewrite-uri. With the above json, a user-defined function is required. And alas, as of now it's not possible in bicep :(

After further research and testing, in the end this is what I came up with:

Call a new module from the main bicep file by passing an array of logic app

module cbUrl 'module_cburl.bicep' = [for (item, i) in apiOperations: {
  name: 'url${i}'
  params: {
    logicAppId: resourceId(logicAppName)
    workflows: item
  }
}]

Create the module that just call listCallbackUrl() and provide the output with my custom format

param logicAppId string
param workflows string

var url = listCallbackURL('${logicAppId}/hostruntime/runtime/webhooks/workflow/api/management/workflows/${workflows}/triggers/manual', '2022-03-01')
var apiVersion = url.queries['api-version']

output url object = {
  basePath: url.basePath
  queries: '?api-version=${uriComponent(apiVersion)}&amp;sv=${uriComponent(url.queries.sv)}&amp;sp=${uriComponent(url.queries.sp)}&amp;sig=${uriComponent(url.queries.sig)}'
}

Use the array of result into API Management policy, iterating it with the same apiOperations variable.

resource apiPolicies 'Microsoft.ApiManagement/service/apis/operations/policies@2021-08-01' = [for (item, i) in apiOperations: {
  name: '${apimSvcRef.name}/${apiName}/${item}/policy'
  properties: {
    value: '<policies>\r\n  <inbound>\r\n    <base />\r\n    <set-backend-service base-url="${cbUrl[i].outputs.url.basePath}" />\r\n    <rewrite-uri template="${cbUrl[i].outputs.url.queries}" />\r\n  </inbound>\r\n  <backend>\r\n    <base />\r\n  </backend>\r\n  <outbound>\r\n    <base />\r\n  </outbound>\r\n  <on-error>\r\n    <base />\r\n  </on-error>\r\n</policies>'
    format: 'xml'
  }
}]

Retrieve Logic App (Standard / Single-tenant) workflow callback URL using Azure bicep

Leonard Soetedjo — Wed, 20 Apr 2022 14:02:11 +0000

Recently I've been researching on Azure components, especially using Logic App (single-tenant) and its deployment automation using Azure bicep. One roadblock I faced was retrieving the callback URLs of the workflows inside the Logic App.

It's pretty easy to retrieve the callback URL of a workflow of Logic App (multi-tenant), since practically they're the same thing. The code below retrieve the callback URL of a workflow:

listCallbackURL('${logicAppWorkflow.id}/triggers/manual', '2017-07-01').value

Logic App (single-tenant) on the other hand, is a different beast altogether. A single Logic App (single-tenant) can house many different workflows. I've been searching around in Microsoft's documentation, in forums to no avail. Finally I got the hint from stackoverflow post here where user ibda provided the REST API sample below:

az rest --method post --uri https://management.azure.com/subscriptions/$subscription/resourceGroups/$ResourceGroupName/providers/Microsoft.Web/sites/$LogicAppName/hostruntime/runtime/webhooks/workflow/api/management/workflows/$MyWorkflow/triggers/manual/listCallbackUrl?api-version=2018-11-01

Essentially, with the above sample we can just call the function as follows:

listCallbackUrl('${logicAppStd.id}/hostruntime/runtime/webhooks/workflow/api/management/workflows/${workflowName}/triggers/manual', '2018-11-01')

Building container image in AWS CodeBuild with buildah

Leonard Soetedjo — Sat, 02 Jan 2021 09:20:40 +0000

While I was trying out AWS CodeBuild, I realised that the available build images are only Amazon Linux 2 & Ubuntu 18.04. Being used to CentOS, I initially tested out using Amazon Linux 2.

However, Amazon Linux 2 is based on RHEL7. I faced issues in building image using buildah in Amazon Linux 2. As such, I decided to take the challenge to use custom image based on centos:latest, which buildah can be easily installed and configured.

When using custom image, there are a couple of things that need to be done as these will not be available in the image:

install aws cli
install buildah

1. Installing AWS CLI v2

As per AWS CLI's guide, we'll download & install the latest version:

curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
unzip -q awscliv2.zip
./aws/install

2. Installing buildah

Installing buildah is simpler in CentOS8:

dnf install -y buildah fuse-overlayfs

However, the build process was giving an error:
failed to mount overlay for metacopy check with "nodev,metacopy=on" options: invalid argument. Googling around, I found this issue

Error: failed to mount overlay for metacopy check with "nodev,metacopy=on" options: invalid argument #8118

boistordu posted on Oct 23, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.1.1

Output of podman info --debug:

Error: failed to mount overlay for metacopy check with "nodev,metacopy=on" options: invalid argument

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

this is last version on fedora repo

Additional environment details (AWS, VirtualBox, physical, etc.): vps at provider of vps

View on GitHub

. What needs to be done is to remove the metacopy=on in /etc/containers/storage.conf. Subsequently, the build is successful :)

Below is a summary of my CodeBuild's buildspec.yml (configured for terraform):

version: 0.2

phases:
  install:
    commands:
      - dnf -y update

      - echo "Install aws cli v2"
      - dnf -y install unzip
      - curl https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip
      - unzip -q awscliv2.zip
      - ./aws/install
      - export PATH=/usr/local/bin:$PATH

      - echo "Install buildah"
      - dnf install -y podman buildah fuse-overlayfs
      - sed -i 's/^mountopt =.*/mountopt = "nodev"/g' /etc/containers/storage.conf
  pre_build:
    commands:
      - echo "Login to ECR"
      - REPO_URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com
      - aws ecr get-login-password --region ${AWS_DEFAULT_REGION} | buildah login --username AWS --password-stdin ${REPO_URI}
  build:
    commands:
      - echo "Build started on `date`"
      - cd ${CODEBUILD_SRC_DIR}
      - cat /etc/os-release
      - sh build-container.sh ${IMAGE_NAME}
  post_build:
    commands:
      - echo "Build completed on `date`"
      - TAG=`date +%Y%m%d_%H%M%S`
      - echo "Pushing image to repository ${IMAGE_NAME}"
      - buildah push ${IMAGE_NAME} ${REPO_URI}/${IMAGE_NAME}:${TAG}

With the above, I'm able to build the container image using buildah and pushing it to my container repo.