Forem: Karthik Ganesan

List app registrations with credentials about to expire

Karthik Ganesan — Mon, 20 Mar 2023 04:37:37 +0000

App registrations is a mechanism in Azure AD allowing to work with an application and its permissions. It’s an object in Azure AD that represents the application, its redirect URI (where to redirect users after they have signed in), its logout URL (where to redirect users after they’ve signed out), API access and custom application roles for managing permissions to users and apps.

As a matter of fact, through an app registration, you can restrict access to an application to only a specific group of users, if needed. An example of this is a solution I built a few years ago where we had two separate apps: a customer-facing app and a management app. Each had its app registration. I’ve restricted access to only a select group of people responsible for managing the system for the management app.

Associated with an app registration is a service principal, which is the identity of that application. As you undoubtedly know, a service principal has credentials. However, you may not know that these credentials have an expiry date (end-date). If you’re not aware of that and don’t monitor and manage that, you may end up with applications and services that stop working.

So, we’ll see how we can define an Azure Automation runbook that we can run periodically to detect and get a list of those credentials that are either expired or about to expire.

Also, suppose you are looking for an out-of-the-box solution rather than building custom solutions. In that case, this article also recommends one that proactively monitors and alerts on Azure app registration credential expiry without any coding or much manual effort.

Setting up the automation runbook

Creating an Azure Automation runbook can be done through the Azure portal or a CLI. We’ll show the portal way here.

We first start by creating an Automation account. In the Azure portal, look for “Automation accounts”, then create a new instance:

Once the account is created, we need to make a runbook (Use an Automation account to do many tasks where each runbook will handle a given task).

Go to the “Runbooks” section, then click “Create a runbook” and enter the requested information

You’re then presented with a screen to enter the code for that runbook. Our code will be in PowerShell. We’ll get to the complete source code in the next section.

For now, I’ve displayed some sample code:

You can notice, in line 3, that we import the “AzureAD” PowerShell module to interact with Azure AD. We use it at line 13 to get the list of all app registrations.
You can notice that, too, between lines 6 and 9, we are authenticating to Azure AD before getting the list of app registrations (again, at line 13).

From the toolbar (above the text editor), you can save the runbook, test it, publish it (you need to do that before you can use it in production), and revert to the previous version (in case the new version doesn’t work as expected).

We need first to install it since we’re importing a module (here, “AzureAD” at line 3).

For that matter, at the Automation account level, we click on “Modules”, and we look for “AzureAD”:

Since that module isn’t installed, we need to install it from the gallery by clicking on “Add a module”. We’ll pick 5.1 as the runtime version:

The code

The PowerShell code to be added to the runbook is listed here. Replace the previous code with this one.

The code is pretty easy to understand. One thing worth mentioning is the $daysToExpire variable that you’ll have to set to an appropriate value for your scenario. It’s intended to detect the service principals whose credentials are about to expire in the x coming days.

Configuring the permissions for the runbook

At this point, if you execute the runbook, you’ll notice that it might not work. That’s because the identity under which the runbook runs doesn’t have permissions to interact with Azure AD.

An Azure Automation account has an associated identity. Find it in the “Connection” section under “Shared resources” in the Azure portal.

I’ll choose the “AzureRunAsConnection”, which is of type “Service principal”, and give it the appropriate

To find that service principal in Azure AD, I need to search for the name of the Automation account in the list of “All applications” under “App registrations”:

Since we want to list app registrations from the Azure AD, we need to assign the directory role “Directory readers” to the service principal associated with our Automation account (the one that will execute the runbook) following the least privileges principle.

So, we go to “Roles and administrators” in our Azure AD tenant select “Directory readers”:

Then, we add an assignment to our service principal:

And we’re done.

Setting up a schedule

Schedule configuration can be done in the runbook blade since each runbook can have its schedule:

The process is straightforward, and the frequency depends on your need. Once every 24 hours can be a reasonable frequency.

*Get proactive alerts on credential expiration with out of box solution
*
Building custom solutions by writing lengthy codes and using various Azure services has been the preference to get notified about the credentials that are about to expire. This is because the native-Azure monitoring tools don’t offer any features to fulfill such App registration credential expiry monitoring needs.

But, following the custom approach involves too many manual tasks, might be error-prone, and if you are from the Azure Operations team, you will have to rely heavily on developers.

So, it’s been a goal of Serverless360 to mitigate this enormous drawback by offering to monitor the expiry dates of credentials associated with App registrations, and here is what it’s currently capable of,

-Manage and monitor any number of credentials with ease.
-Get intelligent alerts before a client secret/credential expires.
-Customize the number of days to notify you before the credential expires.
-Extensively view the properties of your Azure App registrations and their credentials.

To sum it up

Azure Automation is an excellent service that you can use for various management tasks. In this article, we’ve used it to detect and list service principals that are either expired or about to expire so we can act on them and decide whether to renew these credentials.

Azure Application Insights vs Log Analytics: Which one should you choose?

Karthik Ganesan — Wed, 22 Feb 2023 06:57:41 +0000

We often speak to organizations about Azure and one of the common questions is what is the difference between App Insights and Log Analytics.

In this article we will aim to discuss those differences and overlaps.

If we think about the Azure Monitoring Platform / ecosystem, then Log Analytics and App Insights both play a role within that platform as shown below.

Log Analytics plays a role in the storage of Log Data and the analysis of Log Data.

App Insights plays a role in the analysis and insights into log data.

The official line from Microsoft is that in 2018 Log Analytics and Application Insights became a single service to provide powerful end-to-end monitoring for your applications. The 2 resource types are still however separate and have different target use cases and overlaps which we will explore more below.

The key change that Microsoft has introduced was that App Insights used to be a complete standalone service whereas now you have a workspace-enabled App Insights whereby when you configure an App Insights instance you will also have a Log Analytics workspace which it points to.

Rather than having its own separate log storage, App Insights will store the logs in Log Analytics and App Insights provides views of the log data.

It’s probably easiest to think of Log Analytics as being the raw database and tables for Log data and App Insights is a set of views aimed at providing useful views on your applications’ telemetry data.

Log Analytics Use Cases

The most common use cases people will be familiar with for Log Analytics will be a Virtual Machine which will have a logging agent on it which will send telemetry to a Log Analytics workspace.

You can also configure other services to send their telemetry logs to a Log Analytics workspace with some examples such as:

Service Bus
Frontdoor
SQL Azure

For many of the Azure resources you can configure the diagnostics settings to send telemetry logs to Log Analytics and this is like infrastructure level logging of how those resources are being used and how they are performing.

App Insights Use Cases

With App Insights, you are looking at more application-level use cases. Some of the examples would be things like configuring a Web App, a Function App, or API Management to send application-level telemetry to App Insights.

With the workspace-based App Insights the log data is still stored in a Log Analytics workspace but you are streaming your telemetry to App Insights and viewing it through App Insights.

A look under the hood

With Log Analytics the data sent to the logs is a more raw format and is typically in tables like AzureDiagnostics and AzureMetrics for infrastructure level logs discussed previously, but if your using App Insights you will find the Log Analytics workspace under the hood contains the tables shown below.

These are the tables where App Insights is storing the telemetry data in Log Analytics.

If you look at the “logs” through App Insights (remember we said think of the App Insights as a view on the tables in Log Analytics) you will see the following tables.

If you write a query of the requests table in App Insights that queries the log table then it is querying the AppRequests table in Log Analytics.

From the perspective of writing a query the below 2 queries are equivalent:

App Insights:

Log Analytics:

Notice that the key difference is that the filter on the _resourceId field gets to the app insights resource and the TimeGenerated field in Log Analytics is seen as timestamp in App Insights.

What additional value does App Insights provide on top of Log Analytics

Reiterating the point that Log Analytics is the log storage and App Insights is a layer on top aimed at providing friendly views of the Log data to suit application performance management for apps built with App Service, Function Apps or even if your using a Server based IIS app with the telemetry agent, the thing App Insights gives you are added value features which use that log data and present it in ways which help you build and operate awesome apps.

Transaction Search

One of the most handy features of App Insights is the transaction search. This allows you to search for requests and dependencies or traces that have been executed by your application.

You can then open up the item you’re looking for and it will wind through the distributed trace and show you the call stack of services that have been called. You can see below a request to by API Management which was then forwarded to a Function App.

Application Map

The application map allows me to see the dependencies between my app and resources it may use such as an HTTP endpoint or a SQL database. App Insights will look at the telemetry captured and workout these services and their usage and performance levels.

Live Metrics

Live metrics is a great way to see a live stream overview of how your app is working and the requests it is processing.

Availability

Availability allows you to create web tests to ping endpoints in your app and test that they are working and monitor their SLA.

Failures

Failures allows you to begin exploring the errors within your app to identify common problems and areas you need to work on.

Performance

Performance shows you the average response times, usage and useful aggregations of the performance data in your app. It’s a great way to explore and monitor your app usage and to troubleshoot performance issues.

Usage

Usage provides a number of features aimed at helping you explore the user experience and usage of your application.

You can begin to understand user journeys in your app and how it is used in the real world.

There are also lots of deeper dive views and dashboards you can look into.

What do App Insights and Log Analytics mean for Serverless360

Log Analytics and App Insights are awesome features on Azure and they can really help your team. The challenge with them really falls into the area of roles and responsibilities. In an organization, the Azure experts will be really comfortable with using App Insights and Log Analytics and can do deep dive analysis and learn deep insights about application performance.

The problem is that not everyone in the organization who is a stakeholder in the application or IT operations is an Azure expert and, in most cases, Azure experts are a premium and scarce resource for most organizations.

The value proposition for Serverless360 in this case is to be able to leverage App Insights and Log Analytics data but to present it in a much more consumable format for the IT operations team so they can understand the data in the context of an application they are supporting without needing the steep learning curve to use Azure.

Serverless360 is all about lowering the total cost of ownership of your solutions by moving support to the left away from your experts and allowing your support teams to be more productive.

The example below is a dashboard we created for one of our application teams which shows an overview of how the web services from their application which we expose through API Management are performing.

You can learn more about how Serverless360 democratizes support of Azure solutions around App Insights and Log Analytics via the following articles:

Conclusion

Hopefully, this article clarifies your understanding of the relationship between Log Analytics and App Insights. Log Analytics is the log storage and query resource which can be used by both infrastructure-level and application-level solutions. It is about storage and query of log data.

Application Insights is a layer on top of Log Analytics aimed at application-level telemetry and uses the log data stored in Log Analytics to provide an additional bunch of features that make App Insights an Application Performance Management tool.

Both are highly likely to be part of your architecture and provide great features to help you build better apps and operate well used infrastructure.

Using APIM as a Proxy for Serverless360 BAM

Karthik Ganesan — Tue, 07 Feb 2023 04:37:16 +0000

Many companies like all their API calls to external services routed through Azure API Management (APIM). If you are a user of Serverless360 BAM, then one of the options is to use Azure API Management as a proxy to your BAM API and then use APIM from within Logic Apps.

Serverless360 BAM

Serverless360 is an enterprise product that aims to provide simple yet powerful toolsets for all operational needs in Azure Cloud. Serverless360 offers an end-to-end distributed tracing feature called BAM. BAM helps users identify how a transaction flows in a business process and spotlight failures in those transactions. BAM can be instrumented in your business process by containing services like Logic Apps, Function Apps, APIMs, an On-prem system (capable of making an API call), etc.

Azure API Management Proxy

The standard approach for BAM would look something like the below. There are three main areas:

The BAM Service provided by Serverless360
Your database and storage where you want to store your BAM data
The components which you wish to publish BAM events

Depending on the hosting option you choose for BAM, you may choose to host the BAM Service yourself or host it on the Serverless360 SaaS platform.

For the scenario we are discussing here, as a BAM user, you might pass your calls to Serverless360 BAM through your own Azure API Management solution before they go to Serverless360. It would look like the below.

Why might you choose to do this?

There are several different reasons you might consider doing this, which include:

You want to centrally route all API calls through APIM for operational and monitoring purposes.
You might want to connect APIM to BAM, and you might use the advanced subscription and critical management features in APIM to give other teams access to BAM
You might want to have a layer of abstraction from your implementation to a 3rd party service.
You might want to simplify your configuration scenarios for logic apps, so you don’t need to use a custom connector.
There can be some performance benefits by using APIM rather than a custom connector with Logic Apps.

How do we set up this scenario?

It is easy to set up this scenario. The steps to do this are below.

Download the API Specification for Serverless360 BAM

You need to go to the action menu from the BAM home page and access the BAM connection details.

You can get the link to the Swagger for your BAM API to view it in swagger UI.

You can paste the link into the browser, and it will then show you the BAM API.

The swagger link you need for APIM is shown above with the arrow; you can click it if you want and see the Swagger.

It would help if you took this URL to APIM so you could import your API.

Create BAM API in APIM

Next, go to Azure APIM and start to create a new API using the import Open API option.

You can then add the URL for the Swagger, which will import information about your BAM API. You probably want to modify the API suffix so it has an extension that matches the pattern for the standard you are using in your APIM.

Next, click Create, and APIM will import the API and set up BAM for you.

You will now see a new API, as shown below.

Set the host and key for BAM

Next, you need to set the key for calling your BAM API. The best way to do this is to get the key from the BAM portal, and then in the policy for your All Operations on the API, you can add the header to set the key, as shown below.

Most people will probably have a named value which will be either a secret or linked to a Key Vault which will be used to set the BAM API key in the policy.

Your APIM is now set up to be able to send messages to BAM.

Use BAM in your Logic App

Next, I will go to my Logic Apps and look to add an APIM action. If you select your APIM instance and the BAM API you created, you will see the list of operations imported when we set up APIM.

You can now use these operations as we do with the custom connector.

With APIM having all of the schemas for the BAM API, your Logic App will be able to create a rich design-time experience, just like with the custom connector, as shown below.

The main difference is that instead of creating an API connection behind the scenes and inheriting the additional configuration complexity that Logic Apps have in that area, you inherit the cool features for Logic Apps and APIM. You will be able to do more advanced security scenarios such as managed identity, and you can supply a subscription key and other things depending on how you have set up APIM.

You will also avoid the custom connector throttling limits, so we expect to see a performance improvement here too.

Summary

Hopefully, you will find that choosing to use APIM as a proxy is easy to set up, and it will open some more advanced setup scenarios that some BAM users may find helpful. Choosing the right distributed tracing tool plays a vital role in your business.

Business Activity Monitoring with Flat File Messages

Karthik Ganesan — Tue, 07 Feb 2023 04:30:26 +0000

Let us consider that you have implemented an integrated solution and are using Serverless360 Business Activity Monitoring (BAM) to help provide your support users and business users with visibility of what is happening in the business transaction. BAM provides you with distributed tracing to attain maximum visibility on the integration solution that the functional operations team needs. You might have a scenario where you have an unstructured message format that you want to archive in BAM so that the users can see the data which was processed.

Some examples of these might be:

Processing CSV files
Processing plain text files
Processing EDI messages

In this article, we will look at how you can implement BAM with a Logic App solution that will allow you to archive unstructured messages but use data from them within BAM.

Before continuing, if you are unfamiliar with BAM or how it can be used in Logic Apps, please check out the resources below.

https://docs.serverless360.com/docs/checkpointing-in-bam

Sending flat file data to a system

To start, we will look at the scenario where we have an interface that will run on a schedule and look up data from one of our systems and then produce a flat-file CSV message which will be written to the partner’s SFTP site.

The solution will look like the below diagram.

A Logic App implements the solution with a recurrence trigger that will call API Management to query some data from our SaaS system. It will then use the Integration Account to convert the resulting data to a CSV format and deliver the data with the SFTP connector.

We want to share the raw messages with BAM to support the solution effectively. The level 1 support operator can troubleshoot any data issues, and the user can do some self-service actions by checking transactions executed and the data passed between systems.

To implement this, we will start by adding the BAM Start transaction shape, as shown below, to the Logic App.

We will also have a CheckPoint shape after retrieving the API data. In both shapes, we are passing information to BAM to advise where we currently are in the business transaction execution. In these shapes, we can send message body data to BAM, which can be used to extract properties from the data, or we can choose to archive the message so the support user can retrieve it.

When we called the API in the Logic App, we were returned JSON data. We can use the Transform XML shape as shown below to transform the JSON to XML, representing a schema for the CSV file. You would use the Enterprise Integration Pack to create an XSD file for the schema and load it to the integration account.

I can add a BAM checkpoint shape if I want to archive the XML data in BAM.

The critical bit about sending the XML or JSON data to BAM is these structured representations of the data make it easy to promote properties to your BAM transaction so users can search for specific transactions.

Once we have the XML representation of the data, we can use the Flat File Encoding shape, as shown above. It will let us specify the schema and convert the XML to CSV format.

We can then deliver the file to the SFTP site and do a BAM Checkpoint to indicate the interface was successfully processed. The diagram below shows that the CSV data is sent as the message body to BAM.

When the support or business user uses the BAM distributed tracing module, they will not need to worry about the shapes your Logic App may need to process your complex interface. They will see those critical milestones in the process. In a successful BAM transaction, you can see below that each stage lights up green indicating success.

Users can see the JSON data you archived to BAM if they click on the Get CRM Data shape.

If the user clicks on the Complete stage in the BAM transaction, they can see the CSV message body sent to the destination system.

Receiving flat file data from a system

The opposite scenario is when we receive data from a system and then need to decode and parse it before processing it.

In this case, we also want to share the raw data with the support and business super user to help them see the data we are processing for troubleshooting purposes.

In this case, we can implement BAM in our Logic App, and we will use the message archiving feature on the Start Transaction and CheckPoint shapes and pass the data at different stages in the process.

You can see below in this case, and we are using the StartTransaction shape. I have passed the raw CSV file as the message body.

We would then use the Integration Account flat file decoding shape to reference an XSD schema in the integration account, and this would convert the CSV data to XML format.

We can then use a CheckPoint shape after the decoding if we want to send the XML message to BAM.

In the BAM distributed tracing module, as earlier in the article, you will see the key milestones in your interface showing green if they were successful.

You can click in the first stage where I archived the CSV data, and the BAM user can see the message in BAM.

Summary

In the real world, business users, level 1 support operators, and support people outside the integration team are often quite familiar with some data formats and business transactions. Using BAM distributed tracing to give them visibility into the execution and data being processed within your interface can let them participate in your interface’s day-to-day operations, leading to a lower operating cost for your solution.

With Logic Apps, we have some excellent features for decoding, parsing, and transforming message formats like CSV, EDI, and other formats. If we send the raw messages to BAM, we can let the users see what they look like, and if we send the decoded message formats, such as the XML or JSON data, then it’s easy for us to use BAM. It helps to promote metadata from the message to allow users to search for transactions that might have processed specific data.

Tracking IDocs for Integration Scenarios with Serverless360 BAM

Karthik Ganesan — Tue, 07 Feb 2023 04:24:24 +0000

Suppose you are a user of the Microsoft integration stack, and your organization also uses SAP. In that case, you will likely have use cases where an IDOC triggers integration processes in SAP being published.

One of the good things about Logic Apps on Azure is that a connector for SAP allows you to register to receive IDOCS published by SAP, and you can then use them in your integration processes.

We implemented a pattern where we had a generic Logic App registered to receive IDOCS and would then use the Azure Service Bus to implement a pub/sub pattern to allow multiple different interfaces to act upon the receipt of an IDOC into the integration platform.

Below is a representation of this architecture.

It is quite a handy pattern as it allows you to add new IDOCS and new interfaces being triggered via Service Bus in a plug-and-play style.

The Challenge

Because we have several interfaces triggered by IDOCS being published, it is challenging to track which Logic App run processed each IDOC received by the integration platform and which downstream interface processed each IDOC.

The net result of this challenge is that the interfaces are harder to support, lots of time is spent tracking down issues, and the business users and SAP team feel like they have no visibility into the processes that are important to the effective running of their applications.

The Value of BAM in this Scenario

We felt like BAM could help us a lot here. It is beneficial to track the IDOCS being published and received by the integration platform, tracking their participation in interfaces, and then making this accessible. It would help us reduce those support costs and give a better-quality service to the users who depend on our integration platform.

In BAM, we could achieve distributed tracing by creating business processes and transactions that would represent an excellent, more straightforward view of our interfaces. Then, we could create searchable properties that our integrations could use to make it easy to track and trace the information we needed.

BAM Global Properties

In BAM, we have global properties where you declare properties that will be searchable. Interfaces will use common business values, and users may wish to find information using this property. For example, a BAM user may want to search for a transaction via an Order Number.

We started by declaring the properties which are common business properties such as:

CUSTOMER-NO
ORDER-NO

These have business meaning, but in different interfaces, they may be in other messages and message formats. For example, in one interface, they may be in a JSON message, and in another, they could be in XML. Having your interfaces connected to BAM allows you to map properties to the implementation at both design and runtime to ensure you can extract the needed information.

SAP specific properties

Next up, we added some SAP-specific properties. In an IDOC, the EDI_DC40 segment is common to all the IDOCS we receive, and we felt that promoting some properties from this segment would be an excellent way to make helpful information searchable in BAM.

The IDOC properties in the EDI_DC40 segment we are interested in including the following:

DOCNUM is the IDOC number
STATUS is the status of the IDOC
IDOCTYP is the type of IDOC
MESTYP is the message type for the IDOC
CREDAT is the date the IDOC was created
CRETIM is the time the IDOC was created

We create the following list of properties in Serverless360 BAM:

Logic App-Specific Properties

We also need some Logic App properties in BAM so that when we escalate to the IT support team, they can easily see which Logic App they need to check and if they need to study more detail on the technical implementation of the interface.

To do this, we add the below two properties to the BAM Global Properties too:

LOGICAPP-RUN-ID
LOGICAPP-NAME

At this point, we now have all the properties we would want for tracking our IDOCS, and we would begin to create our transaction and Business Process.

BAM Business Process Setup

We chose a dedicated business process for our SAP-focused interfaces so they could easily be managed together. In the business process, we had a transaction called IDoc Published to the EAI Platform, as shown below.

Within the transaction, we created a simple activity diagram, as shown below, representing a simple view of the Logic App. It receives, parses, processes the IDOC, and then publishes it to Service Bus. The Logic App might be around 20 – 30 actions. Still, the BAM user will likely be either an IT support user, SAP application user, or business user. They are not so interested in the implementation detail of the Logic App they want to know about the key milestones, which are:

The IDOC was received
It was successfully published to Service Bus
There was an error

We can configure the properties on the IDOC Received from SAP action, allowing us to map the technical properties from the Logic App to the BAM properties. The below picture shows the properties we configured. The first properties we extract at the start of the transaction.

We promote some additional properties from the body of the IDOC after we have parsed it.

There are two types of properties we are using here:

Message context properties which the Logic App provides at design time
Message Body properties which we are extracting from the message the Logic App sends to BAM at runtime

In this case, the message header properties were the Logic App name and ID and the SAPActionUri from the SAP connector.

The message body properties use XPath against the message body. The Logic App sends the IDOC XML to BAM and allows to extract data from it to show in BAM.

Note:

Sending a message body is optional; if you choose to do it, you also have the option to archive the message body so that the BAM user can access it later if needed. If you do not archive it, it will be discarded when the message body is processed.

When configuring message properties from the SAP IDOC, you would use an XPath like the one below, extracting the DOCNUM, which is the IDOC number.

Receive/idocData/EDI_DC40/DOCNUM/text()

Adding BAM to the Logic App

Most people will probably use the Serverless360 connector JSON with a custom connector in Logic Apps to set up a connector for BAM. Several other ways to set up your Logic Apps to call BAM, such as via API Management.

In the Logic App, you will add an action to start your transaction and then for the checkpoints you want to send to BAM.

Below is the Start Transaction shape I have configured in my Logic App.

It provides information that maps to my BAM transaction and the first step within it. I am also passing the IDOC message as a message body to extract some properties.

Next, in my Logic App, I would do some work to process the IDOC and send a notification that it is received or the message to Service Bus. I would then add a checkpoint shape for BAM, as shown below. It is completing my transaction.

Runtime Tracking

At runtime, I can use the tracking features of BAM to query for the instances of my transactions that have received an IDOC and then publish them into the integration platform. I can create a search like the one below, which uses the friendly columns from my BAM properties to allow me to see the IDOC type, the IDOC number easily and which logic app processed it, as you can see below.

One of the most common support queries will be, “what happened with IDOC [add idoc number here].”

With this query, we can check how we received it and see if it was successful. I can also open up the transaction by clicking and seeing what happened, as shown below.

What if there is an issue?

The BAM tracking view allows your BAM users to have self-service-friendly ideas for complex interfaces. If there was an obvious problem and they needed to escalate it, then they could use the assign feature shown below, where they can link the transaction to another user in Serverless360. It might allow the BAM user to escalate to the Integration support person to take a deeper look at what is happening. It will enable them to find the transaction quickly.

There are also options to add a transaction as a favorite to you can easily find the one you are working with later if you need to take some time to investigate a problem.

Reprocessing

Suppose you wanted to allow the BAM users to reprocess a failed message without needing an Integration Expert to step in. Then it is possible to configure a reprocessing step for the transaction. You might set up a way for the BAM user can push the IDOC back to the Logic App if you need to get it processed again. There is more information about reprocessing in this article.

https://docs.serverless360.com/docs/bam-message-reprocessing

Performance Dashboards

In BAM, I can also create dashboards to give me an easy-to-consume view of how my processes run. Below is an example of one of the widgets I use where I can see how many IDOCS we receive over time.

It is interesting to see we get a few bursts of load in these scenarios.

Downstream Interfaces

Once we have received our IDOCS and published them to Service Bus for other interfaces, we can also add BAM to those interfaces, and we will be able to track and monitor them with similar features, as shown here.

One of the key benefits is mapping properties such as the IDOC number where it is used in other interfaces so we can easily trace the flow of an IDOC to other systems within our Business Process.

Using BAM from Azure Synapse Pipelines

Karthik Ganesan — Tue, 07 Feb 2023 04:11:49 +0000

Many organizations are investing heavily in the data space, and Azure Synapse is one of the technologies in the Microsoft stack which is very popular. Within Synapse, Data Flows are the component of Synapse used to orchestrate the movement of data into and out of your Data Lake and for orchestrating jobs within your Data Platform.

The business heavily depends on your data platform for movements of data within the organization and the analytics-driven from data via Synapse. The challenge, however, is the same old problem that your Data Platform is like a black box that your technology experts can only manage, so you have a bottleneck when someone wants to know if everything is working, how things are running, etc.

With Serverless360, there is an opportunity to leverage the Business Activity Monitoring feature to open up the black box and provide visibility of those critical data flows and orchestrations in an easy-to-digest form, allowing non-experts to see what is going on safely and securely. This visibility will drive confidence in your Data Platform and help it become a key and popular asset to your organization.

In this post, we wanted to demonstrate how you can use BAM from a Synapse Pipeline to help democratize your Data Platform and give users and non-Synapse experts visibility

Azure Synapse Pipeline Data Flow

In this scenario, I have an application database for my rail car management system, which records railcars, what’s in them, and where they are at any time.

I need to pull this data into Synapse to merge it with data from other systems and analyze it in Power BI.

The solution will look something like the below.

In this scenario, we have some data flow pipelines which pull data into the data lake, and then Synapse can query them, and the analytics is served in Power BI.

For this post, we will look at the area in the red box.

The data flow pipeline is the thing we want to create visibility for. We will look at how we can implement BAM in the pipeline to track the processing of this crucial business process without being a Synapse expert.

BAM Design Time

To start with, we need to implement a Business Process and Transaction within Serverless360 BAM, which will be used to represent our data flow.

When you are looking at using BAM in this context, you could have a BAM process that covers multiple data flows or single ones. BAM is the simplified logical representation of distributed tracing and does not have to match the physical implementation precisely. BAM is about creating a friendly and easily understandable view.

I assume you are already slightly familiar with building a business process and transaction for BAM. Please refer to this page for more info on Azure APIM as a proxy for Serverless360 BAM.

In the below picture, you can see I have added a business process for my Synapse demo, and I have added a transaction called “Copy Railcar Data to Synapse.”

In the design for your BAM transaction, you can add shapes to the designer to represent the high-level milestones within the complex data flow. You can see this below.

In this case, we will have a pipeline started milestone, and we will then begin the Importing Railcar Data milestone, which may run for a few mins, and then we will complete it and the transaction.

It will take a few minutes to set up, and you have your BAM transaction ready.

Implement BAM in Synapse Pipeline

Next, we need to go to Synapse and modify our pipeline to include BAM. In this case, we will start with a primary copy data pipeline generated from the Copy Data Tool. The pipeline looks like the below picture.

When it runs, it will go a for each and copy three tables into my data lake.

If I want to add BAM to my data flow pipeline, we can use the Web Activity to make an HTTP call to BAM, and we will end up with some shapes like the below.

Calling BAM API

If you want to call the BAM API from your data flow, it’s as easy as an HTTP call. You will need to supply the URL, some headers, a security key, and a body that maps to your BAM business process, transaction, and milestone stage.

You can call the BAM API directly and get the settings from your BAM configuration settings, as shown below.

The alternative, and also the way id expect many customers to implement this in the real world, would be to connect their internal Azure API Management to the BAM API. Then they can share access to the API with multiple teams within their organization but centrally manage access control and usage.

It would look like the below:

Regardless of which way you implement the pattern, from the perspective of the Data Flow, it’s just a different URL and API key that would be used. In previous articles, we discussed this APIM proxy approach Azure APIM as a proxy for Serverless360 BAM.

Pipeline Variables

If we come back to the pipeline, we will start by needing to add some variables (or parameters) for your pipeline.

I created three variables:

bamUrl will be the url for the BAM API
bamKey will be the API key for my BAM API
bamTransactionInstanceId will be to hold the guid returned by BAM when I start the transaction.

Start Transaction

In the pipeline, the first activity I added is a web activity that will call the BAM API to start a transaction. When this activity executes, it will point to the pipeline began milestone within the BAM diagram and mark it as green to show where we are in the process.

I will pass in the properties shown below:

Body

The body for the request needs to, at a minimum, be the JSON value below. It’s a JSON object with a message body and message header section.

{
"MessageBody": {},
"MessageHeader": {
}
}

You can populate the headers with a key-value pair of properties BAM can capture and promote to be searchable. You might consider passing things like the pipeline name, id, or custom values.

When the pipeline runs, this activity will start the transaction in BAM.

BAM Transaction Instance Id

The next activity in the pipeline was a Set-Variable. It will take the response from the Start Transaction web activity, and it will extract from the response the TransactionInstanceId property in the JSON response, which we will use when we call other BAM milestones to link them all together.

I created the bamTransactionInstanceId variable at the pipeline level earlier, and we will point this set variable activity to that variable and use the following expression.

@activity('BAM - Start Transaction').output['TransactionInstanceId']

Import Started CheckPoint

Next, we are about to start the long-running import process. At this point, I have decided to tell BAM that I am at a checkpoint I have called “Importing Railcar Data.” I will tell BAM that this checkpoint has started, but I expect it to take a little while, so ill come back later and mark it as complete.

To set this checkpoint, we use another web activity in your pipeline. In the case of my demo, it’s pretty simple, but in the real world, you might have some work happening in the pipeline before you get to this checkpoint. Maybe you look up some metadata or similar.

In the pipeline, I am now at this point.

This shape is similar to the previous activity for doing the call to start the transaction. The differences are:

The URL is not the same

The route is to the CheckPoint operation rather than start transaction, so I use the below expression

@concat(variables('bamUrl'),'/api/CheckPoint')

The headers are different

I do not need to supply the SL360-BusinessProcess or SL360-Transaction headers. I pass the SL360-TransactionInstanceId header and use the value from the guid we retrieved from the start transaction response.

I am also setting In Progress as the value for the SL360-StageStatus header.

The body can be different if you want

In my case, I’m just using the same body as earlier, but you can send a different payload if you want to tell BAM about some new or changed metadata.

Simulating Some work Happening

Next in the pipeline, I call the foreach, which will iterate over the configured tables for the import and copy them from my SQL database to the Data Lake.

Completing the Transaction

The final stage in the pipeline tells BAM that the transaction is completed. It marks the Importing RailcarData stage as either successful or failed. When we did the checkpoint earlier to set the Importing RailcarData stage to In Progress, it will show as orange in the BAM runtime view until it is updated to Success or Failure.

In the pipeline, I have set 2 web activities to update BAM. Using the output lines from the ForEach, one will execute in a success condition, and one will execute in a failure.

The only difference between these two shapes is that one sets the Header SL360-StageStatus to Failure and one to Success.

Both of them use the same URL as the previous step to do a checkpoint with the below expression:

@concat(variables('bamUrl'),'/api/CheckPoint')

You can see the below properties for the headers from the success activity:

Running the Data Flow

I have set up my activity to run on a recurring schedule at 8 am each day, like a typically scheduled import to the data lake. If we go to Synapse, we can see in the demo that we have the pipeline run completed.

And we can see in the pipeline history the visual shows us the pipeline execution.

I am returning to the core problem that I need to be a Synapse expert to see this view and to be able to interpret what is happening. Here, BAM comes into the solution to provide a simplified, user-friendly view of the complex implementation.

BAM Runtime Tracking

In the BAM runtime view, I can now see the tracking data each time my pipeline runs. Below you can see the runs for the last few days.

I can quickly tell they were successful, and any that failed would appear here as red on the left-hand side.

If I had sent any custom MessageHeaders from any web activities, I would have the ability to promote that data as searchable columns in the BAM view. An example might be that you promote a specific application property or similar. You can then search for transactions that match that value.

If I click on one of the transactions in the grid, I can open it up, and it will show me the process diagram, and I can see from the key which stages were successful.

If we had sent any metadata or body with the checkpoint actions, we can click on the stages in the BAM diagram and see them.

Other BAM Features

There are several other great features in BAM that you now start opening up for potential users, such as:

Action Required can help you keep on top of managing when issues might have happened and ensuring you have sorted them out
Reprocessing can let you a way to allow the non-expert potentially replay an action
Dashboards can help you get a holistic view of performance and processing across your BAM process

You can find out more about our BAM features in the section of our documentation on this link:

https://docs.serverless360.com/docs/what-is-business-activity-monitoring

Conclusion

I hope this article will show how you can use BAM to help you achieve distributed tracing and improve the visibility and manageability of your data platform processes which you are implementing with Synapse data flows.

We aim to help lower the total cost of ownership through the democratization of support for your solution and improve your users’ experience and confidence in the solution, which will encourage further investment in your platform.

Robotic Process Automation better monitored with Serverless360 BAM

Karthik Ganesan — Mon, 06 Feb 2023 18:06:41 +0000

The critical use case for BAM is to provide a simplified business-friendly view of the integrated processes that are key to your business. In business today, Robotic Process Automation (RPA) is a trending technology. While it has been around for quite a long time, it has gained a significant boost in popularity alongside the popularity of citizen developer and maker use cases. Microsoft Power Automate now includes modules in the RPA space, which has made the technology accessible to many users and scenarios through its association with the Microsoft Power Platform and Office 365.

RPA can let you try out the right ideas in automation, which you may re-make as a fully-fledged automated solution, or you may choose to keep the RPA solution if it just works for your needs.

Due to the popularity of opportunities RPA offers, customers will want to be able to fit RPA use cases into the same business activity monitoring (BAM) approaches they currently use for monitoring other critical business processes. With Serverless360, we wanted to explore how customers might use our BAM product to monitor RPA processes to ensure they can easily track progress, monitor, and troubleshoot.

Scenario

If we consider an example scenario, we will have a sales user meeting a customer. They will discuss a sales deal, and then when the sales user has concluded the deal, they will click a button on their mobile to trigger a cloud flow via the flow button.

The process for this example would be:

The user captures some data on mobile
The user clicks a button on mobile to submit data
Power Automate cloud flow runs and queries additional information from SAP
Cloud flow then runs a desktop flow to integrate with the legacy desktop application and process an order

The below diagram illustrates what this might look like.

This automated process will execute across several places, and we want a consolidated view of what is happening. We want to see what’s happening in a way that is easy to digest so that the support and business user does not need to be an expert to understand it.

By using Serverless360’s distributed tracing capability, BAM, we can send events about significant milestones. We will be able to send BAM events from the Power Automate Cloud flow and the Power Automate Desktop RPA flow, as shown in the diagram below.

BAM Design Time

I described how a real cloud and RPA combined solution might work in the scenario above. In a simplified demo to walk through how you can use BAM, we will design a business transaction that looks like the following diagram.

Here we have four steps. The “Cloud Flow Triggered” and “Cloud Flow Complete” steps will execute in the Power Automate Cloud Flow. The middle two steps will perform in our RPA Power Automate Desktop Flow.

In each stage, you will be able to configure the properties as shown in the image below, and the critical thing you want is the Tracking Name which maps to one of the fields of info you will pass from the Flow.

If you would like to take a step back for a moment and learn more about how to build the BAM transaction at design time, there is more information on this page: https://docs.serverless360.com/docs/bam-end-to-end-tracking

Implement BAM in Cloud Flow

In the cloud flow, we will manually trigger a flow trigger to allow a user to trigger the Flow from a flow button that can accept some user input, as shown below.

The cloud flow will then use the Serverless360 Start Transaction, telling BAM that we have started a transaction. The Flow will then call Power Automate Desktop to run a flow on a machine, and then it will use the BAM CheckPoint shape to indicate the Flow is complete.

The below shape shows calling the Start Transaction. We passed in some information, such as the order and triggered, so that we can see this in BAM.

The below shape shows calling the RPA process. The output from the BAM Start Transaction is a transaction instance id that we will pass down to the RPA flow, which will use this id to correlate BAM events it sends to the same transaction.

Finally, when the desktop flow is complete, we will use another CheckPoint shape to indicate the entire process is complete.

Implement BAM in Desktop Flow

In Power Automate Desktop (PAD), I can now create a flow that will perform actions when it is running on a machine simulating a user performing actions on the desktop.

In PAD, there is the option to use the Invoke Web Service action, allowing you to make an HTTP call from your desktop flow.

Design Decision

One of the design decisions I think you need to make here is how you will allow people to consume your BAM API. From a personal perspective, I would prefer to have RPA use cases call out to an API on my Azure API Management which sits in front of the BAM API. It is how most enterprise customers would likely do this because it gives you rich API Management features for monitoring an API and managing security for multiple users, which will help you support many different RPA projects.

If you decide to use API Management as a proxy to your BAM API, the only difference is that the hostname for your API and the key and header would be different. In this case, I will use APIM as a Proxy which is discussed in one of our previous articles.

View of RPA Desktop Flow

In the desktop flow, I am doing a brief demo below. I will set a variable, the API key, for calling my web service. I will invoke a web service, the BAM CheckPoint, which we also used above from the cloud flow.

I will then wait, simulating and performing some actions to automate the desktop.

I will then execute another checkpoint with the invoke web service action.

The below picture shows the desktop flow.

The first step in my Flow is the key for my API calls to BAM. I can either store this as a variable in my desktop flow or pass it in as a variable from the cloud flow.

If we look at the individual action for invoking web service, you can see that you do an HTTP post to the API Management, which then forwards the call to the BAM API. (Remember that you can call BAM directly if you don’t want to use APIM).

The key bits of information to supply are in the custom headers and body. If you note that the SL-360-TransactionInstanceId is the variable we passed in from the cloud flow.

The SL360-Stage will match the value of the tracking name property from the stage in the BAM designer.

The below picture shows the various settings for calling BAM.

There is one KEY point you need to remember here. When using this action, you need to turn off the encode request body property on the advanced settings for the Invoke Web Service. If you don’t turn this off, the JSON body will be encoded, and you will get an error. The below picture shows this setting.

In the request body, I provided the below JSON.

{
"messageBody":{},
"messageHeader":{
"OrderId":"%orderId%"
}
}

It allows me to pass a message body and an array of headers to BAM. In this case, I’m giving the order id. I could also pass others; maybe something like the machine name from the unattended agent would make sense.

Running the Scenario

When I save and deploy my BAM Desktop flow, I should now have my cloud and desktop flow available. I will see a flow button on my mobile that can run my process.

When I click this, it will open a form created by the Flow button to capture some data, and then I can click to start my Flow, as shown below.

If I want to see what happened to the execution, as a Power Automate user who knows what they are doing, I could go into Power Automate and look at the run history and see how the Flow executed, as shown below.

While this is a pretty good experience, what happens when you move this to the real world and scale it up? What if you have many users triggering processes and need to be able to let your support team, who aren’t Power Automate experts, provide that triage to support your solution to deal with everyday issues? Here is where BAM can help you.

BAM Runtime Tracking

If you are a super user or support user, then you could access BAM and see what is happening because we sent events from our Cloud Flow and Desktop Flow to BAM so we can get visibility of the transactions being processed.

Below is a view from the BAM track where some transactions are in flight.

By promoting metadata from the flows, we can create excellent views which make it easy to see what’s happening. We can also search within these views so that you can search for an instance related to a specific order id.

If you want to check what’s happening to a specific process instance, you can click on it in the above tracking view. It will open up an activity diagram, as shown below, representing the design time diagram you drew earlier in the article.

Here, each shape is shown based on the runtime data to indicate if it is executed and what the results were.

When the transactions finish executing, you can see in the below view both are finished.

Next up, you can click the transaction again and see all shapes are complete.

This tracking view should make it a lot easier to troubleshoot your Power Automate Desktop RPA processes.

Some of the other things you might choose to do when linking Power Automate Desktop or Cloud flows to BAM include:

Send the run id from the Flow to BAM, so it’s easy to work out which Flow ran which process
Send messages or responses to BAM so you can make them more accessible for the support user to see
Promote additional metadata properties so you can search on them

BAM Dashboards

BAM has a dashboard feature that you might find very useful. Once your RPA processes are up and running and you are managing them with BAM, you might be interested to see some analysis of the success and performance of your processes.

With BAM, you will get a generated dashboard for some metrics about your process. You can also add customized widgets based on specific properties or criteria that will provide a rich view of what’s happening.

BAM Action Required

In BAM, we also have a feature called Action Required. The action required aims to help you manage any issues with your processes. If a transaction in BAM is recorded as a failure, it will appear in action needed list.

You can then ignore it and provide a message if you have handled it outside of Serverless360 or configure a reprocessing step that your Serverless360 BAM user can use to reprocess a message. It might allow you to configure a way for your support team to reprocess a message if needed, marking the transaction instance as having been reprocessed.

Action Required is about helping you keep things healthy by knowing what issues haven’t been dealt with yet.

Conclusion

I hope this article shows how easy it is to add BAM to an RPA solution. As customers become more invested in the power of RPA solutions, we believe that plugging them into the modules you use simplifies the management and operations of your other integrated solutions. It will help customers to provide excellent support services to the business functions that depend on IT support for the RPA processes they develop.

Optimizing BAM Performance with API Management

Karthik Ganesan — Mon, 06 Feb 2023 17:45:46 +0000

In a previous article, I discussed how you could use API Management as a proxy for the Serverless360 BAM API. In the real world, I have used this approach a few times because I prefer it to the custom connectors that you can use with Logic Apps which are sometimes painful to use from a DevOps perspective and have known performance limitations that can affect the latency and throughput for connections Logic Apps make to Cloud Services.

Using APIM as a proxy overcomes those bottlenecks and gives me several other benefits such as key management which we have discussed previously.

What a few customers have asked about previously is if they can minimize as much as possible the latency impact in their Logic Apps when they want to use BAM. The latency involves in the BAM API is that when your Logic App (or another component) pushes a message to the BAM API we save the message to a queue on Service Bus behind the scenes so that we can then process the message out of the process and do the work of matching to a transaction, etc. The BAM API can have that latency from the call to your Logic App to the API and then persist the message to the Queue.

One of the things that some customers have done is to use either the HTTP action or APIM Action in the Logic App which takes the custom connector out of the call. This is the first performance optimization you can do if you want to tune your scenario to reduce latency.

If you want to tune things so that you can get even lower latency, then an approach I have used is to take advantage of the send-one-way-request policy in API Management. This allows me to send a normal request from my Logic App to BAM via API Management but inside the policy, I will then use the send-one-way-request to forward the message to BAM and return a response. This means that APIM will respond super quickly to your Logic App so you can continue processing.

What does this look like?

If we consider a sequence diagram showing the typical flow of a Logic App consuming the BAM API, it will look like the below.

In this case, each time the Logic App calls BAM it waits until the message is persisted for processing and then continues.

By putting APIM between the Logic App and BAM we can use a sync to async pattern and the call to the Logic App will return quickly so the Logic App can continue processing. The call to the BAM API will then be done in the background out of the process as shown below.

Design Decision

When you choose to use this sync to async process you do need to be aware that you will be making a trade-off. When you use the API directly you can be sure that the message got persisted and will appear in BAM before you continue processing the Logic App. By using the sync to async approach you accept that if there is an error between APIM and the BAM API then your Logic App will not be aware of this error.

In BAM we chose to use Service Bus for reliability and guaranteed at least one processing of messages and the advanced dead letter and error support, but if you have a scenario where you are willing to accept that in an error condition you don’t mind the risk that you may lose a BAM event then this sync to the async pattern will remove the latency of the BAM API and the Logic App connector from your Logic App processing duration.

Approach

With the approach for this pattern, we will import the API Definition for BAM into API Management as we did in the previous article however this time, we will clone the StartTransaction and CheckPoint operations and create two additional ones called StartTransaction-Async and CheckPoint-Async. This will allow us to use the out-of-the-box approach for the cases where we want to be sure BAM has the message before continuing and we can use the async ones where we are ok to risk potential message loss so we can reduce the latency.

Walk Through

At the start of the walkthrough, I am assuming you have imported your BAM API definition into API Management as demonstrated in the previous article about using APIM as a proxy to BAM.

Add Sync to Async API Operations

Next, we will add the cloned operations. You can use the Azure Portal to clone an operation by right-clicking on it as shown in the picture below.

We will then go and modify the name and path for the operation in the operation settings.

You will want to modify the name that Azure sets by default when cloning the operation and the URL path. I chose to just add a suffix of -Async on them.

You will need to repeat these steps, so you have set up both StartTransaction and CheckPoint with async versions.

Set Policy

For each operation, you will need to modify its inbound policy to do a send-one-way request and then return a response. This is shown in the samples below. Note that I set the BAM URL and API Key as named values, so they are easy to access in the policy.

Start Transaction Async

CheckPoint Async

Logic App Setup

In my Logic App you will see if I access API Management and choose my BAM API there are additional operations available so I can use the async versions of the operations.

When I use the operations the one key difference you need to be aware of is that we have added a new property on the StartTransaction so you can supply your guid when starting a transaction rather than having to wait for the one the BAM API gives you back as shown below.

In this case, I just create a string variable at the start of my Logic App using the Guid() expression and then reference it when calling BAM.

You are now ready to use your BAM actions from APIM in your Logic App and you will find that your logic app will take less time to run using the sync to the async pattern.
Comparison

If we do a basic comparison of the time the Logic App takes for the different approaches with three similar logic apps which do 2 BAM actions and return a response where the logic app looks like the below picture, then the comparison is below.

Logic App with Custom Connector

Logic App with APIM – Sync Pattern

Logic App with APIM – Sync to Async Pattern

Observations

You can see there is a slight performance benefit from the APIM sync approach in terms of latency compared to the custom connector, but it is not that big. Where APIM offers a benefit here however is that when you have a connector that is shared by a lot of Logic Apps there can sometimes be throttling challenges. This is a common Logic App challenge people have come across with connectors in general for Logic Apps at scale and there are patterns people use with multiple instances of connectors.

With the APIM connector, it does not use a cloud connector in the same way, and you do not get the Azure API Connection resource type created, the APIM action is more like a wrapper over the HTTP action which utilizes discovery from Azure APIM to make it easier to consume the API. For this reason, we should not have the throttling challenges we get with API Connectors. There is a small latency improvement in a basic test but as you scale up id expect a more noticeable benefit.

With the Sync to Async pattern, you can now see a noticeable benefit where the API calls to APIM are significantly quicker demonstrating we can make our Logic App run much faster if we need to.

Summary

Hopefully, this article shows it is easy to use APIM to help implement more advanced scenarios with the BAM API where you can tune the performance of your interfaces when you need to have lower latency requirements, but still want to use the cool operational features of our BAM solution.

Azure Documentation tool – the right way to monitor Azure usage

Karthik Ganesan — Mon, 06 Feb 2023 17:07:54 +0000

As the adoption to cloud expand, subscribing and provisioning resources increases day by day. Enterprises have multiple strategies to implement their cloud deployments: single cloud, private cloud, multi-cloud, or hybrid cloud. There is a need to analyse and assess on regular basis to keep a watch on the usage to ensure expenditure on the cloud resources are efficient and there is no wastage.

There are various possibilities but will need some level of technical competence and regular maintenance:

Use Azure portal capabilities to configure asset management and monitoring
Build custom tools using APIs exposed by Microsoft Azure support
Use 3rd party tools and services

Not all stake holders will have access and the necessity to check through these tools, but it will be easy for the decision-makers to investigate the reports, summaries, comparison assessments, and useful charts or diagrammatic representations. This will help them to assess the usage briefly and take strategic decisions.

"A well designed and articulated documentation of your Microsoft Azure environment can help in comprehending the usage easily without accessing too many tools and configurations."

Azure Documentation Generator in Serverless360 helps you to document your Microsoft Azure Subscription. It comes with the capability to aggregate data from disparate resource providers in Microsoft Azure in a single report.

This allows for the creation of comprehensive technical documentation across resources enabling richer insights that would otherwise be impossible. Interpreting your cost and resource information on Microsoft Azure subscription into legible documentation is what Azure Documenter is for.

Microsoft Azure usage is constantly evolving

Microsoft Azure like any other cloud service provider makes it easy to instantiate and deploy resources at ease on a subscription basis. Though limits on usage and restricted user permissions could be set, the resource deployment and usage grow day by day due to the nature of the business demands for various purposes.

Resources could be deployed manually or through automated processes
Resources are deployed across geographical locations
Resources and environments are managed by different teams
Configurations are set or automated to support different workloads/demands
Resources are billed across subscriptions and business units
There is a need for different environments like Development, QA, Staging, Production, etc

Isn’t it necessary to have utmost clarity on Resource usage and their deployments, type, Configuration, SKU of these resources?

Deployment of resources in Microsoft Azure is made so simple that, without proper auditing and governance, this may lead to situations where resource management becomes uncontrollable.

Challenges in maintaining the documentation on your Microsoft Azure Subscriptions

It is very difficult to keep the document up to date with the changes that happen so frequently in the environment/infrastructure. Eventually, the documentation becomes stale. Getting the complete picture of how well Microsoft Azure is being used can be complicated.

It can be quite challenging to consolidate all resource providers, their instances, locations, and cost details into comprehensive documentation.

Subsequently, the reports or documents shared with respective stakeholders and their effectiveness can become too cumbersome. It can also be hard for your team and partners to access and understand them. Many customers experience challenges in documenting the resources deployed to Azure, their configuration, and settings.

How to document Microsoft Azure usage?

It is very helpful to document the whole environment in a way that can be easily exported to a standard format like having the ability to look at what exactly is going on in your Azure environment.

The quantity and quality of the documentation should meet the stakeholders’ needs. Only this can create accurate and just enough documentation.

There are certain aspects to the documentation:

What is to be documented?
How do we document?
Relating the documentation to the deployment and usage?
Keeping the documentation up to date?
Making it transparent and accessible to the relevant stakeholders?

Save time by using our Azure documentation tool

The solution is to use a documentation generator tool like Serverless360 Azure Documenter that helps to document the Azure resources that are running, regardless of how/why they were provisioned, the Azure region and subscription in which they are located.

We have made the configuration simple to get started with generating the documentation with few steps.

There is no separate installation or configuration change required in your Azure subscriptions. Serverless360 Azure Documenter requires Service Principal identity with ‘Reader’ permission to your Azure subscription. After you have configured your account, you can easily generate documentation of your cloud environment.

Our Azure documentation tool will automatically extract information about all the resources in your Microsoft Azure subscription through standard APIs and publish them.

Depending upon the size of your Azure deployment, the time taken for generating the document may differ.

What are the benefits of using Serverless360 Azure Documenter?

Azure Documenter saves a lot of time by automating up-to-date documentation of your Microsoft Azure environments. It generates a report on your assets and reduces manual effort.

Being able to easily document your Azure infrastructure will be most valuable to understand the usage.

It will provide stakeholders at all levels, confidence, and clarity of the adoption to Microsoft Azure cloud.
It aggregates inventory of all your Resource groups. It eliminates manual activity to search and collect data about different resources and deployments.
It eliminates the need for a dedicated documentation team will save you a lot of time considering how little it costs.

Capabilities of Azure Documenter

You would be spending days creating and updating documentation but now you can spend time doing other important activities. Serverless360 Azure Documenter provides capabilities to generate:

Executive Summary

Key decision-makers may not need all the intricate details of the cloud implementation and technical aspects. All they need is a summary at a high level:

What is the usage?
Where are the wastages?
Are the compliance and security guidelines met?
What is the cost associated?
Advisory and guidelines

Billing & Metrics

Generate documentation with vital information like billing and metrics for the overall/group-wise resources in the subscription. Provides a graphical representation of cost incurred resource wise, resource type-wise, location-wise and, resource group-wise. It also provides a split-up of cost consumed at the individual resource level.

Change tracking reports

As the business grows, Azure cloud implementations could expand vastly with thousands of resources and groups cutting across multiple subscriptions and environments. Due to changes in architecture, infrastructure, business logic, upgrades, and migrations, there will be a lot of changes in the resource configuration, addition, and removal, etc on a day-to-day basis. Not all these are done manually, most of them are directed through DevOps and are automated. Also, there are number of teams and people involved.

Keeping a track of all these changes, interpreting the need, etc becomes very important. This will help to understand and check if there were any potential modifications that will impact the application and eventually the business.

Azure Security Compliance reports

Primary need for any infrastructure deployment is to ensure security and compliance of data, practices, access, and use. Adopting to Microsoft Azure is no different. From time to time, it is important to have absolute knowledge and control of data security, data losses due to storage, migration and archives, any security breaches.

These reports provide tracking of implementation of compliance rules, following organisation standards and practices, and for auditing purposes.

Azure cost analysis

Businesses across the world are embracing cloud computing than ever before. They look at migrations, new implementations, and hence the interest towards cloud implementations is only increasing day by day.

“Our commercial cloud revenue grew 36% year over year to $19.5 billion.” said Microsoft Corp. announced the results for the quarter ended June 30, 2021.

This means a lot of enterprises are spending more on the cloud. There could be an overspend if there are no periodic assessments on the expenses. Though cloud vendors promote pay-as-you-go models and various other cost control practices, in reality, the expenditure pattern is different. There are platform costs, most of which are fixed based on the configuration, some cost spiral up exponentially based on usage unless controlled, there are hidden costs as well.

You may also want to compare and analyse cost between two different months.

It is not always easy to use tools like Azure cost calculators for your production loads or interpreting the azure billing data. It becomes imperative to understand the expenses, estimate the growth in future expenditure, determine the wastage, with proper classification and deeper insights.

Schedule documentation and notify stake holders

Generating documentation for an established and evolving Azure estate with thousands of resources may take so long due to data collection, computation, and analysis. It is not practical to manually initiate documentation at the intended time and for many subscriptions and configurations. Relevant stakeholders may have to be notified once the documentation is complete. Azure Documenter provides automation to address these requirements. Notifications are sent to widely used channels like emails, Microsoft Teams etc.

Documentation templates

Documentation needs range from quick summaries and reports that provide high-level overview to comprehensive documentation that covers in-depth details. Create documents that represent not only the Azure resources but its expenses, the exact location of resources, the resource groups, the resource types, and a lot more.

For eg: Executive team may need an expenditure summary, auditing team may be interested in security reports, IT management team may need deeper insights.

Azure Documenter provides default templates to start with easy and insightful documentation and to configure custom templates that will meet your needs.

Publishing options – pdf, online & manage documentation history

The generated documentation can be published and shared with stake holders. Azure Documenter provides two different publishing options:

Publish as PDF

The documents can be published as PDF files and can be stored in default storage provided by Serverless360 or at Azure Storage of your choice. Once the PDF file is created and stored, a unique link for the document will be provided within the portal.

Publish to online platform – Document360

Managing PDF files and maintaining the versions may become difficult after a stage when you deal with multiple environments and too many documents generated over time. There are lot of content publishing tools available online. One such, software-as-a-service is Document360, an online knowledge base authoring and publishing platform.

With Azure Documenter integration to Document360 – Generate, publish and authorize access to documents online hassle free

Archive Documents – PDF of online publishing, Azure Documenter provides document history management for you to access old documents for future reference and comparison.

What are the benefits of integrating Azure Documenter with Document360?

You could of course read any data from your Azure environment and document it to Word or PDF. Once you create the documents you might want to share it with the wider community. You may want to review the document to keep track of the environments and send it to a colleague to review. Though you have SharePoint, office365, or other online platforms to share your files, they demand manual efforts and resources to store, maintain versions, distribute, and access control.

Maintaining versions on word or PDF and sharing them with stakeholders through shared folder access becomes too difficult to manage. As teams grow, managing access control of documents to a wider audience is tedious.

Document360, an online Knowledge Base authoring, and publishing platform is used to publish the documentation generated by Azure Documenter. It provides online publishing, restricted access to the documentation, and archiving capabilities.