Azure Data Factory Automation — Adding Log Analytics for Monitoring

One of the challenges, that I often faced with automating deployment of Azure Data Factory has been to add monitoring to the Azure Data Factory in a scriptable way.

The monitoring is essential to broad system health. For example, to generate alerts failures or if a pipeline is running longer than expected. The preferred and documented way of generating this alert has been to push the logs to Azure Log Analytics and analyzing the logs to generate alerts or provide a broad monitoring capability integrated with other parts of the infrastructure on Azure.

The Azure Data Factory documentation does provide a method using the Azure Monitor REST API. However it is little unwieldy to use, , especially in CI/CD pipelines.

There is another method to do this is actually that is well documented but not often referred to. It is available in the documentation for Azure Monitor.

So, associating an Log Analytics as a diagnostics log and metrics sink for Azure Data Factory using Azure CLI becomes as simple as running the script below:

az monitor diagnostic-settings create \
 - name LogAnalytics02-Diagnostics \
 - resource /subscriptions/(your-subscription)/resourceGroups/(your-resource-group)/providers/Microsoft.DataFactory/factories/(data-factory-name) \
 - logs '[{"category": "PipelineRuns","enabled": true}]' \
 - metrics '[{"category": "AllMetrics","enabled": true}]' \
 - workspace /subscriptions/(your-subscription)/resourcegroups/(your-resource-group)/providers/microsoft.operationalinsights/workspaces/(your-log-analytics-workspace-name)

The pre-requisite to run this scripts are - Azure Data Factory Instance and Log Analytics workspace should already be provisioned.

Just substitute the values for subscriptions and resources with values that applies to you. The example above shows just adding the logs for PipelineRuns but you can add more values in JSON string for logs parameter.