Forem: holger

Transport Sharing | Azure SDK for Python

holger — Fri, 14 Apr 2023 12:35:21 +0000

The code below is based on the blog post 'Transport sharing in the Azure SDK for Python', which was published on Microsoft Developer Blogs.

Further details were gathered from the code samples in the azure-sdk-for-python GitHub repository.

The blog post states:

Sharing a transport refers to creating a single instance of a transport object, such as an HttpTransport, and using it across multiple clients or requests. When you create a new transport object for each client or request, it can lead to increased resource usage and reduced performance.

-- Understanding Transport Sharing

The idea of transport sharing is to re-use existing HTTPTransport objects in order to keep the amount of concurrent connections to an Azure resource at a low level. For example, when performing activities on a particular storage account, the connection to this storage account could be shared.

Below are the steps I performed to test this. I will read containers from a storage account using individual HTTPTransport objects first and then use transport sharing to see the difference between both approaches.

Import the corresponding libraries.

from azure.storage.blob import BlobServiceClient
from azure.identity import AzureCliCredential
from azure.core.pipeline.transport import RequestsTransport

Import constant variables.

This is just so that I don't use the name of the storage account in the main code. One could also read the value from an environment variable or put it into the code directly. The content of the Constants.py file simply looks like this:

STORAGE_ACCOUNT_BLOB_ENDPOINT = 'https://{my_storage_account_name}.blob.core.windows.net/'

The import is then done as usual.

import Constants as CONST

Retrieve credentials from the Azure CLI.

This is just for testing purposes and should not be used in a production environment.

credential = AzureCliCredential()

Individual transports.

Below code can be used to create two clients for the same storage accounts and then list the corresponding containers using both clients individually.

blob_client1 = BlobServiceClient(
    account_url = CONST.STORAGE_ACCOUNT_BLOB_ENDPOINT,
    credential = credential
)

blob_client2 = BlobServiceClient(
    account_url = CONST.STORAGE_ACCOUNT_BLOB_ENDPOINT,
    credential = credential
)

containers_list1 = blob_client1.list_containers()
containers_list2 = blob_client2.list_containers()

for container in containers_list1:
    print(container)

for container in containers_list2:
    print(container)

Depending on the amount of containers in the storage account, the output would be along these lines:

{'name': 'con1', 'last_modified': datetime.datetime(2022, 10, 28, 14, 45, 11, tzinfo=datetime.timezone.utc), 'etag': '"0x1DAF35011161C06"', 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None}, 'public_access': None, 'has_immutability_policy': False, 'deleted': None, 'version': None, 'has_legal_hold': False, 'metadata': None, 'encryption_scope': <azure.storage.blob._models.ContainerEncryptionScope object at 0x7f9bb10632e0>, 'immutable_storage_with_versioning_enabled': False}
{'name': 'con1', 'last_modified': datetime.datetime(2022, 10, 28, 14, 45, 11, tzinfo=datetime.timezone.utc), 'etag': '"0x1DAF35011161C06"', 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None}, 'public_access': None, 'has_immutability_policy': False, 'deleted': None, 'version': None, 'has_legal_hold': False, 'metadata': None, 'encryption_scope': <azure.storage.blob._models.ContainerEncryptionScope object at 0x7f9bb0fa9430>, 'immutable_storage_with_versioning_enabled': False}

When running netstat now, we can see that two TCP connections were established against the storage account in question. One for each blob service client. (For figuring out which IP address the storage account has, one can use nslookup, i.E. : nslookup {my_storage_account_name}.blob.core.windows.net. The IP address that is shown below is completely made up in order to replace the original one.)

$ netstat -tnov | grep 20.41.119.17
tcp        0      0 10.0.0.6:50434          20.41.119.17:443        ESTABLISHED off (0.00/0/0)
tcp        0      0 10.0.0.6:50450          20.41.119.17:443        ESTABLISHED off (0.00/0/0)

Afterwards, the connections need to be closed again.

blob_client1.close()
blob_client2.close()

Shared Transport.

Then, I tried the same using a shared HTTPTransport object. Note that the additional transport and session_owner parameters are passed into BlobServiceClient now.

shared_transport = RequestsTransport()

blob_svc_client1 = BlobServiceClient(
    account_url = CONST.STORAGE_ACCOUNT_BLOB_ENDPOINT,
    credential = credential,
    transport = shared_transport, 
    session_owner = False
)

blob_svc_client2 = BlobServiceClient(
    account_url = CONST.STORAGE_ACCOUNT_BLOB_ENDPOINT,
    credential = credential,
    transport = shared_transport, 
    session_owner = False
)

containers_list_from_shared_transport1 = blob_svc_client1.list_containers()
containers_list_from_shared_transport2 = blob_svc_client2.list_containers()

for cont in containers_list_from_shared_transport1:
    print(container)

for cont in containers_list_from_shared_transport2:
    print(container)

The console output should be similar to the one before, i.E.:

{'name': 'con1', 'last_modified': datetime.datetime(2022, 10, 28, 14, 45, 11, tzinfo=datetime.timezone.utc), 'etag': '"0x1DAF35011161C06"', 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None}, 'public_access': None, 'has_immutability_policy': False, 'deleted': None, 'version': None, 'has_legal_hold': False, 'metadata': None, 'encryption_scope': <azure.storage.blob._models.ContainerEncryptionScope object at 0x7f9bb10632e0>, 'immutable_storage_with_versioning_enabled': False}
{'name': 'con1', 'last_modified': datetime.datetime(2022, 10, 28, 14, 45, 11, tzinfo=datetime.timezone.utc), 'etag': '"0x1DAF35011161C06"', 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None}, 'public_access': None, 'has_immutability_policy': False, 'deleted': None, 'version': None, 'has_legal_hold': False, 'metadata': None, 'encryption_scope': <azure.storage.blob._models.ContainerEncryptionScope object at 0x7f9bb0fa9430>, 'immutable_storage_with_versioning_enabled': False}

However, using netstat again, we can now see that only one TCP connection was opened this time, so the connection was shared with both blob service clients.

$ netstat -tnov | grep 20.41.119.17
tcp        0      0 10.0.0.6:44812          20.41.119.17:443        ESTABLISHED off (0.00/0/0)

As with the previous example, we would finally need to close the connection (this is not required when using the with statement like in the samples in the Azure SDK repository).

shared_transport.close()

What was going on in February 2023 around Python in the Azure ecosystem?

holger — Wed, 01 Mar 2023 11:28:23 +0000

Here are some of the things that I came across in February regarding Python in the Azure ecosystem:

In a blogpost about the State of the Azure SDK 2023 [24], the Azure SDK team has announced the launch of a monthly Azure SDK Community Standup.

[The SDK Team's] goal is to create a space where developers can come together, share their experiences and knowledge, and discuss the latest developments in the Azure SDK ecosystem. [...] Join us every second Tuesday of the month from 11:00 AM to noon Pacific Time (UTC-8) to learn, collaborate, and grow with the community. Check out the recording of our premiere standup!
- The first recording is here. [25]
New Python versions have been released.
- Python 3.10.10 [22] and Python 3.11.2 [23]
Azure Functions support for Python 3.10 is GA. [21]

You can now develop functions using Python 3.10 locally and deploy them to all Azure Functions plans.
Azure SDK for Python - February 2023 Releases [20]
Developing Azure Functions using the v2 programming model for Python | Azure Friday [19]

Gavin Aguiar and Shreya Batra join Scott Hanselman to discuss the v2 programming model for Azure Functions using Python.
- Comprehensive overview over the simplified v2 programming model and how quick deployments can be done through Visual Studio Code.
Announcements of the latest beta release of the Azure Text Analytics client libraries for Python [18]. The blog post highlights Python samples for these features:
- Abstractive summarization
- Named entity recognition (NER) resolutions
- Dynamic classification
- Healthcare FHIR bundles with document type
- Automatic language and script detection
Note

If you follow the example along:
- you may need to create the cognitive services account in the right region, otherwise you might be seein errors like this when attempting to run the code: {'code': 'InvalidRequest', 'message': "Job task: 'AbstractiveSummarization-0' failed with validation error: Document abstractive summarization is not supported in the region {Your-Region}. The supported regions are North Europe, East US, UK South."}
- you may need to request access at https://aka.ms/applyforgatedsummarizationfeatures if the service is part of a gated public preview feature.
Azure Developer CLI (azd) includes new templates for Python [14].
Not really a February update but a reminder:

On 31 March 2023, [Microsoft] will be retiring support for Azure SDK libraries which do not conform to [their] current Azure SDK guidelines.

— Azure Updates[12]
- Here is the list of deprecated Python libraries [13]
February 2023 release of the Python and Jupyter extensions for Visual Studio Code are now available [11].

References

#	Title	URL	Accessed-On
11	Python in Visual Studio Code – February 2023 Release	https://devblogs.microsoft.com/python/python-in-visual-studio-code-february-2023-release/	2023-02-03
12	Update your Azure SDK libraries by 31 March 2023	https://azure.microsoft.com/en-us/updates/support-for-azure-sdk-libraries-that-do-not-conform-to-our-current-azure-sdk-guidelines-will-be-retired-as-of-31-march-2023/	2023-02-06
13	Python - Deprecated Libraries	https://azure.github.io/azure-sdk/releases/deprecated/index.html#python	2023-02-06
14	Azure Developer CLI (azd) – February 2023 Release - Python Templates	https://devblogs.microsoft.com/azure-sdk/azure-developer-cli-azd-february-2023-release/#python	2023-02-09
15	Azure Samples - Deploy a Python (Flask) web app with PostgreSQL in Azure	https://github.com/Azure-Samples/msdocs-flask-postgresql-sample-app	2023-02-09
16	Azure Samples - Deploy a Python (Django) web app with PostgreSQL in Azure	https://github.com/Azure-Samples/msdocs-django-postgresql-sample-app	2023-02-09
17	HTTP API powered by FastAPI with Azure Functions and APIM	https://github.com/pamelafox/fastapi-azure-function-apim	2023-02-09
18	Announcing the new beta release of Azure Text Analytics libraries	https://devblogs.microsoft.com/azure-sdk/announcing-the-new-beta-release-of-azure-text-analytics-libraries/	2023-02-10
19	Developing Azure Functions using the v2 programming model for Python - Azure Friday	https://www.youtube.com/watch?v=KARieaWBxuk	2023-02-11
20	Azure SDK for Python (February 2023)	https://azure.github.io/azure-sdk/releases/2023-02/python.html	2023-02-14
21	Generally available: Azure Functions support for Python 3.10	https://azure.microsoft.com/en-us/updates/generally-available-azure-functions-support-for-python-310/	2023-02-21
22	Python 3.10.10 final	https://docs.python.org/release/3.10.10/whatsnew/changelog.html#python-3-10-10-final	2023-02-23
23	Python 3.11.2 final	https://docs.python.org/release/3.11.2/whatsnew/changelog.html#python-3-11-2	2023-02-23
24	State of the Azure SDK 2023	https://devblogs.microsoft.com/azure-sdk/state-of-the-azure-sdk-2023/	2023-02-23
25	Introducing Azure SDK [February 2023] - Azure SDK Community Standup	https://www.youtube.com/live/HizG8gMR9WI?feature=share	2023-02-23

Python: List all Current and Planned Azure Regions

holger — Thu, 16 Feb 2023 15:17:29 +0000

The list of all current and planned Azure regions can be reviewed at the Azures Global Infrastructure Website which is publicly available [1]. However, it is not a flat list that one could skim through or re-use within a document but it requires interaction by selecting the corresponding regions and countries.

Fortunately, there is Pandas and Python, both of which can be used to extract the various tables from the website and put the data out as a convenient list on the console, into a file or for further processing.

Below, I will go through some lines of code with which this could be done. However, be aware that this approach has significant drawbacks: If the wording on the website or the structure of the table is changed, then the script will not work anymore and requires modification.

I'm pretty sure that these small tasks can soon be handled within a few seconds by AI Companions, such as the new Bing - but doing it manually is still fun, certainly provides an opportunity to learn and you know where the information comes from and how it was processed.

Happy reading. :-)

What are Azure Regions?

Azure operates in multiple datacenters around the world. These datacenters are grouped in to geographic regions, giving [Azure Customers] flexibility in choosing where to build [their] applications.

— Azure Documentation - What are Azure regions? [2]

How are new Regions announced?

New regions, as well as other Azure updates, are typically announced on the Azure Updates website. [3]
It can even be consumed as an RSS Feed.

Using Python and Pandas to extract the various regions and their status

Every now and then I need a list of all current and planned Azure regions. Since navigating the Azures Global Infrastructure Website is a very manual task, I thought about retrieving the data using Python. Here are the steps I took. [1]

First of all, a few libraries would be required:

requests for sending HTTP requests and retrieving data from the web [4]
pandas to filter and present the data [5]
lxml since the read_html() function of Pandas uses this library by default (does not need to be imported explicitly) [6]

The import statements would then look like this (search is actually required within the script to find certain keywords within the data):

import requests
import pandas as pd
from re import search

As next step, the data from the Azure Website could be retrieved and Pandas' readhtml() function could be used to read the various tables into dataframes.

url = 'https://azure.microsoft.com/en-us/explore/global-infrastructure/geographies/'
html = requests.get(url).content
df_list = pd.read_html(html)

When looking at the first slice of the dataframes, one could see that some further polishing might be needed and that there was a lot of information that was not necessarily of interest for a basic list.

for df in df_list[:1]:
    print(df)

Output:

                       Regions  \
0                     Location   
1                  Year opened   
2  Availability Zones presence   
3                   Compliance   
4               Data residency   
5            Disaster Recovery   
6           Products by region   
7                 Available to   

                               East Asia  Start free  \
0                                          Hong Kong   
1                                               2010   
2                             Available with 3 zones   
3  Global Compliance  CIS Benchmark, CSA STAR Att...   
4  Stored at rest in the Asia Pacific region  Lea...   
5  Cross-region options:  Azure Site Recovery  Re...   
6                        See products in this region   
7                         All customers and partners   

                          Southeast Asia  Start free  
0                                          Singapore  
1                                               2010  
2                             Available with 3 zones  
3  Global Compliance  CIS Benchmark, CSA STAR Att...  
4  Stored at rest in the Asia Pacific region  Lea...  
5  Cross-region options:  Azure Site Recovery  Re...  
6                        See products in this region  
7                         All customers and partners

From above output, it is possible to derive some intermediate conclusions though:

The dataframe containing the original table headers starts with Regions might not be of interest.
The header row of the other dataframes contains the Region, the first row contains the location.

This can be confirmed when looking at the header rows only:

for df in df_list[:1]:
    for dc in list(df):
        print(dc)

Regions
East Asia  Start free
Southeast Asia  Start free

One could check whether a row contains Regions and if so, just ignore it. Then if it contains Coming soon, one could set the state variable to planned and otherwise consider it active.

The appropriate location is contained in the first row, which is why it could be derived from df[dc][0] and then the region fields could need some clean-up by removing strings that are not of interest, like Start free or Get started.

regions_list = []
locations_list = []

for df in df_list:
    for dc in list(df):
        if search('Regions', dc):
            pass
        else:
            if search('Coming soon', dc):
                state = 'planned'
            else:
                state = 'active'

            az_location = df[dc][0]
            region = dc.removesuffix('  Start free')
            region = region.removesuffix('  Get started')
            region = region.removesuffix('  Coming soon')

Since there are duplicates in the list of dataframes (wich I assume are coming from the fact that there is a selection for nearby datacenters on the website, that is based on the currently selected region), it may be required to skip some of them, which is why a list could be useful that contains all regions that have already been processed.

Then, a dictionary can be created that holds the corresponding values and is finally appended to a list.

            if region in regions_list:
                pass
            else:
                regions_list.append(region)
                locations_list.append(
                    dict({
                        'az_display_name': region,
                        'az_short_name': region.replace(' ','').lower(),
                        'az_location': az_location,
                        'az_state': state
                    })
                )

As last step, the region details could be put into one (or separate) dataframes. Since the list of regions is relatively large, Pandas needs to be configured to display them all when the output is primarily for the console.

pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)

Once this is done, the dataframes can be put out to the console.

df = pd.DataFrame(locations_list)

df_planned_regions = df[df.values == 'planned']
df_active_regions = df[df.values == 'active']

# Print only active regions
print(df_active_regions)

# Print only planned regions
print('\n') # Newline
print(df_planned_regions)

# Print all regions, regardless of their status
print('\n') # Newline
print(df)

The list of planned regions would look something like this:

         az_display_name       az_short_name      az_location az_state
13     Indonesia Central    indonesiacentral          Jakarta  planned
17         Malaysia West        malaysiawest     Kuala Lumpur  planned
18     New Zealand North     newzealandnorth         Auckland  planned
19  Saudi Arabia Central  saudiarabiacentral     Saudi Arabia  planned
20          Taiwan North         taiwannorth           Taipei  planned
21          Austria East         austriaeast           Vienna  planned
23          Denmark East         denmarkeast       Copenhagen  planned
29        Greece Central       greececentral           Athens  planned
30           Italy North          italynorth            Milan  planned
32        Poland Central       polandcentral           Warsaw  planned
33         Spain Central        spaincentral           Madrid  planned
41         Chile Central        chilecentral         Santiago  planned
42        Mexico Central       mexicocentral  Querétaro State  planned
46             East US 3             eastus3          Georgia  planned
60   US Sec West Central    ussecwestcentral      Undisclosed  planned
62        Israel Central       israelcentral           Israel  planned

Through the to_markdown() function, we could even create it in markdown format right away.

    print(df_planned_regions.to_markdown())

|    | az_display_name      | az_short_name      | az_location     | az_state   |
|---:|:---------------------|:-------------------|:----------------|:-----------|
| 13 | Indonesia Central    | indonesiacentral   | Jakarta         | planned    |
| 17 | Malaysia West        | malaysiawest       | Kuala Lumpur    | planned    |
| 18 | New Zealand North    | newzealandnorth    | Auckland        | planned    |
| 19 | Saudi Arabia Central | saudiarabiacentral | Saudi Arabia    | planned    |
| 20 | Taiwan North         | taiwannorth        | Taipei          | planned    |
| 21 | Austria East         | austriaeast        | Vienna          | planned    |
| 23 | Denmark East         | denmarkeast        | Copenhagen      | planned    |
| 29 | Greece Central       | greececentral      | Athens          | planned    |
| 30 | Italy North          | italynorth         | Milan           | planned    |
| 32 | Poland Central       | polandcentral      | Warsaw          | planned    |
| 33 | Spain Central        | spaincentral       | Madrid          | planned    |
| 41 | Chile Central        | chilecentral       | Santiago        | planned    |
| 42 | Mexico Central       | mexicocentral      | Querétaro State | planned    |
| 46 | East US 3            | eastus3            | Georgia         | planned    |
| 60 | US Sec West Central  | ussecwestcentral   | Undisclosed     | planned    |
| 62 | Israel Central       | israelcentral      | Israel          | planned    |

The table renders nicely in markdown.

	az_display_name	az_short_name	az_location	az_state
13	Indonesia Central	indonesiacentral	Jakarta	planned
17	Malaysia West	malaysiawest	Kuala Lumpur	planned
18	New Zealand North	newzealandnorth	Auckland	planned
19	Saudi Arabia Central	saudiarabiacentral	Saudi Arabia	planned
20	Taiwan North	taiwannorth	Taipei	planned
21	Austria East	austriaeast	Vienna	planned
23	Denmark East	denmarkeast	Copenhagen	planned
29	Greece Central	greececentral	Athens	planned
30	Italy North	italynorth	Milan	planned
32	Poland Central	polandcentral	Warsaw	planned
33	Spain Central	spaincentral	Madrid	planned
41	Chile Central	chilecentral	Santiago	planned
42	Mexico Central	mexicocentral	Querétaro State	planned
46	East US 3	eastus3	Georgia	planned
60	US Sec West Central	ussecwestcentral	Undisclosed	planned
62	Israel Central	israelcentral	Israel	planned

Please find the cohesive code below. It may not be very elegant but shows how tables from websites could be extracted using Python and Pandas.

import requests
import pandas as pd
from re import search

def list_azure_regions():
    url = 'https://azure.microsoft.com/en-us/explore/global-infrastructure/geographies/'
    html = requests.get(url).content
    df_list = pd.read_html(html)

    regions_list = []
    locations_list = []

    for df in df_list:
        for dc in list(df):
            if search('Regions', dc):
                pass
            else:
                if search('Coming soon', dc):
                    state = 'planned'
                else:
                    state = 'active'

                az_location = df[dc][0]
                region = dc.removesuffix('  Start free')
                region = region.removesuffix('  Get started')
                region = region.removesuffix('  Coming soon')

                if region in regions_list:
                    pass
                else:
                    regions_list.append(region)
                    locations_list.append(
                        dict({
                            'az_display_name': region,
                            'az_short_name': region.replace(' ','').lower(),
                            'az_location': az_location,
                            'az_state': state
                        })
                    )

    return locations_list

if __name__ == '__main__':
    azure_regions = list_azure_regions()

    pd.set_option('display.max_rows', None)
    pd.set_option('display.max_columns', None)

    df = pd.DataFrame(azure_regions)

    df_planned_regions = df[df.values == 'planned']
    df_active_regions = df[df.values == 'active']

    # Print only active regions
    print(df_active_regions)

    # Print only planned regions
    print('\n') # Newline
    print(df_planned_regions)

    # Print all regions, regardless of their status
    print('\n') # Newline
    print(df)

References

#	Title	URL	Accessed-On
1	Azure geographies	https://azure.microsoft.com/en-us/explore/global-infrastructure/geographies/#overview	2023-02-16
2	What are Azure regions?	https://learn.microsoft.com/en-us/azure/virtual-machines/regions#what-are-azure-regions	2023-02-16
3	Azure Updates	https://azure.microsoft.com/en-us/updates/	2023-02-16
4	requests 2.28.2	https://pypi.org/project/requests/	2023-02-16
5	pandas 1.5.3	https://pypi.org/project/pandas/	2023-02-16
6	lxml 4.9.2	https://pypi.org/project/lxml/	2023-02-16

How to enable 'Soft Delete' for Blobs in Azure Storage Accounts using Terraform

holger — Thu, 26 Jan 2023 14:21:17 +0000

What is Blob Soft Delete?

Blob soft delete protects an individual blob, snapshot, or version from accidental deletes or overwrites by maintaining the deleted data in the system for a specified period of time. During the retention period, you can restore a soft-deleted object to its state at the time it was deleted. After the retention period has expired, the object is permanently deleted.

— Azure Documentation - Soft delete for blobs [1]

Besides enabling Blob soft delete, Microsoft recommends enabling further data protection features:

Container soft delete, to restore a container that has been deleted [2].
Blob versioning, to automatically maintain previous versions of a blob [3].

How can I configure Blob Soft Delete through Terraform?

The Azure Portal has the settings under Data Protection -> Enable soft delete for blobs.

However, the Azure Storage Account REST API has the same configuration as DeleteRetentionPolicy [4] and terraform uses a similar wording with delete_retention_policy with their azurerm_storage_account resource [5].

The setting can be configured as part of the blob_properties block of the azurerm_storage_account resource. For example, if we wanted to enable soft delete with a retention time of 5 days, the setting could be:



  blob_properties {
    delete_retention_policy {
      days = 5
    }
  }

The full configuration with a resource group and storage account could be like this:



terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "random_id" "deployment_id" {
  byte_length = 8
}

resource "azurerm_resource_group" "rg" {
  location = "westeurope"
  name     = "rg-${lower(random_id.deployment_id.hex)}"
  tags = {
    environment = "test"
  }
}

resource "azurerm_storage_account" "storage_acct" {
  name = "stg${lower(random_id.deployment_id.hex)}"
  resource_group_name = azurerm_resource_group.rg.name
  location = azurerm_resource_group.rg.location

  account_kind             = "StorageV2"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  access_tier              = "Hot"

  min_tls_version           = "TLS1_2"
  enable_https_traffic_only = true

  blob_properties {
    last_access_time_enabled = true
    delete_retention_policy {
      days = 5
    }
  }

  tags = {
    environment = "test"
  }
}

output "storage_account_blob_uri" {
  value = azurerm_storage_account.storage_acct.primary_blob_endpoint
  description = "Primary Blob Endpoint"
}

Above configuration would create a resource group and storage account with a random number as suffix. Furthermore, the primary blob endpoint is added as output to the console.

Once applied, the setting should be reflected in the Azure Portal as well.

References

#	Title	URL	Accessed-On
1	Soft delete for blobs	https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-blob-overview	2023-01-26
2	Enable and manage soft delete for containers	https://learn.microsoft.com/en-us/azure/storage/blobs/soft-delete-container-enable?tabs=azure-portal	2023-01-26
3	Enable and manage blob versioning	https://learn.microsoft.com/en-us/azure/storage/blobs/versioning-enable?tabs=portal	2023-01-26
4	Azure REST API - Set Blob Service Properties	https://learn.microsoft.com/en-us/rest/api/storageservices/set-blob-service-properties?tabs=azure-ad	2023-01-26
5	azurerm_storage_account	https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account	2023-01-26

Azure SDK for Python: How to create a Python Azure Web App and Deploy Code from a GitHub Repository

holger — Tue, 24 Jan 2023 16:06:33 +0000

Here is a quick read on how I created a Python 3.11 Web App and deployed code directly from a GitHub repository.

In order to be able to deploy an App Service Plan and the corresponding Web App, I installed and imported two libraries:

azure-identity [1]
azure-mgmt-web [2]

Installation through PIP

pip install azure-identity
pip install azure-mgmt-web

Import Libraries

Since I was using the AzureCLI for authentication anyways, I was importing AzureCliCredential from the azure.identity library. This is just my personal preference for testing. Other classes from that library would work as well. From azure.mgmt.web, we'd need the WebSiteManagementClient class as well as multiple models for creating the App Service Plan and Web App configuration.

from azure.identity import AzureCliCredential
from azure.mgmt.web import WebSiteManagementClient
from azure.mgmt.web.models import AppServicePlan, SkuDescription, Site, SiteConfig, SiteSourceControl

The following two libraries are not required for functionality but they help during testing. I used uuid to generate a unique (well, not 100%ish) name and json to prettify the output a little bit:

import uuid
import json

Create the App Service Plan

First of all I created a random, unique number and provided appropriate subscription and resource group details.

# Create a random number
unique_deployment_number = str(uuid.uuid4().time_low)

# Provide Subscription / Resource Group details
my_subscription_id = '{subscription-id}'
my_resource_group_name = '{resource-group-name}'

Secondly, I needed to authenticate to Azure and to create the appropriate client:

# Authenticate to Azure and create client
my_credential = AzureCliCredential()
webapp_client = WebSiteManagementClient(credential=my_credential, subscription_id=my_subscription_id)

Then, I created a Linux App Service Plan. This required some specific settings, such as:

the App Service Plan name, for which I'm just using the term 'plan' and add aforementioned unique number to it.
the Azure location, where the Plan should be created (I'm using westeurope for testing, as this region usually has all the SKUs available)
the Azure Resource Group into which the Plan should be deployed
the Plan type, which must be linux since I wanted to create a Python Web App
the Plan must be reserved as this is a requirement for linux App Service Plans
the SKU details, which determine the costs of the App Service Plan (I was using the free tier in order to avoid high costs during testing)

# Create the App Service Plan
create_plan = webapp_client.app_service_plans.begin_create_or_update(
    name = 'plan-'+unique_deployment_number,
    resource_group_name = my_resource_group_name,
    app_service_plan = AppServicePlan(
        location = 'westeurope',
        kind = 'linux',
        reserved = True,
        sku = SkuDescription(
            name = 'F1',
            capacity = 1,
            tier = 'Free'
        )
    )
)

Lastly, I used the json library to put the results out to the console.

# Output of the results as JSON
print(json.dumps(create_plan.result().as_dict(), sort_keys=True, indent=2))

Result:

{
  "elastic_scale_enabled": false,
  "geo_region": "West Europe",
  "hyper_v": false,
  "id": "/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Web/serverfarms/plan-4074503405",
  "is_spot": false,
  "is_xenon": false,
  "kind": "linux",
  "location": "westeurope",
  "maximum_elastic_worker_count": 1,
  "maximum_number_of_workers": 0,
  "name": "plan-4074503405",
  "number_of_sites": 0,
  "number_of_workers": 1,
  "per_site_scaling": false,
  "provisioning_state": "Succeeded",
  "reserved": true,
  "resource_group": "{resource-group-name}",
  "sku": {
    "capacity": 1,
    "family": "U",
    "name": "U13",
    "size": "U13",
    "tier": "LinuxFree"
  },
  "status": "Ready",
  "subscription": "{subscription-id}",
  "target_worker_count": 0,
  "target_worker_size_id": 0,
  "type": "Microsoft.Web/serverfarms",
  "zone_redundant": false
}

Create the Web App

Next, I needed to create the web app on top of the newly created plan.

As first step, I was getting the App Service Plan Id.

try:
    app_service_plan = webapp_client.app_service_plans.get(
        name = 'plan-'+unique_deployment_number,
        resource_group_name = my_resource_group_name
    )
except Exception:
    print(f'Error: App Service Plan {my_app_service_plan_name} not found.')

Secondly, I checked whether the name that I wanted to use for my web app was already in use or not (similar to the plan name I just built it from the same random number):

availability_check = webapp_client.check_name_availability(
    name = 'webapp-'+unique_deployment_number,
    type = 'Microsoft.Web/sites'
)

And then finally, I was able to deploy the Web App. Note that both site configurations must be in place for the Web App to be properly created as Python App:

python_version
linux_fx_version

if availability_check.name_available == True:
    create_website = webapp_client.web_apps.begin_create_or_update(
        name = 'webapp-'+unique_deployment_number,
        resource_group_name = my_resource_group_name,
        site_envelope = Site(
            location = 'westeurope',
            server_farm_id = app_service_plan.id,
            site_config = SiteConfig(
                python_version = '3.11',
                linux_fx_version = 'PYTHON|3.11'
            )
        )
    )
    # Output of the results as JSON
    print(json.dumps(create_website.result().as_dict(), sort_keys=True, indent=2))
else:
    print(f'Error: Name {my_webapp_name} is already in use.')

Since the name was not in use yet, the output confirmed the successful creation of the web app:

{
  "availability_state": "Normal",
  "client_affinity_enabled": true,
  "client_cert_enabled": false,
  "client_cert_mode": "Required",
  "container_size": 0,
  "custom_domain_verification_id": "{redacted}",
  "daily_memory_time_quota": 0,
  "default_host_name": "webapp-4074503405.azurewebsites.net",
  "enabled": true,
  "enabled_host_names": [
    "webapp-4074503405.azurewebsites.net",
    "webapp-4074503405.scm.azurewebsites.net"
  ],
  {...readacted...}
  "state": "Running",
  "storage_account_required": false,
  "type": "Microsoft.Web/sites",
  "usage_state": "Normal",
  "vnet_content_share_enabled": false,
  "vnet_image_pull_enabled": false,
  "vnet_route_all_enabled": false
}

A check through the Web App in the Azure Portal confirmed the successful deployment.

Deploy Code from GitHub

As last step, I wanted to deploy sample code from GitHub into the newly created Web App. Since this post is more about the process of creating the infrastructure and deploying the app and not about developing the app itself, I have simply forked this hello-world sample app [3] from the Azure Samples GitHub repository [4].

Since I did not want to use continuous integration but rather go for a one-off deployment, I used is_manual_integration = True for the source control settings.

source_control = webapp_client.web_apps.begin_create_or_update_source_control(
    resource_group_name = my_resource_group_name,
    name = 'webapp-'+unique_deployment_number,
    site_source_control = SiteSourceControl(
        is_manual_integration = True,
        repo_url = 'https://github.com/{repository-url}',
        branch = 'master'
    )
)

In a proper script, I could now have checked the status through the source_control.status() variable to become 'Succeeded'. Once the deployment was done, the portal would reflect the External Repository Project:

Finally, I could confirm the deployment by following the link to the Web App URL and see the corresponding Hello, World! message in the browser.

Conclusions

I was able to use Python and the Azure SDK for Python for a step-by-step deployment of an App Service Plan, a Web App and a sample Python Flask application from a GitHub repository. I had some issues with creating the App Service Plan and the Web App with the correct settings for Python, which resulted in the following learnings:

When creating an App Service Plan of kind linux, the parameter reserved = True must be set, otherwise the App Service Plan will show as Windows in the Azure Portal.

When creating the Web App, both python_version and linux_fx_version must be set - otherwise the stack will not be configured for Python and App deployments would fail for me:

        site_config = SiteConfig(
            python_version = '3.11',
            linux_fx_version = 'PYTHON|3.11'
        )

The deployment from a GitHub repository was very straightforward.

References

#	Title	URL	Accessed-On
1	azure-identity 1.12.0	https://pypi.org/project/azure-identity/	2023-01-24
2	azure-mgmt-web 7.0.0	https://pypi.org/project/azure-mgmt-web/	2023-01-24
3	Python Flask sample for Azure App Service (Linux)	https://github.com/Azure-Samples/python-docs-hello-world	2023-01-24
4	Azure Samples	https://github.com/Azure-Samples	2023-01-24
5	Azure Web Apps libraries for Python	https://learn.microsoft.com/en-us/python/api/overview/azure/app-service?view=azure-python	2023-01-24

Azure SDK for Python: Retrieve Virtual Machine Image Details

holger — Fri, 13 Jan 2023 12:16:45 +0000

The Azure SDK for Python includes capabilities that allow users to retrieve Virtual Machine image details, such as the publisher names, offers, SKUs and the image details itself. [4] I was reading a thread on Stack Overflow and got interested in this functionality - and here is the corresponding write-up.

The corresponding library is the Azure Management Compute library which includes the ComputeManagementClient class which in turn has a function called virtual_machine_images.list. [1][2] However, this function requires further parameters:

location
publisher_name
offer
skus

Except for location, we may not have all the required details at hand to retrieve the image details. Further relevant functions include:

list_publishers - which has location as a required parameter
list_offers - which requires location and publisher_name
list_skus - which requires location, publisher_name and offer parameters

Now that looks like a few for-loops would be helpful until we could finally list all VM images. But would we even want this? Maybe. Given that there are thousands of images available in each region, maybe rather not as it would take a significant amount of time. So, maybe build our own function to be flexible?

Let's try and go through this step by step and take a look at each of those functions.

First, we would need the appropriate Azure SDK for Python libraries for authentication and compute management:

azure-identity [3]
azure-mgmt-compute [1]

from azure.identity import AzureCliCredential
from azure.mgmt.compute import ComputeManagementClient

We'd need to define our Credentials, Subscription ID and Azure Location.

my_credential = AzureCliCredential()
my_subscription_id = "{subscription-id}"
my_location = "{azure-location}"

Now we could create the client and collect the list of Publishers.

compute_client = ComputeManagementClient(credential=my_credential, subscription_id=my_subscription_id)
img_publishers = compute_client.virtual_machine_images.list_publishers(location=my_location)

At this point, if we were to display the content of the img_publishers variable, it would show loads of objects:

[...<azure.mgmt.compute.v2022_08_01.models._models_py3.VirtualMachineImageResource object at 0x7fb8d85c0790>, <azure.mgmt.compute.v2022_08_01.models._models_py3.VirtualMachineImageResource object at 0x7fb8d85c07c0>, <azure.mgmt.compute.v2022_08_01.models._models_py3.VirtualMachineImageResource object at 0x7fb8d85c07f0>, <azure.mgmt.compute.v2022_08_01.models._models_py3.VirtualMachineImageResource object at 0x7fb8d85c0820>, <azure.mgmt.compute.v2022_08_01.models._models_py3.VirtualMachineImageResource object at 0x7fb8d85c0850>, <azure.mgmt.compute.v2022_08_01.models._models_py3.VirtualMachineImageResource object at 0x7fb8d85c0880>...]

Just to get an idea of how many publishers there are (I used northeurope as Azure location):

>>> len(img_publishers)
1871

Let's just pick one randomly, to see what details are contained within these objects:

>>> print(img_publishers[1200])
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/MicrosoftSQLServer', 'name': 'MicrosoftSQLServer', 'location': 'northeurope', 'tags': None, 'extended_location': None}

The item of interest seems to be the name. We can now use that to go further and query the corresponding offers that belong to this publisher. Let's stick to above example in order to keep it simple for the moment.

offers = compute_client.virtual_machine_images.list_offers(location=my_location, publisher_name=img_publishers[1200].name)

Let's see how many offers are there from MicrosoftSQLServer:

>>> len(offers)
37

We'll again pick one of them to see what properties might be important - and it's, again, name.

>>> print(offers[36])
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/MicrosoftSQLServer/ArtifactTypes/VMImage/Offers/sql2022-ws2022', 'name': 'sql2022-ws2022', 'location': 'northeurope', 'tags': None, 'extended_location': None}

Next stop: SKUs

skus = compute_client.virtual_machine_images.list_skus(location=my_location,publisher_name=img_publishers[1200].name, offer=offers[36].name)

Telling by the length of the list, there are only four SKUs for this particular offer:

>>> len(skus)
4

And picking one of them reveals that, again, the name property is what we need.

>>> print(skus[0])
{'additional_properties': {'properties': {'automaticOSUpgradeProperties': {'automaticOSUpgradeSupported': False}}}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/MicrosoftSQLServer/ArtifactTypes/VMImage/Offers/sql2022-ws2022/Skus/enterprise-gen2', 'name': 'enterprise-gen2', 'location': 'northeurope', 'tags': None, 'extended_location': None}

Since we now have everything together we can finally call the list() function.

images = compute_client.virtual_machine_images.list(location=my_location, publisher_name=img_publishers[1200].name, offer=offers[36].name, skus=skus[0].name)

We now have identified 2 images:

>>> len(images)
2

And from that we can get an idea about what the outcome will be:

>>> print(images[0])
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/MicrosoftSQLServer/ArtifactTypes/VMImage/Offers/sql2022-ws2022/Skus/enterprise-gen2/Versions/16.0.221025', 'name': '16.0.221025', 'location': 'northeurope', 'tags': None, 'extended_location': None}

Picking items from the individual lists is a) not much fun and b) does not scale very well. As next step we might want to wrap some for-loops around these.
So, let's take a step back and assume we were back at a point where we imported our libraries and defined my_credential, my_subscription_id and my_location variables. We would also have created the ComputeManagementClient and dumped the list of publishers into the img_publishers variable.

from azure.identity import AzureCliCredential
from azure.mgmt.compute import ComputeManagementClient

my_credential = AzureCliCredential()
my_subscription_id = "{subscription-id}"
my_location = "{azure-location}"

compute_client = ComputeManagementClient(credential=my_credential, subscription_id=my_subscription_id)
img_publishers = compute_client.virtual_machine_images.list_publishers(location=my_location)

From here on we could iterate through the Publishers, Offers, SKUs and finally Images.

for publisher in img_publishers:
    offers = compute_client.virtual_machine_images.list_offers(location=my_location, publisher_name=publisher.name)
    for offer in offers:
        skus = compute_client.virtual_machine_images.list_skus(location=my_location,publisher_name=publisher.name, offer=offer.name)
        for sku in skus:
            images = compute_client.virtual_machine_images.list(location=my_location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
            for image in images:
                print(image)

It shouldn't take long for us to see some results on screen (cancelled the processing using CTRL-C, otherwise this would take ages to get all the images):

...
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/aod/ArtifactTypes/VMImage/Offers/win2019azpolicy/Skus/win2019azpolicy/Versions/0.0.1', 'name': '0.0.1', 'location': 'northeurope', 'tags': None, 'extended_location': None}
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/aod/ArtifactTypes/VMImage/Offers/win2019azpolicy/Skus/win2019azpolicy/Versions/0.0.3', 'name': '0.0.3', 'location': 'northeurope', 'tags': None, 'extended_location': None}
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/apigee/ArtifactTypes/VMImage/Offers/apigee-edge/Skus/apigee-edge-4-15-07/Versions/1.0.0', 'name': '1.0.0', 'location': 'northeurope', 'tags': None, 'extended_location': None}
{'additional_properties': {}, 'id': '/Subscriptions/{subscription-id}/Providers/Microsoft.Compute/Locations/northeurope/Publishers/apigee/ArtifactTypes/VMImage/Offers/apigee-edge/Skus/apigee-edge-private-cloud/Versions/4.16.05', 'name': '4.16.05', 'location': 'northeurope', 'tags': None, 'extended_location': None}
...

While this works, it might not be very helpful for further processing. We might want to add the details to a custom dictionary? If so, the loops could be put out like this:

for publisher in img_publishers:
    offers = compute_client.virtual_machine_images.list_offers(location=my_location, publisher_name=publisher.name)
    for offer in offers:
        skus = compute_client.virtual_machine_images.list_skus(location=my_location,publisher_name=publisher.name, offer=offer.name)
        for sku in skus:
            images = compute_client.virtual_machine_images.list(location=my_location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
            for image in images:
                image_dict = dict({
                    'publisherName' : publisher.name,
                    'offerName' : offer.name,
                    'skuName': sku.name,
                    'imageName': image.name
                })
                print(image_dict)

The dictionary would look something like this (again CTRL-C'd after a few seconds to stop the processing as it would take too long):

...
{'publisherName': 'aod', 'offerName': 'win2019azpolicy', 'skuName': 'win2019azpolicy', 'imageName': '0.0.1'}
{'publisherName': 'aod', 'offerName': 'win2019azpolicy', 'skuName': 'win2019azpolicy', 'imageName': '0.0.3'}
{'publisherName': 'apigee', 'offerName': 'apigee-edge', 'skuName': 'apigee-edge-4-15-07', 'imageName': '1.0.0'}
{'publisherName': 'apigee', 'offerName': 'apigee-edge', 'skuName': 'apigee-edge-private-cloud', 'imageName': '4.16.05'}
...

If we wanted to add those individual dictionary to a large one to process in the end, we could append the image details rather then printing them out on screen.

results = []

for publisher in img_publishers:
    offers = compute_client.virtual_machine_images.list_offers(location=my_location, publisher_name=publisher.name)
    for offer in offers:
        skus = compute_client.virtual_machine_images.list_skus(location=my_location,publisher_name=publisher.name, offer=offer.name)
        for sku in skus:
            images = compute_client.virtual_machine_images.list(location=my_location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
            for image in images:
                image_dict = dict({
                    'publisherName' : publisher.name,
                    'offerName' : offer.name,
                    'skuName': sku.name,
                    'imageName': image.name
                })
                results.append(image_dict)

This would still run a very long time though. So maybe we would want to add some query functionality? Let's create a function.

def get_vm_images(credential, subscription_id, location, image_publisher):
    results = []

    compute_client = ComputeManagementClient(credential=credential, subscription_id=subscription_id)
    img_publishers = compute_client.virtual_machine_images.list_publishers(location=location)

    for publisher in img_publishers:
        if image_publisher in publisher.name:
            offers = compute_client.virtual_machine_images.list_offers(location=location, publisher_name=publisher.name)
            for offer in offers:
                skus = compute_client.virtual_machine_images.list_skus(location=location,publisher_name=publisher.name, offer=offer.name)
                for sku in skus:
                    images = compute_client.virtual_machine_images.list(location=location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
                    for image in images:
                        image_dict = dict({
                            'publisherName' : publisher.name,
                            'offerName' : offer.name,
                            'skuName': sku.name,
                            'imageName': image.name
                        })
                        results.append(image_dict)
        else:
            pass
    return results

By checking whether the value of image_publisher (which we pass to the function) is part of the publisher.name value, we could make sure that only if there is a match, Images would be queried and added to the results dictionary: if image_publisher in publisher.name.

We could try this:

>>> get_vm_images(credential=my_credential, subscription_id=my_subscription_id, location=my_location, image_publisher="canonical")

Hm, we get nothing:

[]

So, maybe capitalization matters.

>>> get_vm_images(credential=my_credential, subscription_id=my_subscription_id, location=my_location, image_publisher="Canonical")

And indeed, this seems to work better:

[...{'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202006110'}, {'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202007020'}, {'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202007030'}, {'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202007070'}, {'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202007080'}, {'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202007090'}, {'publisherName': 'Canonical', 'offerName': 'UbuntuServer', 'skuName': '19_10-daily-gen2', 'imageName': '19.10.202007100'}]

So, something else to account for I presume... since there might be multiple variations. Maybe some publishers are typed all lowercase, others require capitalization - we could replace above if-clause (I'm sure there are more elegant ways though) by: if image_publisher.lower() in publisher.name or image_publisher.capitalize() in publisher.name:

The function code would now look like this

def get_vm_images(credential, subscription_id, location, image_publisher):
    results = []

    compute_client = ComputeManagementClient(credential=credential, subscription_id=subscription_id)
    img_publishers = compute_client.virtual_machine_images.list_publishers(location=location)

    for publisher in img_publishers:
        if image_publisher.lower() in publisher.name or image_publisher.capitalize() in publisher.name:
            offers = compute_client.virtual_machine_images.list_offers(location=location, publisher_name=publisher.name)
            for offer in offers:
                skus = compute_client.virtual_machine_images.list_skus(location=location,publisher_name=publisher.name, offer=offer.name)
                for sku in skus:
                    images = compute_client.virtual_machine_images.list(location=location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
                    for image in images:
                        image_dict = dict({
                            'publisherName' : publisher.name,
                            'offerName' : offer.name,
                            'skuName': sku.name,
                            'imageName': image.name
                        })
                        results.append(image_dict)
        else:
            pass

    return results

With this in place, we should be able to get results even when passing something like canonical instead of Canonical.

Now we have this very basic function in place. This could be used in various ways. I.E. to produce json output...

import json

canonical_images = get_vm_images(credential=my_credential, subscription_id=my_subscription_id, location=my_location, image_publisher="canonical")
print(json.dumps(canonical_images))

[ ... {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202006110"}, {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202007020"}, {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202007030"}, {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202007070"}, {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202007080"}, {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202007090"}, {"publisherName": "Canonical", "offerName": "UbuntuServer", "skuName": "19_10-daily-gen2", "imageName": "19.10.202007100"}]

...or to load into a Pandas DataFrame:

import pandas as pd

canonical_images = get_vm_images(credential=my_credential, subscription_id=my_subscription_id, location=my_location, image_publisher="canonical")
df = pd.DataFrame(canonical_images)

print(df)

>>> print(df)
     publisherName                                     offerName           skuName        imageName
0        Canonical  0001-com-ubuntu-confidential-vm-experimental             18_04   18.04.20210309
1        Canonical  0001-com-ubuntu-confidential-vm-experimental        18_04-gen2   18.04.20210309
2        Canonical  0001-com-ubuntu-confidential-vm-experimental             20_04   20.04.20210309
3        Canonical  0001-com-ubuntu-confidential-vm-experimental        20_04-gen2   20.04.20210309
4        Canonical         0001-com-ubuntu-confidential-vm-focal     20_04-lts-cvm  20.04.202111100
...            ...                                           ...               ...              ...
4546     Canonical                                  UbuntuServer  19_10-daily-gen2  19.10.202007030
4547     Canonical                                  UbuntuServer  19_10-daily-gen2  19.10.202007070
4548     Canonical                                  UbuntuServer  19_10-daily-gen2  19.10.202007080
4549     Canonical                                  UbuntuServer  19_10-daily-gen2  19.10.202007090
4550     Canonical                                  UbuntuServer  19_10-daily-gen2  19.10.202007100

[4551 rows x 4 columns]

Finally, here is the full script for testing. I've added an additional if-clause so that the function would also accept None as image publisher - however, this is not recommended really as it takes a lot of time.

from azure.identity import AzureCliCredential
from azure.mgmt.compute import ComputeManagementClient

def get_vm_images(credential, subscription_id, location, image_publisher=None):
    results = []

    compute_client = ComputeManagementClient(credential=credential, subscription_id=subscription_id)
    img_publishers = compute_client.virtual_machine_images.list_publishers(location=location)

    if image_publisher:
        for publisher in img_publishers:
            if image_publisher.lower() in publisher.name or image_publisher.capitalize() in publisher.name:
                offers = compute_client.virtual_machine_images.list_offers(location=location, publisher_name=publisher.name)
                for offer in offers:
                    skus = compute_client.virtual_machine_images.list_skus(location=location,publisher_name=publisher.name, offer=offer.name)
                    for sku in skus:
                        images = compute_client.virtual_machine_images.list(location=location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
                        for image in images:
                            image_dict = dict({
                                'publisherName' : publisher.name,
                                'offerName' : offer.name,
                                'skuName': sku.name,
                                'imageName': image.name
                            })

                            results.append(image_dict)
            else:
                pass
    else:
        for publisher in img_publishers:
            offers = compute_client.virtual_machine_images.list_offers(location=location, publisher_name=publisher.name)
            for offer in offers:
                skus = compute_client.virtual_machine_images.list_skus(location=location,publisher_name=publisher.name, offer=offer.name)
                for sku in skus:
                    images = compute_client.virtual_machine_images.list(location=location, publisher_name=publisher.name, offer=offer.name, skus=sku.name)
                    for image in images:
                        image_dict = dict({
                            'publisherName' : publisher.name,
                            'offerName' : offer.name,
                            'skuName': sku.name,
                            'imageName': image.name
                        })

                        results.append(image_dict)

    return results


my_credential = AzureCliCredential()
my_subscription_id = "{subscription-id}"

get_vm_images(credential=my_credential, subscription_id=my_subscription_id, location="eastus", image_publisher="Canonical")

I hope this might be helpful for someone who wants to test and try the Azure SDK for Python - I certainly enjoyed looking into it. Let me just add a few bullet points:

Above code samples are just for fun and learning and should not be used in production since there are a lot of important things missing such as error-handling and logging.
There are surely more elegant ways of doing the same with Python.
The Azure CLI is capable of listing all images out of the box: az vm image list --all

That's it for the moment - thanks for reading.

References

#	Title	URL	Accessed-On
1	azure-mgmt-compute 29.0.0	https://pypi.org/project/azure-mgmt-compute/	2023-01-12
2	VirtualMachineImagesOperations Class - list	https://learn.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2022_08_01.operations.virtualmachineimagesoperations?view=azure-python#azure-mgmt-compute-v2022-08-01-operations-virtualmachineimagesoperations-list	2023-01-12
3	azure-identity 1.12.0	https://pypi.org/project/azure-identity/	2023-01-13
4	Use the Azure libraries (SDK) for Python	https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-overview	2023-01-12
5	VirtualMachineImagesOperations Class - list_publishers	https://learn.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2022_08_01.operations.virtualmachineimagesoperations?view=azure-python#azure-mgmt-compute-v2022-08-01-operations-virtualmachineimagesoperations-list-publishers	2023-01-12
6	VirtualMachineImagesOperations Class - list_offers	https://learn.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2022_08_01.operations.virtualmachineimagesoperations?view=azure-python#azure-mgmt-compute-v2022-08-01-operations-virtualmachineimagesoperations-list-offers	2023-01-12
7	VirtualMachineImagesOperations Class - list_skus	https://learn.microsoft.com/en-us/python/api/azure-mgmt-compute/azure.mgmt.compute.v2022_08_01.operations.virtualmachineimagesoperations?view=azure-python#azure-mgmt-compute-v2022-08-01-operations-virtualmachineimagesoperations-list-skus	2023-01-12

Azure SDK for Python: WAM Support in Beta/Pre-Release

holger — Wed, 11 Jan 2023 14:32:18 +0000

Azure Identity 1.13.0b1 is in Pre-Release state

I saw on my GitHub Feed yesterday that the Azure SDK Releases have the azure-identity beta library 1.13.0b1 in pre-release now [2][3].
The Azure Identity beta library 1.13.0b1 should be able to call Web Account Manager (WAM), a component that ships with Windows 10+ and Windows Server 2019+ and acts as an authentication broker allowing users to benefit from integration with accounts known to Windows [1][4].

The beta library can be tested and installed through pip:

pip install azure-identity==1.13.0b1

The class in question is InteractiveBrowserCredential, which does now accept an allow_broker keyword as per the class's documentation below (full details here) [5]:

class InteractiveBrowserCredential(InteractiveCredential):
    """Opens a browser to interactively authenticate a user.

    ...snip...

    :keyword bool allow_broker: Brokers provide single sign-on, device identification, and application identification
        verification. If this parameter is set to True, the broker will be used when possible. Defaults to False.
        Check https://learn.microsoft.com/azure/active-directory/develop/scenario-desktop-acquire-token-wam
        for more WAM information.

    ...snip...

    """

When using allow_broker=True on Windows 10+ or Windows Server 2019+ ...

from azure.identity import InteractiveBrowserCredential
credential = InteractiveBrowserCredential(allow_broker=True)

...the authentication process should try to leverage WAM.
On older Windows versions or Mac/Linux, it should revert to using the browser.

I'm not sure if this functionality will be widely used with the Python SDK, however, according to this GitHub issue, it seems to be part of the Azure SDK design guideline as "the support for WAM must exist in all Tier-1 languages (C#, Java, JS, Python) for Azure Identity SDK" [6].

References

#	Title	URL	Accessed-On
1	Desktop app that calls web APIs: Acquire a token using WAM	https://learn.microsoft.com/en-us/azure/active-directory/develop/scenario-desktop-acquire-token-wam	2023-01-11
2	Azure SDK Releases	https://azure.github.io/azure-sdk/releases/latest/all/python.html	2023-01-11
3	azure-identity 1.13.0b1	https://pypi.org/project/azure-identity/1.13.0b1/	2023-01-11
4	WAM	https://github.com/AzureAD/microsoft-authentication-library-for-dotnet/wiki/wam	2023-01-11
5	class InteractiveBrowserCredential(InteractiveCredential)	https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/identity/azure-identity/azure/identity/_credentials/browser.py	2023-01-11
6	Add WAM Support to MSAL for Python, for cross-language consistency #451	https://github.com/AzureAD/microsoft-authentication-library-for-python/issues/451	2023-01-11

How to build an Azure Function for checking available names across Azure resources

holger — Wed, 04 Jan 2023 13:51:41 +0000

Introduction

In a previous post we've looked at a Python script to query available names across Azure resources [5]. The script uses the Azure Resource Graph to find resources with a given name across all subscriptions in scope of the principal that is running the script [6]. Within this post we are going to use an Azure Function to run this script using a Managed Identity that holds the corresponding Reader role against a certain scope [7].

Important

This post contains some code for testing and playing around with Python in Azure Functions.

The code presented here must not be used in production environments since it is lacking important functionality such as security settings, logging, error handling and more. It is only intended for testing and learning.

Resources should be removed right away after testing in order to avoid costs.

Also, the code is leveraging the Python v2 programming model in Azure Functions, which is currently in preview [1].

Building the Azure Infrastructure

We are using terraform and the corresponding azurerm provider to build the Azure infrastructure for the Azure Function App [2]. The terraform code is also located in a GitHub Repository.

See the Terraform documentation if you need details on how to install terraform itself on the system of your choice [9].

Resources required:

Storage Account
App Service Plan
Function App
App Insights

Terraform Code

Terraform Provider

File: provider.tf

As for the provider block, only the azurerm provider is required.

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

provider "azurerm" {
  features {}
}

Resource Group

File resource_group.tf

We will use the West Europe region in this example - however, this can be amended to whatever region suits best. Subsequently created resources will simply refer to the resource group region in order to make sure that the same region is used for all resources.

resource "azurerm_resource_group" "rg" {
  location = "westeurope"
  name     = "{resource-group-name}"
  tags = {
    owner       = "me"
    environment = "test"
  }
}

Storage Account

Files:

Azure Functions require a storage account since they rely on Azure Storage for operations such as managing triggers and logging function executions. See Storage considerations for Azure Functions if more details are required around this topic [3].

resource "azurerm_storage_account" "storage_acct" {
  name                = "{storage-account-name}"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location

  account_kind             = "StorageV2"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  access_tier              = "Hot"

  min_tls_version           = "TLS1_2"
  enable_https_traffic_only = true

  tags = {
    owner       = "me"
    environment = "test"
  }
}

Since I wanted to check the primary access key when required for troubleshooting, I've added it as an output, however, this is not required. If added, it should be marked as sensitive = true for obvious reasons.

output "sas_connection_string" {
  sensitive = true
  value     = azurerm_storage_account.storage_acct.primary_connection_string
}

If we want to put it out to the console, we can use this this terraform command to do so (after it has been deployed successfully):

terraform output sas_connection_string

Application Insights

File: app_insights.tf

For logging purposes we are also deploying App Insights, which will allow us to monitor executions in Azure Functions [4]. This will be helpful for troubleshooting.

resource "azurerm_application_insights" "func_app_insights" {
  name                = "{app-insights-name}"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  application_type    = "other"
}

Function App (and App Service Plan)

Files:

Since we are running Python Code, the App Service Plan must be of os_type = "Linux". Furthermore, in this example we're deploying it in a Consumption Plan [8]. This means it is in the Dynamic tier and we therefore need to use the sku_name = "Y1" configuration in terraform.

As we are using the aforementioned Python v2 programming model in Azure Functions, it is important that we add AzureWebJobsFeatureFlags = "EnableWorkerIndexing" to the app_settings section. Additionally, we need ENABLE_ORYX_BUILD = true and SCM_DO_BUILD_DURING_DEPLOYMENT = true - otherwise we won't be able to use the Remote build feature [10].

resource "azurerm_service_plan" "consumption_plan" {
  name                = "{service-plan-name}"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  os_type             = "Linux"
  sku_name            = "Y1"
}

resource "azurerm_linux_function_app" "naming_func" {
  name                       = "{function-app-name}"
  location                   = azurerm_resource_group.rg.location
  resource_group_name        = azurerm_resource_group.rg.name
  service_plan_id            = azurerm_service_plan.consumption_plan.id
  storage_account_name       = azurerm_storage_account.storage_acct.name
  storage_account_access_key = azurerm_storage_account.storage_acct.primary_access_key

  identity {
    type = "SystemAssigned"
  }

  app_settings = {
    ENABLE_ORYX_BUILD              = true
    SCM_DO_BUILD_DURING_DEPLOYMENT = true
    AzureWebJobsFeatureFlags       = "EnableWorkerIndexing"
    APPINSIGHTS_INSTRUMENTATIONKEY = azurerm_application_insights.func_app_insights.instrumentation_key
  }

  site_config {
    application_stack {
      python_version = "3.9"
    }
  }
}

Lastly, it is required to assign the Function Apps Managed Identity to a scope. For testing, I'm adding it as a reader role to the resource group scope, however, if we wanted to check for available names across multiple subscriptions, it would need to be added to the corresponding subscription or management group scopes.

resource "azurerm_role_assignment" "func_mi_assignment" {
  scope                = azurerm_resource_group.rg.id
  role_definition_name = "Reader"
  principal_id         = azurerm_linux_function_app.naming_func.identity[0].principal_id
}

Deployment

We can now run terraform init, terraform plan and terraform deploy. If all went well, we would have terraform confirm the successful deployment.

Using the Azure CLI, we can also confirm that the resources exist as expected.

az resource list --resource-group {resource-group-name} --output table

Creating the Function App

Modifying the Code

File: function_app.py

First of all, we need to make sure that the original Python script would be able to run within the Azure Function as there are a few things that are different now.

Managed Identity

When running the script locally, we were simply using the AzureCliCredential class from the azure.identity library, which would use the context of the user that is locally logged on through the Azure CLI. Obviously, this won't work with a managed identity. Fortunately, this part can be easily replaced through a different class from the same library: the DefaultAzureCredential Class [11].

We would therefore import the following libraries:

from azure.identity import DefaultAzureCredential
from azure.mgmt.resource import SubscriptionClient
import azure.mgmt.resourcegraph as arg
import json

import azure.functions as func

import azure.functions as func is required for classes specific to Function Apps and we are also adding import json as we want to return json values to the caller of the Function App.

We can then just add the authentication part like this and as we were using the credential variable already in the original code, nothign else needs to change from an authentication perspective.

# Authenticate
credential = DefaultAzureCredential()

Python Functions Declaration

In the original script, the code was wrapped around these two functions, which can remain unchanged:

def resource_graph_query( query ):
    # Get your credentials from Azure CLI (development only!) and get your subscription list
    subs_client = SubscriptionClient(credential)
    subscriptions_dict = []

    for subscription in subs_client.subscriptions.list():
        subscriptions_dict.append(subscription.as_dict())

    subscription_ids_dict = []

    for subscription in subscriptions_dict:
        subscription_ids_dict.append(subscription.get('subscription_id'))

    # Create Azure Resource Graph client and set options
    resource_graph_client = arg.ResourceGraphClient(credential)
    resource_graph_query_options = arg.models.QueryRequestOptions(result_format="objectArray")

    # Create query
    resource_graph_query = arg.models.QueryRequest(subscriptions=subscription_ids_dict, query=query, options=resource_graph_query_options)

    # Run query
    resource_graph_query_results = resource_graph_client.resources(resource_graph_query)

    # Show Python object
    return resource_graph_query_results

def check_name_availability(resource_name, resource_type=None):

    if(resource_type):
        rg_query = f"Resources | where name =~ '{resource_name}' | where type =~ '{resource_type}'"
    else:
        rg_query = f"Resources | where name =~ '{resource_name}'"


    rg_results = resource_graph_query(rg_query)

    results_dict = []

    if(rg_results.data):
        availability = False
    else:
        availability = True

    results_dict = dict({
        'resource_name': resource_name,
        'available': availability
    })

    return results_dict

Code specific to the Function App

The following code had to be modified to suit the Function App.
We are basically defining the name and the route:

app = func.FunctionApp()

@app.function_name(name="CheckNameAvailability")
@app.route(route="checkNameAvailability")

This will create the Function App with a name of CheckNameAvailability and the route, which defines the URL - in the case of above example this is https://{function-app-name}.azurewebsites.net/api/checknameavailability.

Lastly, we need the main function containing the code. The function accepts two parameters, resourceName and resourceType, which are added to the corresponding variables r_name and r_type. These can be passed as parameters or json payload. Then we do check whether only resourceName or bothm resourceName and resourceType, or none of these are populated and return the corresponding result to the caller.

def main(req: func.HttpRequest) -> func.HttpResponse:
    r_name = req.params.get('resourceName')
    r_type = req.params.get('resourceType')
    if not r_name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            r_name = req_body.get('resourceName')

    if not r_type:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            r_type = req_body.get('resourceType')

    if r_name and r_type:
        result = check_name_availability(resource_name=r_name, resource_type=r_type)
        result_as_json = json.dumps(result)
        return func.HttpResponse(result_as_json)
    elif r_name:
        result = check_name_availability(resource_name=r_name)
        result_as_json = json.dumps(result)
        return func.HttpResponse(result_as_json)
    else:
        return func.HttpResponse(
            "This HTTP triggered function executed successfully. Pass a resourceName in the query string or in the request body for a proper response.",
            status_code=200
        )

Creating the Function Locally

For creating the function locally, we need to make sure the appropriate prerequisites are met [12].
We can then following along with the Azure Function documentation on creating a local function project [13].

func init CheckNameAvailability --python -m V2

This will create a new folder along with files required for the Function App.

Add the Code

Files:

We can then add the previously created code into the function_app.py file. Furthermore, the requirements.txt file needs to be populated since we are using the remote build feature. A sample can be found in here, that includes the following:

# Do not include azure-functions-worker in this file
# The Python Worker is managed by the Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues

azure-functions
azure-common
azure-core
azure-identity
azure-mgmt-core
azure-mgmt-resource
azure-mgmt-resourcegraph
certifi
cffi
charset-normalizer
cryptography
idna
isodate
msal
msal-extensions
msrest
oauthlib
portalocker
pycparser
PyJWT
requests
requests-oauthlib
six
typing-extensions
urllib3

Publishing the Function

For publishing the code to our Azure Function App, we can run the following command in order to trigger the remote build:

func azure functionapp publish {function-app-name}

After some time, we should see the notification Remote build succeeded! and the Syncing triggers... is successfully displaying the new function URL. If this is running into a timeout, then there might be missing dependencies in the requirements.txt or another issue with the code.

Testing the Functionality

In order to test the Function App, we would need to know the Function Key. We could either look it up through the portal or use the command line:

func azure functionapp list-functions {function-app-name} --show-keys

Also from the command line, we can use curl to test the newly created function:

curl -X POST https://{function-app-name}.azurewebsites.net/api/checknameavailability \
     -H 'Content-Type: application/json' \
     -H 'x-functions-key: {FUNCTION-KEY}' \
     -d '{"resourceName": "{storage-account-name}", "resourceType": "Microsoft.Storage/storageAccounts"}' | jq

In the case of above example, the result should be false, since I used the same name for the Function Apps Storage Account.

{
  "resource_name": "{storage-account-name}",
  "available": false
}

If we tried with a different name, it should return true:

curl -X POST https://{function-app-name}.azurewebsites.net/api/checknameavailability \
     -H 'Content-Type: application/json' \
     -H 'x-functions-key: {FUNCTION-KEY}' \
     -d '{"resourceName": "{unused-storage-account-name}", "resourceType": "Microsoft.Storage/storageAccounts"}' | jq

{
  "resource_name": "{unused-storage-account-name}",
  "available": true
}

Cleaning Up

After successful learning and testing, we can clean everything up by running terraform destroy and remove the previously created resources.

References

#	Title	URL	Accessed-On
1	Azure Functions Python developer guide	https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python	2023-01-04
2	Terraform: azurerm	https://registry.terraform.io/providers/hashicorp/azurerm/3.37.0	2023-01-04
3	Storage considerations for Azure Functions	https://learn.microsoft.com/en-us/azure/azure-functions/storage-considerations	2023-01-04
4	Monitor executions in Azure Functions	https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring	2023-01-04
5	Azure SDK for Python - How to check for Available Resource Names	https://dev.to/holger/azure-sdk-for-python-how-to-check-for-available-resource-names-1lg5	2023-01-04
6	What is Azure Resource Graph?	https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview	2023-01-04
7	What are managed identities for Azure resources?	https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview	2023-01-04
8	Azure Functions Consumption plan hosting	https://learn.microsoft.com/en-us/azure/azure-functions/consumption-plan	2023-01-04
9	Install Terraform	https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli	2023-01-04
10	Azure Functions - Remote build	https://learn.microsoft.com/en-us/azure/azure-functions/functions-deployment-technologies#remote-build	2023-01-04
11	DefaultAzureCredential Class	https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential	2023-01-04
12	Quickstart: Create a Python function in Azure from the command line - Prerequisite check	https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?pivots=python-mode-decorators&tabs=azure-cli%2Cbash#prerequisite-check	2023-01-04
13	Quickstart: Create a Python function in Azure from the command line - Create a local function project	https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?pivots=python-mode-decorators&tabs=azure-cli%2Cbash#create-a-local-function-project	2023-01-04

Azure SDK for Python - How to check for Available Resource Names

holger — Mon, 02 Jan 2023 13:52:39 +0000

If we wanted to check in Azure whether or not a given resource name is still available, we can do so relatively easily within the Azure Portal by navigating to the All resources blade, making sure the subscription filter has all relevant subscriptions in scope and typing in the name that we want to check.

This is fine for a manual approach - but how could we do this programmatically?

Azure REST API

One option would be to use the Azure REST API [1]. Most Azure services include a REST API capability for checking for available resource names - let's take Storage Accounts as an example [2].

We could create a POST request against https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Storage/checkNameAvailability?api-version=2022-09-01 in order to verify the availability:

from azure.identity import AzureCliCredential
import requests

credential = AzureCliCredential()
access_token = credential.get_token("https://management.core.windows.net/")
bearer_token = f"Bearer {access_token.token}"

r_url = 'https://management.azure.com/subscriptions/{subscription_id}/providers/Microsoft.Storage/checkNameAvailability?api-version=2022-09-01'
r_header = {'Content-Type': 'application/json', 'Authorization' : bearer_token}
r_body = {'name': '{storage_account_name}','type': 'Microsoft.Storage/storageAccounts'}

result = requests.post(headers=r_header,url=r_url,json=r_body)

print(result.json())

If the storage account name was already used, the response should be:

{'nameAvailable': False, 'reason': 'AlreadyExists', 'message': 'The storage account named storage_account_name is already taken.'}

Otherwise, the response would simply be:

{'nameAvailable': True}

As you can see, this works well, however, there are drawbacks:

Not all Azure services offer the capability for checking the available name through the Azure REST API.
If they do, we need to know the REST API URL.
We can only run the query against a single subscription.

To resolve the third point, we could first receive a list of available subscriptions and then iterate through each subscription and query the REST API on a per-subscription basis. This would work as well, however the other two drawbacks would still apply.

Azure SDK for Python

A simpler approach - since we are using Python anyways in this example - would be to use the Azure SDK for Python [6].

from azure.identity import AzureCliCredential
from azure.mgmt.storage import StorageManagementClient
credential = AzureCliCredential()

subscription_id = '{subscription_id}'
storage_account_name = '{storage_account_name}'

storage_client = StorageManagementClient(credential, subscription_id)



availability_result = storage_client.storage_accounts.check_name_availability(
    { 
        "name": storage_account_name
    }
)

print(availability_result)

Similar to what we saw when querying the Azure REST API, the result indicating that the name was already used would look like this:

{'additional_properties': {}, 'name_available': False, 'reason': 'AlreadyExists', 'message': 'The storage account named storage_account_name is already taken.'}

If the name was still available, the response would be:

{'additional_properties': {}, 'name_available': True, 'reason': None, 'message': None}

This approach includes similar drawbacks as using the REST API directly.

Not all Azure services offer the capability for checking the available name.
If they, do we need to import the corresponding Python library.
We can only run the query against a single subscription.

Again, the last point can be remediated by using the SubscriptionClient Class [4], generating a list of subscription id's and iterating through it - however, the approach might not be very flexible.

Azure Resource Graph

Fortunately, Azure offers a great capability called Azure Resource Graph [3], which we can also use through the Azure SDK for Python.

Below, we see two functions created. resource_graph_query is used to run a query against Azure Resource Graph [5]. check_name_availability is then used to create and execute the query against a specific name - and optionally, resource type.

from azure.identity import AzureCliCredential
from azure.mgmt.resource import SubscriptionClient
import azure.mgmt.resourcegraph as arg

credential = AzureCliCredential()

def resource_graph_query( query ):
    # Get your credentials from Azure CLI (development only!) and get your subscription list
    subs_client = SubscriptionClient(credential)
    subscriptions_dict = []

    for subscription in subs_client.subscriptions.list():
        subscriptions_dict.append(subscription.as_dict())

    subscription_ids_dict = []

    for subscription in subscriptions_dict:
        subscription_ids_dict.append(subscription.get('subscription_id'))

    # Create Azure Resource Graph client and set options
    resource_graph_client = arg.ResourceGraphClient(credential)
    resource_graph_query_options = arg.models.QueryRequestOptions(result_format="objectArray")

    # Create query
    resource_graph_query = arg.models.QueryRequest(subscriptions=subscription_ids_dict, query=query, options=resource_graph_query_options)

    # Run query
    resource_graph_query_results = resource_graph_client.resources(resource_graph_query)

    # Show Python object
    return resource_graph_query_results

def check_name_availability(resource_name, resource_type=None):

    if(resource_type):
        rg_query = f"Resources | where name =~ '{resource_name}' | where type =~ '{resource_type}'"
    else:
        rg_query = f"Resources | where name =~ '{resource_name}'"


    rg_results = resource_graph_query(rg_query)

    results_dict = []

    if(rg_results.data):
        availability = False
    else:
        availability = True

    results_dict = dict({
        'resource_name': resource_name,
        'available': availability
    })

    return results_dict

r_name = '{storage_account_name}'
r_type = 'Microsoft.Storage/storageAccounts'

result = check_name_availability(resource_name=r_name, resource_type=r_type)

print(result)

Thanks to Azure Resource Graph, we can just pass a list of subscription id's and query them all at once with a single call. Particularly when checking hundreds or thousands of subscriptions at once, Resource Graph improves performance significantly.

Furthermore, this approach prevents us from having to pass in exact REST API URLs or import Python libraries for the individual Azure resources that we are querying.

The results of above code would be like this (but could obviously be improved or extended) if the resource existed already:

{'resource_name': 'storage_account_name', 'available': False}

Contrarily, if the resource did not exist yet, the response would be True:

{'resource_name': 'storage_account_name', 'available': True}

References

#	Title	URL
1	Azure REST API reference	https://learn.microsoft.com/en-us/rest/api/azure/
2	Storage Accounts - Check Name Availability	https://learn.microsoft.com/en-us/rest/api/storagerp/storage-accounts/check-name-availability?tabs=HTTP
3	What is Azure Resource Graph?	https://learn.microsoft.com/en-us/azure/governance/resource-graph/overview
4	SubscriptionClient Class	https://learn.microsoft.com/en-us/python/api/azure-mgmt-subscription/azure.mgmt.subscription.subscriptionclient?view=azure-python
5	Quickstart: Run your first Resource Graph query using Python	https://learn.microsoft.com/en-us/azure/governance/resource-graph/first-query-python
6	Use the Azure libraries (SDK) for Python	https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-overview

How to retrieve a list of available services from the Azure Retail Rates Prices API

holger — Thu, 22 Dec 2022 16:33:10 +0000

Introduction

You probably know about the Azure Retail Rates Prices API [1] at https://prices.azure.com/api/retail/prices. It provides a "programmatic way to retrieve retail prices for all Azure services" [1] and does not require authentication - which makes it easy to query it from any kind of code (which was probably the idea behind creating it this way).
As a test, you can just head over to https://prices.azure.com/api/retail/prices in your browser and the API will provide the first 100 datasets of the price list.

The samples within this article will be based on Python but it could be done in a lot of other languages as well (technically, probably even plain curl within a bash script). However, due to the huge amount of pricing data that is returned by the API, Python has some advantages as there are some powerful libraries, such as Pandas, which is really beneficial when working with large datasets.

Retrieve a list of Available Services in a given Region

The API Sample Calls [2] reveal that the API allows for some filter queries to be passed which allows us to narrow the retrieved data sets down by a bit - which certainly makes it easier to work with.

As per the documentation, potential filters [3] are:

armRegionName
Location
meterId
meterName
productid
skuId
productName
skuName
serviceName
serviceId
serviceFamily
priceType
armSkuName

To start somewhere, the armRegionName could be relatively straightforward. For example, we could check for the available regions in the documentation [4] or use Az CLI or Azure PowerShell to query them (only when being authenticated though).

However, what about the serviceName? It is probably a common use case to retrieve prices for, let's say, Storage Services or Virtual Machine sizes. If we wanted to specify those, then it's necessary that we know what the API is expecting.
One way would be to try and browse through the text but that surely isn't fun.

What we could do though is retrieve price datasets for a given region and then filter out the unique values for the serviceName property.

Create the Python Script

Note that the only library required here is the requests library. We could then define the function as outlined below.

import requests

def fetch_azure_services(region_name='westeurope'):
    '''Function fetches Service Names from the Azure Retail Price API based on a given region. Region must be lowercase.'''

    filter = f"?&$filter=armRegionName eq '{region_name}'"
    url = f"https://prices.azure.com/api/retail/prices{filter}"

    all_price_records = []

    while True:
        if not url:
            break
        response = requests.get(url)
        if response.status_code == 200:
            json_data = response.json()
            url = json_data['NextPageLink'] # Fetch next link
            all_price_records = all_price_records + json_data['Items']

    service_names = []

    for record in all_price_records:
        service_names.append(record['serviceName'])

    unique_service_names = list(set(service_names))

    return unique_service_names

What above code does is:

Construct the url including the filter value for a given region
Send a request
Add the response to a python dictionary
Extract the NextPageLink and continue sending requests with all subsequent values of NextPageLink until it does not contain values any more. NextPageLink looks something like "NextPageLink":"https://prices.azure.com:443/api/retail/prices?$skip=100", with the next value using $skip=200 and so on, until there is nothing to skip any more and all values that match the filter query have been returned.
The function then iterates through all returned records and extracts all serviceName values
Lastly, it removes all duplicates and returns a list of unique values

Let's Try

When we execute the function, we can just pass in any region for which we want to retrieve the list of available services. Let's say we wanted to retrieve the service names for the East US region. The armRegionName for this region would be eastus - basically the region name with all lowercase and no spaces. In this case the function would be executed as below:

eastus_service_names = fetch_azure_services('eastus')

This will return a python list of 117 services, which you can check by getting the length of the list.

print(len(eastus_service_names))

We can then sort the list and just print it out in alphabetical order.

eastus_service_names.sort()
print(eastus_service_names)

The output should then be like this:

['API Management', 'Advanced Data Security', 'Advanced Threat Protection', 'App Configuration', 'Application Gateway', 'Application Insights', 'Automation', 'Azure API for FHIR', 'Azure Active Directory B2C', 'Azure Active Directory Domain Services', 'Azure Active Directory for External Identities', 'Azure Analysis Services', 'Azure App Service', 'Azure Applied AI Services', 'Azure Arc Enabled Databases', 'Azure Bastion', ...]

I've shortened it since there is no need to print the whole list here.

Conclusion

We can query the Azure Retail Rates Prices API [1] through Python to get pricing details. In order to speed up potential requests against the API, it is beneficial to create a specific query. One of the commonly used values to query might be the serviceName property. Aforementioned example shows that we can also use the API to get all possible values for the serviceName property and we could then use these values in subsequent queries.

The method used in this example certainly has drawbacks - mainly performance, as we are always retrieving the whole price list for a given region, just to filter out the serviceName.

The example has only scratched the surface of the possibilities that are available with the API and we may be looking into it a little more in subsequent articles.

References

#	Title	URL
1	Azure Retail Rates Prices API	https://learn.microsoft.com/en-us/rest/api/cost-management/retail-prices/azure-retail-prices
2	API Sample Calls	https://learn.microsoft.com/en-us/rest/api/cost-management/retail-prices/azure-retail-prices#api-sample-calls
3	API filters	https://learn.microsoft.com/en-us/rest/api/cost-management/retail-prices/azure-retail-prices#api-filters
4	Azure geographies	https://azure.microsoft.com/en-us/explore/global-infrastructure/geographies/
5	Requests Library	https://pypi.org/project/requests/

How to use the Azure SDK for Python to retrieve a list of Azure Resources

holger — Thu, 22 Dec 2022 08:19:52 +0000

Introduction

The Azure SDK for Python [1] is one of many ways to connect to Azure REST APIs and extract information. Here is an example on how to retrieve all resources from an Azure subscription, put the data into a Pandas DataFrame [2] and plot a simple pie chart using matplotlib [3].

Import Libraries

We need to import the following libraries.



# Import libraries
from azure.identity import AzureCliCredential
from azure.mgmt.resource import ResourceManagementClient
import pandas as pd
import matplotlib

Acquire Credentials and obtain the Management Object

When using AzureCliCredential(), Python will use the Azure CLI context of the currently logged-in user for executing the queries.



# Acquire credential
credential = AzureCliCredential()

# Define Scope
subscription_id = "subscription_id"

# Obtain the management object for resources.
resource_client = ResourceManagementClient(credential, subscription_id)

Create and Populate a Dictionary

Using resource_client.resources.list() through the resource client will retrieve an interable object with all Azure resources in the given subscription. We can now populate a Python dictionary with the resources and add them to a Pandas DataFrame.



resources = resource_client.resources.list()
dict = []
for resource in resources:
    dict.append(resource.as_dict())

Create a DataFrame from the Dictionary



df = pd.DataFrame.from_dict(dict)

Select only the Top 10 rows based on the Resource Type Count



type_df = df['type'].value_counts()
type_df_top = type_df.head(10)

Finally plot the Pie Chart



plot = type_df_top.plot.pie(figsize=(10, 10), autopct= lambda x: '{:.0f}'.format(x*type_df_top.sum()/100))

... which might look like something along these lines.

Summary

This short article has scratched the surface on what is possible when using the Azure SDK for Python since it allows as to combine the power of Python with the capabilities of the Azure REST API.
We have extracted a list of Azure resources based on a given context, added them to a Pandas DataFrame and plotted a simple pie chart based on the top 10 resource count.

References

#	Title	URL
1	Use the Azure libraries (SDK) for Python	https://learn.microsoft.com/en-us/azure/developer/python/sdk/azure-sdk-overview
2	pandas.DataFrame	https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html
3	Matplotlib: Visualization with Python	https://matplotlib.org/

How to use the Azure SDK for Python to list Storage Account Lifecycle Policies

holger — Wed, 21 Dec 2022 19:52:05 +0000

The code samples and information provided within this document are solely for testing and learning purposes and should not be used in production environments.

I was looking into a way to list Storage Accounts that have (or do not have) Lifecycle Policies enabled and output them as JSON.
The Azure Samples Repository does have a sample for polices, however, I couldn't figure out how to list policies since the sample only includes examples for Creating, Updating, Deleting and Getting a specific policy.
The latter requires the Resource Group Name, Storage Account Name and Management Policy Name as parameter which did not seem helpful.

A check against the REST API documentation for Management Policies (Get) reveals that the URI Parameter managementPolicyName should always be default.

Based on aforementioned sample, the following should work.

Import Libraries

from azure.identity import AzureCliCredential
from azure.mgmt.resource import SubscriptionClient
from azure.mgmt.storage import StorageManagementClient
import azure.mgmt.resourcegraph as arg
import json

Create the Credential Client

credential = AzureCliCredential()

Create the Resource Graph Query Function

This is taken from: https://learn.microsoft.com/en-us/azure/governance/resource-graph/first-query-python

def resource_graph_query( query ):
    # Get your credentials from Azure CLI (development only!) and get your subscription list
    subs_client = SubscriptionClient(credential)
    subscriptions_dict = []

    for subscription in subs_client.subscriptions.list():
        subscriptions_dict.append(subscription.as_dict())

    subscription_ids_dict = []

    for subscription in subscriptions_dict:
        subscription_ids_dict.append(subscription.get('subscription_id'))

    # Create Azure Resource Graph client and set options
    resource_graph_client = arg.ResourceGraphClient(credential)
    resource_graph_query_options = arg.models.QueryRequestOptions(result_format="objectArray")

    # Create query
    resource_graph_query = arg.models.QueryRequest(subscriptions=subscription_ids_dict, query=query, options=resource_graph_query_options)

    # Run query
    resource_graph_query_results = resource_graph_client.resources(resource_graph_query)

    # Show Python object
    return resource_graph_query_results

Query all Storage Accounts

In order to narrow down the results by a bit, I only queried Storage Accounts from one particular region. If there are a lot of storage accounts in scope this should be tested against a single account first to a) see if it works and b) get an idea about the performance.

query = "resources | where type =~ 'Microsoft.Storage/storageAccounts' | where location =~ 'eastus'"

storage_accounts = resource_graph_query(query).data

Retrieve the Storage Account Lifecycle Policies

for storage_account in storage_accounts:
    storage_client = StorageManagementClient(credential=credential, subscription_id=storage_account.get('subscriptionId'))
    try:
        print(json.dumps(storage_client.management_policies.get(
            account_name=storage_account.get('name'), 
            resource_group_name=storage_account.get('resourceGroup'), 
            management_policy_name='default').as_dict()))
    except Exception:
        continue

If this worked, the output would be something like this:

{
    "id": "/subscriptions/{subscription-id}/resourceGroups/resource_group_name/providers/Microsoft.Storage/storageAccounts/storageaccountname/managementPolicies/default",
    "name": "DefaultManagementPolicy",
    "type": "Microsoft.Storage/storageAccounts/managementPolicies",
    "last_modified_time": "2022-09-23T09:00:20.182687Z",
    "policy": {
        "rules": [
            {
                "enabled": true,
                "name": "Move Blobs to Cool After 1 Day",
                "type": "Lifecycle",
                "definition": {
                    "actions": {
                        "base_blob": {
                            "tier_to_cool": {
                                "days_after_modification_greater_than": 1.0
                            }
                        }
                    },
                    "filters": {
                        "blob_types": [
                            "blockBlob"
                        ]
                    }
                }
            }
        ]
    }
}

References

#	Title	URL
1	Optimize costs by automatically managing the data lifecycle	https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview
2	Azure Samples Repository	https://github.com/Azure-Samples
3	Management Policies - Get	https://learn.microsoft.com/en-us/rest/api/storagerp/management-policies/get
4	Quickstart: Run your first Resource Graph query using Python	https://learn.microsoft.com/en-us/azure/governance/resource-graph/first-query-python