Getting started with Windows Containers on Azure Kubernetes Service

Rudy van Sloten — Thu, 20 Jan 2022 22:40:24 +0000

Windows support has finally arrived in Kubernetes and AKS. Learn how to migrate your workloads and what pitfalls to avoid in this short and sweet introduction to Windows Containers.

Windows support is relatively new to AKS and has been stable for about a year at the time of this article's writing. I've recently had to do a deep dive into the technology for a project, migrating legacy applications that still require the full .NET 4.8 Framework. While Kubernetes deployment and image building are quite similar to Linux-based containers, there are several key differences.

Kubernetes on Windows is a great way to make older .NET/IIS applications, Windows console apps and services more flexible, highly available and scalable using native Kubernetes and cloud technologies. Azure Kubernetes Service takes away the overhead of managing servers or VM images, and gives you a single pane of glass solution for both your Linux and Windows deployments.

Image Versions

Microsoft offers four different base images on Docker Hub for Windows Server containers, which they also use in their collection of IIS and ASP.NET images:

windows/nanoserver is ultra-light at a mere 112MB. It only runs .NET Core out of the box, but PowerShell Core can be installed during the image build.
windows/servercore is 1.2GB in size and a good option for "lifting and shifting" Windows Server apps. It supports everything regular Windows Server 2019 Core offers.
windows/server weighs in at a hefty 3.1GB, has full Windows API support, and allows you to use all Windows Server features. This image can only be pulled on Windows 11 and Windows Server 2022.
windows is the largest image at 3.4GB. It has full Windows API support, but lacks native server features. This is Windows 10 under the hood. RDP connections are not possible out of the box with this image, so it cannot be used as an Azure Virtual Desktop.

Note: These are compressed sizes, unpacked, the size may be up to 2–5 times larger.

Important to remember is that major version tags such as ltsc2019, 1909, and 20H2 are updated automatically every second Tuesday of the month at 00:00 UTC. For granular control over your Windows Update cycle, refer to KB or version-specific tags.

Memory allocation

Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. Windows nodes do not overcommit memory for processes running in containers. The net effect is that Windows won’t reach out of memory conditions the same way Linux does, and processes page to disk instead of being subject to out-of-memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance.

Prerequisites

To follow along with the practical section, you’ll need a couple of tools:

A Windows 10/11 PC/VM with Virtualization enabled
Docker Desktop, with Windows Containers enabled
An Azure account
Azure CLI
kubectl
PowerShell

Every resource used in this article is available in this git repository:
rdvansloten/windows-containers-demo. You can run the scripts and Dockerfiles by git cloning it to your local machine/server:

git clone https://github.com/rdvansloten/windows-containers-demo.git

Creating Azure resources

Quick version

If you don’t want to explore and create the resource setup step-by-step, there’s an all-inclusive, run-once PowerShell script to set up all required resources. cd into windows-containers-demo/powershell and execute deployment.ps1. Check out the brief README.md in that folder for the script's arguments. A login window may pop up to verify your Azure credentials.

Skip to Creating your Dockerfile after the deployment script is finished.

Long version

Let’s start off by logging into Azure via the CLI. To execute the scripts in this article, you’ll need to open PowerShell as an Administrator. This is required for interacting with the Docker daemon. macOS/Linux users may use the command pwsh before attempting to create these resources.

# Login to Azure
az login
# List and select the right Subscription
az account show
az account set --subscription "<YOUR SUBSCRIPTION ID/NAME HERE>"

Set the below variables in your terminal if you intend to run all the other snippets below. Replace the $AZUREUSER value with your Azure account’s email address.

# Generate a random number for uniqueness
$RANDOM = Get-Random -Minimum -100000 -Maximum 999999

$AZUREUSER      = "yourEmail@domain.com" # Your Azure account email
$NODEPOOLNAME   = "win"                  # Max 6 characters
$LOCATION       = "westeurope"           # The region you prefer
$RESOURCEGROUP  = "rg-aks"               # Name of your Resource Group
$CLUSTERNAME    = "myaks$($RANDOM)"      # Must be globally unique

After logging in, you start off by creating the basics; a Resource Group to hold your infrastructure and an Identity that AKS will use to talk to other services, such as the Virtual Network, Container Registry, and VM Scale Sets.

# Create Resource Group
az group create `
  --name $RESOURCEGROUP `
  --location $LOCATION

# Create Managed Identity
az identity create `
  --name $CLUSTERNAME-id `
  --resource-group $RESOURCEGROUP `
  --location $LOCATION

# Store Managed Identity ID in variable for AKS creation
$AKSIDENTITY = $(
  az identity show `
    --name $CLUSTERNAME-id `
    --resource-group $RESOURCEGROUP `
    --query id
)

Virtual Network

Let’s create a Virtual Network with enough IP addresses to accommodate a lot of Pods, Services, and Ingresses. Also, make sure that the AKS identity can create network resources, or your components inside Kubernetes will not be able to be provisioned.

# Create VNET and subnet
az network vnet create `
  --name $CLUSTERNAME-vnet `
  --resource-group $RESOURCEGROUP `
  --location $LOCATION `
  --address-prefixes 172.10.0.0/16 `
  --subnet-name kubernetes `
  --subnet-prefixes 172.10.128.0/17

# Store subnet ID in variable for AKS creation
$SUBNETID = $(
  az network vnet subnet list `
    --resource-group $RESOURCEGROUP `
    --vnet-name $CLUSTERNAME-vnet `
    --query "[0].id"
)

# Grant AKS identity VNET access for Azure CNI
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
    )`
  --scope $(
    az network vnet show `
      --resource-group $RESOURCEGROUP `
      --name $CLUSTERNAME-vnet `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "Network Contributor"

Creating the AKS cluster

Now that the prerequisites have been deployed, you can finally build your AKS cluster.

Linux && Windows

Even though we want a Windows cluster, AKS still requires a Linux node for the control plane. Let’s set the single Linux node to one of the cheapest VM sizes, Standard_B2s (as it won’t be doing much work), and assign the Windows node pool a Standard_D2s_v4 node, sporting 2 vCPU cores and 7GB of RAM.

Networking

You’ll select the virtual network you’ve created in the previous step and define a few private ranges for AKS to use. Pods and Nodes will be assigned an address in the 172.10. range. Services and system resources, such as CoreDNS, will operate on a private 10. range. For testing purposes, let’s let Azure worry about public DNS and Ingress. The addon http_application_routing will provide these resources for you. You will only need to add an annotation to your application to get it published on the internet.

Please note that the http application routing addon is not meant for production usage. Check out alternatives like Traefik for production workloads.

Azure CNI

In order for each Pod to have its own address, you will want to deploy Azure CNI rather than Kubenet for your networking layer. Some benefits of Azure CNI over Kubenet:

No NAT, so the source of the packets is retained.
Workloads can be scaled out to Virtual Nodes.
Virtual Networks can be managed separately/independently.
Each Pod gets an IP address from the subnet, making debugging easier.
Most importantly, Windows node pools support Azure CNI only.

Azure CNI also forces you to plan out your Virtual Network beforehand, which helps enormously with network architecture and capacity planning.

# Assign AKS Kubelet Identity permissions on the AKS Resource Group
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
  )`
  --scope $(
    az group show `
      --name $(
        az group show `
          --resource-group $RESOURCEGROUP `
          --query name
      ) `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "Contributor"

# Create AKS cluster
az aks create `
  --resource-group $RESOURCEGROUP `
  --name $CLUSTERNAME `
  --location $LOCATION `
  --assign-identity "$AKSIDENTITY" `
  --assign-kubelet-identity "$AKSIDENTITY" `
  --node-vm-size Standard_B2s `
  --node-count 1 `
  --enable-aad `
  --enable-azure-rbac `
  --network-plugin azure `
  --vnet-subnet-id "$SUBNETID" `
  --docker-bridge-address 172.17.0.1/16 `
  --dns-service-ip 10.2.0.10 `
  --service-cidr 10.2.0.0/24 `
  --generate-ssh-keys `
  --enable-addons http_application_routing

# Add Windows node pool
az aks nodepool add `
  --name $NODEPOOLNAME `
  --resource-group $RESOURCEGROUP `
  --cluster-name $CLUSTERNAME `
  --node-vm-size Standard_D2s_v4 `
  --node-count 1 `
  --os-type Windows `
  --max-surge 33%

# Assign yourself Admin permissions on the AKS cluster
az role assignment create `
  --assignee $(
    az ad user show `
      --id $(
        az ad user list `
          --query "[?contains(@.otherMails,'$AZUREUSER')].userPrincipalName " -o tsv
      ) `
      --query userPrincipalName 
  ) `
  --scope $(
    az aks show `
      --name $CLUSTERNAME `
      --resource-group $RESOURCEGROUP `
      --query id
  )`
  --role "Azure Kubernetes Service RBAC Cluster Admin"

az role assignment create `
  --assignee $(
    az ad user show `
      --id $(
        az ad user list `
          --query "[?contains(@.userPrincipalName,'$AZUREUSER')].userPrincipalName " -o tsv
      ) `
      --query userPrincipalName 
  ) `
  --scope $(
    az aks show `
      --name $CLUSTERNAME `
      --resource-group $RESOURCEGROUP `
      --query id
  )`
  --role "Azure Kubernetes Service RBAC Cluster Admin"

# Assign AKS identity permissions to node pool Resource Group
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
  )`
  --scope $(
    az group show `
      --name $(
        az aks show `
          --name $CLUSTERNAME `
          --resource-group $RESOURCEGROUP `
          --query nodeResourceGroup
      ) `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "Contributor"

Permissions, RBAC

The AKS deployment script finishes off by assigning the AKS identity permissions to create resources in the special “MC_” Resource Group that AKS generates during creation. This is required for it to interactively spawn resources. Contributor is a very broad permission set and should not be used lightly in production scenarios.

Consult Microsoft Docs: AKS Cluster Identity Permissions for an exhaustive list of permissions you may want to apply/deny when setting up AKS.

During the last script, you’ve also applied the Azure Kubernetes Service RBAC Cluster Admin role to your own Azure AD user. This is required because of the --enable-aad and --enable-azure-rbac arguments passed to az aks create. These arguments allow Azure AD/RBAC objects to be added to Kubernetes (Cluster)Roles.

Deploying a Windows Server container image

If you don’t have your own container repository, let’s create one and give the AKS identity permissions to pull images from it. The Azure Container Registry is a private container repository that includes security scanning, image management, and authentication. It also has experimental support for Helm chart storage.

# Create Container Registry
az acr create `
  --resource-group $RESOURCEGROUP `
  --name "$($CLUSTERNAME)acr" `
  --sku Basic

# Add AKS to Container Registry
az role assignment create `
  --assignee-object-id $(
    az identity show `
      --name $CLUSTERNAME-id `
      --resource-group $RESOURCEGROUP `
      --query principalId
    )`
  --scope $(
    az acr show `
      --name "$($CLUSTERNAME)acr" `
      --query id
    )`
  --assignee-principal-type ServicePrincipal `
  --role "AcrPull"

Creating your Dockerfile

With AKS up and running, you can now run Windows containers in the cloud. Start off by creating a Dockerfile and uploading the resulting image to an Azure Container Registry.

For this demo, I’ve created a simple PowerShell script for the container to loop. cd into the docker/powershell/ folder inside the cloned windows-containers-demo repo.

Then, run az acr build inside that folder to upload it to your ACR:

az acr build `
  --image myapp:latest `
  --platform Windows `
  --registry "$($CLUSTERNAME)acr" .

This command combines several Docker and Azure commands to save time/effort. You can tag it however you want, and use the tag when deploying your app. For this example, I use latest.

Optionally, you may also test your newly uploaded image locally on your Windows machine using docker run:

# Uses the docker CLI with your Azure login
az acr login --name "$($CLUSTERNAME)acr"

# Run the container locally on port 5000
docker run `
  --publish 5000:5000 `
  --interactive `
  --detach "$($CLUSTERNAME)acr.azurecr.io/myapp:latest"

Deploying your app

Now that your Windows nodes are up and running, and the image is uploaded to the Container Registry, you can deploy it to AKS using kubectl. For this demo’s purposes, I kept it simple with a the deploy.yaml file in the kubernetes/ folder you can feed into kubectl.

Before applying this file, cd into kubernetes/ and open deploy.yaml in a text editor like Visual Studio Code or Notepad, and modify lines 20 and 68:

host: myapp.<CLUSTER_DNS_ZONE>
image: <ACR_NAME>.azurecr.io/myapp:latest

Fill out the placeholder values with the resources you’ve created. For me, it looked something like this:

host: myapp.a430e1209ded44679aab.westeurope.aksapp.io
image: myaks321123.azurecr.io/myapp:latest

The value for image: <ACR_NAME> can be found by running:

az acr list `
  --resource-group $RESOURCEGROUP `
  --query "[0].name" -o tsv

The value for host: <CLUSTER_DNS_ZONE> can be found by running:

az network dns zone list `
  --resource-group $(
    az group show `
      --name $(
        az aks show `
          --name $CLUSTERNAME `
          --resource-group $RESOURCEGROUP `
          --query nodeResourceGroup
      ) `
      --query name
  )`
--query "[0].name" -o tsv

cd into the kubernetes/ folder. Then, use the following snippet to log in to your AKS cluster, and deploy your image:

# Load AKS details into ~/.kube/config
az aks get-credentials `
  --name $CLUSTERNAME `
  --resource-group $RESOURCEGROUP

# Check if AKS is reachable and Windows node is up
kubectl get nodes

# Deploy yaml manifest to AKS
kubectl apply -f deploy.yaml

# Get the URL for your app
kubectl get ingress myapp -o=jsonpath='{.spec.rules[0].host}'

Due to the size of the servercore:ltsc2019 image, the first pull might take 5–10 minutes. Wait for a while and watch for logs or errors using kubectl describe deployment myapp. I recommend tracking the progress using a GUI app, such as Lens Desktop.

Open your browser and go to the URL (something.some-region.aksapp.io) presented after running the kubectl get ingress command above. You will now be able to view your first Windows application running on Kubernetes!

Caveats

In contrast to the the benefits of Kubernetes on Windows, there are a few drawbacks to using Windows Containers in AKS:

Node pools are prohibitively more expensive than Linux node pools due to licensing. Bringing your own license with AHUB may alleviate some pain.
Windows-based Pods have limited CSI Driver support.
Windows versions of your DaemonSets are required. (monitoring, drivers)

Be mindful of the costs of running this setup. The above configuration will cost around $10/day, meaning you can only run it for 2–3 weeks on an Azure free account.

Service	Size	$/day
Azure Kubernetes Service	Linux, Standard_B2s	$1.15
Azure Container Registry	Basic, 25GB Bandwidth	$1.37
Virtual Machine Scale Set	Windows, Standard_D2s	$7.13

Run the az aks stop command to stop AKS from draining your wallet, or delete the cluster in its entirety when you are done with it.

Conclusion

With this, you are running your first basic Windows application in Kubernetes. I hope I’ve equipped you with some basic knowledge and tools to start tackling these issues. For a more production-ready environment, I would suggest looking into leveraging Terraform or ARM for infrastructure deployment, enterprise-grade Ingress Controllers, and setting your cluster, registry, and all its dependencies to private.

For any questions or inquiries, feel free to reach out to me on LinkedIn or in the comments!

Sources
kubernetes.io: Windows containers in Kubernetes
Microsoft Windows Server Blog: Windows Server container support in AKS is now generally available
Docker Hub: Windows base OS images
blog.sixeyed.com: Getting Started with Kubernetes on Windows

Forem: Rudy van Sloten