Forem: Tommy Falgout

GitOps for Presentations

Tommy Falgout — Fri, 04 Aug 2023 16:54:59 +0000

Yes, I work for Microsoft. No, I do not like PowerPoint. Here’s my alternative with the source code which I’ll explain here.

For 20+ years I’ve done UNIX/Linux development and have worked at Microsoft for 6 years. And I’ve learned that Microsoft will typically build the all-encompassing Enterprise-ready solution and the OSS ecosystem will build a narrow-focused tool that you can piece together with others.

Each have their own benefits and constraints. There is No Silver Bullet.

A common set of requirements I encounter are:

I need to easily present to a public audience
I might have to use someone else’s computer
I want to share the slides afterwards
I need to quickly update the slides
I just want to display text and images. (PowerPoint is an absurdly impressive tool with lots of features that I rarely use.)

Internal Microsoft SharePoint policy prevents sharing slides with external visitors. This often results in emailing 10-100MB PPTs or PDF files around. Blah!

Piecing bits of OSS, I present to you “GitOps for Presentations”. It involves:

Git + GitHub - Version control of content
Markdown - Easy styling of content
MARP - Converts CommonMark to HTML, PDF, PPT
VSCode - Edit the content (There’s even a MARP extension which allows you to preview in real-time!)
GitHub Actions - Build the presentation from Markdown
GitHub Pages - Host the presentation

Benefits:

Free (as in beer)
Simple to setup
You can style your presentations. For example, I’ve created a CSS which models the Microsoft styling guides for PPT’s
Easy to share and update

Limitations:

MARP’s formatting is basic. Especially if you’re coming from PowerPoint

That’s cool, but why didn’t you …

use Remark or Reveal.js?
- There’s many great presentations frameworks, but I wanted something really simple. KISS
- You should be able to replace MARP with any of those other frameworks and still get the same results.
just present your PPT and email it?
- That requires work and time. At conferences, I don’t have time/might forget to follow-up with everyone. I create a QR code and put it at the end of the slides. This enables self-service, discovery and also saves me previous keystrokes.
use slides.com or Google Slides?
- Microsoft has embraced OSS and purchased GitHub, so I wanted to find a way to explore integrating all of this. I’ve been very happy with the results!

I’m sold! How do I get started?

I’ve made it easy for anyone to get started by creating a GitHub template for this project (which is also a presentation)
Click “Use this template” and create a new repository
Enable GitHub Actions to auto-publish to GitHub Pages
- In your new Repo, click Settings -> Pages
  - Set Source to GitHub Actions
You’re done!

PEDANTIC DISCLAIMER:

I’m quite familiar with GitOps, and while this is outside of running Kubernetes clusters as IaC, there are some similarities with the top-level concept of using Git to set my desired state of my presentation.
MARP technically uses CommonMark. It’s close enough for what most people will need

Managing Azure Subscription Quota and Throttling Issues

Tommy Falgout — Wed, 21 Jun 2023 19:50:28 +0000

As Azure customers and partners build bigger and more complex solutions in their subscriptions, you might hit quota and throttling issues. These can be irksome and cause confusion. This article will walkthrough some of the scenarios I’ve seen and how to design with them in mind.

Let’s make sure we’re on the same page regarding terminology used in this article:

Azure Resource Manager (ARM) - The management layer and API behind all Azure resources
Resource Provider (RP) - Each resource type inside Azure has a RP which allows you to manage that resource (e.g. Storage, Key Vault, VMSS, etc.)
Quota - the maximum number of a specific resource available for your subscription. Similar to a credit card limit
- Examples:
- Some quotas have adjustable and non-adjustable quotas
  - Some adjustable quotas can be managed programmatically using the Azure Quota Service API
Throttling - maximum number of API requests you can make in a certain period. Similar to bandwidth throttling
- NOTE: There are subscription and tenant level throttling limits. Each Storage, Networking, Compute and Azure Resource Graph also have throttling limits
- NOTE: Throttling for RP’s are per subscription per region
- Examples:
  - Rate limit of writes to a subscription per hour
  - Rate limit of Deleting a VMSS in 3 min

Managing Quotas

Because quotas are mostly static, viewing your quotas is pretty simple. Simply to go the Azure Portal and click on “My quotas”.

If you need to increase your quota, you might need to open an Azure Support ticket. For example, if you need to start deploying in a new region, you might need to open a ticket to increase the “Total Regional vCPUs” and “VMSS” quotas in “West Central US”. Once the ticket has been approved, the quota will be available to you.

Managing Throttling

For the most part, you won’t need to worry about throttling, but if you’re doing very large scale deployments with LOTS of constant churning of resources, you might hit throttling limits.

These limits are less about the number of resources, but HOW you use the resources. For example:

You can have 5000 AKS cluster in one subscription, each AKS cluster can have a maximum of 100 node pools. If you try creating the max # of AKS clusters with the max # of node pools simultaneously, then you’ll definitely hit the throttling limit.
Some OSS projects aggressively call ARM and the RP API’s in a reconciliation loop. Multiple instances of these projects will also hit the throttling limit.

Since throttling is specific to the current time window, it can be trickier. There’s no “hard formula” for when you’ll hit a threshold. But when you do, you’ll probably start seeing 429 HTTP status responses.

Throttling Examples

Thankfully, you can get insights into your current throttling status by looking at response headers for the requests.

x-ms-ratelimit-remaining-subscription-reads - # of read operations to this subscription remaining
x-ms-ratelimit-remaining-subscription-writes - # of writes operations to this subscription remaining
x-ms-ratelimit-remaining-resource - Compute RP specific header, which could show multiple policy statuses. (see “Read Request for GETting a VMSS” below for details)

Let’s dig into this deeper using the Azure CLI.

Example: Create a Resource Group (Write Request)

Because this request creates a RG, it will count against our subscription writes:

➜  az group create -n $RG --location $LOCATION --verbose --debug --debug 2>&1 | grep 'x-ms'

DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '<GUID>'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-writes': '1199'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '<GUID>'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '<GUID>'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'SOUTHCENTRALUS:20230512T163152Z:<GUID>'

NOTE: The key point is how the x-ms-ratelimit-remaining-subscription-writes is now 1199 (instead of the standard 1200 per hour as per the Subscription and Tenant limits)

Example: GET a VMSS (Read Request)

This request performs a GET (read) request on an existing VMSS. This is similar to the write request for the RG, but since Compute RP also has a separate set of throttling policies, it also counts against the Compute RP limits.

➜  az vmss show -n $VMSS_NAME -g $RG --debug 2>&1 | grep x-ms
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '<GUID>'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-resource': 'Microsoft.Compute/GetVMScaleSet3Min;197,Microsoft.Compute/GetVMScaleSet30Min;1297'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '<GUID>'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '11999'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '<GUID>'
DEBUG: cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'SOUTHCENTRALUS:20230512T162738Z:<GUID>'

NOTE: The key point is how x-ms-ratelimit-remaining-resource has two key-value pairs:

Microsoft.Compute/GetVMScaleSet3Min;197 - I ran this command before, so I have 197 requests available in the 3 minute window for performing GET requests on the VMSS resource
Microsoft.Compute/GetVMScaleSet30Min;1297 - I now have 1297 requests available in the 30 minute window for performing GET requests on VMSS resources

NOTE: x-ms-ratelimit-remaining-subscription-reads doesn’t seem to decrease (11999). Even if I run the same command again. I haven’t figured that out yet.

Designing with quotas and throttling in mind

Most Azure deployments won’t need this type of fine tuning, but just in case, there’s some documented Throttling Best Practices as well as my personal pro-tips:

Use the Azure SDK, as many services have the recommended retry guidance built-in
Instead of creating and deleting VMSS (which consume multiple VMSS API requests), scale the VMSS to 0 (which only consumes 1 VMSS API request)
Any type of Kubernetes cluster auto-scaler will perform a reconciliation loop with Azure Compute RP. This could eat into your throttling limits
Use the Azure Quota Service API to programmatically request quota increases

If you’re unable to workaround the throttling limits, then the next step is to look at the Deployment Stamp pattern using multiple subscriptions. You can programmatically create subscriptions using Subscription vending.

Hopefully this article has helped you understand quotas limits and throttling limits in Azure, and how to work around them. Let me know if you have any additional questions and/or feedback and I can follow-up with additional details.

When, How and Where to use ClusterAPI (CAPI) and ClusterAPI for Azure (CAPZ)

Tommy Falgout — Mon, 18 Apr 2022 14:21:02 +0000

This article explains why, when, and how to use self-managed Kubernetes clusters in Azure for testing custom scenarios.

Kubernetes has gotten so large and complex that most companies prefer to use the managed service (e.g. AKS) instead of running it themselves. By using a managed Kubernetes service, this frees up the operations team to focus on their core competency instead of optimizing, backing up and upgrading of Kubernetes.

While this reduces the operational burden, you lose the ability to modify the platform. Sometimes these are acceptable tradeoffs, sometimes you need to manage it yourself.

Historically, AKS-engine was the OSS tool for creating unmanaged Kubernetes clusters on Azure, but it had some limitations. CAPI/CAPZ is the go-forward solution for creating and operating self-managed clusters declaratively.

I highly recommend reading Scott Lowe’s article on An introduction to CAPI. It covers a lot of terminology and concepts used here.

One of the reasons for using CAPI/CAPZ is as a testing and development tool for Kubernetes on Azure. For example, you might need to build and test the following scenarios:

A kernel change to the worker nodes
A modification to the K8S config on control plane nodes
An installation of a different CNI
The use of K8S to manage K8S

This diagram represents a high level architecture of a starter CAPI/CAPZ cluster.

The rest of this article will explain how to implement the above scenarios utilizing the CAPI quickstart. Because the command arguments will change over time, this article will describe the steps and provide a link to the full details like this:

Link to CAPI Quick Start with details: base command to run

Create the KIND Cluster

Similar to RepRap, CAPI uses a Kubernetes cluster to make more Kubernetes clusters. The easiest way is with Kubernetes IN Docker (KIND). As the name implies, it’s a Kubernetes cluster which runs as a Docker container. This is our starting point for what we call “Bootstrap Cluster”.

Create Kind Cluster: kind create cluster

Initialize cluster for Azure

We will use this bootstrap cluster to initialize the “Management Cluster” which contains all of the CRDs and runs the CAPI controllers. This is where we will apply all of our changes to meet our scenarios.

Initialize cluster for Azure: clusterctl init --infrastructure azure

Generate cluster configuration

Now that our management cluster is ready, we want to define what our workload cluster will look like. Thankfully, there are different flavors we can pick from. By using the default, we will get an unmanaged K8S cluster using virtual machines.

Generate cluster configuration: clusterctl generate cluster capi-quickstart > capi-quickstart.yaml

We now have a file which contains the CRDs which will define our workload cluster. We will modify capi-quickstart.yaml and edit the CRDs to implement each of our scenarios.

Full documentation is available for CAPI (baseline) CRDs and CAPZ (Azure specific resources) CRDs.

Scenario: Worker node kernel change

If we want to modify the worker nodes, we likely want to add a preKubeadmCommands and postKubeadmCommands directive in the KubeadmConfigTemplate.

preKubeadmCommands allows a list of commands to run on the worker node BEFORE joining the cluster.

postKubeadmCommands allows a list of commands to run on the worker node AFTER joining the cluster.

apiVersion: bootstrap.cluster.x-k8s.io/v1alpha4
kind: KubeadmConfigTemplate
metadata:
  name: capi-quickstart-md-0
  namespace: default
spec:
  template:
    spec:
      preKubeadmCommands:
        - wget -P /tmp https://kernel.ubuntu.com/<path>.deb
        - dpkg -i /tmp/<package name>.deb
      postKubeadmCommands:
        - reboot

After you’ve made these changes, you can proceed to the rest of the steps by applying the resources to your management cluster which will then create your workload cluster and deploy the CNI.

Scenario: Modify Kubernetes components

If we want to modify the control plane, we can make changes to the KubeadmControlPlane. This allows us to leverage the kubeadm API to customize various components.

For example, to enable a Feature Gate on the kube-apiserver:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha4
kind: KubeadmControlPlane
metadata:
  name: capi-quickstart-control-plane
  namespace: default
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          feature-gates: MyFeatureGate=true

The above example omits some fields for brevity. Make sure that you keep any existing args and configurations that you are not modifying in-place.

After you’ve made these changes, you can proceed to the rest of the steps by applying the resources to your management cluster which will then create your workload cluster and deploy the CNI.

Apply the Workload Cluster

Now that we have defined what our cluster should look like, apply the resources to the management cluster. The CAPZ operator will detect the updated resources and talk to Azure Resource Manager.

Apply the workload cluster: kubectl apply -f capi-quickstart.yaml

Monitor the Cluster Creation

After you’ve made the changes to the capi-quickstart.yaml resources and applied them, you’re ready to watch the cluster come up.

Watch the cluster creation:

kubectl get cluster
clusterctl describe cluster capi-quickstart
kubectl get kubeadmcontrolplane – Verify the Control Plane is up

Now that the workload cluster is up and running, it’s time to start using it!

Get the Kubeconfig for the Workload Cluster

Now that we’re dealing with two clusters (management cluster in Docker and workload cluster in Azure), we now have two kubeconfig files. For ease, we will save it to the local directory.

Get the Kubeconfig for the workload cluster: clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

Install the CNI

By default, the workload cluster will not have a CNI and one must be installed.

Deploy the CNI: kubectl --kubeconfig=./capi-quickstart.kubeconfig apply -f https://...calico.yaml

Scenario: Install a different CNI

If you want to use flannel as your CNI, then you can apply the resources to your management cluster which will then create your workload cluster.

However, instead of Deploying the CNI, you can follow the steps in the Install Flannel walkthrough.

Cleanup

When you’re done, you can cleanup both the workload and management cluster easily.

Delete the workload cluster kubectl delete cluster capi-quickstart

If you want to create the workload cluster again, you can do so by re-applying capi-quickstart.yaml

Delete the management cluster kind delete cluster

If you want to create the management cluster again, you must start from scratch. If you delete the management cluster without deleting the workload cluster, then the workload cluster and Azure resources will remain.

Summary

Similar to how Kubernetes allows you to orchestrate containers using a declarative syntax, CAPI/CAPZ allows you to do the same, but for Kubernetes clusters in Azure.

This article covered example scenarios for when to use CAPI/CAPZ as well as a walkthrough on how to implement them.

I’m especially excited for the future of CAPI/CAPZ and how it can integrate with other Cloud Native methodologies like GitOps to declaratively manage clusters.

P.S. I am extremely grateful to Cecile Robert Michon’s (Twitter & Github) technical guidance for this article. Without her support, I wouldn’t have gotten this far and definitely would have missed a few key scenarios. Thanks Cecile!

Getting started with Loki and AKS

Tommy Falgout — Fri, 01 Apr 2022 17:02:55 +0000

Searching through application logs is a critical part of any operations team. And as the Cloud Native ecosystem grows and evolves, more modern approaches for this use case are emerging.

The thing about retaining logs is that the storage requirements can get big. REALLY big.

One of the most common log search and indexing tools is ElasticSearch. ElasticSearch is exceptionally good at finding a needle in the haystack (e.g. When did the string "Error message #123" occur in any copy of your application on March 17th). It does this by indexing the contents of the log message which can significantly increase your storage consumption.

The enthusiastic team at Grafana created Loki to address this problem. Instead of indexing the full log message, Loki only indexes the metadata (e.g. label, namespace, etc.) of the log, significantly reducing your storage needs. You can still search for the content of the log messages with LogQL, but it's not indexed.

The UI for Loki is Grafana, which you might already be familiar with if you're using Prometheus.

Getting started with Loki on Azure Kubernetes Service (AKS) is pretty easy. These instructions are inspired by the official Loki Getting Started steps with some modifications streamlined for AKS.

# Set some starter env-vars
AKS_RG=loki-rg
AKS_LOCATION=southcentralus
AKS_NAME=loki-aks

# Create the AKS cluster
az group create -n $AKS_RG -l $AKS_LOCATION
az aks create -n $AKS_NAME -g $AKS_RG
az aks get-credentials -n $AKS_NAME -g $AKS_RG


# Helm update and install
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Create a Helm release of Loki with Grafana + Prometheus using a PVC
# NOTE: This diverges from the Loki docs as it uses storageClassName=default instead of "standard" 
helm upgrade --install loki grafana/loki-stack --namespace grafana --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=default,loki.persistence.size=5Gi

# The Helm installation uses a non-default password for Grafana.  This command fetches it.
# Should look like gtssNbfacGRYZFCa4f3CFmMuendaZzrf9so9VgLh
kubectl get secret loki-grafana -n grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

# Port-forward from the Grafana service (port 80) to your desktop (port 3000)
kubectl port-forward -n grafana svc/loki-grafana 3000:80

# In your browser, go to http://127.0.0.1:3000/
# User: admin
# Password: Output of the "kubectl get secret" command.

Now you're ready to start exploring Loki!

We'll start by using Loki to look at Loki's own logs.

Hover over the "Explore" icon (Looks like a compass)
Select "Loki" from the Data Sources menu
Click "Log Browser", which will open up a panel
Under "1. Select labels to search in", click "app"
Under "2. Find values for the selected labels", click "loki"
Under "3. Resulting selector", click "Show logs"

You should now have a view of the Loki logs as such:

Congrats! You've now created an AKS cluster, deployed Loki and Grafana on it, exposed the Grafana endpoint to your desktop and browsed Loki logs using Loki.

Azure Private Link Service + Load Balancer + AKS Limitation

Tommy Falgout — Mon, 21 Mar 2022 18:26:43 +0000

As a Cloud Solution Architect for Microsoft, I'm privileged to work with some great companies which have unique challenges.

One of our large partners was migrating their solution from AWS to Azure. Their configuration exposes 10+ services inside Azure Kubernetes Service (AKS) to their customer inside a different Azure Tenant and Subscription through Private Link Service and Private Endpoints.

The issue is that at this time is:

A single AKS cluster can only have 1 Internal Standard Load Balancer
A single Load Balancer can only have 8 Private Links

This means that they could expose 8 services, but not the rest of them.

Unfortunately, the feature to enable Multiple LB's is not currently available in AKS.

After talking to other AKS experts, we proposed the following:

Use 1 PLS
Use 1 LB
Specify the SAME IP ADDRESS as part of spec.loadBalancerIP in the Service YAML and use different ports for each service

This allowed them to reduce the number of Private Endpoints, reduce their operational complexity as well as use Kubernetes native Port Mapping with minimal architectural change.

We reviewed this with the partner and after some Helm chart + Terraform work, this met their needs swimmingly.

Mission Accomplished.

Walkthrough of AKS + Private Link Service + Private Endpoint

Tommy Falgout — Fri, 11 Mar 2022 19:50:56 +0000

This walkthrough shows how to setup a Private Link Service with an AKS cluster and create a Private Endpoint in a separate Vnet.

While many tutorials might give you a full ARM template, this is designed as a walkthrough which completely uses the CLI so you can understand what's happening at every step of the process.

It focuses on an "uninteresting" workload and uses podinfo as the sample app. This is because it's easy to deploy and customize with a sample Helm chart.

This is inspired and leans heavily on the Azure Docs for creating a Private Link Service.

If you're interested in more samples, check out my Kubernetes Examples repo in GitHub.

Architecture

Prerequisites

Azure CLI
jq

Assumptions

This walkthrough assumes you let Azure create the Vnet when creating the AKS cluster. If you manually created the Vnet, then the general steps are the same, except you must enter the AKS_MC_VNET, AKS_MC_SUBNET env vars manually.

Setup Steps

First, create a sample AKS cluster and install Podinfo on it.

# Set these values
AKS_NAME=
AKS_RG=
LOCATION=

# Create the AKS cluster
az aks create -n $AKS_NAME -g $AKS_RG

# Get the MC Resource Group
AKS_MC_RG=$(az aks show -n $AKS_NAME -g $AKS_RG | jq -r '.nodeResourceGroup')
echo $AKS_MC_RG

# Get the Vnet Name
AKS_MC_VNET=$(az network vnet list -g $AKS_MC_RG | jq -r '.[0].name')
echo $AKS_MC_VNET

AKS_MC_SUBNET=$(az network vnet subnet list -g $AKS_MC_RG --vnet-name $AKS_MC_VNET | jq -r '.[0].name')
echo $AKS_MC_SUBNET

AKS_MC_LB_INTERNAL=kubernetes-internal

AKS_MC_LB_INTERNAL_FE_CONFIG=$(az network lb rule list -g $AKS_MC_RG --lb-name=$AKS_MC_LB_INTERNAL | jq -r '.[0].frontendIpConfiguration.id')
echo $AKS_MC_LB_INTERNAL_FE_CONFIG

# Deploy a sample app using an Internal LB
helm upgrade --install --wait podinfo-internal-lb \
    --set-string service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"=true \
    --set service.type=LoadBalancer \
    --set ui.message=podinfo-internal-lb \
    podinfo/podinfo

Install Steps - Create the Private Link Service

These steps will be done in the MC_ resource group.

# Disable the private link service network policies
az network vnet subnet update \
    --name $AKS_MC_SUBNET \
    --resource-group $AKS_MC_RG \
    --vnet-name $AKS_MC_VNET \
    --disable-private-link-service-network-policies true


# Create the PLS
PLS_NAME=aks-pls
az network private-link-service create \
    --resource-group $AKS_MC_RG \
    --name $PLS_NAME \
    --vnet-name $AKS_MC_VNET \
    --subnet $AKS_MC_SUBNET \
    --lb-name $AKS_MC_LB_INTERNAL \
    --lb-frontend-ip-configs $AKS_MC_LB_INTERNAL_FE_CONFIG

Install Steps - Create the Private Endpoint

These steps will be done in our private-endpoint-rg resource group.

PE_RG=private-endpoint-rg
az group create \
    --name $PE_RG \
    --location $LOCATION

PE_VNET=pe-vnet
PE_SUBNET=pe-subnet

az network vnet create \
    --resource-group $PE_RG \
    --name $PE_VNET \
    --address-prefixes 10.0.0.0/16 \
    --subnet-name $PE_SUBNET \
    --subnet-prefixes 10.0.0.0/24

# Disable the private link service network policies
az network vnet subnet update \
    --name $PE_SUBNET \
    --resource-group $PE_RG \
    --vnet-name $PE_VNET \
    --disable-private-endpoint-network-policies true


PE_CONN_NAME=pe-conn
PE_NAME=pe
az network private-endpoint create \
    --connection-name $PE_CONN_NAME \
    --name $PE_NAME \
    --private-connection-resource-id $PLS_ID \
    --resource-group $PE_RG \
    --subnet $PE_SUBNET \
    --manual-request false \
    --vnet-name $PE_VNET

# We need the NIC ID to get the newly created Private IP
PE_NIC_ID=$(az network private-endpoint show -g $PE_RG --name $PE_NAME -o json | jq -r '.networkInterfaces[0].id')
echo $PE_NIC_ID

# Get the Private IP from the NIC
PE_IP=$(az network nic show --ids $PE_NIC_ID -o json | jq -r '.ipConfigurations[0].privateIpAddress')
echo $PE_IP

Validation Steps - Create a VM

Lastly, validate that this works by creating a VM in the Vnet with the Private Endpoint.

VM_NAME=ubuntu
az vm create \
    --resource-group $PE_RG \
    --name ubuntu \
    --image UbuntuLTS \
    --public-ip-sku Standard \
    --vnet-name $PE_VNET \
    --subnet $PE_SUBNET \
    --admin-username $USER \
    --ssh-key-values ~/.ssh/id_rsa.pub

VM_PIP=$(az vm list-ip-addresses -g $PE_RG -n $VM_NAME | jq -r '.[0].virtualMachine.network.publicIpAddresses[0].ipAddress')
echo $VM_PIP

# SSH into the host
ssh $VM_IP

$ curl <Copy the value from $PE_IP>:9898

# The output should look like:
$ curl 10.0.0.5:9898
{
  "hostname": "podinfo-6ff68cbf88-cxcvv",
  "version": "6.0.3",
  "revision": "",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "podinfo-internal-lb",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.16.9",
  "num_goroutine": "9",
  "num_cpu": "2"
}

Multiple PLS/PE

To test a specific use case, I wanted to create multiple PLS and PE's. This set of instructions lets you easily loop through and create multiple instances.

# podinfo requires a high numbered port, eg 9000+

SUFFIX=9000
helm upgrade --install --wait podinfo-$SUFFIX \
    --set-string service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"=true \
    --set service.type=LoadBalancer \
    --set service.httpPort=$SUFFIX \
    --set service.externalPort=$SUFFIX \
    --set ui.message=podinfo-$SUFFIX \
    podinfo/podinfo

# This might be easier to hard-code
AKS_MC_LB_INTERNAL_FE_CONFIG=$(az network lb rule list -g $AKS_MC_RG --lb-name=$AKS_MC_LB_INTERNAL -o json | jq -r ".[] | select( .backendPort == $SUFFIX) | .frontendIpConfiguration.id")
echo $AKS_MC_LB_INTERNAL_FE_CONFIG

PLS_NAME=aks-pls-$SUFFIX
PE_CONN_NAME=pe-conn-$SUFFIX
PE_NAME=pe-$SUFFIX

az network private-link-service create \
    --resource-group $AKS_MC_RG \
    --name $PLS_NAME \
    --vnet-name $AKS_MC_VNET \
    --subnet $AKS_MC_SUBNET \
    --lb-name $AKS_MC_LB_INTERNAL \
    --lb-frontend-ip-configs $AKS_MC_LB_INTERNAL_FE_CONFIG

PLS_ID=$(az network private-link-service show \
    --name $PLS_NAME \
    --resource-group $AKS_MC_RG \
    --query id \
    --output tsv)
echo $PLS_ID

az network private-endpoint create \
    --connection-name $PE_CONN_NAME \
    --name $PE_NAME \
    --private-connection-resource-id $PLS_ID \
    --resource-group $PE_RG \
    --subnet $PE_SUBNET \
    --manual-request false \
    --vnet-name $PE_VNET

PE_NIC_ID=$(az network private-endpoint show -g $PE_RG --name $PE_NAME -o json | jq -r '.networkInterfaces[0].id')
echo $PE_NIC_ID

PE_IP=$(az network nic show --ids $PE_NIC_ID -o json | jq -r '.ipConfigurations[0].privateIpAddress')
echo $PE_IP

echo "From your Private Endpoint VM run: curl $PE_IP:$SUFFIX"

Hi, I'm Tommy Falgout

Tommy Falgout — Sat, 14 Jan 2017 16:01:59 +0000

I have been coding for 20+ years.

You can find me on GitHub as lastcoolnameleft

I live in Plano, TX.

I work for Microsoft

Nice to meet you.