Forem: Dariusz Parys

Just Your Commands

Dariusz Parys — Mon, 31 Mar 2025 10:07:15 +0000

Just is a simple command runner. You can get if from https://just.systems. I use it meanwhile regularly, and it improves consistency between my local environment and my CI/CD pipelines. Let me explain this with a trivial walkthrough.

Deploying Infrastructure

A common task is to deploy infrastructure. As I use mostly Azure for infrastructure deployments, I use bicep to do them. For this scenario a simple storage deployment will act as the canonical Hello, World. example. You can find the code for the scripts at my GitHub.

dariuszparys / just-your-commands

Sample source code for the blog post https://www.dariuszparys.com/just-your-commands/

Azure Bicep Deployment with Just

This project uses Just as a command runner to manage Azure Bicep deployments.

Prerequisites

Install Just
Install Azure CLI
Install PowerShell
Log in to Azure CLI:
```
az login
```

Environment Variables

Set the following environment variables before running the deployment:

export AZURE_SUBSCRIPTION_ID="your-subscription-id"
export DEPLOYMENT_ENVIRONMENT="dev"  # Optional, defaults to "dev"
export AZURE_LOCATION="northeurope"  # Optional, defaults to "northeurope"
export DEPLOYMENT_NAME="just-demo"   # Optional, defaults to "just-demo"
export RESOURCE_GROUP_NAME="rg-just-demo"  # Optional, defaults to "rg-just-demo"

Available Commands

Validate Deployment

To validate the Bicep template and see what changes would be made:

just validate-all

This command will:

Lint the Bicep files
Validate the deployment
Show what-if changes

Deploy

To deploy the infrastructure:

just deploy

Clean Up

To delete the deployment and resource group:

just destroy

Additional Commands

just help - Lists all available commands
just lint -…

View on GitHub

The workflow usually consists of these steps to make the deployment

Lint the file az bicep lint --file main.bicep
Validate the script az deployment sub validate --template-file main.bicep --parameters environments/dev.bicepparam
Make a what-if analysis az deployment sub what-if --template-file main.bicep --parameters environments/dev.bicepparam
Deploy to Infrastructure az deployment sub create --template-file main.bicep --parameters environments/dev.bicepparam

This can now be scripted or in this case I use just so I will write recipes to execute those steps.

Simple Just Anatomy

I don't want to explain things that are already explained in the Just manuals, but it helps to have a quick context without digging into this.

environment := env("DEPLOYMENT_ENVIRONMENT", "dev")

# Lists all available recipies
help:
  @just --list

_hello name="world":
  @echo "Hello, {{name}}"

# Deploys infrastructure to defined environment
deploy: _hello
  @echo "Deploying to {{environment}}"

A lot happens here

First, the variable environment gets either assigned an available environment variable called DEPLOYMENT_ENVIRONMENT or if not available, the value dev. So just has functions that can be used in recipes

Second, the first function will be executed when no parameter is applied to the just command. Therefore, I often use help as my first recipe.

❯ just 
Available recipes:
    deploy # Deploys infrastructure to defined environment
    help   # Lists all available recipies

Third, just can have private recipes like _hello which also contains a parameter with a default value

Fourth, deploy leverages the private recipe when called and executes it first without providing any parameter. Therefore, the default is taken.

Please note that comments applied before recipes are used as description in the --list parameter of just and private recipes are not listed.

Calling now just deploy outputs

❯ just deploy
Hello, world
Deploying to dev

My Just Approach

In my working environment and especially in the teams I am working with a lot of people still use Windows. PowerShell is set as the de-facto shell across all operating systems.

PowerShell Shebang

To enable just now to execute all commands using PowerShell I always define at the very top of my justfile

shebang := if os() == 'windows' {
  'pwsh.exe'
} else {
  '/usr/bin/env pwsh'
}
set shell := ["pwsh", "-c"]
set windows-shell := ["pwsh.exe", "-NoProfile", "-Command"]

This will instruct the execution engine to use PowerShell as the execution shell. There are two variables depending on the environment shell and windows-shell that are used for that.

The shebang is used when a recipe will spawn multiple lines and needs to keep the context. If you don't specify this, the commands will be executed in their own environment without knowing anything about the previous line.

# Test without
without:
  Write-Host "Without"
  $some_var = "Without"
  Write-Host $some_var

When executing without you see in the output that the variable is set, but it is not available when writing to console anymore, as each line is in its own context

❯ just without
Write-Host "Without"
Without
$some_var = "Without"
Write-Host $some_var

Using now the shebang defined above, it works

# Test with
with:
  #!{{shebang}}
  Set-PSDebug -Trace 1
  Write-Host "With"
  $some_var = "With"
  Write-Host $some_var
  Set-PSDebug -Off

The output does keep the variable because it executes it with the defined shell environment in one context

 just with
DEBUG:   32+  >>>> Write-Host "With"

With
DEBUG:   33+  >>>> $some_var = "With"

DEBUG:   34+  >>>> Write-Host $some_var

With
DEBUG:   35+  >>>> Set-PSDebug -Off

The First Recipe is Help

I always define this as my very first recipe

# Lists all available recipies
help:
  @just --list

So, when I invoke just every recipe is listed.

❯ just
Available recipes:
    deploy  # Deploys infrastructure to defined environment
    help    # Lists all available recipies
    with    # Test with
    without # Test without

Use Short Recipe Scripts

Using a shebang allows writing scripts as complex as you want. Usually if I tend to use more than one branching path in my script, I extract them into my own PowerShell modules. I invoke them e.g.

module_path := "path/to/psmd"

# Install k3s using Ansible
k3s-install:
  #!{{ shebang }}
  Import-Module "./{{module_path}}" -Force
  Install-K3sCluster -Verbose

No Here Strings

I haven't found a way to use here strings with just. test. For instance, if I have a recipe like this

# Initializes the configuration toml
init:
  #!{{shebang}}
  $toml = @"
[package]
name = "my-project"
version = "0.1.0"
edition = "2021"

[dependencies]
serde = { version = "1.0", features = ["derive"] }
toml = "0.7"
"@

  $toml | Out-File -FilePath "Cargo.toml"
  Write-Host "Created Cargo.toml"

It would just complain about the content inside the here string.

❯ just --list
error: Unknown attribute `package`
 ——▶ justfile:5:2
  │
5 │ [package]
  │  ^^^^^^^

So, I put things like this in an external PowerShell function.

Use Just Recipes in CI/CD

I aim to have my recipes agnostic to the environment. That said, I use them as well in the CI/CD environment. One drawback to using just is that the current GitHub runners or Azure DevOps agent don't have just pre-configure on their environments. So before using them, e.g. in Azure DevOps I have a task installing just

  - task: Bash@3
    inputs:
      targetType: 'inline'
      script: |
        wget -qO - 'https://proget.makedeb.org/debian-feeds/prebuilt-mpr.pub' | gpg --dearmor | sudo tee /usr/share/keyrings/prebuilt-mpr-archive-keyring.gpg 1> /dev/null
        echo "deb [arch=all,$(dpkg --print-architecture) signed-by=/usr/share/keyrings/prebuilt-mpr-archive-keyring.gpg] https://proget.makedeb.org prebuilt-mpr $(lsb_release -cs)" | sudo tee /etc/apt/sources.list.d/prebuilt-mpr.list
        sudo apt update
        sudo apt install -y just
    displayName: 'Install just'

I also only use either AzureCLI@2 or Bash@3 tasks in my piplines (on Azure DevOps). To invoke a recipe e.g.

  - task: Bash@3
    displayName: 'Build APIs'
    inputs:
      targetType: 'inline'
      script: |
        just containerize-apis
    env:
      REGISTRY: $(registry)
      TAG: $(buildTag)

The recipe (simplified version) looks like this

targetRegistry := env("REGISTRY", "localhost:5050")
buildTag := env("TAG", "latest-dev")

# Build all API containers
containerize-apis: (containerize-facilitymanagement targetRegistry) (containerize-storymanagement targetRegistry) (containerize-queuemanagement targetRegistry) (containerize-videostreaming targetRegistry)

# Build Docker container for Queue Management API
containerize-queuemanagement registry:
  #!{{ shebang }}
  docker build `
    --tag {{registry}}/api-queuemanagement:{{buildTag}} `
    --file src/Api.QueueManagement/Dockerfile `
    .

Back Deploying Infrastructure

I started to discuss how I deploy infrastructure using just. When you have a look at the sample justfile I published, you will see that my approach is applied to this. You could argue that destroy is too many lines of code. Yes, this can be extracted to an external script or module. I usually like to use PowerShell modules, but one function is not enough for me to consider this.

shebang := if os() == 'windows' {
  'pwsh.exe'
} else {
  '/usr/bin/env pwsh'
}
set shell := ["pwsh", "-c"]
set windows-shell := ["pwsh.exe", "-NoProfile", "-Command"]

subscription-id := env("AZURE_SUBSCRIPTION_ID")
environment := env("DEPLOYMENT_ENVIRONMENT", "dev")
location := env("AZURE_LOCATION", "northeurope")
deployment-name := env("DEPLOYMENT_NAME", "just-demo")
resourceGroupName := env("RESOURCE_GROUP_NAME", "rg-just-demo")

# Lists all available recipies
help:
    @just --list

# Lint the bicep files
lint:
    az bicep lint --file main.bicep

# Validates the bicep deployment
validate: _ensure-subscription
    #!{{ shebang }}
    az deployment sub validate `
        --name {{deployment-name}} `
        --location {{location}} `
        --template-file main.bicep `
        --parameters environments/{{environment}}.bicepparam `
        --parameters resourceGroupName={{resourceGroupName}}

# Shows what changes would be made by the deployment
what-if: _ensure-subscription
    #!{{ shebang }}
    az deployment sub what-if `
        --name {{deployment-name}} `
        --location {{location}} `
        --template-file main.bicep `
        --parameters environments/{{environment}}.bicepparam `
        --parameters resourceGroupName={{resourceGroupName}}

# Validates and shows what changes would be made by the deployment
validate-all: lint validate what-if

_ensure-subscription:
    az account set --subscription {{subscription-id}}

# Deploys the bicep deployment
deploy: _ensure-subscription
    #!{{ shebang }}
    az deployment sub create `
        --name {{deployment-name}} `
        --location {{location}} `
        --template-file main.bicep `
        --parameters environments/{{environment}}.bicepparam `
        --parameters resourceGroupName={{resourceGroupName}}

# Deletes the deployment and the resource group
destroy: _ensure-subscription
    #!{{ shebang }}
    Write-Host "Deleting deployment..."
    az deployment sub delete `
        --name {{deployment-name}} `
        --no-wait
    Write-Host "Checking if resource group exists..."
    $rgExists = az group exists --resource-group {{resourceGroupName}}
    if ($rgExists -eq "true") {
        Write-Host "Resource group '{{resourceGroupName}}' exists. Deleting..."
        az group delete --resource-group {{resourceGroupName}} --yes --no-wait
        Write-Host "Resource group deletion initiated"
    } else {
        Write-Host "Resource group '{{resourceGroupName}}' does not exist"
    }

The only environment variable you have to provide is AZURE_SUBSCRIPTION_ID and you are good to execute the commands. I assume that you have subscription ownership, else you have to tweak the bicep scripts.

❯ just
Available recipes:
    deploy       # Deploys the bicep deployment
    destroy      # Deletes the deployment and the resource group
    help         # Lists all available recipies
    lint         # Lint the bicep files
    validate     # Validates the bicep deployment
    validate-all # Validates and shows what changes would be made by the deployment
    what-if      # Shows what changes would be made by the deployment

Conclusion

There are many other runners you could use, starting from good old make to a python-based one like tox and many others out there. You can achieve similar things with all of them, I stick to Just. I like it.

Azure Container Registry Cache: Updated Guide with Docker Hub Integration

Dariusz Parys — Wed, 29 Jan 2025 12:20:30 +0000

Some time ago, I wrote an article about Docker Hub rate limits and described ways to mitigate them. Unfortunately, the method I suggested using Azure Container Registry cache no longer works without authentication.

You can find more details about this in the Artifact cache in Azure Container Registry documentation, specifically in the Upstream support section.

Azure Container Registry supports only authenticated pulls to Docker Hub.

How to Enable Caching, the correct way

Let me walk you through enabling the cache. Here are the steps:

Create a free Docker Hub account if you don't have one yet

❗ Check your eligibility for a personal account first. You may need a paid subscription tier for Docker if you're not using it for personal or open source projects.

Create a personal access token in Docker Hub
Store your Docker Hub username and token credentials in Azure KeyVault
Create an Azure Container Registry credential cache to retrieve the KeyVault credentials for Docker Hub
Create an Azure Container Registry caching rule using that credential set

To simplify this process, I've created a GitHub repository with all the necessary automation.

dariuszparys / acr-cache-docker-images

Project Documentation for Bicep Project

Overview

This project configures an Azure Container Registry with Docker Hub credentials stored securely in Azure Key Vault, using managed identities for secure access.

Project Structure

The project is organized into the following directories and files:

/src: Contains the main Bicep file and modules for resource deployment.
/src/modules: Includes reusable Bicep modules for creating and managing Azure resources.
deploy.sh: Script to automate the deployment process.
.env: File for environment variables
README.md: Project documentation and instructions.

Prerequisites

Azure subscription
Azure CLI installed
Bicep CLI installed (if necessary)
Docker Hub account

Deployment

Set environment variables:

export DOCKER_HUB_USERNAME="your-username"
export DOCKER_HUB_PAT="your-personal-access-token"

Configure parameters

Update main.bicepparam to your liking.

Run deployment:

chmod +x deploy.sh
./deploy.sh

Caching Repositories

Once this is set up we can now cache any public repository from docker hub in the Azure Container Registry. The following…

View on GitHub

Bicep Automation

The GitHub repository uses Bicep to automate resource creation in Azure. Before running it, you'll need a personal access token from Docker Hub.

Feel free to choose access permissions that make sense for your needs. Since I'm only working with public repositories, I selected that option.

Example Deployment

To test this, you can either export the variables DOCKER_HUB_USERNAME and DOCKER_HUB_PAT and run the deploy.sh script, or modify the src/main.bicepparam file.

The credentials will be stored in a newly created Azure KeyVault within the example resource group. You can modify the resource group name and location in the parameter file.

After deployment, you can link your public Docker Hub repository to Azure Container Registry by creating a cache with the credentials set from the Bicep deployment.

For example, to cache the public Python repository, you can

az acr cache create \
  --registry acrdemoxtpogsljytqrc \
  --name mypython \
  --source-repo docker.io/library/python \
  --target-repo python \
  --cred-set dockerhub

Once the cache rule is created we can evaluate it in the Azure Portal

To pull new images, simply reference your Azure Container Registry repository wherever you need it—whether in Azure Pipelines, GitHub Actions, or directly in the shell like

❯ az acr login --name acrdemoxtpogsljytqrc
Login Succeeded

~ took 5s
❯ docker pull acrdemoxtpogsljytqrc.azurecr.io/python
Using default tag: latest
latest: Pulling from python
Digest: sha256:d57ec66c94b9497b9f3c66f6cdddc1e4e0bad4c584397e0b57a721baef0e6fdc
Status: Image is up to date for acrdemoxtpogsljytqrc.azurecr.io/python:latest
acrdemoxtpogsljytqrc.azurecr.io/python:latest
~
❯ docker pull acrdemoxtpogsljytqrc.azurecr.io/python:alpine3.20
alpine3.20: Pulling from python
c4f340f38ec0: Download complete
0152682790bb: Download complete
bb4d68a0c460: Download complete
05da92a20133: Download complete
Digest: sha256:9ab3b6ef4afb7582afaa84e97d40a36f192595bb0578561c282cecc22a45de49
Status: Downloaded newer image for acrdemoxtpogsljytqrc.azurecr.io/python:alpine3.20
acrdemoxtpogsljytqrc.azurecr.io/python:alpine3.20

Thanks to the cache rule, the images automatically transfer to your Azure Container Registry and appear in your ACR repository.

Notes

In the shared GitHub project, I use a system-assigned identity, which requires splitting KeyVault creation and KeyVault Access Policies. This setup can be simplified once the corresponding issue is resolved to allow user-assigned managed identities.

I hope this helps!

Cover image by Photo by Guillaume Bolduc / Unsplash

Generate Sequence Diagrams for Azure B2C User Journeys with ChatGPT

Dariusz Parys — Tue, 06 Aug 2024 07:43:04 +0000

Working with Azure B2C is an XML battle, especially when it comes to customizing user journeys using custom policies. A starter pack you can get from GitHub helps you set up your own policy.

Azure B2C custom policy starter pack

The readme is comprehensive and has all the necessary information, but sometimes, it helps to have a flow sequence diagram. Sequence diagrams can be created using Mermaid, given a syntax that can be included in your Markdown documentation. The renderer will then visualize the sequences for you.

Generative AI can help here

There is an excellent way to do this with Generative AI. In my case, I use ChatGPT. The steps are as follows.

First, upload the XML policy files you want to analyze. In my example, I'm interested in the sign-in and sign-up user journey, so I'm only uploading TrustFrameworkBase.xml, TrustFrameworkExtensions.xml, and SignUpOrSignIn.xml.

Next, prompt ChatGPT to generate the sequence diagram based on the uploaded files. In the case I provide here, I used the following prompt:

Create a sequence diagram for the <User Journey Name> user journey in Azure AD B2C. The diagram should integrate the flow of claims directly into the main sequence and use notes and background colors to distinguish between different flows. 

### Sequence Diagram Details:
1. The initial request where the user opens the page and requests the policy.
2. The user decision point where they choose between different options (e.g., Sign Up or Sign In).
3. The detailed flow for each option, including:
    - Collecting claims
    - Validating and transforming claims
    - Interactions with Azure AD for validation and user data retrieval
    - Displaying success messages to the user
4. The token refresh flow if applicable.
5. Use notes to highlight claims collection and validation.
6. Use different background colors to distinguish between different options (e.g., Sign Up and Sign In).

Replace with the user journey I uploaded, in the example Sign In or Sign Up. The chatbot will start to analyze the files, and as a result, I get back this Mermaid sequence diagram.

sequenceDiagram
    participant User
    participant Browser
    participant AzureAD_B2C as Azure AD B2C
    participant IDP as Identity Provider
    participant App as Relying Party Application

    Note over User, Browser: Step 1: Access Sign Up or Sign In page
    User->>Browser: Access Sign Up/Sign In Page
    Browser->>AzureAD_B2C: Request Sign Up/Sign In Page
    AzureAD_B2C->>Browser: Return Sign Up/Sign In Page

    Note over User, Browser: Step 2: Enter credentials or select IDP
    User->>Browser: Enter credentials/select IDP
    Browser->>AzureAD_B2C: Submit credentials/IDP selection

    alt Local Account
        AzureAD_B2C->>AzureAD_B2C: Validate credentials
        AzureAD_B2C->>Browser: Prompt for additional claims (if needed)
        Browser->>User: Display additional claims form
        User->>Browser: Submit additional claims
        Browser->>AzureAD_B2C: Submit additional claims
    else Social IDP
        AzureAD_B2C->>IDP: Redirect to IDP
        IDP->>User: Display IDP login page
        User->>IDP: Enter credentials
        IDP->>AzureAD_B2C: Return user claims
    end

    Note over AzureAD_B2C: Step 4: Process user claims
    AzureAD_B2C->>AzureAD_B2C: Validate and transform claims
    AzureAD_B2C->>App: Send claims to relying party application

    Note over App, User: Step 5: User is authenticated
    App->>User: Welcome Page or Application Landing Page

You can look at the result in GitHub, which supports rendering Mermaid diagrams.

https://gist.github.com/dariuszparys/904d64978f9448f23bfb7698fee0cf6a

As with all generative models, the output may vary depending on how you prompt it and what model you use. It is more about the basic idea that we can transform one format to another, which helps us create documentation we don't have out of the box. Give it a try.

How to Avoid Docker Hub Rate Limits (and Improve Your Workflow)

Dariusz Parys — Tue, 14 May 2024 13:27:35 +0000

Docker Hub has a rate limit that is effectively in place since November 2020 Checking Your Current Docker Pull Rate Limits and Status | Docker. The detailed documentation about this rate limit is described here Docker Hub rate limit | Docker Docs

There is nothing new, but somehow, there is. I recently received an email from Azure stating that I should mitigate the effects of rate limiting by June 30th, 2024.

This date made me think, and I did some investigations. First, I started to look up precisely what the limits are.

Checking Local Rate Limits

If you don't have a docker subscription you pay for, you might hit the docker rate limit, which is for the free account - 200 image pulls per 6 hours. Our anonymous pulls were already just 100 image pulls per 6 hours.

I never log in to Docker Hub, so I run under the anonymous tier. I also never hit this limit locally. Also, I have had builds often on certain days, and 100 pulls per 6 hours seems not too much. It turns out that the pull only counts if the image is not already cached on your system. That is good, as you usually don't wipe your image cache each time.

Retrieving Rate Limits on Your System

Checking my local pulled docker images shows only one python base image

❯ docker images
REPOSITORY   TAG                IMAGE ID       CREATED         SIZE
python       3.11-slim-buster   db841a2e8ab3   11 months ago   120MB

Following the guidance I retrieve with the anonymous bearer token the current limits I have.

❯ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Tue, 07 May 2024 10:31:48 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 94;w=21600
docker-ratelimit-source: 0.0.0.0

I pulled now an additional ubuntu:latest image and re-executed the curl call to the rate limit endpoint. The output states correctly that my limit was reduced by one.

❯ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Tue, 07 May 2024 10:32:19 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 93;w=21600
docker-ratelimit-source: 0.0.0.0

Pulling again an image that is now locally cached, in this case the ubuntu:latest again, doesn't affect the rate limit at all.

❯ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Tue, 07 May 2024 10:32:36 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 93;w=21600
docker-ratelimit-source: 0.0.0.0

Azure & Azure DevOps MS Hosted Agents: Handling Rate Limits

But what about the Azure and Azure DevOps environments with Microsoft-hosted agents? Are they supposed to have the same issue?

I created a simple test pipeline to see how Azure DevOps handles rate limits.

The docker images cached in the Ubuntu:22.04 runner image, can be found here runner-images/images/ubuntu/Ubuntu2204-Readme.md. It doesn't have any of the images cached that I just pulled in the pipeline.

The exciting thing about this pipeline run is that the output of the "Check rate limit" step doesn't show any rate limits.

HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Tue, 07 May 2024 10:05:56 GMT
strict-transport-security: max-age=31536000
docker-ratelimit-source: 13.79.31.179

I did the same for Azure, opened a CLI in the Portal, and entered the same rate limit query.

dariusz [ ~ ]$ TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5429    0  5429    0     0  20653      0 --:--:-- --:--:-- --:--:-- 20721

dariusz [ ~ ]$ curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Tue, 07 May 2024 11:00:31 GMT
strict-transport-security: max-age=31536000
docker-ratelimit-source: 0.0.0.0

I inspected the retrieved token with https://jwt.ms and received the anonymous tier output, with an untouched rate limit of 100.

{
  "alg": "RS256",
  "typ": "JWT",
  "x5c": [
    "MIIEFj...Cx5Q3"
  ]
}.{
  "access": [
    {
      "actions": [
        "pull"
      ],
      "name": "ratelimitpreview/test",
      "parameters": {
        "pull_limit": "100",
        "pull_limit_interval": "21600"
      },
      "type": "repository"
    }
  ],
  "aud": "registry.docker.io",
  "exp": 1715079853,
  "iat": 1715079553,
  "iss": "auth.docker.io",
  "jti": "dckr_jti_CjpzWM-iQQaK_7QXf8wlUkz8kJI=",
  "nbf": 1715079253,
  "sub": ""
}.[Signature]

Based on this claim Testing Dockerhub pull limits in Azure - Docker Community Forums there must have been some agreement that was already known back in 2022 when this was written, and probably this agreement will now run out by end of June 2024.

Starting Rate Limit Mitigation

The rate limits will likely affect future CI pipeline runs. In our case, we have a CI pipeline that builds images on each push to the Azure DevOps git repository using the Azure Container Registry build task.

Current State: CI Pipeline Configuration in Azure DevOps

Below is a snippet from a pipeline step in Azure DevOps to build our image.

- task: AzureCLI@2
displayName: "Build and push container image"
inputs:
  azureSubscription: $(serviceConnection)
  addSpnToEnvironment: true
  workingDirectory: $(projectRoot)
  scriptType: bash
  scriptLocation: inlineScript
  inlineScript: |
    az acr build --registry "${REGISTRY}" --image "${REGISTRY}.azurecr.io/backend:dev-${BUILD_NUMBER}" --image "${REGISTRY}.azurecr.io/backend:dev-latest" --platform linux --file Dockerfile .
env:
  REGISTRY: $(containerRegistryName)
  BUILD_NUMBER: $(Build.BuildNumber)

The corresponding Dockerfile still refers to the base image on the Docker hub.

FROM python:3.11-slim-buster
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir -r /code/requirements.txt
COPY app /code/app
CMD ["uvicorn", "app.main:app", "--host=0.0.0.0", "--port=8000"]

Using Azure Container Registry Caching

As the Azure Container Registry is in place, we changed the behavior using the Azure Container Registry Cache feature.

NOTE: In its basic tier, Azure Container Registry costs nearly as much as the Pro subscription for Docker Hub. We are using Azure Container Registry for our project anyway, so we mitigate the additional costs. Your mileage may vary.

This feature allows the ACR to pull and cache an image from another registry based on your defined rules. In the case of the Python-based images, a rule might look like this.

az acr cache create \
    --registry ${REGISTRY} \
    --name python-base-images \
    --source-repo docker.io/library/python

We changed the Dockerfile to pull from our ACR instead of the Docker hub where the base image originally lives.

ARG REGISTRY=your-acr-registry-name
FROM $REGISTRY.azurecr.io/python:3.11-slim-buster
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir -r /code/requirements.txt
COPY app /code/app
CMD ["uvicorn", "app.main:app", "--host=0.0.0.0", "--port=8000"]

The first time this runs, ACR will pull this image from Docker Hub and cache it under the python repository, as specified in the shell command above.

Once the pipeline runs and pulls the image through the Azure Container Registry, you can evaluate the cache rule and look up the cached repository.

az acr cache list --registry ${REGISTRY}
az acr repository show --name ${REGISTRY} --repository python

From now on, using Azure Container Registry Cache will mitigate rate limits that might affect our pipeline runs on Microsoft-hosted Agents and offer benefits like improved build performance, reduced network traffic, enhanced reliability, and simplified management.

Conclusion: Optimizing Development Workflows in the Face of Rate Limits

Docker Hub rate limits present an opportunity to optimize your development workflow and improve performance.

Take Action

Explore caching strategies: Understand how effective caching can reduce your reliance on Docker Hub.
Leverage your cloud provider's registry: Cloud registries often offer integrated solutions that streamline image management and can boost build speeds.
If you need a high volume of pulls, consider a paid Docker Hub subscription: This ensures reliable access and support.

By proactively addressing Docker Hub's limits, you can enhance the efficiency and resilience of your CI/CD pipelines.