Forem: Paul Yu

Installing VMWare Workstation Pro on Ubuntu Desktop

Paul Yu — Wed, 22 Oct 2025 13:00:00 +0000

This is a guide that will walk you through the steps to install VMWare Workstation Pro on Ubuntu Desktop. I am currently using Ubuntu 24.10 with version 6.11.x of the Linux kernel on a UEFI-based system with secure boot enabled. This guide is based on my experience with this setup and may not apply to all systems.

Download VMWare Workstation Pro

To start, browse to https://support.broadcom.com and login or create an account if you don't already have one.

Once you have logged in, navigate to the free downloads page and search for "VMWare Workstation Pro".

In the search results, click on the VMWare Workstation Pro link then click on VMware Workstation Pro 17.0 for Linux to expand the download options.

Download the 17.6.2 release.

Install VMWare Workstation Pro

With the download complete, open a terminal and navigate to the directory where the installer was downloaded.

Run the following command to make the installer executable.

sudo chmod +x VMware-Workstation-Full-17.6.2-24409262.x86_64.bundle

Install some pre-requisite packages before you attempt to install VMWare Workstation Pro.

sudo apt update
sudo apt install build-essential linux-headers-$(uname -r)

Run the following command to install the VMware bundle.

sudo ./VMware-Workstation-Full-17.6.2-24409262.x86_64.bundle

Manually install virtual machine modules

Run the following command to get the status of the vmware service.

systemctl status vmware

If you see that the service is inactive (dead), you will need to install the vmmon and vmnet modules manually.

As mentioned above, I am using a UEFI-based system with secure boot enabled. So I need to install the modules, sign the files, and enroll the MOK key.

There are a few open-source repositories that provide the modules. I am using the vmware-host-modules by @bytium because it offers a build for the 17.6.2 release using newer kernel versions... Thanks @bytium 🎉

Run the following command to clone the repository and checkout the 17.6.2 branch.

git clone https://github.com/bytium/vm-host-modules.git
cd vm-host-modules
git checkout 17.6.2

Run the following commands to build and install the module.

make
sudo make install

Run the following commands to sign the module, enter a password for the MOK key and import it.

openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -nodes -days 36500 -subj "/CN=VMware/"
sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo -n vmmon)
sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 ./MOK.priv ./MOK.der $(modinfo -n vmnet)
sudo mokutil --import MOK.der

Reboot your system.

Pay close attention to the boot process. You will see a blue screen that says "Press any key to perform MOK management". Press any key to enter the MOK management utility.

In the MOK management utility, select "Enroll MOK" then select "Continue". You will be prompted to enter the password you set when signing the module.

Once you have entered the password, select "Reboot" to reboot your system again.

After your system has rebooted, run the following command to check the status of the vmware service.

systemctl status vmware

You should see that the service is now active (running). At this point you should be good create virtual machines.

Optionally, you can restart the vmware service to verify that all the services are running.

sudo /etc/init.d/vmware restart

You should see output similar to the following.

Stopping VMware services:
   VMware Authentication Daemon                                        done
   Virtual machine monitor                                             done
Starting VMware services:
   Virtual machine monitor                                             done
   Virtual machine communication interface                             done
   VM communication interface socket family                            done
   Virtual ethernet                                                    done
   VMware Authentication Daemon                                        done
   Shared Memory Available                                             done

Configure VMWare Workstation Pro

There are some additional configurations that you should make to VMWare Workstation Pro.

Open a terminal and run the following command to start VMWare Workstation Pro as root.

sudo vmware

Memory settings

In the VMWare Workstation Pro window, click on Edit then Preferences.

Click on Memory, select Fit all virtual machine memory into reserved host RAM under Additional Memory settings, then click Close.

Network settings

By default, VMWare Workstation Pro uses DHCP to assign IP addresses to virtual machines. If you want to use a static IP address, you will need to know network settings for the NAT network.

With VMWare Workstation Pro open as root, click on Edit then Virtual Network Editor.

Select the NAT network and click on NAT Settings.

Here you will see the Subnet IP, Subnet Mask, and Gateway IP. You will need these values to configure static IP addresses for your virtual machines.

Summary

In this guide, you learned how to install VMWare Workstation Pro on Ubuntu Desktop. You also learned how to install the vmmon and vmnet modules manually and sign them for use a UEFI-based system with secure boot enabled. The manual installation, signing, and enrollment of the MOK key may not be necessary for all systems. But if you are in a similar situation, you should be able to get VMWare Workstation Pro up and running with these steps.

Hope this helps and let me know in the comments below if this guide was helpful or if you have any questions.

Peace ✌️

Uninstall VMWare Workstation Pro

If you need to uninstall VMWare Workstation Pro, you can run the following command.

sudo vmware-installer -u vmware-workstation

You will be prompted to keep or remove the configuration files. Type yes to remove the configuration files or quit to cancel the uninstallation process.

Local LLMs: Running Ollama and Open WebUI in Docker on Ubuntu

Paul Yu — Wed, 08 Oct 2025 13:00:00 +0000

Ollama is a popular tool for running large language models (LLMs) locally on your machine. It provides a simple interface to interact with various models without needing an internet connection. Open WebUI is a web-based user interface that allows you to interact with LLMs through a browser.

I am working on an Ubuntu 24.04.3 LTS Desktop machine with decent hardware to run models locally, so my preference is to run Ollama as a local service rather than confining it to a container. This way, Ollama can take full advantage of my machine's capabilities, especially the GPU.

Open WebUI is also an application that I'd like constantly running in the background so I can access it anytime without remembering to start it up. Naturally, running the web application in a container makes sense for this use case.

Getting these two components to work together came with a few challenges. In this post, I'll walk you through the steps I took to set up Ollama as a local service with Open WebUI running in a Docker container.

Prerequisites

To get started, you'll need to have Docker installed on your machine. If you haven't done so already, you can download and install Docker Desktop by following the Get Docker guide. However, I prefer to run Docker Engine directly so I followed this Install Docker Engine guide.

Step 1: Install Ollama

Depending on your operating system, you can follow the instructions on the Ollama Quickstart page to install Ollama. For Ubuntu, I followed the Linux installation instructions. This can take a while as it downloads the necessary files and GPU accelerators if applicable.

I tried this using both AMD and NVIDIA GPUs and they both worked fine. The GPUs were detected automatically by Ollama, which is nice!

Step 2: Configure Ollama

Once Ollama is installed, it runs as a local service on your machine. You can verify that it's running by executing the following command in your terminal:

systemctl status ollama

You should see output indicating that the Ollama service is active and running.

But before we can use it in a way that it can be called from a container, we need to make a small configuration change. By default, Ollama listens on 127.0.0.1 which means it can only accept connections from the local machine.

To verify this, you can run:

ss -tuln | grep 11434

You should see output similar to this:

tcp   LISTEN 0      4096        127.0.0.1:11434      0.0.0.0:*

To allow connections from other containers, we need to change this to listen on 0.0.0.0.

To do this, we need to modify the Ollama service configuration file. Run the following command to open the configuration file in a text editor:

sudo systemctl edit ollama.service

You will see a file that looks something like this:

### Editing /etc/systemd/system/ollama.service.d/override.conf
### Anything between here and the comment below will become the contents of the drop-in file

<HERE_IS_WHERE_YOU_SHOULD_ADD_YOUR_CHANGES>

### Edits below this comment will be discarded

There will be a big blank space where you can add your changes. Add the following lines to the file:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_CONTEXT_LENGTH=32768"

Note the OLLAMA_CONTEXT_LENGTH setting. I plan to use models that support long context lengths, so I set this to 32k. You should adjust this value based on the models you plan to use or omit it if you want to stick with the default. You can always change this in other ways. You can find a lot more helpful tips on configuring this in the Ollama FAQ.

Save and exit the editor. Then, reload the systemd configuration and restart the Ollama service to apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart ollama

You can verify that Ollama is now listening on all interfaces by running:

ss -tuln | grep 11434

You should now see the following output:

tcp   LISTEN 0      4096                *:11434            *:*

Now Ollama is configured to accept connections on all interfaces!

Note: By configuring Ollama to listen on 0.0.0.0, it becomes accessible to your entire local network. This is intentional and beneficial for several reasons:

Allows multiple containerized applications to connect to Ollama (not just Open WebUI)

Enables testing AI applications from other devices on your network

Supports development workflows where you might have multiple services that need LLM access

Makes it easy to experiment with different AI tools and interfaces without reconfiguring Ollama each time

If you're concerned about security, ensure you're on a trusted network or configure your firewall to restrict access to port 11434. For production environments, consider setting up authentication or using a reverse proxy.

To put it back to the way it was, just delete the file /etc/systemd/system/ollama.service.d/override.conf and restart the Ollama service.

Finally, run the following command to pull a model to use with Open WebUI:

ollama pull gpt-oss:20b

This will download the gpt-oss:20b model to your local machine. I chose this model because it's a capable open-source model that runs well on consumer hardware while providing good performance for general-purpose tasks. Feel free to use any other model available in the Ollama model registry such as llama3, mistral, or gemma based on your hardware capabilities and use case.

Step 3: Run Open WebUI in a Docker Container

Now that Ollama is set up and running the way we want, we can proceed to run Open WebUI in a Docker container.

Run the following command to start the Open WebUI container:

docker run -d \
-p 9090:8080 \
--name open-webui \
--restart unless-stopped \
--add-host=host.docker.internal:host-gateway \
-v $HOME/.open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:v0.6.33

This command does the following:

-d: Runs the container in detached mode (in the background).
-p 9090:8080: Maps port 8080 in the container to port 9090 on your host machine.
--name open-webui: Names the container "open-webui".
--restart unless-stopped: Ensures the container restarts automatically unless explicitly stopped.
--add-host=host.docker.internal:host-gateway: Adds a host entry to allow the container to access services running on the host.
-v $HOME/.open-webui:/app/backend/data: Mounts a volume to persist data so that your settings and chat history are saved even if the container is removed.
ghcr.io/open-webui/open-webui:v0.6.33: Specifies the image to use for the container. You can check for the latest version on the Open WebUI releases page.

After initially running the container, run the following command to check the status of the container:

docker ps -a | grep open-webui

Wait until the STATUS column shows Up and (healthy) before proceeding.

Once you've confirmed the container is running, you can access the Open WebUI interface by navigating to http://localhost:9090 in your web browser.

Proceed to set up Open WebUI by following the on-screen instructions.

Step 4: Use Open WebUI with Ollama

At this point you should be on an empty Open WebUI interface. To connect it to your local Ollama service, click the Select a model drop down at the top left of the page. If all went well, you should see a list of models available from your local Ollama installation.

Select the model you pulled earlier (e.g., gpt-oss:20b) and start interacting with it through the web interface!

When you get the notification from Open WebUI that a new version is available, all you need to do is stop the container, and re-run the docker run command from Step 3 with the new version number.

Alternative options

Another option to run the Open WebUI container and avoid making changes to the Ollama service is to run the container with the following command:

docker run -d \
--network=host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434/ \
-v $HOME/.open-webui:/app/backend/data \
--name open-webui \
--restart unless-stopped \
ghcr.io/open-webui/open-webui:v0.6.33

This command uses the --network=host option to allow the container to share the host's network stack, making it easier to connect to services running on the host machine. The -e OLLAMA_BASE_URL=http://127.0.0.1:11434/ environment variable is set to point to the Ollama service running on the host.

While this approach is simpler, I chose the primary method (exposing Ollama on 0.0.0.0) for better container isolation. With --network=host, the Open WebUI container can access all network services on my host machine, including services that might be running on localhost that weren't intended to be exposed. By using bridge networking with --add-host=host.docker.internal:host-gateway, the container can only access services that are explicitly listening on the network, giving me more control over what the container can reach. This follows the principle of least privilege - the container only has access to what it needs (Ollama) rather than unrestricted access to my entire host network.

Conclusion

In this post, we walked through the steps to set up Ollama and Open WebUI using Docker containers on a Linux machine. By configuring Ollama to listen on all interfaces and running Open WebUI in a container, we created a flexible and accessible environment for interacting with large language models locally.

This Ollama setup also allows you to build and test your own AI applications against Ollama models running locally.

Cleanup

When you want to stop and remove the Open WebUI container, you can run:

docker rm open-webui --force

Note that this removes the container but preserves your data in $HOME/.open-webui. Your chat history, settings, and configurations will remain intact if you decide to run the container again later.

If you want to completely remove all Open WebUI data as well, run:

rm -rf $HOME/.open-webui

To stop the Ollama service, you can run:

sudo systemctl stop ollama

Or to disable it from starting automatically on boot:

sudo systemctl disable ollama

Feel free to explore different models and configurations to suit your needs. Happy chatting!

References

Build a Weather MCP Server with Rust: Complete Tutorial for AI Integration

Paul Yu — Tue, 10 Jun 2025 13:00:00 +0000

Building AI tools that can access real-time data requires bridging the gap between language models and external APIs. The Model Context Protocol (MCP) makes this possible by providing a standard way for AI assistants to interact with data sources.

In this tutorial, we'll build a weather MCP server using Rust that connects to the National Weather Service API, giving any MCP-compatible AI assistant the ability to fetch live weather alerts and forecasts.

This walkthrough uses the soon-to-be-released rust-sdk also known as the rmcp crate, and builds on examples from the rust-sdk MCP server examples and the official quickstart guides for MCP server developers.

Prerequisites

Before you begin, ensure you have the following installed:

Rust and Cargo installed on your machine
Node.js 22+ and npm installed for running the MCP inspector.

If you are just getting started with MCP, I recommend checking out the Model Context Protocol (MCP) for Beginners course and join the community on the Microsoft AI Foundry Discord server or the Microsoft AI Foundry GitHub Discussions.

Project setup

Create a new Rust project:

cargo new weather
cd weather

Add these dependencies to your Cargo.toml:

rmcp: The MCP SDK for Rust.
tokio: For asynchronous runtime and I/O operations.
serde: For serializing and deserializing data structures.
serde_json: For JSON serialization and deserialization.
anyhow: For error handling.
tracing: For logging and diagnostics.
tracing-subscriber: For subscribing to tracing events and filtering logs.
reqwest: For making HTTP requests to the National Weather Service API.

You can add these dependencies manually to your Cargo.toml file, but I like to use the cargo add command to add them easily.

cargo add rmcp --features server,transport-io
cargo add tokio --features macros,rt-multi-thread
cargo add serde --features derive
cargo add serde_json
cargo add anyhow
cargo add tracing
cargo add tracing-subscriber --features env-filter
cargo add reqwest --features json

After running the commands above, here is how your Cargo.toml should look:

[package]
name = "weather"
version = "0.1.0"
edition = "2024"

[dependencies]
anyhow = "1.0.98"
reqwest = { version = "0.12.19", features = ["json"] }
rmcp = { version = "0.1.5", features = ["server", "transport-io"] }
serde = { version = "1.0.219", features = ["derive"] }
serde_json = "1.0.140"
tokio = { version = "1.45.1", features = ["macros", "rt-multi-thread"] }
tracing = "0.1.41"
tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }

Open the project in your favorite code editor.

Building the weather server

The MCP server will expose two tools:

get_alerts: Returns weather alerts for a given state.
get_forecast: Returns the weather forecast for a given location (latitude and longitude coordinates).

Open the src/main.rs file and add the following code to the top. This will import the necessary crates and modules for building the MCP server.

use anyhow::Result;
use reqwest;
use rmcp::{
    ServerHandler, ServiceExt,
    model::{ServerCapabilities, ServerInfo},
    schemars, tool,
    transport::stdio,
};
use tracing_subscriber::{self, EnvFilter};

const NWS_API_BASE: &str = "https://api.weather.gov";
const USER_AGENT: &str = "weather-app/1.0";

The src/main.rs file includes a main function that you will implement later to run the server. Leave it as is for now and add code above the main function.

Testing NWS API endpoints

To retrieve weather data, the server will make HTTP requests to the National Weather Service (NWS) API.

This RESTful API has several endpoints that allow you to access weather data without requiring an API key. All that is required is to set a user agent in the HTTP request headers.

The following endpoints will be used in this project:

Alerts Endpoint: https://api.weather.gov/alerts/active?area={state} - This endpoint returns active weather alerts for a given state.
Points Endpoint: https://api.weather.gov/points/{latitude},{longitude} - This endpoint returns the forecast URL for a specific latitude and longitude.
Forecast Endpoint: https://api.weather.gov/gridpoints/{office}/{gridX},{gridY}/forecast - This endpoint returns the weather forecast for a specific grid point.

To test these endpoints manually, you can use the curl command or any HTTP client to make requests to the NWS API.

For example, to get weather alerts for a state, you can use the following command to make a request to the alerts endpoint:

curl "https://api.weather.gov/alerts/active?area=CA"

To get the weather forecast, you would need to make a request to the points endpoint for a specific location. In the response, you will receive a forecast URL which you can use to get the forecast data.

For example, to get the forecast for Los Angeles, you can use the following command:

curl -L "https://api.weather.gov/points/34.0499998,-118.249999"

The NWS API redirects precise latitude and longitude to a standardized grid point covering a larger region. Use the -L flag to follow this redirect and access the normalized endpoint.

Within the points response, you will find a forecast field that contains the URL for the forecast data.

Use the forecast URL to get the forecast data:

curl "https://api.weather.gov/gridpoints/LOX/155,45/forecast"

Within the forecast response, you will find a properties field that contains the forecast data, which includes an array of periods with details about the weather forecast for different times of the day.

Modeling the weather data

To work with the weather data returned by the NWS API, define Rust structs that match the structure of the JSON data returned by the API. This will allow the server to deserialize the JSON data into Rust types using the serde crate.

Add the following code to the src/main.rs file to define the structs for the weather data:

#[derive(Debug, serde::Deserialize)]
pub struct AlertResponse {
    pub features: Vec<Feature>,
}

#[derive(Debug, serde::Deserialize)]
pub struct Feature {
    pub properties: FeatureProps,
}

#[derive(Debug, serde::Deserialize)]
pub struct FeatureProps {
    pub event: String,
    #[serde(rename = "areaDesc")]
    pub area_desc: String,
    pub severity: String,
    pub status: String,
    pub headline: String,
}

#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct PointsRequest {
    #[schemars(description = "latitude of the location in decimal format")]
    pub latitude: String,
    #[schemars(description = "longitude of the location in decimal format")]
    pub longitude: String,
}

#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct PointsResponse {
    pub properties: PointsProps,
}

#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct PointsProps {
    pub forecast: String,
}

#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct GridPointsResponse {
    pub properties: GridPointsProps,
}

#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct GridPointsProps {
    pub periods: Vec<Period>,
}

#[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
pub struct Period {
    pub name: String,
    pub temperature: i32,
    #[serde(rename = "temperatureUnit")]
    pub temperature_unit: String,
    #[serde(rename = "windSpeed")]
    pub wind_speed: String,
    #[serde(rename = "windDirection")]
    pub wind_direction: String,
    #[serde(rename = "shortForecast")]
    pub short_forecast: String,
}

These structs represent the data returned by the NWS API for weather alerts and forecasts.

Adding helper functions

The server will return weather alerts and forecasts in a human-readable format. To achieve this, add the following helper functions to format the data.

fn format_alerts(alerts: &[Feature]) -> String {
    if alerts.is_empty() {
        return "No active alerts found.".to_string();
    }

    let mut result = String::with_capacity(alerts.len() * 200);

    for alert in alerts {
        result.push_str(&format!(
            "Event: {}\nArea: {}\nSeverity: {}\nStatus: {}\nHeadline: {}\n---\n",
            alert.properties.event,
            alert.properties.area_desc,
            alert.properties.severity,
            alert.properties.status,
            alert.properties.headline
        ));
    }
    result
}

fn format_forecast(periods: &[Period]) -> String {
    if periods.is_empty() {
        return "No forecast data available.".to_string();
    }

    let mut result = String::with_capacity(periods.len() * 150);

    for period in periods {
        result.push_str(&format!(
            "Name: {}\nTemperature: {}°{}\nWind: {} {}\nForecast: {}\n---\n",
            period.name,
            period.temperature,
            period.temperature_unit,
            period.wind_speed,
            period.wind_direction,
            period.short_forecast
        ));
    }
    result
}

Implementing the weather tools

Add the following code to define a Weather struct that will hold the HTTP client used to make requests to the NWS API.

#[derive(Debug, Clone)]
pub struct Weather {
    client: reqwest::Client,
}

Next, implement the Weather struct with a constructor that initializes the HTTP client with a user agent.

#[tool(tool_box)]
impl Weather {
    #[allow(dead_code)]
    pub fn new() -> Self {
        let client = reqwest::Client::builder()
            .user_agent(USER_AGENT)
            .build()
            .expect("Failed to create HTTP client");
        Self { client }
    }
}

This code creates a new instance of the Weather struct with an HTTP client that has the user agent set to weather-app/1.0. The client is a reusable instance that will be used to make requests to the NWS API.

As demonstrated in the section above, there will be a few HTTP requests made to the NWS API. To make the code cleaner and more reusable, create a make_request function within the Weather struct.

async fn make_request<T>(&self, url: &str) -> Result<T, String>
where
    T: serde::de::DeserializeOwned,
{
    tracing::info!("Making request to: {}", url);

    let response = self
        .client
        .get(url)
        .send()
        .await
        .map_err(|e| format!("Request failed: {}", e))?;

    tracing::info!("Received response: {:?}", response);

    match response.status() {
        reqwest::StatusCode::OK => response
            .json::<T>()
            .await
            .map_err(|e| format!("Failed to parse response: {}", e)),
        status => Err(format!("Request failed with status: {}", status)),
    }
}

This function will handle making HTTP GET requests and deserializing the JSON response into the specified type. It takes a URL as input, makes an HTTP GET request to that URL, and returns the deserialized response as the specified type T. If the request fails or the response cannot be parsed, it returns an error message.

Add the following code to implement the get_alerts tool. This function retrieves weather alerts for a specified US state. It constructs the URL for the NWS API, makes the request, and formats the alerts into a human-readable string.

#[tool(description = "Get weather alerts for a US state")]
async fn get_alerts(
    &self,
    #[tool(param)]
    #[schemars(description = "the US state to get alerts for")]
    state: String,
) -> String {

    tracing::info!("Received request for weather alerts in state: {}", state);

    let url = format!("{}/alerts/active?area={}", NWS_API_BASE, state);

    match self.make_request::<AlertResponse>(&url).await {
        Ok(alerts) => format_alerts(&alerts.features),
        Err(e) => {
            tracing::error!("Failed to fetch alerts: {}", e);
            "No alerts found or an error occurred.".to_string()
        }
    }
}

Next, add the following code to implement the get_forecast tool. This function retrieves the weather forecast for a specific location using latitude and longitude coordinates. As demonstrated in the section above, it first makes a request to the points endpoint to get the forecast URL, then uses that URL to retrieve the actual forecast data.

#[tool(description = "Get forecast using latitude and longitude coordinates")]
async fn get_forecast(
    &self,
    #[tool(aggr)] PointsRequest {
        latitude,
        longitude,
    }: PointsRequest,
) -> String {
    tracing::info!(
        "Received coordinates: latitude = {}, longitude = {}",
        latitude,
        longitude
    );

    let points_url = format!("{}/points/{},{}", NWS_API_BASE, latitude, longitude);

    let points_result = self.make_request::<PointsResponse>(&points_url).await;

    let points = match points_result {
        Ok(points) => points,
        Err(e) => {
            tracing::error!("Failed to fetch points: {}", e);
            return "No forecast found or an error occurred.".to_string();
        }
    };

    match self
        .make_request::<GridPointsResponse>(&points.properties.forecast)
        .await
    {
        Ok(forecast) => format_forecast(&forecast.properties.periods),
        Err(e) => {
            tracing::error!("Failed to fetch forecast: {}", e);
            "No forecast found or an error occurred.".to_string()
        }
    }
}

Finally, add the following code to implement the ServerHandler trait for the Weather struct. This trait is part of the rmcp crate and allows the server to define its capabilities and instructions for clients.

#[tool(tool_box)]
impl ServerHandler for Weather {
    fn get_info(&self) -> ServerInfo {
        ServerInfo {
            instructions: Some("A simple weather forecaster".into()),
            capabilities: ServerCapabilities::builder().enable_tools().build(),
            ..Default::default()
        }
    }
}

This code provides basic server information and capabilities, indicating that the server supports tools.

Finally, implement the main function to run the server. Replace the existing main function in src/main.rs with the following code:

#[tokio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt()
        .with_env_filter(EnvFilter::from_default_env().add_directive(tracing::Level::DEBUG.into()))
        .with_writer(std::io::stderr)
        .with_ansi(false)
        .init();

    tracing::info!("Starting MCP server");

    let service = Weather::new().serve(stdio()).await.inspect_err(|e| {
        tracing::error!("serving error: {:?}", e);
    })?;

    service.waiting().await?;

    Ok(())
}

This code initializes logging, creates the Weather service, and starts the MCP server using stdio transport.

A reference to the final src/main.rs file can be found here: src/main.rs.

Run the following commands to format the code and build the project:

cargo fmt
cargo build

If all goes well, you should see no errors, and the project will build successfully.

Testing with MCP Inspector

The Model Context Protocol Inspector is handy web tool for testing MCP servers. Run the following command to start the MCP Inspector:

npx @modelcontextprotocol/inspector cargo run

Once the MCP Inspector is started, navigate to http://127.0.0.1:6274 in your web browser, connect to your MCP server, and test the tools.

Summary

You now have a working MCP server that provides weather data from the National Weather Service API. The server handles the multi-step API calls (points → forecast URL → actual forecast) and formats the data for AI consumption.

From here, you could add error handling improvements, caching, or support for additional NWS endpoints. Check out the Rust SDK examples for other patterns and features.

MCP is changing the way AI assistants can access real-time data, and building servers like this one is a great way to get started. As mentioned above, if you are new to MCP, I highly recommend you work through the Model Context Protocol (MCP) for Beginners course and join the community on the Microsoft AI Foundry Discord server or the Microsoft AI Foundry GitHub Discussions.

Learn more

Deploying AKS Automatic clusters with Pulumi: A step-by-step guide

Paul Yu — Mon, 10 Mar 2025 13:00:00 +0000

Yo! Let's build an AKS Automatic cluster with Pulumi 🚀

If you don't already know, Pulumi is a modern Infrastructure as Code (IaC) tool that allows you to use your favorite programming language to deploy and manage cloud resources. I like it because I can use Go and the Pulumi Go SDK to write my infrastructure code. There are other languages supported like Python, TypeScript, .NET, and more so be sure to check out their docs for more information.

This is fairly long article, so let's dive right in.

Prerequisites

Before we start, you need to have an Azure subscription and the following tools installed:

Azure CLI version 2.67.0 or later
Pulumi CLI version 3.143.0 or later
Go version 1.23.4 or later
VSCode version 1.96.2 or later
Go extension for VSCode version 0.45.0 or later

Login to Azure and Pulumi

As mentioned above, I'll be using the Go SDK and using Pulumi's Azure Native provider to deploy Azure resources and will need Azure credentials to authenticate. I'll use the Azure CLI to authenticate because it's the easiest way to get started, but there are other ways to authenticate with Azure including OpenID Connect, Service Principal, and Managed Identity.

Open a terminal and run the following command to login to Azure.

az login --use-device-code

Pulumi uses its own cloud-based service to store state information by default. You can setup a an account here, but if you don't want to create a Pulumi account, you can store state information locally in a file. I prefer this for quick demos and getting started as it's just easier for me and I don't have to worry about setting up or logging into an account. In a production scenario, you should consider using the cloud-based service or another backend options. Refer to the Pulumi documentation for more information.

Run the following command to login to Pulumi using a local file.

pulumi login file://~/.pulumi

Initialize Pulumi project

Now that we got the prerequisites out of the way, let's create a new Pulumi project for our AKS deployment.

Run the following command to create a new folder for the project.

mkdir aks-pulumi-demo
cd aks-pulumi-demo

Run the following command to create a new Pulumi project.

pulumi new azure-go \
--name aks-pulumi-demo \
--description "sample aks deployment with pulumi" \
--stack dev \
--secrets-provider default

Here is a breakdown of the command:

pulumi new azure-go creates a new Pulumi project using the Azure Go template
--name sets the name of the project to aks-pulumi-demo
--description sets the description of the project
--stack sets the stack name to dev; think of stack as a way to manage different environments like dev, staging, and production
--secrets-provider sets the secrets provider to default which uses the local file to store secrets; you can use other providers like default, passphrase, awskms, azurekeyvault, gcpkms, or hashivault

You will be prompted to enter a passphrase to encrypt the secrets. You can leave it blank for now and hit enter to proceed.

You will also be prompted to select a region to deploy the resources. Just be careful here and make sure you set the location to a region that supports all the Azure resources you intend to deploy. You can refer to the Azure documentation for more information on product availability by region. I'll use swedencentral for this demo.

As noted in the Azure docs, AKS Automatic tries to dynamically select a virtual machine SKU for the system node pool based on the capacity available in the subscription. Make sure your subscription has quota for 16 vCPUs of any of the following SKUs in the region you're deploying the cluster to: Standard_D4pds_v5, Standard_D4lds_v5, Standard_D4ads_v5, Standard_D4ds_v5, Standard_D4d_v5, Standard_D4d_v4, Standard_DS3_v2, Standard_DS12_v2. You can view quotas for specific VM-families and submit quota increase requests through the Azure portal.

You can also set the location by running the following command.

pulumi config set azure-native:location swedencentral

Setup main.go file

At this point you should have a new Go project with a main.go file and some sample code.

Here is what the directory structure should look like:

.
├── go.mod
├── go.sum
├── main.go
├── Pulumi.dev.yaml
└── Pulumi.yaml

1 directory, 5 files

Open the folder with VSCode and open the main.go file. Here you will see sample code to create an Azure storage account and export its keys. Delete the sample code and replace it with the code below to create an Azure resource group.

package main

import (
    "github.com/pulumi/pulumi-azure-native-sdk/resources/v2"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create an Azure Resource Group
        resourceGroup, err := resources.NewResourceGroup(ctx, "resourceGroup", nil)
        if err != nil {
            return err
        }

        return nil
    })
}

This code will import the Pulumi SDK and Pulumi Azure Native SDK to create resource groups and sets up the main function to make a pulumi.Run function call to run the Pulumi program. This function accepts a RunFunc type which is where you define the body of the Pulumi program to run.

You've just added code to create an Azure resource group using the resources.NewResourceGroup function. This function is available via the resources module in the Azure Native SDK. The NewResourceGroup function accepts a pulumi.Context and a string to set the name of the resource group object.

It is important to note that the value of resourceGroup in the NewResourceGroup function above is not the actual name of the resource group that will be created in Azure. It's the name of the object within the Pulumi state file. More on that in a bit...

If you are using VSCode with the Go extension, you might notice a squiggly line under the resourceGroup variable. That's because Go does not like it when variables are declared but not used. To fix this, we can export the value of the resource group by adding the following code just before the return nil line inside the pulumi.Run function. This will allow us to see the actual resource group name once the deployment is complete.

Add the following code to the code fille.

ctx.Export("resourceGroupName", pulumi.All(resourceGroup.Name).ApplyT(
    func(args []interface{}) string {
        return args[0].(string)
    }))

Running Pulumi programs

We now have a basic Pulumi program that creates an Azure resource group and exports the value of the resource group name. Let's deploy the program and see the output.

Save the main.go file then open the VSCode terminal and run the following command to provision the infrastructure.

pulumi up

I like to deploy as I go, but you can also run pulumi preview to see what resources will be created without actually creating them. This is useful for checking your work before deploying.

You will be prompted to enter a passphrase to decrypt the secrets. If you left it blank earlier, you can just hit enter to proceed.

Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your passphrase to unlock config/secrets
Previewing update (dev):
     Type                                     Name                 Plan
 +   pulumi:pulumi:Stack                      aks-pulumi-demo-dev  create
 +   └─ azure-native:resources:ResourceGroup  resourceGroup        create

Outputs:
    resourceGroupName: output<string>

Resources:
    + 2 to create

Within a few seconds you will be prompted to perform the update. Use the arrow keys to select yes and hit enter to confirm you want to deploy Azure resources.

Do you want to perform this update?  [Use arrows to move, type to filter]
> yes
  no
  details

Once the deployment is complete, you will see the outputs of the resources that were created. Here is what the output should look like:

Updating (dev):
     Type                                     Name                 Status
 +   pulumi:pulumi:Stack                      aks-pulumi-demo-dev  created (1s)
 +   └─ azure-native:resources:ResourceGroup  resourceGroup        created (0.93s)

Outputs:
    resourceGroupName: "resourceGroup3f007f94"

Resources:
    + 2 created

Duration: 3s

Randomizing resource names

Did you notice that the resource group name was randomized by Pulumi? It did so by taking the name of the resourceGroup object and appending some random text to the end. If you have naming conventions that you like to follow you will want to set the actual resource group name yourself. To do this, you can pass in a resources.ResourceGroupArgs struct to the NewResourceGroup function. This struct has fields that you can set to control the resource group name and other properties of the resource group.

I don't have strict naming conventions to follow so I often randomize resource names myself to meet global naming requirements for some Azure resources. To implement my own randomization, I'll use Pulumi's random provider to generate a random integer then pass in the name that we want the resource group name to use.

Add the following import statement to the top of the file to import the random package.

"github.com/pulumi/pulumi-random/sdk/v4/go/random"

Pulumi docs are your friend. You can find the documentation for any resource and find the latest version of the provider to use by viewing the code sample on the provider's page. For example, the RandomInteger resource page will show you the proper import path to use in the example code block.

Next, add this code to the top of the pulumi.Run function, just above the resource group creation block.

// Create a 4 digit random integer to be used for resource names
randomInt, _ := random.NewRandomInteger(ctx, "randomInt", &random.RandomIntegerArgs{
    Min: pulumi.Int(1000),
    Max: pulumi.Int(9999),
})

Now we have a random integer that we can use to generate unique resource names.

Update the resource group creation block to include a new resources.ResourceGroupArgs struct that will set the ResourceGroupName field using the random integer to create a unique name. Here is the updated code to create the resource group.

// Create an Azure Resource Group
resourceGroup, err := resources.NewResourceGroup(ctx, "resourceGroup", &resources.ResourceGroupArgs{
    ResourceGroupName: pulumi.Sprintf("rg-akspulumidemo%v", randomInt.Result),
})
if err != nil {
    return err
}

Save the main.go file then run the following command in a terminal to install any missing dependencies.

pulumi install

Now run the following command to update the infrastructure.

pulumi up

You can run export PULUMI_CONFIG_PASSPHRASE="" to avoid entering the passphrase every time you run pulumi up.

After you confirm the changes, you will see the output of the deployment and see that the original resource group was replaced with a new one since renaming resources is "not a thing" in Azure. Here is what the output should look like:

Previewing update (dev):
     Type                                     Name                 Plan        Info
     pulumi:pulumi:Stack                      aks-pulumi-demo-dev
 +   ├─ random:index:RandomInteger            randomInt            create
 +-  └─ azure-native:resources:ResourceGroup  resourceGroup        replace     [diff: ~resourceGroupName]

Resources:
    + 1 to create
    +-1 to replace
    2 changes. 1 unchanged

Do you want to perform this update? yes
Updating (dev):
     Type                                     Name                 Status              Info
     pulumi:pulumi:Stack                      aks-pulumi-demo-dev
 +   ├─ random:index:RandomInteger            randomInt            created (0.00s)
 +-  └─ azure-native:resources:ResourceGroup  resourceGroup        replaced (16s)      [diff: ~resourceGroupName]

Outputs:
  ~ resourceGroupName: "resourceGroup3f007f94" => "rg-akspulumidemo8622"

Resources:
    + 1 created
    +-1 replaced
    2 changes. 1 unchanged

Duration: 19s

Great job! We now have a resource group with a custom and unique name. Let's move on to creating an a few more resources before we deploy the AKS cluster.

Deploy a container registry

With the resource group in place, we can now add other resources based on the workload requirements. I don't have any specific requirements for this demo, but I'll add an Azure Container Registry to show you how you can attach the registry to the AKS cluster.

Add the following import statement to the top of the file to import the azure-native.containerregistry package.

"github.com/pulumi/pulumi-azure-native-sdk/containerregistry/v2"

Next, add the following code in the main function to create an Azure Container Registry and export the name of the registry.

// Create Azure Container Registry
containerRegistry, err := containerregistry.NewRegistry(ctx, "containerRegistry", &containerregistry.RegistryArgs{
    ResourceGroupName: resourceGroup.Name,
    RegistryName:      pulumi.Sprintf("acrakspulumidemo%v", randomInt.Result),
    Sku: &containerregistry.SkuArgs{
        Name: pulumi.String("Standard"),
    },
})
if err != nil {
    return err
}

ctx.Export("containerRegistry", pulumi.All(containerRegistry.Name).ApplyT(
    func(args []interface{}) string {
        return args[0].(string)
    }))

Whether you keep all the resource creation code together and all the export code together is up to you. I like to keep the resource creation code together and the export code together so I can easily see what resources are being created and what values are being exported. Feel free to organize the code in a way that makes sense to you.

Deploy observability tools

You may also want to add observability resources like Azure Log Analytics Workspace for application logging, Azure Monitor Workspace for Prometheus metrics, and Azure Managed Grafana to visualize resources in your AKS cluster.

Add Azure Log Analytics Workspace

Add the following import statement to the top of the file to import the azure-native.operationalinsights package.

"github.com/pulumi/pulumi-azure-native-sdk/operationalinsights/v2"

Next, add the following code to create an Azure Log Analytics Workspace and export the name of the workspace.

// Create Azure Log Analytics Workspace for Container Insights
logAnalyticsWorkspace, err := operationalinsights.NewWorkspace(ctx, "logAnalyticsWorkspace", &operationalinsights.WorkspaceArgs{
    ResourceGroupName: resourceGroup.Name,
    RetentionInDays:   pulumi.Int(30),
    Sku: &operationalinsights.WorkspaceSkuArgs{
        Name: pulumi.String("PerGB2018"),
    },
    WorkspaceName: pulumi.Sprintf("law-akspulumidemo%v", randomInt.Result),
})
if err != nil {
    return err
}

ctx.Export("logAnalyticsWorkspace", pulumi.All(logAnalyticsWorkspace.Name).ApplyT(
    func(args []interface{}) string {
        return args[0].(string)
    }))

This code will create an Azure Log Analytics Workspace with a retention period of 30 days and export the name of the workspace. We'll also randomize the name of the workspace to meet Azure's global naming requirements.

Add Azure Managed Prometheus

Add the following import statement to the top of the file to import the azure-native.monitor package.

"github.com/pulumi/pulumi-azure-native-sdk/monitor/v2"

Now add the following code to create an Azure Monitor Workspace and export the name of the workspace.

// Create Azure Monitor Workspace for managed Prometheus
azureMonitorWorkspace, err := monitor.NewAzureMonitorWorkspace(ctx, "azureMonitorWorkspace", &monitor.AzureMonitorWorkspaceArgs{
    ResourceGroupName:         resourceGroup.Name,
    AzureMonitorWorkspaceName: pulumi.Sprintf("prom-akspulumidemo%v", randomInt.Result),
})
if err != nil {
    return err
}

ctx.Export("azureMonitorWorkspace", pulumi.All(azureMonitorWorkspace.Name).ApplyT(
    func(args []interface{}) string {
        return args[0].(string)
    }))

This code will create an Azure Monitor Workspace for managed Prometheus and export the name of the workspace.

Add Azure Managed Grafana

Add the following import statement to the top of the file to import the azure-native.dashboard package.

"github.com/pulumi/pulumi-azure-native-sdk/dashboard/v2"

Then add the following code to create an Azure Managed Grafana resource with Azure Monitor Workspace integration, and export the name of the dashboard.

// Create Azure Managed Grafana with Azure Monitor Workspace integration
grafanaDashboard, err := dashboard.NewGrafana(ctx, "grafanaDashboard", &dashboard.GrafanaArgs{
    Identity: &dashboard.ManagedServiceIdentityArgs{
        Type: pulumi.String("SystemAssigned"),
    },
    Properties: dashboard.ManagedGrafanaPropertiesArgs{
        ApiKey:              pulumi.String("Enabled"),
        PublicNetworkAccess: pulumi.String("Enabled"),
        GrafanaIntegrations: dashboard.GrafanaIntegrationsArgs{
            AzureMonitorWorkspaceIntegrations: dashboard.AzureMonitorWorkspaceIntegrationArray{
                &dashboard.AzureMonitorWorkspaceIntegrationArgs{
                    AzureMonitorWorkspaceResourceId: azureMonitorWorkspace.ID(),
                },
            },
        },
    },
    ResourceGroupName: resourceGroup.Name,
    Sku: &dashboard.ResourceSkuArgs{
        Name: pulumi.String("Standard"),
    },
    WorkspaceName: pulumi.Sprintf("graf-akspulumidemo%v", randomInt.Result),
})
if err != nil {
    return err
}

ctx.Export("grafanaDashboard", pulumi.All(grafanaDashboard.Name).ApplyT(
    func(args []interface{}) string {
        return args[0].(string)
    }))

This code will create an Azure Managed Grafana with a system-assigned managed identity and Azure Monitor Workspace integration and export the name of the dashboard.

Provision Azure resources

That should do it for the observability resources. Let's save the file and run the program to deploy what we have so far.

Run the following commands to update the infrastructure.

pulumi install
pulumi up

Once the deployment is complete, you will see the outputs of the new resources that were added. Here is what the output should look like:

Previewing update (dev):
     Type                                           Name                   Plan
     pulumi:pulumi:Stack                            aks-pulumi-demo-dev
 +   ├─ azure-native:monitor:AzureMonitorWorkspace  azureMonitorWorkspace  create
 +   ├─ azure-native:containerregistry:Registry     containerRegistry      create
 +   ├─ azure-native:dashboard:Grafana              grafanaDashboard       create
 +   └─ azure-native:operationalinsights:Workspace  logAnalyticsWorkspace  create

Outputs:
  + azureMonitorWorkspace: output<string>
  + containerRegistry    : output<string>
  + grafanaDashboard     : output<string>
  + logAnalyticsWorkspace: output<string>

Resources:
    + 4 to create
    3 unchanged

Do you want to perform this update? yes
Updating (dev):
     Type                                           Name                   Status
     pulumi:pulumi:Stack                            aks-pulumi-demo-dev
 +   ├─ azure-native:containerregistry:Registry     containerRegistry      created (11s)
 +   ├─ azure-native:operationalinsights:Workspace  logAnalyticsWorkspace  created (14s)
 +   ├─ azure-native:monitor:AzureMonitorWorkspace  azureMonitorWorkspace  created (12s)
 +   └─ azure-native:dashboard:Grafana              grafanaDashboard       created (139s)

Outputs:
  + azureMonitorWorkspace: "prom-akspulumidemo4266"
  + containerRegistry    : "acrakspulumidemo4266"
  + grafanaDashboard     : "graf-akspulumidemo4266"
  + logAnalyticsWorkspace: "law-akspulumidemo4266"
    resourceGroupName    : "rg-akspulumidemo4266"

Resources:
    + 4 created
    3 unchanged

Duration: 2m35s

Deploy AKS Automatic cluster

Now, the moment you've been waiting for! Let's add the AKS Automatic cluster.

Add the following import statement to the top of the file to import the azure-native.containerservice package.

"github.com/pulumi/pulumi-azure-native-sdk/containerservice/v2"

Next, add the following code to create an Azure Kubernetes Service Automatic cluster and export the name of the cluster.

// Create AKS Automatic cluster
managedCluster, err := containerservice.NewManagedCluster(ctx, "managedCluster", &containerservice.ManagedClusterArgs{
    ResourceGroupName: resourceGroup.Name,
    ResourceName:      pulumi.Sprintf("aks-%v", randomInt.Result),
    Sku: &containerservice.ManagedClusterSKUArgs{
        Name: pulumi.String("Automatic"),
        Tier: pulumi.String("Standard"),
    },
    AgentPoolProfiles: containerservice.ManagedClusterAgentPoolProfileArray{
        &containerservice.ManagedClusterAgentPoolProfileArgs{
            Mode:   pulumi.String("System"),
            Name:   pulumi.String("systempool"),
            VmSize: pulumi.String("Standard_D4pds_v6"),
            Count:  pulumi.Int(3),
        },
    },
    AddonProfiles: containerservice.ManagedClusterAddonProfileMap{
        "omsagent": &containerservice.ManagedClusterAddonProfileArgs{
            Enabled: pulumi.Bool(true),
            Config: pulumi.StringMap{
                "logAnalyticsWorkspaceResourceID": logAnalyticsWorkspace.ID(),
                "useAADAuth":                      pulumi.String("true"),
            },
        },
    },
    AzureMonitorProfile: containerservice.ManagedClusterAzureMonitorProfileArgs{
        Metrics: containerservice.ManagedClusterAzureMonitorProfileMetricsArgs{
            Enabled: pulumi.Bool(true),
            KubeStateMetrics: containerservice.ManagedClusterAzureMonitorProfileKubeStateMetricsArgs{
                MetricAnnotationsAllowList: pulumi.String(""),
                MetricLabelsAllowlist:      pulumi.String(""),
            },
        },
    },
    Identity: &containerservice.ManagedClusterIdentityArgs{
        Type: containerservice.ResourceIdentityTypeSystemAssigned,
    },
})
if err != nil {
    return err
}

ctx.Export("aksName", pulumi.All(managedCluster.Name).ApplyT(
    func(args []interface{}) string {
        return args[0].(string)
    }))

This code will create an AKS Automatic cluster with the following configuration:

Sets the SKU and Tier to Automatic and Standard respectively
Adds a system pool with 3 nodes of size Standard_D4pds_v6
Enables the Azure Monitor Addon with Log Analytics Workspace integration for Container Insights
Enables the Azure Monitor Profile with metrics enabled for Prometheus

As noted above, AKS Automatic can automatically select the VM size based on its requirements and SKU availability in the region you selected. If you have specific VM requirements, you can set the VmSize field to the desired SKU but it's not required for AKS Automatic.

Let's save the file and run the following command to update the infrastructure.

pulumi install
pulumi up

After about 10-15 minutes, the deployment should be complete and you will see the output of the AKS cluster name. Here is what the output should look like:

Previewing update (dev):
     Type                                             Name                 Plan
     pulumi:pulumi:Stack                              aks-pulumi-demo-dev
 +   └─ azure-native:containerservice:ManagedCluster  managedCluster       create

Outputs:
  + aksName              : output<string>

Resources:
    + 1 to create
    7 unchanged

Do you want to perform this update? yes
Updating (dev):
     Type                                             Name                 Status
     pulumi:pulumi:Stack                              aks-pulumi-demo-dev
 +   └─ azure-native:containerservice:ManagedCluster  managedCluster       created (767s)

Outputs:
  + aksName              : "aks-4266"
    azureMonitorWorkspace: "prom-akspulumidemo4266"
    containerRegistry    : "acrakspulumidemo4266"
    grafanaDashboard     : "graf-akspulumidemo4266"
    logAnalyticsWorkspace: "law-akspulumidemo4266"
    resourceGroupName    : "rg-akspulumidemo4266"

Resources:
    + 1 created
    7 unchanged

Duration: 12m50s

Deploy role assignments

At this point, all the resources are in place, but now we need to add a bunch of role permissions like AcrPull for the Azure Container Registry, Azure Monitor Reader for the Azure Monitor Workspace, and give your user access to the AKS cluster and Azure Managed Grafana.

Add the following import statement to the top of the file to import the azure-native.authorization package.

"github.com/pulumi/pulumi-azure-native-sdk/authorization/v2"

Grant cluster permissions for pulling images

Granting an AKS cluster access to an Azure Container Registry has always been a little tricky. Since the kubelet is the one pulling images from the registry, we need to assign the AcrPull role to the kubelet's principal ID which is available once the AKS cluster is created.

Add the following code to retrieve the kubelet's principal ID.

// Get the kubelet's principal ID for AcrPull role assignment
kubeletPrincipalId := managedCluster.IdentityProfile.MapIndex(pulumi.String("kubeletidentity")).ObjectId()

Next add the following code to create the AcrPull role assignment for the kubelet.

// Create a role assignment so that the kubelet can pull images from ACR
_, err = authorization.NewRoleAssignment(ctx, "kubeletRoleAssignment", &authorization.RoleAssignmentArgs{
    PrincipalId:      kubeletPrincipalId.Elem().ToStringOutput(),
    RoleDefinitionId: pulumi.String("/providers/Microsoft.Authorization/roleDefinitions/7f951dda-4ed3-4680-a7ca-43fe172d538d"),
    Scope:            containerRegistry.ID(),
    PrincipalType:    pulumi.String("ServicePrincipal"),
})
if err != nil {
    return err
}

Grant permissions for monitoring

In order to view cluster metrics from the Azure Managed Grafana, the Monitoring Reader role must be assigned to the system-assigned managed identity of the Azure Managed Grafana resource.

Add the following code to create the Azure Monitor Reader role assignment.

// Create a role assignment so that Azure Managed Grafana can query the Azure Monitor Workspace and Log Analytics Workspace
_, err = authorization.NewRoleAssignment(ctx, "azureMonitorWorkspaceRoleAssignment1", &authorization.RoleAssignmentArgs{
    PrincipalId:      grafanaDashboard.Identity.Elem().PrincipalId(),
    RoleDefinitionId: pulumi.String("/providers/Microsoft.Authorization/roleDefinitions/43d0d8ad-25c7-4714-9337-8ba259a9fe05"),
    Scope:            resourceGroup.ID(),
    PrincipalType:    pulumi.String("ServicePrincipal"),
})
if err != nil {
    return err
}

Grant yourself permissions for access

Finally, you'll need to grant yourself access to the AKS cluster and Azure Managed Grafana.

Add the following code to retrieve the your user principal ID.

// Get current user principal
client, err := authorization.GetClientConfig(ctx, pulumi.CompositeInvoke())
if err != nil {
    return err
}

The AKS Automatic cluster authorization is managed by Microsoft Entra ID, so we need to assign the Azure Kubernetes Service RBAC Cluster Admin role to your user principal ID. Without this, you would not be able to run kubectl commands against the AKS cluster.

// Create a role assignment so I can access the kubeapiserver
_, err = authorization.NewRoleAssignment(ctx, "managedClusterRoleAssignment", &authorization.RoleAssignmentArgs{
    PrincipalId:      pulumi.String(client.ObjectId),
    RoleDefinitionId: pulumi.String("/providers/Microsoft.Authorization/roleDefinitions/b1ff04bb-8a4e-4dc4-8eb5-8693973ce19b"),
    Scope:            managedCluster.ID(),
    PrincipalType:    pulumi.String("User"),
})
if err != nil {
    return err
}

Finally, give yourself access to the Azure Monitor Workspace and Azure Managed Grafana.

// Create a role assignment so I can query the Azure Monitor Workspace
_, err = authorization.NewRoleAssignment(ctx, "azureMonitorWorkspaceRoleAssignment2", &authorization.RoleAssignmentArgs{
    PrincipalId:      pulumi.String(client.ObjectId),
    RoleDefinitionId: pulumi.String("/providers/Microsoft.Authorization/roleDefinitions/43d0d8ad-25c7-4714-9337-8ba259a9fe05"),
    Scope:            azureMonitorWorkspace.ID(),
    PrincipalType:    pulumi.String("User"),
})
if err != nil {
    return err
}

// Create a role assignment so I can access Azure Managed Grafana dashboards
_, err = authorization.NewRoleAssignment(ctx, "grafanaRoleAssignment", &authorization.RoleAssignmentArgs{
    PrincipalId:      pulumi.String(client.ObjectId),
    RoleDefinitionId: pulumi.String("/providers/Microsoft.Authorization/roleDefinitions/22926164-76b3-42b3-bc55-97df8dab3e41"),
    Scope:            grafanaDashboard.ID(),
    PrincipalType:    pulumi.String("User"),
})
if err != nil {
    return err
}

Provision role assignments

Now that we have all the role assignments in place, let's save the file and run the following command to update the infrastructure one last time.

pulumi install
pulumi up

After a few seconds, the deployment should be complete and you will see the output of the role assignments. Here is what the output should look like:

Previewing update (dev):
     Type                                          Name                                  Plan
     pulumi:pulumi:Stack                           aks-pulumi-demo-dev
 +   ├─ azure-native:authorization:RoleAssignment  azureMonitorWorkspaceRoleAssignment1  create
 +   ├─ azure-native:authorization:RoleAssignment  kubeletRoleAssignment                 create
 +   ├─ azure-native:authorization:RoleAssignment  managedClusterRoleAssignment          create
 +   ├─ azure-native:authorization:RoleAssignment  azureMonitorWorkspaceRoleAssignment2  create
 +   └─ azure-native:authorization:RoleAssignment  grafanaRoleAssignment                 create

Resources:
    + 5 to create
    8 unchanged

Do you want to perform this update? yes
Updating (dev):
     Type                                          Name                                  Status
     pulumi:pulumi:Stack                           aks-pulumi-demo-dev
 +   ├─ azure-native:authorization:RoleAssignment  azureMonitorWorkspaceRoleAssignment1  created (3s)
 +   ├─ azure-native:authorization:RoleAssignment  kubeletRoleAssignment                 created (4s)
 +   ├─ azure-native:authorization:RoleAssignment  grafanaRoleAssignment                 created (3s)
 +   ├─ azure-native:authorization:RoleAssignment  azureMonitorWorkspaceRoleAssignment2  created (3s)
 +   └─ azure-native:authorization:RoleAssignment  managedClusterRoleAssignment          created (3s)

Outputs:
    aksName              : "aks-4266"
    azureMonitorWorkspace: "prom-akspulumidemo4266"
    containerRegistry    : "acrakspulumidemo4266"
    grafanaDashboard     : "graf-akspulumidemo4266"
    logAnalyticsWorkspace: "law-akspulumidemo4266"
    resourceGroupName    : "rg-akspulumidemo4266"

Resources:
    + 5 created
    8 unchanged

Duration: 7s

Test the deployment

Now that you've given yourself the proper permissions to the Azure resources, you can test access to the AKS cluster using the kubectl command line tool.

If you don't have kubectl installed, you can to install it using the az aks install-cli command.

To configure kubectl to use the AKS cluster, run the following command.

export PULUMI_CONFIG_PASSPHRASE="" #<-- set this to whatever you set when you created the stack
az aks get-credentials -n $(pulumi stack output aksName) -g $(pulumi stack output resourceGroupName)

You can now run kubectl commands against the AKS cluster. For example, you can run the following command to get the status of the nodes in the cluster.

kubectl cluster-info

You should be presented with a URL to authenticate with Microsoft Entra ID. Once you authenticate, you can run the following command to get the status of the nodes in the cluster.

From there, feel free to browse to the Azure Portal and check out the resources that were created. You can also access the Azure Managed Grafana dashboard to view the metrics of the AKS cluster.

What's next?

This was a long but simple example of deploying an AKS Automatic cluster with Pulumi. You can extend this example by adding more resources like Prometheus data collection endpoints, rule groups, and alerts. I won't cover that in this article because it would make this article way too long, but you can view the code sample in my GitHub repo here.

Clean up

After you're done with the resources, delete them by running the following command.

pulumi destroy

After the resources are deleted, remove the stack by running the following command.

pulumi stack rm dev

Finally, remove the Pulumi project by running the following command.

rm -rf ~/.pulumi/.pulumi/stacks/aks-pulumi-demo*

Summary

In this article, I showed you how to deploy an AKS Automatic cluster with Pulumi. I used the Pulumi Go SDK and the Azure Native provider to create Azure resources like resource groups, Azure Container Registry, Azure Log Analytics Workspace, Azure Monitor Workspace, and Azure Managed Grafana.

Working with Pulumi to deploy Azure resources is a good alternative to using the other IaC tools out there like Terraform and Azure Bicep. Pulumi allows you to use your favorite programming language to create resources for your workload. The pattern of creating resources and exporting their values is the same for all resources as shown in the example above.

It is also worth nothing that Terraform also offers a Cloud Development Kit (CDK) that allows you to use your favorite programming language to create resources. Let me know in the comments if you would like to see a similar example using Terraform CDK.

Until then check out some of the resources below and let me know if you have any questions

Happy coding!

Resources

Certified Argo Project Associate (CAPA) Exam Study Guide

Paul Yu — Thu, 06 Feb 2025 13:00:00 +0000

I recently passed the Certified Argo Project Associate (CAPA) exam and wanted to share a study guide to help you prepare for the exam. If you are not already aware, the Cloud Native Computing Foundation (CNCF) offers a suite of certifications that validate your knowledge and expertise in cloud-native technologies, with Argo being one of them.

The CAPA exam is designed for beginners who are new to Argo and its suite of tools which includes Argo Workflows, Argo CD, Argo Rollouts, and Argo Events.

But be warned... even though it says it's for beginners, it can be a bit intimidating if you've never worked with these tools before. So in this study guide, I'll try to cover key topics you should be familiar and include some resources and practical exercises to help you prepare for the exam

Good news is the exam is multiple choice so it's not as rigorous as the Kubernetes exams where you have to perform tasks in a live server environment; however, you are expected to have a good bit of understanding on how to use these tools including detailed knowledge of some of their Custom Resource Definition (CRD) specifications.

You have 90 minutes to complete the exam, and when you pass (I know you will ☺️), the certification is valid for 2 years. If you don't pass on the first try... no sweat, you do get one free retake 😅

As documented on the CAPA exam page, here is a breakdown of the exam topics and the percentage of questions you can expect from each section:

Let's now dig into each of the topics and provide high-level summaries of what you should know for each section. I'll also include some resources and practical exercises to help you prepare for the exam 💪

Argo Workflows 36%

Argo Workflows is an open-source cloud-native workflow engine for orchestrating jobs on Kubernetes. Like all the other projects in the Argo suite it's implemented as an extension to Kubernetes following the Operator pattern and provides a powerful way to define complex workflows using YAML.

Understand Argo Workflow Fundamentals

When you install Argo Workflows, you get a Workflow Controller which is reponsible for reconciling the desired state of the Workflow CRD and an Argo Server which is responsible for serving the API requests.

Here is a high-level overview of the Argo Workflow architecture:

Source: https://argo-workflows.readthedocs.io/en/latest/architecture/

The Workflow CRD is the primary resource you will interact with to define your workflows. It allows you to define a sequence of tasks and dependencies between tasks. It also is reponsible for storing the state of the workflow.

The Workflow spec contains the following key attributes:

entrypoint is the name of the template that will be executed first
templates is a list of templates that define the tasks in the workflow

Check out this example of a basic workflow with some defaults.

The template is where tasks are defined to be run in the workflow. A template can be one of the following task types:

container is a task that runs a container. This is useful when you want to run a task that is already containerized.
script is a task that gives you a bit of flexibility to run a script inside a container. This is useful when you need to run a script but don't want to create a separate container image.
resource is a task that allows you to perform operations on cluster resources. This can be used to get, create, apply, delete, replace, or patch resources. This is useful when you need to interact with Kubernetes resources.
suspend is a task that allows you to pause the workflow until it is manually resumed. This is useful when you need to wait for a manual approval or intervention.
plugin is a task that allows you to run an external plugin. This is useful when you need to run a task that is not supported by the built-in templates.
containerset is a task that allows you to run multiple containers within a single pod. This is useful when you need to run multiple containers that share the same network namespace, IPC namespace, and PID namespace.
http is a task that allows you to make HTTP requests. This is useful when you need to interact with external services.

Be sure to read through each of the template type field references to understand how to define and take a look at some of the examples to see how they are used.

You can also control the order in which tasks are run with template invocators. This defines the execution flow of the workflow and there are two types of invocators:

steps is a of tasks that are executed in order. Steps can be nested to create a hierarchy of tasks and outer steps are run sequentially while inner steps are run in parallel. However, you can get more control over the execution of the inner steps by using the synchronization field.
dag is a directed acyclic graph of templates that are executed based on dependencies. This is useful when you need to run tasks in parallel and define dependencies between tasks; especially when you tasks that rely on the output of other tasks.

Generating and Consuming Artifacts

Artifacts can be consumed by tasks or be outputs of tasks which in turn can be consumed by other tasks.

Check out this example of a workflow that generates and consumes artifacts

To output an artifact, you can use the outputs.artifacts field in the task template. This field is a list of artifacts that includes the name of the artifact and the path to the artifact.

To consume an artifact, you can use the inputs.artifacts field in the task template. Like the outputs field, this field is a list of artifacts that includes the name of the artifact and the path to the artifact. If you are consuming an artifact that is an output of another task, you can reference the artifact by the name of the task that produced it along with the name of the artifact.

There are also ways to reduce the amount of data that is stored in the workflow by using artifact repositories and garbage collection.

Finally, you can get a bit more advanced control on when to output an artifact by using conditional artifacts based on expressions.

Understand Argo Workflow Templates

Workflow Templates are reusable templates that can be used to define Workflows. They are useful when you have a common set of tasks that you want to reuse across multiple workflows typically scoped to a namespace. They build on the same concepts as Workflows but are defined as a separate resource.

Check out this example of workflow templates and this example of how workflow templates can be referenced in workflow definitions.

Note how the Workflow in the sample calls the random-fail-template which is defined in the templates.yaml file.

There is also a concept of cluster scoped workflow templates with the ClusterWorkflowTemplate resource. This is useful when you want to define a template that can be used across multiple namespaces.

Understand the Argo Workflow Spec

There will be questions on the Workflow spec so be sure to read through the documentation to understand the various fields that can be used to configure the behavior of the workflow. Some of the key fields to pay attention to include:

Also, be sure to understand how to configure variables in the workflow spec and how to configure metrics which will be useful when you want to monitor the performance of the workflow and its tasks.

Finally, it is important to understand how to configure retries, retry strategy, and the retry policies that are available to you to handle task failures, because they will eventually fail 😉

Work with DAG (Directed-Acyclic Graphs)

DAG is important to understand as it allows you to define dependencies between tasks in a workflow. Tasks without dependencies (steps) are run in parallel while tasks with dependencies are run sequentially. If you have tasks that rely on the output of other tasks, you'll want to use a DAG.

Check out this sample workflow that uses a DAG.

Pay special attention to how tasks within a DAG workflow are executed based on dependencies. You may be faced with questions on what outputs would look like based on the dependencies defined in the workflow.

Run Data Processing Jobs with Argo Workflows

Data processing jobs are common use cases for Argo Workflows. You can use Argo Workflows to process data from various sources, transform the data, and store the results.

Check out the data processing and data sourcing and transformation documentation to understand how to configure workflows to process data.

Data sources essentially point to artifact repositories and can be external like s3 buckets. The data fetched from these sources can be further transformed using expressions.

Check out this example of a workflow that processes data

Practice

To see Argo Workflows in action, follow the quick start guide to deploy a simple workflow that processes data.

Argo CD 34%

Argo CD is a declarative, continuous delivery tool for Kubernetes that enables you to implement GitOps principles. It allows you to manage the full lifecycle of applications in a Kubernetes cluster using Git repositories as the source of truth.

Understand Argo CD Fundamentals

To understand how you can best leverage Argo CD, you need to be familiar with basic concepts around containerized applications, Kubernetes, and tools commonly used to deploy and manage applications like Helm and Kustomize.

To quickly get up to speed, make sure you read through the Understanding The Basics section of the Argo CD documentation.

Here are some of the basic core concepts you should be familiar with:

Application is the collection of Kubernetes resources that represent your application. It is represented by an Application CRD in Argo CD and commits to a Git repository.
Application source type is the tool used to manage the application like Helm or Kustomize
Target state is the desired state of the application that is defined in the Git repository
Live state is the actual state of the application in the Kubernetes cluster
Sync is the action to applying the target state to the live state
Sync status indicates whether the target state and live state are in sync
Sync operation status indicates the status of the last sync operation
Refresh is the action of comparing the live state with the target state

Here is the high-level architecture of Argo CD. It too is implemented as an extension to Kubernetes following the Operator pattern.

Source: https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/

One of the major components of Argo CD is the API Server which allows various external systems to interact with it like the CLI, UI, and CI/CD systems. It also takes care of ensuing communication between Git repositories and the Kubernetes cluster and acts as the security gatekeeper for the cluster.

The second major component is the Repository Server which is responsible for syncing the target state of the application with the live state in the Kubernetes cluster.

Lastly, the Application Controller is responsible for managing the lifecycle of applications in the cluster.

Synchronize Applications with Argo CD

Argo CD can keep your applications in sync automatically with the desired state defined in the Git repository. You can control how the synchronization happens using various sync options which can be configured via resource annotations or via the spec.syncPolicy.syncOptions attribute in the Application CRD. Some of the options include controlling how the sync is performed, what resources are ignored, and how to handle out-of-sync situations like pruning resources or not pruning resources, performing selective syncs, server-side apply, and more. So be sure you understand how to configure these options.

If you need to control the order in which applications are synced, you can use sync waves. This is useful when you have dependencies between applications and need to ensure that they are synced in a specific order.

If you need to control when a sync can occur, you can use sync windows to define a time window when a sync can or cannot occur. This is useful when you need to prevent a sync from happening during a maintenance window or when you need to ensure that a sync happens during a specific time.

You can also control what resources get synced by using a selective sync or apply special annotations to resources to ignore them.

Lastly, synchronization can be controlled using resource hooks which gives you the flexibility to run custom logic at various stages of the synchronization process like pre-sync, post-sync, pre-sync failure, and post-sync failure.

Knowing when to hook into the synchronization process is important so be sure to read through the documentation to understand how to use resource hooks.

Use Argo CD Application

Before we get into the Argo CD Application, you should know that Argo Projects are used to group applications and manage access control. This is useful when you need to manage multiple applications and control who has access to them. Argo CD comes with a default project called default and most beginners will deploy applications to this project. However, as your team grows and you have more applications, you may want to create additional projects to manage access control and restrict where applications can be deployed from, what applications can be deployed, and what clusters and namespaces they can be deployed to.

Check out this example of how to define a project and the various fields that can be configured.

You can also work with Projects and most other Argo CD resources using the argocd CLI or the Argo CD UI.

The Application resource is the primary resource you will interact with to define your applications that Argo CD will manage. Two key attributes to pay attention to include:

source is the source of the application that points to the Git repository
destination is the destination of the application that points to the Kubernetes cluster

Check out this example of an application to see how it is defined and the various fields that can be configured and this list of annotations and labels that can be used to control how the application is managed.

If you need to pass parameters to your application, you can use parameters to define them in the application spec. This is useful when you need to pass configuration values to your application that are stored in the Git repository.

Typically your application will be sourced from a single repo; however, you can also source your application from multiple sources.

Here is a sample of how to define multiple sources in an application.

If you are managing applications that needs to be deployed across multiple clusters, you can use the ApplicationSet resource which is managed by the ApplicationSet controller. This feature extends the capabilities of the Application resource and focuses on cluster administration scenarios including multi-cluster application management, multi-tenancy, and more.

Prior to Argo CD v2.3, this did require a separate installation but now it is included in the main Argo CD installation.

The ApplicationSet resource gives you powerful tools to template your applications using the Templates field and the ability to use Go Templates to generate the unique values for fields.

It is also worth noting that you should be a bit familiar with the Generators that are available to you when using the ApplicationSet resource. This includes the List, Git, Helm, and Kustomize generators that can be used to generate the resources that will be deployed to the cluster.

In some advanced scenarios like cluster bootstrapping, you may need to use an App of Apps pattern that can be used to nest applications within a parent application. This is useful when you need to create applications that in turn create other applications.

Prior to Argo CD v2.5, applications were managed out of the argocd namespace. Since then Argo CD administrators now have the ability to define namespaces where applications can be deployed to. To enable this app in any namespace feature checkout the namespace-based configuration documentation to see which flags must be set.

Deleting an application is a common task that you will need to perform. When you delete an application, you can use either the argocd CLI or kubectl and with both, you have control on whether or not you want cascading deletion of resources. This is useful when you need to control how resources are deleted when an application is deleted. Also be sure to understand how the deletion finalizer can be used with the App of Apps pattern.

Configure Argo CD with Helm and Kustomize

In the section above, you saw how Application resources can be sourced from Helm and Kustomize.

When using Helm to define your application it is important to know that Argo CD inflates the Helm chart and stores the resources in the Kubernetes cluster. So don't expect to see the Helm installation using the typical Helm commands. Be sure to take a look though the Helm guide to understand how to pass in parameter values, configure hooks, and even install custom Helm plugins.

For Kustomize based applications, be sure to read through the Kustomize guide to understand how to configure your application using patches, components, and build options.

Argo CD supports multiple versions of kustomize. See this sample on how to set the version of kustomize to use but note that you must also have the kustomize versions set in the ConfigMap.

Lastly, you can use kustomize to customize a Helm chart deployment by enabling the --enable-helm flag. This is useful when you need to customize a Helm chart deployment using kustomize patches.

Identify Common Reconciliation Patterns

Argo CD uses a reconciliation loop to ensure that the target state of the application matches the live state in the Kubernetes cluster. This loop is responsible for detecting changes in the target state, comparing it with the live state, and taking the necessary actions to ensure that the target state is applied to the live state.

Be sure to read through the reconciliation optimization documentation to understand how to optimize the reconciliation process and improve the performance of Argo CD.

You can also configure rate limits on application reconciliations to control the rate at which applications are reconciled and configure retry and backoff strategies. This is useful when you need to prevent the reconciliation loop from consuming too many resources.

Practice

To see Argo CD in action, follow the quick start guide to deploy a simple application and manage its lifecycle.

Argo Rollouts 18%

Argo Rollouts is a tool that enables you to manage and automate the deployment of applications on Kubernetes. It takes the concept of Kubernetes Deployment to the next level by providing advanced deployment strategies and features that help you achieve more controlled and reliable application updates.

Like all the other Argo projects it is implemented as a Kubernetes controller with its own set of CRDs that provide advanced deployment capabilities such as blue-green deployments, canary deployments, and progressive delivery.

Understand Argo Rollouts Fundamentals

In the context of software development, Continuous Integration (CI), Continuous Delivery (CD), and Progressive Delivery (PD) are three key practices that help teams deliver software more efficiently and reliably.

CI is the process of integrating code changes in a shared repository and making sure it can be built and tested with the intent of detecting issues early.
CD is the ability to deploy apps to production, reliably, confidently, and on demand.
PD is the evolution of CD that focuses on the gradual and controlled delivery of new features to users; which ultimately reduces the risk of deploying new features. In addition to gradual delivery, PD also gives you the ability to implement feature flags, A/B testing, and phased rollouts

Argo Rollouts enables PD by providing a new controller called the Argo Rollouts Controller to manage the lifecycle of Pods and ReplicaSets.

Source: https://argoproj.github.io/argo-rollouts/architecture/

The Rollout resource is the primary resource that you will interact with. This can be a drop in replacement for the Deployment resource or used alongside it.

The ReplicaSet is the same as the one used by the Deployment controller. The difference is that the Rollout controller will manage the ReplicaSet and its Pods.

Argo Rollouts also has the ability to integrate with Ingress, Service and/or service meshes to manage traffic routing and direct traffic to the new versions of an application.

The traffic can be managed manually using the UI or CLI or automatically using the Analysis feature. This feature allows you to integrate with various metrics providers to monitor and analyze the performance of the new version of an application. If the metrics do not meet the defined thresholds, the Rollout controller will automatically rollback to the previous version.

Use Common Progressive Rollout Strategies

The most commonly used progressive rollout strategies within Argo Rollouts Blue-Green and Canary.

With Blue-Green, both the old and new versions of an application are deployed simultaneously. Once testing is complete, traffic is switched to the new version and the old version is deleted.

Source: https://argoproj.github.io/argo-rollouts/concepts/#blue-green

With Canary, a new version of an application is deployed alongside the old version. Only a subset of users will be directed to the new version and monitored for any issues. If no issues are detected, the new version will be gradually rolled out to all users by increasing the percentage of traffic directed to the new version.

Source: https://argoproj.github.io/argo-rollouts/concepts/#canary

The strategy you choose will depend on your use case and requirements. Be sure to read through this comparison to understand the differences.

One important thing to consider when using Argo Rollouts is pod placement on nodes which can trigger unwanted disruptions as node autoscalers look to optimize and scale-down underutilized nodes. To prevent this, you can use PodAntiAffinity to ensure that new versions of pods are not scheduled on the same node as old versions.

Describe Analysis Template and Analysis Run

As mentioned above, Argo Rollouts can automatically manage traffic routing using the Analysis feature. To do this, you will need to define AnalysisTemplate and AnalysisRun resources.

The AnalysisTemplate resource defines the metrics and frequency that will be used to monitor the new version of an application. It will also include success and failure thresholds that will be used to determine whether the new version is performing as expected.

The AnalysisRun resource defines the actual analysis that will be performed. Upon completion, it will return a status of Successful, Failed, or inconclusive and the Rollout controller will use this information to determine whether to continue with the rollout or rollback to the previous version.

Think of AnalysisTemplate as a template and AnalysisRun as an instance of that template.

There is quite a bit to learn about this feature when it comes to resource specification, so be sure to read through the Analysis & Progressive Delivery documentation as well as its various metric provider integrations to understand how to define and use the resources.

Practice

To see Argo Rollouts in action, install the controller and kubectl plugin then follow the quick start guide to deploy a simple application and manage its lifecycle.

Argo Events 12%

Argo Events enables you to implement event-driven workflow automation at scale within Kubernetes clusters.

Understand Argo Events Fundamentals

Event-Driven Architecture (EDA) is the foundation of modern software development. It allows for the creation of highly responsive and adaptable systems that can react to real-time events and changes in the environment.

Below is an overview of Argo Events:

Source: https://argoproj.github.io/argo-events

With Argo Events, you can build event-driven workflows that respond to a wide range of events, such as messages in a queue, changes in a database, or even webhook notifications from external services. Currently Argo Events supports over 20 event sources and can trigger actions in response to those events including creation of Kubernetes resources, triggering of workflows, and more.

Understand Argo Event Components and Architecture

Understanding the architecture of Argo Events is essential for grasping how it operates and how its components interact with each other.

Source: https://argoproj.github.io/argo-events/concepts/architecture/

Let's quickly go over the components:

Event Source is the external system that generates events.
Sensor listens to event sources and triggers actions to respond to those events.
EventBus is backbone for managing delivery of events from event sources to sensors
Trigger responds to events by performing actions such as starting workflows, creating Kubernetes resources, or sending notifications.

For Webhook event sources, you can use a simple Kubernetes Secret to store the authentication token to protect the webhook endpoint.

You can also further validate the event data that is received by the event source using filters and expressions. This is useful when you need to ensure that the event data meets certain criteria before triggering an action.

Practice

To see Argo Events in action, follow the quick start guide to deploy a simple event-driven workflow.

Summary

That was a lot of information to digest but I hope it gives you a good starting point to prepare for the CAPA exam. Be sure to read through the official documentation for each of the projects and try out some of the examples to get a better understanding of how they work.

In addition to all the URLs provided above, be sure to check out the awesome-argo site maintained by friends at Akuity for a ton of great resources on the Argo projects.

If this guide was helpful, please share it with your friends and colleagues. If you have any questions or feedback, feel free to leave a comment below or reach out on LinkedIn, BlueSky, or X.

Good luck with your exam and happy learning! 🚀

Using AKS-managed Istio External Ingress Gateway with Gateway API

Paul Yu — Tue, 13 Aug 2024 14:06:38 +0000

Introduction

Kubernetes is great at orchestrating containers, but it can be a bit tricky to manage traffic routing. There are many options and implementations that you, as a cluster operator have probably had to deal with. We have the default Service resource that can be used to expose applications, but it is limited to routing based on layer 4 (TCP/UDP) and does not support more advanced traffic routing use cases. There's also the Ingress controller, which enabled layer 7 (HTTP) routing, and securing the North-South traffic using TLS, but it was not standardized and each vendor implementation required learning a new set of resource annotations. When it comes to managing and securing East-West traffic between services, there's Service Mesh which is yet another layer of infrastructure to manage on top of Kubernetes. And we're in the same boat when it comes to resource management with each vendor having their own ways of doing things.

Rise of the Gateway API

There's no doubt that Ingress and service mesh have greatly improved the way we manage and secure traffic in Kubernetes, but the subtle differences between implementations has made it difficult to standardize on a single way of doing things. This is where the Gateway API comes in. It is a Kubernetes add-on that aims to bring users an extensible, role-oriented, and protocol-aware services configuration mechanism. It is a new API that both Ingress and service mesh vendors can standardize on.

The API has been designed from the ground up, taking into account the lessons learned from the Ingress and service mesh APIs. To me, it's a game-changer because it ensures that you can express traffic routing configurations that were previously only possible via custom Ingress annotations. It also offers role-oriented configuration, which means that cluster operators can configure gateways and app developers can manage routes through the gateway. This is a big win for organizations that have a separation of duties between cluster operators and app developers. And finally, the API is portable, meaning that the API specification is supported by many implementations including Istio.

Istio on AKS

If you haven't heard of Istio yet, it is a CNCF-graduated service mesh project that provides a uniform way to connect, manage, and secure microservices. Istio is a fully-featured service mesh that supports managing traffic flows between services, enforcing access policies, and aggregating telemetry data. It not only provides East-West traffic management and security but also North-South traffic management and security through its Ingress Gateway. We can also thank Istio for much of the inspiration behind the Gateway API - if you've worked with Istio, you'll see the similarities.

AKS is a managed Kubernetes offering and as such many popular open-source projects are available as managed add-ons and extensions. Istio is one of those add-ons that can be enabled on AKS. AKS also provides managed deployments of Istio Ingress Gateway, which can be used to route traffic into the cluster from both internal origins within a virtual network as well as external traffic from the public internet. In this article, I'll show you how you can experiment with the Gateway API to manage the external Istio Ingress Gateway on AKS.

The Gateway API is not fully supported yet supported for the Istio-based managed add-on on AKS and this should be considered an experiment for now.

Gateway API in action with Istio on AKS

Here's a high-level overview of what we'll be doing to leverage the Gateway API to manage the Istio external ingress gateway on AKS:

Provision AKS cluster with Istio enabled
Provision AKS managed Istio external ingress gateway
Install Gateway API CRDs
Deploy a new Gateway resource in the aks-istio-ingress namespace and configure it to program the existing AKS-managed Istio Ingress Gateway - this is the key!
Deploy HTTPRoute resource that uses the new Gateway

Before you begin, you'll need to make sure you have access to an Azure subscription and have the Azure CLI and kubectl installed.

Provision AKS cluster with Istio enabled

Start by setting some local variables.

RG_NAME=rg-istio-gtw-demo
AKS_NAME=aks-istio-gtw-demo
LOCATION=westus3

Create a resource group and an AKS cluster with Istio enabled. The --enable-asm flag enables the managed Istio add-on and the --revision asm-1-22 flag specifies the version of Istio to install.

az group create \
--name $RG_NAME \
--location $LOCATION

az aks create \
--resource-group $RG_NAME \
--name $AKS_NAME \
--enable-asm \
--kubernetes-version 1.30 \
--revision asm-1-22

az aks get-credentials \
--resource-group $RG_NAME \
--name $AKS_NAME

Enable the Istio external ingress gateway with the following command. This can take a few minutes to complete.

az aks mesh enable-ingress-gateway \
--resource-group $RG_NAME \
--name $AKS_NAME \
--ingress-gateway-type external

Deploy a demo application

To demo the Gateway API, we'll deploy the AKS store demo app, a simple e-commerce application which consists of two services: store-front and store-admin. The store-front service is the customer facing e-commerce store, and the store-admin service is a back-end service that serves an admin dashboard for managing the store.

Create a namespace and optionally label it for Istio injection.

kubectl create namespace pets
kubectl label namespace pets istio.io/rev=asm-1-22

Deploy the AKS store demo app into the pets namespace using the aks-store-all-in-one YAML manifest.

kubectl apply -n pets -f https://raw.githubusercontent.com/Azure-Samples/aks-store-demo/main/aks-store-all-in-one.yaml

The aks-store-all-in-one.yaml manifest exposes the two services using LoadBalancer which assigns a public IP for each service. We want to use the Istio external ingress gateway as the application's point of entry, so we can patch the store-front and store-admin services to use ClusterIP instead of LoadBalancer.

kubectl patch service -n pets store-admin -p '{"spec":{"type":"ClusterIP"}}'
kubectl patch service -n pets store-front -p '{"spec":{"type":"ClusterIP"}}'

Make sure all the pods and services are running before moving on.

kubectl get all -n pets

Implement Gateway API for Istio external ingress

The Gateway API is not yet included in the Kubernetes distribution by default, so you will need to install the Gateway API CRDs and controllers before you can use it. You can always find the latest releases on the Gateway API GitHub releases page.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/standard-install.yaml

To use the Gateway API, we need to create a Gateway resource in the aks-istio-ingress namespace. It's important that the Gateway resource is deployed into the same namespace as the Istio external ingress gateway; otherwise, the Gateway resource will not be programmed correctly.

The Gateway resource is a top-level resource that acts as a load balancer operating at the edge of the mesh. It can be used to leverage a particular GatewayClass, in our case, we will be using the Istio external ingress gateway.

Source: https://gateway-api.sigs.k8s.io

Deploy the Gateway for our sample application in the aks-istio-ingress namespace.

kubectl apply -n aks-istio-ingress -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
spec:
  gatewayClassName: istio
  addresses:
  - value: aks-istio-ingressgateway-external
    type: Hostname
  listeners:
  - name: default
    hostname: "*.aks.rocks"
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
EOF

The critical implementation detail here is the gatewayClassName field with a value of istio and the addresses field with a value of aks-istio-ingressgateway-external. By default, the Gateway resource will automatically provision a Service and Deployment but we don't need in this scenario. Instead, we can manually specify the DNS name of the Service that was deployed when AKS provisioned the Istio ingress gateway for you. Also, the allowedRoutes field specifies that the Gateway will accept routes from all namespaces. This can be restricted to specific namespaces if needed.

Next, deploy an HTTPRoute for the store-admin and store-front services in the pets namespace.

kubectl apply -n pets -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: store-admin
spec:
  parentRefs:
  - name: gateway
    namespace: aks-istio-ingress
  hostnames: ["admin.aks.rocks"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: store-admin
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: store-front
spec:
  parentRefs:
  - name: gateway
    namespace: aks-istio-ingress
  hostnames: ["store.aks.rocks"]
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: store-front
      port: 80
EOF

Here we have two routes that will be routed through the gateway. The store-admin route will route traffic to the store-admin service and the store-front route will route traffic to the store-front service. The hostnames field specifies the hostname that the route will match on and the rules field specifies the path that the route will match on.

Being able to deploy HTTPRoutes in application namespaces is a powerful feature of the Gateway API because it allows app developers to manage their own routes.

Let's validate that the Gateway and HTTPRoute resources were created successfully. Grab the public IP address of the Istio external ingress gateway and test the store app using a curl command.

INGRESS_PUBLIC_IP=$(kubectl get svc -n aks-istio-ingress aks-istio-ingressgateway-external -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -IL "http://${INGRESS_PUBLIC_IP}" -H "Host: store.aks.rocks"

Note the -H "Host: store.aks.rocks" flag is used to specify the hostname in the request header. This is important because the Istio gateway is configured to route based on hostname.

Optionally, you could also add an entries to your /etc/hosts file.

echo "${INGRESS_PUBLIC_IP} admin.aks.rocks store.aks.rocks" | sudo tee -a /etc/hosts

With the host entries updated, you can test the app in your browser. Just be sure to remove the entries from your /etc/hosts file when you're done.

Also, don't forget to clean up the Azure resources when you're done.

az group delete --name $RG_NAME

Summary

We can see that the Gateway API is a powerful new API that provides a standardized way to configure traffic routing in Kubernetes. It is a big step forward from the Ingress and service mesh APIs and provides a way to express traffic routing configurations that were previously only possible via custom Ingress annotations. The Gateway API is portable and can be used with many different implementations including Istio. In this article, you learned how you can install the Gateway API CRDs into your AKS cluster and use the Gateway API to manage the AKS-managed external Istio Ingress Gateway.

Many resource types and features of Gateway API have graduated to GA with other parts of the API still evolving. If you'd like to get involved in the Gateway API project, head over to the Gateway API GitHub repository for more information.

Happy traffic routing!

Resources

Adding a GitHub Codespace button to your README

Paul Yu — Thu, 21 Mar 2024 13:00:00 +0000

GitHub Codespaces is a great way to make it easier for people to contribute to your project. With a few clicks, folks can spin up a Codespace environment with all necessary tooling installed and be productive right away. But it does take a few clicks and this quick post is to show how you can save developers a click or two because every click matters 😆

With one line of markdown in your README, you can add a button that looks like this...

To add the button, you'll first need to get your GitHub repo's ID. The GitHub repo ID is not obvious to find, but you can use the GitHub CLI to retrieve it.

Run a command like this from the root of your repo and make sure to replace Azure-Samples/aks-store-demo with your org/repo.

gh api repos/Azure-Samples/aks-store-demo --jq '.id'

From there, all you need to do is add the following line to your README, like this...

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://github.com/codespaces/new?hide_repo_select=true&ref=main&repo=648726487)

Notice the query parameters ref and repo in the URL above. Make sure you replace 648726487 in the code above with your repo's ID and set ref to whatever branch you want to open the Codespace in as it will currently open in the main branch.

That's it! Now when someone visits your repo, they'll see a button at the top of the README that they can simply click to open a Codespace for your repo. 🎉

Soaring to New Heights with Kaito: The Kubernetes AI Toolchain Operator

Paul Yu — Wed, 20 Mar 2024 14:00:00 +0000

Earlier today at KubeCon Europe 2024, Jorge Palma of the AKS team gave a keynote talk on Kaito, the Kubernetes AI Toolchain Operator.

// Detect dark theme var iframe = document.getElementById('tweet-1770375558607647132-590'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1770375558607647132&theme=dark" }

This tool has been released as an open-source project a few months back and you may or may not have heard of it.

So if you don't know, now you know...

You're probably thinking, "but what is it, and what can it do for me?" 🤔

What’s in a name?

Kaito is a cool name and the word rolls off the tongue quite nicely. After doing a bit of searching I came across a definition of the name Kaito, which is a Japanese name commonly given for boys and has deep meaning and symbolism within the Japanese culture.

According to Japanese Board, the name is made up of two kanji characters: 海 and 斗

海 (kai) means "sea" or "ocean"
斗 (to) has several meanings and references like a "measuring cup" or "big dipper" used in Chinese astrology

Ultimately the word symbolizes guidance and direction, and aligning oneself with the stars over the vastness of the sea. This is a fitting name for a tool that aims to help you navigate through what could be a sea of complexity that is, AI and Kubernetes 😅

What is Kaito?

Okay, enough with the philosophical stuff. Let's get to the meat of it.

Kaito is a Kubernetes Operator that aims to help you with your AI workloads on Kubernetes. It's a tool that in it's current form, helps you reduce your time to inference when it comes to deploying open-source AI models with inferencing endpoints on Kubernetes.

Why use Kaito?

If you're working with AI today, especially OpenAI or Azure OpenAI, you're probably familiar with the fact the model is hosted in the cloud and you primarily interact with it via REST API. The Model-as-a-Service (MaaS) approach is great for a lot of use cases, but there are times when you need to run the model closer to the data due to a variety of reasons like data sovereignty, compliance, or latency.

Also, you might not want to run OpenAI's models because they are not truly open-source and may feel like a black box when it comes to understanding how they were built and how they work. You might want to run your own models, or models from the open-source community, many of which can be found on Hugging Face's model hub.

Kaito aims to help you with this by providing a way to deploy open-source AI models as inferencing services on Kubernetes. It cuts through all the minutiae of deploying, scaling, and managing AI and GPU-based workloads on Kubernetes.

More specifically it helps you with the following:

Automated GPU node provisioning and configuration Kaito will automatically provision and configure GPU nodes for you. This can help reduce the operational burden of managing GPU nodes, configuring them for Kubernetes, and tuning model deployment parameters to fit GPU profiles.
Reduced cost: Kaito can help you save money by splitting inferencing across lower end GPU nodes which may also be more readily available and cost less than high-end GPU nodes.
Support for popular open-source LLMs: Kaito offers preset configurations for popular open-source LLMs. This can help you deploy and manage open-source LLMs on AKS and integrate them with your intelligent applications.
Fine-grained control: You can have full control over data security and privacy, model development and configuration transparency, and the ability to fine-tune the model to fit your specific use case.
Network and data security: You can ensure these models are ring-fenced within your organization's network and/or ensure the data never leaves the Kubernetes cluster.

Kaito Architecture

Kaito is built using the Kubernetes controller design pattern. Here is a high-level overview of the architecture of Kaito which can be found in the Kaito GitHub repo.

The operator that consists of two controllers.

Workspace controller: This controller reconciles the Workspace custom resource which creates a machine custom resource to trigger node auto provisioning and creates the inference workload.
Node provisioner controller: This controller uses the machine custom resource to interact with the Workspace controller. It leverages Karpenter APIs to add new GPU nodes to the AKS cluster.

As a cluster operator, you only need to be concerned with a couple of helm install commands to install the controllers and a Workspace custom resource. The Workspace controller will reconcile the Workspace custom resource which creates a machine custom resource to trigger node auto provisioning and creates the inference workload based on a preset configuration.

The Kaito team has done quite a bit of work to ensure some of the most popular open-source models like Meta's Llama2, TII's Falcon, Mistral AI's Mistral, Microsoft's Phi-2, and others are containerized properly and work well on Kubernetes. So they've created what they call presets for these models. This means you can deploy these models with a few lines of YAML and Kaito will take care of the rest.

Here is an example of a Workspace custom resource:

apiVersion: kaito.sh/v1alpha1
kind: Workspace
metadata:
  name: workspace-falcon-7b-instruct
resource:
  instanceType: "Standard_NC12s_v3"
  labelSelector:
    matchLabels:
      apps: falcon-7b-instruct
inference:
  preset:
    name: "falcon-7b-instruct"

Only 12 lines of YAML 🎉

If you're wondering if this works on different cloud providers, unfortunately the answer right now is "no". Kaito is currently only supported on Azure. But with this being an open-source project, you are welcome to contribute and help make it work on other cloud providers 🤝

Getting Started

Getting started with Kaito is pretty straightforward. You can visit the Kaito GitHub repo and follow the installation instructions to start.

I've also recorded a "Learn Live" session on @MicrosoftReactor YouTube channel with Ishaan Seghal from the Kaito team where we did an in-depth walk-through on how to get started. Be sure to check that video out below.

The workshop material for the session can be found here: Open Source Models on AKS with Kaito

As with any big event like KubeCon comes a slew of announcements. One of the announcements was the release of the Kaito add-on for AKS (currently in Preview) which simplifies this even more by having AKS install and manage Kaito in your AKS cluster for you. Documentation for that can be found here: Deploy an AI model on Azure Kubernetes Service (AKS) with the AI toolchain operator (preview)

Lastly, there is a neat blog article which walks you through some of the steps listed above but includes an interesting use case of using Kaito for forecasting energy usage using the LLaMA2 model. Be sure to check that out as well. 2.1 Forecasting Energy Usage with Intelligent Apps Part 1: Laying the Groundwork with AKS, KAITO, and LLaMA

More Presets

The Kaito team publishes all the models they support as presets here and is working on adding more for popular open-source AI models. If you have a model you'd like to see supported, you can open an issue on the Kaito GitHub repo and let the team know. They are always looking for feedback and contributions from the community.

Conclusion

By now, you should have a good understanding of what Kaito is and what it aims to help you with as a Kubernetes Operator. Being able to deploy open-source AI models as inferencing services on Kubernetes will be a game-changer for a lot of folks. But inferencing is just one part of the story and probably and that's where this Kaito journey begins. The team is also looking at how to help you with fine-tuning these open-source AI models on Kubernetes in the future, so stay tuned for that.

So, leverage Kaito for your AI workloads in Kubernetes. The name says it all.

If you have any questions or feedback, please feel free to leave a comment below or reach out to me on Twitter or LinkedIn. Also be sure to follow the @TheAKSCommunity YouTube channel for more on this project as it evolves.

Until next time,

Peace ✌️

Strengthening the Secure Supply Chain

Paul Yu — Sun, 17 Mar 2024 13:00:00 +0000

This post will walk you through a demo I presented at the SCaLE21X conference. The session is titled, Strengthening the Secure Supply Chain with Project Copacetic, Eraser, and FluxCD and this step-by-step guide will enable you do it on your own.

Prerequisites

To begin, you will need to have the following:

Docker Desktop to run a Kubernetes cluster locally
Git to clone the demo repository
GitHub account

We will also be using the following tools:

But don't worry about installing all of these tools right now. I will walk you through the installation process as we go. All you need to start is Docker Desktop and a Bash shell.

Note: I used a Mac on Apple Silicon for this demo. If you are using a different operating system, you may need to adjust the commands accordingly.

Install KIND and kubectl

First, you will need to create a Kubernetes cluster. You can use KIND which stands for Kubernetes in Docker. It is a tool for running local Kubernetes clusters using containers as “nodes”.

Head over to the KIND documentation to install KIND on your local machine. On my Mac, I used Homebrew to install KIND:

brew install kind

Next we need to install kubectl to interact with the cluster. Head over to the kubectl documentation to install the tool on your local machine. This is how I installed kubectl on my Mac using the curl command:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/arm64/kubectl"

Create a local Kubernetes cluster

Once you have KIND installed, you can use the following commands to create a single-node Kubernetes cluster:

kind create cluster --name scale21x-demo

With the local Kubernetes cluster running and kubectl installed, run the following command to verify the cluster is running:

kubectl cluster-info --context kind-scale21x-demo

Install GitHub CLI

We'll be working with a GitHub repository and configuring GitHub Actions. I like to use the GitHub CLI which makes it realy easy to work with GitHub using the command line.

Head over to the GitHub CLI documentation and install the tool on your local machine. I installed the GitHub CLI with the following command:

brew install gh

With the GitHub CLI installed, run the following command to authenticate with GitHub:

gh auth login --scopes repo,workflow,read:packages,write:packages,delete:packages

Note: The login scopes listed above are required for the demo.

Export your GitHub username as an environment variable:

export GITHUB_USER=$(gh api user --jq .login)

We also want to be able to push container images to the GitHub Container Registry, so we need to authenticate with the GitHub Container Registry. You can use the following command to authenticate with the GitHub Container Registry:

gh auth token | docker login ghcr.io -u $GITHUB_USER --password-stdin

Fork and clone sample app repo

We will be using a sample application that I wrote for this demo. You can fork and clone the Azure-Samples/aks-store-demo repository by running the following command:

gh repo fork Azure-Samples/aks-store-demo --clone
cd aks-store-demo

When working with a forked repository, we need to set the default repo, so that when we execute workflow commands it will use the forked repo and not the original. You can use the following command to set the default repo:

gh repo set-default

Note: When prompted, select the forked repository.

Build sample app container

The sample app contains multiple applications, but we'll only focus on the store-front app. Let's build the store-front container and push it to the GitHub Container Registry.

Run the following command to build the container image:

docker build --label "org.opencontainers.image.source=https://github.com/$GITHUB_USER/aks-store-demo" -t ghcr.io/$GITHUB_USER/aks-store-demo/store-front:1.2.0 -t ghcr.io/$GITHUB_USER/aks-store-demo/store-front:latest ./src/store-front

Note: This may take a few minutes to complete.

Run the following commands to push the tagged container image to the GitHub Container Registry:

docker push ghcr.io/$GITHUB_USER/aks-store-demo/store-front:latest
docker push ghcr.io/$GITHUB_USER/aks-store-demo/store-front:1.2.0

Note: You may need to link the package registry to the repository. You can do this by following the instructions listed here.

Install Kustomize CLI

Kustomize is a neat tool for customizing Kubernetes configurations. It makes it very easy to manage and customize Kubernetes configurations including container images and tags. Head over to the Kustomize documentation to install Kustomize on your local machine. I installed Kustomize with the following command:

brew install kustomize

Edit store-front image source

Using the kustomize CLI, we can update the kustomization.yaml file to use the container image you just pushed to the GitHub Container Registry.

First, make sure you are in the root directory of the cloned repository then change into the directory where the kustomization.yaml file is located:

cd kustomize/overlays/dev

Now you can use the following command to update the kustomization.yaml file:

kustomize edit set image ghcr.io/azure-samples/aks-store-demo/store-front=ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.0

Commit the changes to the kustomization.yaml file and push the changes to the repository:

cd -
git add ./kustomize/overlays/dev/kustomization.yaml
git commit -m 'feat: update store-front image'
git push

Install Flux CLI

Next, you will need to install the Flux CLI so that you can bootstrap your Kubernetes cluster for GitOps. Head over to the Flux documentation to install Flux on your local machine. Again, I'm using Homebrew so I installed with the following command:

brew install fluxcd/tap/flux

Bootstrap the Kubernetes Cluster for Flux

We're ready to bootstrap the Kubernetes cluster. But you need to ensure that you have the following environment variables set so that Flux can authenticate with GitHub on your behalf:

export GITHUB_USER=$(gh api user --jq .login)
export GITHUB_TOKEN=$(gh auth token)

Run the following command to bootstrap your Kubernetes cluster for GitOps:

flux bootstrap github create \
  --owner=$GITHUB_USER \
  --repository=aks-store-demo \
  --personal \
  --private false \
  --path=./kustomize/overlays/dev \
  --branch=main \
  --reconcile \
  --read-write-key \
  --author-name=fluxcdbot \
  --author-email=fluxcdbot@users.noreply.github.com \
  --components-extra=image-reflector-controller,image-automation-controller

We don't need the GitHub token anymore, so you can unset the environment variable:

unset GITHUB_TOKEN

After a few minutes, the cluster will be reconciled, you can run the following command to see the app running in the Kubernetes cluster:

kubectl get pods -n pets

Note: If you see a status of ImagePullBackOff for your store-front pod, it may be due to package visibility. In which case, you may need to link the package registry to the repository. You can do this by following the instructions listed here.

Install Trivy CLI

Next, you will need to install Trivy on your local machine. Trivy is a simple and comprehensive vulnerability scanner for containers. Head over to the Trivy documentation to install Trivy on your local machine. I installed Trivy with the following command:

brew install aquasecurity/trivy/trivy

Run a Trivy scan

Now that you have Trivy installed, you can use the following command to run a Trivy scan on the container images you pushed to the GitHub Container Registry:

trivy image --vuln-type os --ignore-unfixed ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.0

You should see Trivy output that looks something like this:

Total: 2 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 1, CRITICAL: 0)

┌──────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│ Library  │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │                            Title                            │
├──────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ libexpat │ CVE-2023-52425 │ HIGH     │ fixed  │ 2.5.0-r0          │ 2.6.0-r0      │ expat: parsing large tokens can trigger a denial of service │
│          │                │          │        │                   │               │ https://avd.aquasec.com/nvd/cve-2023-52425                  │
│          ├────────────────┼──────────┤        │                   │               ├─────────────────────────────────────────────────────────────┤
│          │ CVE-2023-52426 │ MEDIUM   │        │                   │               │ expat: recursive XML entity expansion vulnerability         │
│          │                │          │        │                   │               │ https://avd.aquasec.com/nvd/cve-2023-52426                  │
└──────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┴─────────────────────────────────────────────────────────────┘

Now, let's re-run the command to output the results in JSON format:

trivy image --vuln-type os --ignore-unfixed -f json -o /tmp/store-front.1.2.0.json ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.0

Install Copacetic CLI

So we have an OS vulnerability in the container image. We can patch the vulnerability with Project Copacetic. Copacetic is a tool for automating the scanning, patching, deployment, and deletion of container images in a Kubernetes cluster. Head over to the Copacetic documentation to install Copacetic on your local machine. I installed Copacetic with the following command:

brew install copa

Patch the vulnerability with copa

With Copacetic installed, you can use the following command to patch the vulnerability in the container image:

copa patch -i ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.0 -r /tmp/store-front.1.2.0.json -t 1.2.1

Now if you re-run the Trivy scan, you should see that the vulnerability has been patched:

trivy image --vuln-type os --ignore-unfixed ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.1

And if you run the following command, you should see the history of the container image with the patched layer:

docker history ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.1

Push the patched container image to the GitHub Container Registry:

docker push ghcr.io/${GITHUB_USER}/aks-store-demo/store-front:1.2.1

Configure Flux image update automation

Now that you have patched the vulnerability in the container image, you can configure Flux to automatically update the container image when a new version is available. You can use the following commands to configure Flux to automatically update the container image:

# tells flux where the container image is stored
flux create image repository store-front \
  --image=ghcr.io/$GITHUB_USER/aks-store-demo/store-front \
  --interval=1m

# tells flux how to find latest version of the container image
flux create image policy store-front \
  --image-ref=store-front \
  --select-semver='>=1.0.0'

# tells flux where to make edits based on new version of the container image
flux create image update store-front \
  --git-repo-ref=flux-system \
  --git-repo-path="./kustomize/overlays/dev" \
  --checkout-branch=main \
  --author-name=fluxcdbot \
  --author-email=fluxcdbot@users.noreply.github.com \
  --commit-template="{{range .Updated.Images}}{{println .}}{{end}}"

One last step is to "mark" the manifest so that Flux will be able to update the image in the right spot within the kustomization.yaml when a new version is available. You can use the following command to mark the manifest:

sed -i '' -e "s^newName: ghcr.io/${GITHUB_USER}/aks-store-demo/store-front^newName: ghcr.io/${GITHUB_USER}/aks-store-demo/store-front # {\"\$imagepolicy\": \"flux-system:store-front:name\"}^g" ./kustomize/overlays/dev/kustomization.yaml
sed -i '' -e "s^newTag: 1.2.0^newTag: 1.2.0 # {\"\$imagepolicy\": \"flux-system:store-front:tag\"}^g" ./kustomize/overlays/dev/kustomization.yaml

Commit the changes to the kustomization.yaml file and push the changes to the repository:

git add ./kustomize/overlays/dev/kustomization.yaml
git commit -m "feat: adding flux image update markers"
git push

Run the following commands to force a reconciliation of the Flux controllers:

flux reconcile image repository store-front
flux reconcile image update store-front
flux reconcile kustomization flux-system

If all went well, you should see that the image has been updated to version 1.2.1:

kubectl get deploy store-front -n pets -o yaml | grep image:

Great! Now you have Flux configured to automatically update the container image when a new version is available. Now we need to make sure the image is automatically patched when a vulnerability is found.

Automatically patch vulnerabilities with GitHub Actions

You can use the following commands to configure Copacetic to automatically patch vulnerabilities with GitHub Actions. We'll rely on the copa-action that is available in the GitHub Marketplace.

Create a new file called .github/workflows/patch-container-images.yaml.

touch .github/workflows/patch-container-images.yaml

Open the file and add the following content:

name: patch-container-images

on:
  schedule:
    - cron: "30 0 * * 2"
  workflow_dispatch:

permissions:
  contents: read
  packages: write

jobs:
  test:
    runs-on: ubuntu-latest

    strategy:
      fail-fast: false
      matrix:
        apps:
          - "store-front"

    steps:
      - name: Authenticate with GitHub CLI
        run: |
          gh auth login --with-token <<< "${{ github.token }}"

      - name: Get the latest tag
        id: semver_tag
        run: |
          tag=$(gh api user/packages/container/aks-store-demo%2F${{ matrix.apps }}/versions --jq '.[0] | .metadata.container.tags[0]')
          echo "tag=$tag" >> $GITHUB_OUTPUT

      - name: Bump the tag
        id: bump_tag
        run: |
          tag=$(echo ${{ steps.semver_tag.outputs.tag }} | awk -F. -v OFS=. '{$NF = $NF + 1;} 1')
          echo "tag=$tag" >> $GITHUB_OUTPUT

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: "image"
          format: "table"
          ignore-unfixed: true
          vuln-type: "os"
          image-ref: ghcr.io/${{ github.repository }}/${{ matrix.apps }}:latest

      - name: Generate Trivy Report
        uses: aquasecurity/trivy-action@062f2592684a31eb3aa050cc61e7ca1451cecd3d # v0.18.0
        with:
          scan-type: "image"
          format: "json"
          output: "report.json"
          ignore-unfixed: true
          vuln-type: "os"
          image-ref: ghcr.io/${{ github.repository }}/${{ matrix.apps }}:latest

      - name: Check Vuln Count
        id: vuln_count
        run: |
          report_file="report.json"
          vuln_count=$(jq 'if .Results then [.Results[] | select(.Class=="os-pkgs" and .Vulnerabilities!=null) | .Vulnerabilities[]] | length else 0 end' "$report_file")
          echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT

      - name: Copa Action
        if: steps.vuln_count.outputs.vuln_count != '0'
        id: copa
        uses: project-copacetic/copa-action@3843e22efdca421adb37aa8dec103a0f1db68544 # v1.2.1
        with:
          image: ghcr.io/${{ github.repository }}/${{ matrix.apps }}:latest
          image-report: "report.json"
          patched-tag: "patched"

      - name: Login to GHCR
        if: steps.copa.conclusion == 'success'
        id: login
        uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d # v3.0.0
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ github.token }}

      - name: Push patched image
        if: steps.login.conclusion == 'success'
        run: |
          docker tag ${{ steps.copa.outputs.patched-image }} ghcr.io/${{ github.repository }}/${{ matrix.apps }}:${{ steps.bump_tag.outputs.tag }}
          docker push ghcr.io/${{ github.repository }}/${{ matrix.apps }}:${{ steps.bump_tag.outputs.tag }}

This workflow will run every Tuesday (do you remember "Patch Tuesdays"? 😆) at 12:30 AM UTC and can also be triggered manually. It looks for the most recent image tag and bumps the semver which will be used then tagging the next container image. When scanning container images, it is important that copa does not patch from previously patched versions so we'll always patched from the latest version for this demo. Copa will scan the container image for vulnerabilities and if any are found, it will patch and push the image to the GitHub Container Registry.

Commit and push the changes to the repository:

git add .github/workflows/patch-container-images.yaml
git commit -m "ci: add patch-container-images workflow"
git push

Run the workflow

gh workflow run patch-container-images.yaml

View the workflow run

gh run watch

Note: If you run into an error with the workflow where the message states denied: permission_denied: write_package, you will need to configure the store-front package settings to allow Actions repository access. See here for additional information.

A few minutes after the workflow completes, you should see that the container image has been patched and pushed to the GitHub Container Registry and the deployment has been updated in the Kubernetes cluster.

kubectl get deploy store-front -n pets -o yaml | grep image:

Cleaning up cluster images with Eraser

We're almost done. We just need to clean up the vulnerable container images from the Kubernetes nodes. If you run the following command, yo uwill see that the vulnerable container image is still present on the Kubernetes nodes:

kubectl get nodes -o json | jq '.items[].status.images[].names | last' | grep store-front

This is where we install Eraser, a tool for automating the deletion of vulnerable container images from Kubernetes nodes, into the Kubernetes cluster.

Run the following command to install Eraser into the Kubernetes cluster:

kubectl apply -f https://raw.githubusercontent.com/eraser-dev/eraser/v1.3.1/deploy/eraser.yaml

After a few minutes, you should see that the vulnerable container image has been deleted from the Kubernetes nodes:

kubectl get nodes -o json | jq '.items[].status.images[].names | last' | grep store-front

You should only see the patched container image on the Kubernetes nodes.

Clean up local machine

When you are done, you can delete the Kubernetes cluster by running the following command:

kind delete cluster --name scale21x-demo

Conclusion

In this post, you learned how to strengthen the secure supply chain with Trivy, Copacetic, Eraser, Flux, and GitHub Actions. It is easy to get lost in the sea of tools and technologies available to secure your supply chain, but with the right tools and processes in place, you can ensure that your container images are secure and up-to-date and in an automated fashion 🚀

I hope you found this post helpful and that you are able to use the steps outlined here within in your own environments. If you have any questions or feedback, please feel free to leave a comment below or reach out to me on Twitter or LinkedIn.

Peace ✌️

Resources

Bootstrap your GitOps-enabled AKS cluster with Terraform: A code sample using the Flux v2 K8s Extension

Paul Yu — Thu, 28 Sep 2023 13:00:00 +0000

In my previous posts, we learned how to get started with GitOps on AKS using the K8s extension for AKS.

Then, we took a look at the Flux CLI and explored how it can be used to bootstrap your cluster and generate FluxCD manifests so that we can use GitOps to implement GitOps 🤯, and implemented Flux's image update automation capability.

From there, we built on the concept of image update automation, and showed you how you can use Flagger to automate canary deployments.

Much of this was done using the Flux CLI to generate Flux CRD manifests, but you might be thinking...

"All that Flux CLI stuff is cool, but I want to bootstrap my AKS cluster to use GitOps, using the AKS extension!"

In this post, I'll show you how to do just that... with my favorite IaC tool, Terraform!

What's in the box?

To get started, head over to my GitHub repo and fork/clone it. This is where I have all the Terraform code you need to deploy and bootstrap your AKS cluster using a single terraform apply command.

At a high-level, the Terraform code will do the following:

Create a resource group
Create an AKS cluster with the Istio service mesh add-on and external ingress gateway enabled
Create a Kubernetes Secret in the flux-system namespace to store GitHub credentials for private Git repository access
Install the Flux v2 AKS extension with the following components enabled:
- image-automation-controller
- image-reflector-controller
- notification-controller
Configure the Flux v2 AKS extension with the following Kustomizations:
- The dev-app Kustomization deploys the GitRepository resource which points to the flux-k8s-ext branch in my AKS Store Demo Manifests repo, and Kustomization resource to deploy the AKS Store Demo application
- The dev-image Kustomization deploys the ImageRepository, ImagePolicy and ImageUpdateAutomation resources to automate image updates for the store-front application; this Kustomization has a dependency on the dev-app Kustomization
- The dev-flagger Kustomization deploys the HelmRepository, HelmChart, and HelmRelease to install Flagger along with the OCIRepository and Kustomization resource to install the Flagger Loadtester; this Kustomization also has a dependency on the dev-app Kustomization
- The dev-canary-store-front Kustomization deploys the Canary resource that is used to automate progressive delivery of the store-front application; this Kustomization has a dependency on the dev-flagger Kustomization

As an added bonus, I've also included Azure Managed Grafana and Azure Managed Prometheus in case you wanted to explore the monitoring capabilities Flux on AKS.

To deploy, simply follow the instructions found in the README of the repo.

Kustomizations, Kustomizations, Kustomizations...

There are some differences between the way Flux CLI bootstraps your GitOps cluster and the way you can bootstrap your AKS cluster using the AKS extension.

When Flux CLI bootstraps a cluster, it creates folders called /clusters/dev and /clusters/dev/flux-system. The /clusters/dev/flux-system folder contains manifests that are used to install Flux and store GitRepository source used for bootstrapping.

Here is what the directory structure looked like when we bootstrapped using Flux CLI.

.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
├── clusters
│   └── dev
│       ├── aks-store-demo-kustomization.yaml
│       ├── aks-store-demo-source.yaml
│       ├── aks-store-demo-store-front-image-policy.yaml
│       ├── aks-store-demo-store-front-image-update.yaml
│       ├── aks-store-demo-store-front-image.yaml
│       ├── flagger-helmrelease.yaml
│       ├── flagger-loadtester-kustomization.yaml
│       ├── flagger-loadtester-source.yaml
│       ├── flagger-source.yaml
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           └── kustomization.yaml
└── overlays
    └── dev
        ├── kustomization.yaml
        └── namespace.yaml

7 directories, 25 files

The /clusters/dev/flux-system/gotk-sync.yaml file is the Kustomization resource that Flux uses to configure the cluster. It tells Flux to deploy all resources found in the /clusters/dev directory.

With the AKS extension, the gotk-* manifests are not saved or used from our repo, so we can discard them. However, in order to deploy the resources, we do need to make Flux aware that we want these resources deployed.

The files in the /clusters/dev directory needs to be split out and installed as separate Kustomizations.

Here is what the directory structure ends up looking like to bootstrap a cluster using the AKS extension.

.
├── README.md
├── base
│   ├── kustomization.yaml
│   ├── makeline-service.yaml
│   ├── mongodb.yaml
│   ├── order-service.yaml
│   ├── product-service.yaml
│   ├── rabbitmq.yaml
│   ├── store-admin.yaml
│   ├── store-front.yaml
│   ├── virtual-customer.yaml
│   └── virtual-worker.yaml
├── clusters
│   └── dev
│       ├── flagger
│       │   ├── flagger-helmrelease.yaml
│       │   ├── flagger-loadtester-kustomization.yaml
│       │   ├── flagger-loadtester-source.yaml
│       │   ├── flagger-source.yaml
│       │   └── kustomization.yaml
│       └── image-update
│           ├── aks-store-demo-store-front-image-policy.yaml
│           ├── aks-store-demo-store-front-image-update.yaml
│           ├── aks-store-demo-store-front-image.yaml
│           └── kustomization.yaml
└── overlays
    └── dev
        ├── canary
        │   ├── kustomization.yaml
        │   └── store-front-canary.yaml
        ├── ingressgateway.yaml
        ├── kustomization.yaml
        └── namespace.yaml

9 directories, 25 files

As you can see, the image-update and flagger components have been split into their own Kustomization directories. I also created a canary Kustomization to deploy the Canary resource and moved it down to the overlays/dev directory since it is more specific to my app deployment.

By taking this directory structure approach, we can link the Kustomizations together using the dependsOn property and ensure things get installed and deployed in the right order.

So in my fluxcd.tf I am first deploying the dev-app Kustomization, then the dev-image and dev-flagger Kustomizations (which depends on dev-app), and finally the dev-canary-store-front Kustomization (which depends on dev-flagger).

When you run the Terraform, it will take roughly 15-20 minutes end-to-end, but at the end of it all, you'll have a fully functional AKS Store Demo app fully configured with image update automation and canary deployments 🥳

Conclusion

Using the Flux CLI to generate Flux manifests is a great way to get started with GitOps on AKS. It gives you a way to understand exactly what CRDs need to be deployed and how the manifests are written. However, if you are looking for an Azure native (and supported way) of bootstrapping your cluster, you'll need to plan for a few more Kustomizations and nest them using Flux dependencies. Just a little more planning but not much more.

Now go and bootstrap your GitOps cluster, the Azure way! 🚀

Continue the conversation

If you have any feedback or suggestions, please feel free to reach out to me on Twitter or LinkedIn.

You can also find me in the Microsoft Open Source Discord, so feel free to DM me or drop a note in the cloud-native channel where my team hangs out!

Peace ✌️

Progressive Delivery on AKS: A Step-by-Step Guide using Flagger with Istio and FluxCD

Paul Yu — Tue, 26 Sep 2023 13:00:00 +0000

In my previous post, we setup an Azure Kubernetes Service (AKS) cluster to automatically update images based on new image tags in a container registry. As soon as a new image was pushed to the registry the image was immediately updated.

But what if you don't want an agent automatically pushing out new images without some sort of testing? 🤔

In this article, we'll build upon Flux's image update automation capability and add Flagger to implement a canary release strategy.

Flagger is a progressive delivery tool that enables a Kubernetes operator to automate the promotion or rollback of deployments based on metrics analysis. It supports a variety of metrics including Prometheus, Datadog, and New Relic to name a few. It also works well with Istio service mesh, and can implement progressive traffic splitting between primary and canary releases.

The goal here is to harness the power of image update automation while implementing some sort of gating process around it.

Here is the intended workflow:

Modify application code, then commit and push the change to the repo.
Create a new release in GitHub which kicks off a release workflow to build and push an updated container image to a GitHub Container Registry.
FluxCD detects the new image and updates the image tag in a YAML manifest.
FluxCD rolls out the new image to the cluster.
Flagger detects a new deployment revision and starts a canary deployment.
Flagger progressively routes traffic to the new deployment based on metrics.
Flagger promotes the new deployment to production if the metrics are within the threshold.

We'll move really fast through the AKS cluster provisioning, bootstrapping process, and deploying the AKS Store Demo sample app.

If you want a closer look at how image update automation is configured using FluxCD, check out my previous post.

Let's go!

Prerequisites

Before you begin, you need to have the following:

Create an AKS cluster and bootstrap FluxCD

Run the following command to log into Azure and make sure you have the AzureServiceMeshPreview feature enabled.

az login
az feature register --namespace "Microsoft.ContainerService" --name "AzureServiceMeshPreview"

Next run the following command to setup some variables for your deployment.

RG_NAME=rg-flagger
AKS_NAME=aks-flagger
LOC_NAME=westus2

We'll deploy an AKS cluster with the Istio service mesh add-on enabled. If you are unfamiliar with service mesh in general, check out the Istio documentation and my previous post on Service Mesh Considerations for more information.

If you still have the AKS cluster from the previous post, you might want to delete it and start fresh.

Run the following commands to create the resource group and AKS cluster with the Istio add-on enabled.

az group create -n $RG_NAME -l $LOC_NAME
az aks create -n $AKS_NAME -g $RG_NAME --enable-azure-service-mesh --generate-ssh-keys -s Standard_B4s_v2

Istio offers internal and external ingress capabilities which controls traffic coming into the cluster. We'll use the external ingress as our entry point for the sample app.

Run the following command to enable the external ingress gateway.

az aks mesh enable-ingress-gateway \
  -n $AKS_NAME \
  -g $RG_NAME \
  --ingress-gateway-type external

After the cluster and Istio external ingress gateway are deployed, run the following command to connect to the cluster.

az aks get-credentials -n $AKS_NAME -g $RG_NAME

Let's move on to bootstrapping FluxCD and deploying the AKS Store Demo app.

Bootstrap cluster using FluxCD

We'll use the GitHub CLI to work with GitHub and Flux CLI generate new Flux manifests so be sure you have these tools installed.

Connect to GitHub using the GitHub CLI

Run the following command to log into GitHub.

gh auth login --scopes repo,workflow

Fork and clone the AKS Store Demo repo

If you're continuing from the previous post, you should already have the AKS Store Demo repo forked and cloned. If not, run the following commands to fork and clone the repo.

gh repo fork https://github.com/azure-samples/aks-store-demo.git --clone
cd aks-store-demo
gh repo set-default

Create a release workflow

If you're continuing from the previous post, you should already have a release workflow in the AKS Store Demo repo. If not, make sure you are in the root of the aks-store-demo repository and run the following commands to create a release workflow.

# download the releae workflow
wget -O .github/workflows/release-store-front.yaml https://raw.githubusercontent.com/pauldotyu/aks-store-demo/main/.github/workflows/release-store-front.yaml

# download the TopNav.vue file which we'll be modifying
wget -O src/store-front/src/components/TopNav.vue https://raw.githubusercontent.com/pauldotyu/aks-store-demo/main/src/store-front/src/components/TopNav.vue

# commit and push
git add -A
git commit -m "feat: add release workflow"
git push

# back out to the previous directory
cd -

Fork and clone the AKS Store Demo Manifests repo

The great thing about Flux is that it can be bootstrapped using GitOps. So we'll point to a branch in my AKS Store Demo Manifests repo which has everything we need to get the cluster setup quickly.

If you're continuing from the previous post, you should already have the AKS Store Demo Manifests repo forked and cloned. If not, run the following commands to fork and clone it.

gh repo fork https://github.com/pauldotyu/aks-store-demo-manifests.git --clone
cd aks-store-demo-manifests
gh repo set-default

I have updated manifests to include Istio resources in the istio branch. We'll use this branch to the cluster.

git fetch
git checkout --track origin/istio

# get the latest from upstream
git fetch upstream istio
git rebase upstream/istio

Secrets for FluxCD Image Update Automation

As mentioned in my previous post, we'll need to create a Flux secret to allow Flux to write to our GitHub repo.

Run the following command to create a namespace to land the Kubernetes secret into.

kubectl create namespace flux-system

Run the following commands set your GitHub info.

# make sure you are in the aks-store-demo-manifests repo
export GITHUB_USER=$(gh api user --jq .login)
export GITHUB_TOKEN=$(gh auth token)
export GITHUB_REPO_URL=$(gh repo view --json url | jq .url -r)

Run the following command to create the secret.

flux create secret git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --username=$GITHUB_USER \
  --password=$GITHUB_TOKEN

Update the GitRepository URL in a couple of Flux manifests to point to your repo.

sed "s/pauldotyu/${GITHUB_USER}/g" clusters/dev/flux-system/gotk-sync.yaml > tmp && mv tmp clusters/dev/flux-system/gotk-sync.yaml
sed "s/pauldotyu/${GITHUB_USER}/g" clusters/dev/aks-store-demo-source.yaml > tmp && mv tmp clusters/dev/aks-store-demo-source.yaml
sed "s/pauldotyu/${GITHUB_USER}/g" clusters/dev/aks-store-demo-store-front-image.yaml > tmp && mv tmp clusters/dev/aks-store-demo-store-front-image.yaml

git add -A
git commit -m 'feat: update git sync url'
git push

We can now bootstrap our cluster with FluxCD using the manifests in the istio branch of the AKS Store Demo Manifests repo.

flux bootstrap github create \
  --owner=$GITHUB_USER \
  --repository=aks-store-demo-manifests \
  --personal \
  --path=./clusters/dev \
  --branch=istio \
  --reconcile \
  --network-policy \
  --components-extra=image-reflector-controller,image-automation-controller

After a minute or two, run the following command to watch the bootstrap process.

flux logs --kind=Kustomization --name=aks-store-demo -f
# press ctrl-c to exit

Once the Kustomization reconciliation process is complete, run the following command to retrieve the public IP address of the Istio ingress gateway.

echo "http://$(kubectl get svc -n aks-istio-ingress aks-istio-ingressgateway-external -o jsonpath='{.status.loadBalancer.ingress[0].ip}')"

You should see the AKS Store Demo app running in your browser.

Install Flagger

Time to install Flagger, the GitOps way!

We'll use the Flux CLI to generate the Flagger manifests and commit them to our repo.

Run the following command to generate a HelmRepository for Flagger's Helm chart.

flux create source helm flagger \
  --url=oci://ghcr.io/fluxcd/charts \
  --export > ./clusters/dev/flagger-source.yaml

Run the following command to create a values.yaml file which will be used to configure Flagger.

cat <<EOF > values.yaml
meshProvider: istio
prometheus:
  install: true
EOF

Here, we are telling Flagger to use Istio as the service mesh provider and to install Prometheus to collect metrics.

Next we need to create a HelmRelease resource to install Flagger and pass in the values.yaml file we just created to configure Flagger.

flux create helmrelease flagger \
  --target-namespace=flagger-system \
  --create-target-namespace=true \
  --crds CreateReplace \
  --source=HelmRepository/flagger \
  --chart=flagger \
  --values=values.yaml \
  --export > ./clusters/dev/flagger-helmrelease.yaml

You don't need the values.yaml file anymore, so run the following command to delete it.

rm values.yaml

Flagger can also run load tests against your application to generate metrics. We'll use its load testing service to generate load against our application.

Flagger's load testing service can be installed via a Kustomization resource based on manifests packaged as an artifact in an Open Container Initiative (OCI) registry

Run the following command to create an OCIRepository pointing to an OCI registry.

flux create source oci flagger-loadtester \
  --url=oci://ghcr.io/fluxcd/flagger-manifests \
  --tag-semver=1.x \
  --export > ./clusters/dev/flagger-loadtester-source.yaml

Run the following command to create a Kustomization resource for the installation manifests.

flux create kustomization flagger-loadtester \
  --target-namespace=dev \
  --prune=true \
  --interval=6h \
  --wait=true \
  --timeout=5m \
  --path=./tester \
  --source=OCIRepository/flagger-loadtester \
  --export > ./clusters/dev/flagger-loadtester-kustomization.yaml

We're ready to commit our changes to our repo.

# pull the latest changes from the repo
git pull

# add the new files and commit the changes
git add -A
git commit -m 'feat: add flagger'
git push

This will trigger a FluxCD reconciliation and install Flagger in our cluster.

After a minute or two, run any of the following commands to see the status of the new resources.

flux get source helm
flux get source chart
flux get source oci
flux get helmrelease
flux get kustomization

Confirm that Flagger is installed and running.

kubectl get deploy -n flagger-system

Deploy a Canary

With Flagger installed, a Canary Custom Resource Definition (CRD) is available to us.

A Canary resource will automate some of our Kubernetes resources. It will create a Service, VirtualService, and a Canary deployment for us. So we don't need to create these resources ourselves.

Open the ./base/store-front.yaml manifest file using your favorite editor and remove the Service resource.

The Service resource looks like this.

---
apiVersion: v1
kind: Service
metadata:
  name: store-front
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: store-front

Next remove the VirtualService resource.

The VirtualService resource looks like this.

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: store-front
spec:
  hosts:
    - "*"
  gateways:
    - store-front
  http:
    - route:
        - destination:
            host: store-front
            port:
              number: 80

Finally, add the following Canary resource to the end of the manifest file.

---
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: store-front
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: store-front
  progressDeadlineSeconds: 60
  service:
    port: 80
    targetPort: 8080 # make sure this matches container port
    portDiscovery: true
    hosts:
      - "*"
    gateways:
      - store-front
  analysis:
    interval: 1m
    threshold: 10
    maxWeight: 20
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 30s
      - name: error-rate
        thresholdRange:
          max: 10
        interval: 30s
    webhooks:
      - name: acceptance-test
        type: pre-rollout
        url: http://flagger-loadtester.dev/
        timeout: 60s
        metadata:
          type: bash
          cmd: "curl -s http://store-front-canary.dev"
      - name: load-test
        url: http://flagger-loadtester.dev/
        timeout: 5s
        metadata:
          cmd: "hey -z 1m -q 10 -c 2 http://store-front-canary.dev"

There's a lot going on here, but essentially we are telling Flagger to create a Canary deployment for our store-front app. We are also telling Flagger to use the Istio ingress gateway to route traffic to our app and to use the load testing service to generate load against our app to generate metrics and analyze them to determine if the canary deployment should be promoted to production.

Note the threshold, maxWeight, and stepWeight values in the Canary manifest. We are going to start with 10% traffic directed to the canary and increase by 10%. Once the test is successful at 20% traffic, the Canary will promote the canary to primary and receive 100% of traffic. 20% is a low number here an intentionally set to speed up the testing process. Normally you would set this number closer to 100.

Commit and push the changes to your repo.

git add ./base/store-front.yaml
git commit -m 'feat: add store-front canary'
git push

Get Flux to reconcile the changes.

flux reconcile kustomization aks-store-demo --with-source

After Flux has finished reconciling, wait a minute or two then run the following command to watch the canary deployment and wait until the status is Initialized.

kubectl get canary -n dev store-front -w
# press ctrl-c to exit

You can view Flagger logs with the following command:

kubectl logs -n flagger-system deployment/flagger-system-flagger

Run the following command to view important resources.

kubectl get service,destinationrule,virtualservice -n dev

Flagger has created two Service resources (one for canary and one for primary), two DestinationRule resources (one to route traffic to canary and one to route to primary) and one VirtualService resource that can be configured for traffic shifting between the two services. Right now, the primary service is weighted to receive 100% of the traffic and you can confirm this by running the following command.

kubectl get virtualservice -n dev store-front -o yaml

With all this in place, ensure you can still access the AKS Store Demo app in your browser.

Test the Canary

Now we can test the progressive deployment of our canary.

Flip back to the aks-store-demo repo so that we can make another change to the TopNav.vue file.

Run the following commands that will update the version number in the TopNav.vue file from 1.0.0 to 2.0.0.

# set the version number
PREVIOUS_VERSION=1.0.0
CURRENT_VERSION=2.0.0

# make sure you are in the aks-store-demo directory
sed "s/Azure Pet Supplies v${PREVIOUS_VERSION}/Azure Pet Supplies v${CURRENT_VERSION}/g" src/store-front/src/components/TopNav.vue > TempTopNav.vue
mv TempTopNav.vue src/store-front/src/components/TopNav.vue

Commit and push the changes to your repo.

git add -A
git commit -m "feat: update title again"
git push

Create a new release in GitHub and watch the magic happen!

gh release create $CURRENT_VERSION --generate-notes

Wait a few seconds then run the following command to watch the release build.

gh run watch

With the new image built, the Flux ImagePolicy resource will reconcile, detect a new image tag and trigger the ImageUpdateAutomation resource reconciliation process.

The new image tag will be written to the kustomization.yaml manifest and the sample app's Kustomization resource will reconcile and update the its Deployment.

Here is where Flagger picks up the baton. Flagger will detect a new deployment revision and trigger a canary deployment. It will progressively route traffic to the new deployment based on the metrics we defined in the Canary resource. If the metrics are within the threshold, Flagger will promote the new deployment as the primary.

You can run the following commands to watch the image update process.

# watch image policy
flux logs --kind=ImagePolicy --name=store-front -f

# watch kustomization
flux logs --kind=Kustomization --name=aks-store-demo -f

# confirm the image tag was updated
kubectl get deploy -n dev store-front -o yaml | grep image:

You can then run the following command to watch the canary deployment.

kubectl logs -n flagger-system deployment/flagger-system-flagger -f | jq .msg

By the end of the Canary deployment process, you should see the following messages.

"New revision detected! Scaling up store-front.dev"
"Starting canary analysis for store-front.dev"
"Pre-rollout check acceptance-test passed"
"Advance store-front.dev canary weight 10"
"Advance store-front.dev canary weight 20"
"Copying store-front.dev template spec to store-front-primary.dev"
"Routing all traffic to primary"
"Promotion completed! Scaling down store-front.dev"

Now you can refresh the AKS Store Demo app in your browser and see the new version of the app 🥳

Conclusion

Image update automation is cool, but it's even cooler when you can implement some sort of gating process around it. Flagger is a great tool to help you implement progressive delivery strategies in your Kubernetes cluster. It works well with Istio and can be configured to use a variety of metrics providers and if it detects an issue, it will rollback the deployment. It's a great tool to have in your GitOps tool belt and will help you automate your deployments with confidence.

We've covered a lot of ground when it comes to GitOps and AKS but we have only scratched the surface. So stay tuned for more GitOps goodness!

Continue the conversation

If you have any feedback or suggestions, please feel free to reach out to me on Twitter or LinkedIn.

You can also find me in the Microsoft Open Source Discord, so feel free to DM me or drop a note in the cloud-native channel where my team hangs out!

Peace ✌️

Resources

Automating Image Updates on AKS: A Step-by-Step Guide using open source FluxCD

Paul Yu — Fri, 22 Sep 2023 13:00:00 +0000

In my previous post, we walked through the setup of FluxCD on AKS via AKS extensions. In this article, we'll go a bit deeper and take a look at how you can use FluxCD to automate image updates in your AKS cluster.

The goal here is to streamline the process of updating your application deployments in your cluster.

Here is our intended workflow:

Modify application code, then commit and push the change to the repo.
Create a new release in GitHub which kicks off a release workflow to build and push an updated container image to a GitHub Container Registry.
FluxCD detects the new image and updates the image tag in the cluster.
FluxCD rolls out the new image to the cluster.

We'll use same AKS store demo app we used in the previous post, but this time we'll go a bit faster.

If you need a refresher on how to deploy the application, you can refer to the previous post.

Prerequisites

Before you begin, you need to have the following:

Create an AKS cluster

Use the following Azure CLI commands to create a new resource group and AKS cluster.

RG_NAME=rg-imageupdate
AKS_NAME=aks-imageupdate
LOC_NAME=westus3

az group create -n $RG_NAME -l $LOC_NAME
az aks create -n $AKS_NAME -g $RG_NAME

When the cluster is created, run the following command to connect to the cluster.

az aks get-credentials -n $AKS_NAME -g $RG_NAME

Bootstrapping FluxCD

In my previous post, I used the AKS extension to install FluxCD. This time, I'll be using the Flux CLI to bootstrap FluxCD onto the AKS cluster. This process will enable us to have a bit more control over the installation and gives us the ability to save the Flux resources to our GitHub repo.

So we're going to implement GitOps even on top of our GitOps tooling 🤯

There's a few things we need to do before we can bootstrap the cluster. First, we need the Flux CLI. Once you have the CLI installed, run the following command to ensure your cluster is ready for bootstrapping.

flux check --pre

We also need a repo to land the Flux resources into. So let's fork and clone the aks-store-demo-manifests repository. This with be our repo for both application deployment and our GitOps configurations.

We'll be doing a bunch of things within GitHub. I prefer using the GitHub CLI over clicking around in the GitHub website.

Run the following commands to login to GitHub, clone the repo and prep it for our exercise.

# navigate to the home directory or wherever you want to clone the repo
cd ~

# login to GitHub
gh auth login --scopes repo,workflow

# fork and clone the repo to your local machine
gh repo fork https://github.com/pauldotyu/aks-store-demo-manifests.git --clone

# make sure you are in the aks-store-demo-manifests directory
cd aks-store-demo-manifests

# since we are in a forked repo, we need to set the default to be our fork
gh repo set-default

# merge the "kustomize" branch into "main"
git merge origin/kustomize

# push the change up to your fork
git push

We need to set some environment variables to for the Flux bootstrapping process. Run the following commands to set the environment variables.

# set the repo url
export GITHUB_REPO_URL=$(gh repo view --json url | jq .url -r)

# set your GitHub username
export GITHUB_USER=$(gh api user --jq .login)

# set your GitHub personal access token
export GITHUB_TOKEN=$(gh auth token)

Now we're ready to bootstrap FluxCD. Run the following command to bootstrap FluxCD. This command will install FluxCD with additional components to enable image automation. It will also generate the Flux resources and commit them to our Git repo in the clusters/dev directory.

flux bootstrap github create \
  --owner=$GITHUB_USER \
  --repository=aks-store-demo-manifests \
  --personal \
  --path=./clusters/dev \
  --branch=main \
  --reconcile \
  --network-policy \
  --components-extra=image-reflector-controller,image-automation-controller

After a few minutes, run the following commands and you should see the Flux resources in the repo.

git fetch
git rebase

# show the Flux resources
tree clusters/dev

In order for the image-automation-controller to write commits to our repo, we need to create a Kubernetes secret to store our GitHub credentials. Run the following command to create the secret.

flux create secret git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --username=$GITHUB_USER \
  --password=$GITHUB_TOKEN

This command will create Kubernetes secrets in your cluster but you could also use Azure Key Vault with the Secret Store CSI driver to store your GitHub credentials.

We don't need the GitHub PAT token anymore, so run the following command to discard it.

unset GITHUB_TOKEN

Next we need to create a GitRepository resource. This resource will point to our fork of the aks-store-demo-manifests repo where we have our app deployment and GitOps manifests stored.

Run the following command to create the configuration and export it to a YAML file which we'll commit to our repo.

flux create source git aks-store-demo \
  --url=$GITHUB_REPO_URL \
  --branch=main \
  --interval=1m \
  --secret-ref=aks-store-demo \
  --export > ./clusters/dev/aks-store-demo-source.yaml

We also need to specify the Kustomization resource to tell FluxCD where to find the app deployment manifests in our repo. Run the following command to create the configuration and export it to a YAML file which we'll also commit to our repo.

flux create kustomization aks-store-demo \
  --source=aks-store-demo \
  --path="./overlays/dev" \
  --prune=true \
  --wait=true \
  --interval=1m \
  --retry-interval=2m \
  --health-check-timeout=3m \
  --export > ./clusters/dev/aks-store-demo-kustomization.yaml

We have two new Flux resource manifests. Run the following command to commit and push the files to the repo so Flux can begin the reconciliation process.

git add -A
git commit -m "feat: add source and kustomization" 
git push

As soon as the code is pushed, Flux will do it's thing; reconcile. Run the following command to watch the Kustomization reconciliation process.

flux get kustomizations --watch

After a few minutes you should see something like this...

aks-store-demo  main@sha1:95de7bde  False   True    Applied revision: main@sha1:95de7bde

Confirm the GitRepository and Kustomization resources have been created.

flux get sources git 
flux get kustomizations

Run the following command to ensure all the Pods are running.

kubectl get po -n dev

Once you see all the Pods are running, run the following command to grab the public IP of the store-front service.

kubectl get svc store-front -n dev

Using a web browser, navigate to the public IP and confirm the application is up and running.

Setup the release workflow

The source code for the aks-store-demo application is located in a different repository than where our manifests are.

To give you a bit of a warning, we'll be flipping back and forth between the aks-store-demo and aks-store-demo-manifests repository directories.

Hint: you can use the cd - command to flip back and forth between the last two directories you were in.

The application code is in the aks-store-demo repository. Run the following commands to fork and clone the repo.

# navigate to the home directory or wherever you want to clone the repo
cd -

# fork and clone the repo to your local machine
gh repo fork https://github.com/azure-samples/aks-store-demo.git --clone

# make sure you are in the aks-store-demo directory
cd aks-store-demo

# since we are in a forked repo, we need to set the default to be our fork
gh repo set-default

Currently the aks-store-demo repository has Continuous Integration (CI) workflows to build and push container images using GITHUB_SHA as the image tag. We need to create a new release workflow that will build and push a new container image using semantic versioning. Flux's image update automation policy which is used to determine the most recent image tag can use various methods including numerical tags, alphabetical tags, and semver tags. So in this case, as much as I'd like to use the GITHUB_SHA, it does not provide a conducive way to determine "latest".

Therefore, we need to tag our releases with semantic version numbers. GitHub offers a way to create tagged releases. Additionaly, GitHub Actions have the ability to trigger workflows when a new release is published.

Run the following command to create a new workflow file.

# make sure you are in the aks-store-demo directory
touch .github/workflows/release-store-front.yaml

Open the release-store-front.yaml file using your favorite editor and paste the following code.

name: release-store-front

on:
  release:
    types: [published]

permissions:
  contents: read
  packages: write

jobs:
  publish-container-image:

    runs-on: ubuntu-latest

    steps:
      - name: Set environment variables
        id: set-variables
        run: |
          echo "REPOSITORY=ghcr.io/$(echo ${{ github.repository }} | tr '[:upper:]' '[:lower:]')" >> "$GITHUB_OUTPUT"
          echo "IMAGE=store-front" >> "$GITHUB_OUTPUT"
          echo "VERSION=$(echo ${GITHUB_REF#refs/tags/})" >> "$GITHUB_OUTPUT"
          echo "CREATED=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> "$GITHUB_OUTPUT"

      - name: Env variable output
        id: test-variables
        run: |
          echo ${{ steps.set-variables.outputs.REPOSITORY }}
          echo ${{ steps.set-variables.outputs.IMAGE }}
          echo ${{ steps.set-variables.outputs.VERSION }}
          echo ${{ steps.set-variables.outputs.CREATED }}

      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v1
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ github.token }}

      - name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: src/store-front
          file: src/store-front/Dockerfile
          push: true
          tags: |
            ${{ steps.set-variables.outputs.REPOSITORY }}/${{ steps.set-variables.outputs.IMAGE }}:latest
            ${{ steps.set-variables.outputs.REPOSITORY }}/${{ steps.set-variables.outputs.IMAGE }}:${{ steps.set-variables.outputs.VERSION }}
          labels: |
            org.opencontainers.image.source=${{ github.repositoryUrl }}
            org.opencontainers.image.created=${{ steps.set-variables.outputs.CREATED }}
            org.opencontainers.image.revision=${{ steps.set-variables.outputs.VERSION }}

This workflow will be triggered each time a new release is published and the ${GITHUB_REF#refs/tags/} bit allows us to extract the release version to use as an image tag.

Save the file, then commit and push the changes to the repo.

git add .github/workflows/release-store-front.yaml
git commit -m "feat: add release workflow"
git push

Update the application

Since we're in the app repo, let's make a change to it.

We'll make a very small edit to the src/store-front/src/components/TopNav.vue file and update the title of the application to include a version number.

Open the file and change the title on line 4 from Azure Pet Supplies to Azure Pet Supplies v1.0.0.

If you don't want to use an editor, you can run the following command to make the change using sed (I love and hate sed at the same time).

# make sure you are in the aks-store-demo directory
sed -i -e "s/Azure Pet Supplies/Azure Pet Supplies v1.0.0/g" src/store-front/src/components/TopNav.vue

Commit and push the changes to the repo.

git add src/store-front/src/components/TopNav.vue
git commit -m "feat: update title"
git push

Using GitHub CLI, create and publish a new release tagged as 1.0.0. This will trigger the release workflow we just created.

gh release create 1.0.0 --generate-notes

Wait about 10 seconds then run the following command to watch the workflow run.

# you can select the release workflow from the list
gh run watch

With the store-front container image release workflow setup, let's configure image update automation in Flux.

Configure image update automation

To setup FluxCD to listen for changes in the GitHub Container Registry, we need to create a new ImageRepository resource. You need to create an ImageRepository resource for each image you want to automate. In this case, we're only automating the store-front image.

⚠️ Flip back over to the aks-store-demo-manifests repo

Using the FluxCLI, run the following command to create the manifest for the ImageRepository resource.

flux create image repository store-front \
  --image=ghcr.io/$GITHUB_USER/aks-store-demo/store-front \
  --interval=1m \
  --export > ./clusters/dev/aks-store-demo-store-front-image.yaml

Run the following command to create an ImagePolicy resource to tell FluxCD how to determine the newest image tags. We'll use the semver filter to only allow image tags that are valid semantic versions and equal to or greater than 1.0.0.

flux create image policy store-front \
  --image-ref=store-front \
  --select-semver='>=1.0.0' \
  --export > ./clusters/dev/aks-store-demo-store-front-image-policy.yaml

There are other filters you can use as well. You can read more about them here.

Finally, run the following command to create an ImageUpdateAutomation resource which enables FluxCD to update images tags in our YAML manifests.

flux create image update store-front \
  --interval=1m \
  --git-repo-ref=aks-store-demo \
  --git-repo-path="./base" \
  --checkout-branch=main \
  --author-name=fluxcdbot \
  --author-email=fluxcdbot@users.noreply.github.com \
  --commit-template="{{range .Updated.Images}}{{println .}}{{end}}" \
  --export > ./clusters/dev/aks-store-demo-store-front-image-update.yaml

If you are uncomfortable with FluxCD making changes directly to the checkout branch (in this case main), you can create a separate branch for FluxCD (using the --push-branch parameter) to specify where commits should be pushed to. This will then enable you to follow your normal Git workflows (e.g., create a pull request to merge the changes into the main branch).

Run the following commands to commit and push the new Flux resources to the repo.

git add -A
git commit -m "feat: add image automation"
git push

After a few minutes, run the following command to see the status of the image update automation resources.

flux get image repository store-front
flux get image policy store-front
flux get image update store-front

The image update automation is setup but it won't do anything until we tell it which Deployments to update.

Update the manifest

We're going to sick with updating the store-front only. So we'll need to update the store-front deployment manifest to use the ImagePolicy resource we created earlier. This is done by marking the manifest with a comment.

Open the base/store-front.yaml file using your favorite editor.

On line 19, update the image to use your GitHub Container Registry. Then at the end of the line, add a comment to include the namespace and name of your ImagePolicy resource. This marks the deployment for Flux to update.

The comment is super important because Flux will not implement the image tag policy if it doesn't have this comment.

The line should look something like this:

image: ghcr.io/<REPLACE_THIS_WITH_YOUR_GITHUB_USERNAME>/aks-store-demo/store-front:latest # {"$imagepolicy": "flux-system:store-front"}

Setting the marker in the deployment manifest is fine for this demo, but ideally you'd want to set it in the kustomization manifest instead.

Commit and push the changes to the repo.

git add base/store-front.yaml
git commit -m "feat: add imagepolicy to store-front manifest"
git push

At this point, we've successfully setup image automation in our cluster. Time to test.

Test the image automation

We're going to test the developer workflow we laid out at the beginning of this article.

Make another change

Flip back over to the aks-store-demo repo

Make another change to the TopNav.vue file. This time, change the title to Azure Pet Supplies v2.0.0.

# make sure you are in the aks-store-demo directory
sed -i -e "s/Azure Pet Supplies v1.0.0/Azure Pet Supplies v2.0.0/g" src/store-front/src/components/TopNav.vue

Commit and push the changes to the repo.

git add src/store-front/src/components/TopNav.vue
git commit -m "feat: update title again"
git push

Create a new 2.0.0 release. This will trigger the release workflow.

gh release create 2.0.0 --generate-notes

# wait about 5 seconds then run the following command
gh run watch

Verify the image update

After a few minutes, you should see the new image tag in the aks-store-demo repo and the store-front deployment being updated in the cluster.

The reconcile interval for ImagePolicy was set to 1 minute. So after 1 minute or so, we should see that the update was successful.

flux get image policy store-front

Now check the store-front deployment to see the new image tag.

kubectl get deploy store-front -n dev -o yaml | grep image:

You should see that the image tag has been updated to 2.0.0.

Sweet! We've successfully automated image updates in our cluster using FluxCD and GitOps 🚀

Get the public IP of the store-front service by running the following command.

kubectl get svc store-front -n dev

Using a web browser, navigate to the public IP address of the store-front service. You should see the application running with our new title that includes v2.0.0 🥳

Conclusion

The power of GitOps is on full display here 🤖

In this article, we took a look at how you can use leverage an innovative feature of FluxCD to automate image updates in your AKS cluster. As an app developer, I can simply make my code changes and push it through a PR process. Once the PR is approved and merged and a new release is created, the image update automation kicks in and updates the image tag in the cluster for us without any manual intervention. This is a great way to streamline the deployment process and keep your cluster up to date with the latest images.

As an aside, I did all this open source FluxCD. This could also be done using the AKS extension for FluxCD but you do need to enable the image-automation-controller and image-reflector-controller components. You can find more information on how to do that here).

If you have any feedback or suggestions, please feel free to reach out to me on Twitter or LinkedIn.

Peace ✌️

Forem: Paul Yu

Installing VMWare Workstation Pro on Ubuntu Desktop

Download VMWare Workstation Pro

Install VMWare Workstation Pro

Manually install virtual machine modules

Configure VMWare Workstation Pro

Memory settings

Network settings

Summary

Uninstall VMWare Workstation Pro

Local LLMs: Running Ollama and Open WebUI in Docker on Ubuntu

Prerequisites

Step 1: Install Ollama

Step 2: Configure Ollama

Step 3: Run Open WebUI in a Docker Container

Step 4: Use Open WebUI with Ollama

Alternative options

Conclusion

Cleanup

References

Build a Weather MCP Server with Rust: Complete Tutorial for AI Integration

Prerequisites

Project setup

Building the weather server

Testing NWS API endpoints

Modeling the weather data

Adding helper functions

Implementing the weather tools

Testing with MCP Inspector

Summary

Learn more

Deploying AKS Automatic clusters with Pulumi: A step-by-step guide

Prerequisites

Login to Azure and Pulumi

Initialize Pulumi project

Setup main.go file

Running Pulumi programs

Randomizing resource names

Deploy a container registry

Deploy observability tools

Add Azure Log Analytics Workspace

Add Azure Managed Prometheus

Add Azure Managed Grafana

Provision Azure resources

Deploy AKS Automatic cluster

Deploy role assignments

Grant cluster permissions for pulling images

Grant permissions for monitoring

Grant yourself permissions for access

Provision role assignments

Test the deployment

What's next?

Clean up

Summary

Resources

Certified Argo Project Associate (CAPA) Exam Study Guide

Argo Workflows 36%

Understand Argo Workflow Fundamentals

Generating and Consuming Artifacts

Understand Argo Workflow Templates

Understand the Argo Workflow Spec

Work with DAG (Directed-Acyclic Graphs)

Run Data Processing Jobs with Argo Workflows

Practice

Argo CD 34%

Understand Argo CD Fundamentals

Synchronize Applications with Argo CD

Use Argo CD Application

Configure Argo CD with Helm and Kustomize

Identify Common Reconciliation Patterns

Practice

Argo Rollouts 18%

Understand Argo Rollouts Fundamentals

Use Common Progressive Rollout Strategies

Describe Analysis Template and Analysis Run

Practice

Argo Events 12%

Understand Argo Events Fundamentals

Understand Argo Event Components and Architecture

Practice

Local LLMs: Running Ollama and Open WebUI in Docker on Ubuntu