Setting Up Side-by-Side Python Versions on Azure DevOps Self-Hosted Agents

Rich Evans — Wed, 12 Nov 2025 01:18:33 +0000

Setting Up Side-by-Side Python Versions on Azure DevOps Self-Hosted Agents

You've set up an Azure DevOps self-hosted agent and need Python for your pipelines. You install Python, configure your pipeline with UsePythonVersion@0, and the job fails - the agent can't find it.

The ADO documentation isn't clear on how to properly set up Python so the agent can use it, especially when you need multiple versions side-by-side. This guide shows you exactly how to do it using uv and symlinks.

The Problem

ADO agents won't use your system-installed Python. The UsePythonVersion task only looks in _work/_tool/Python/ with this exact structure:

_work/_tool/
  └── Python/
      ├── 3.12.12/
      │   ├── x64/
      │   │   └── bin/
      │   └── x64.complete
      └── 3.12/
          ├── x64/
          │   └── bin/
          └── x64.complete

The agent needs both the full version (3.12.12) and major.minor (3.12) so pipelines can request either. Miss the .complete file or put it in the wrong place, and ADO ignores the installation entirely.

Note: Examples use Python 3.12.12 (latest 3.12.x at time of writing).

The Solution & Why It Works

Instead of manually installing Python or fighting with system packages, use uv (the fast Python installer) and symlinks:

Why symlinks instead of copies?

Instant setup (no file copying)
Zero disk space overhead
Atomic updates (change one symlink)
Multiple versions coexist without conflicts

Why separate full and major.minor versions?

Production pipelines pin to 3.12.12 (never changes unexpectedly)
Dev pipelines use 3.12 (gets security updates automatically)
Explicit control via --update flag prevents accidental breaking changes

Why this matters: Without the --update flag, installing Python 3.12.12 creates that specific version but leaves 3.12 pointing to 3.12.11. With --update, it creates 3.12.12 and updates the 3.12 symlink to point to it. This means you can test new patch versions before promoting them.

Quick Start

# Install uv if needed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Run the script (see full script at end of post)
cd /path/to/ado-agent
./setup-python-ado.sh 3.12

# Restart agent
cd /path/to/agent && sudo ./svc.sh restart

# Later, when updating to a new patch version:
./setup-python-ado.sh 3.12 --update

Key Implementation Details

Here are the tricky parts worth understanding—these are where things typically break:

1. Finding uv's Python installation

PYTHON_PATH=$(uv python find $PYTHON_VERSION)
UV_PYTHON_DIR=$(dirname $(dirname $PYTHON_PATH))

uv python find returns the path to the Python binary (e.g., /home/user/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/bin/python3.12). We go up two directories to get the installation root we need to symlink.

2. Symlinking to ADO's expected structure

mkdir -p $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64
ln -sf $UV_PYTHON_DIR/* $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64/

This creates the directory structure ADO expects and symlinks the entire Python installation into it. Result: _work/_tool/Python/3.12.12/x64/bin/python3.12 points to uv's installation.

3. Creating the completion marker

touch $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64.complete

ADO only recognizes a Python installation if there's an x64.complete file next to the x64/ directory. Without it, the UsePythonVersion task ignores the installation entirely.

4. Validating the symlink worked

[[ ! -e "python${PYTHON_MAJOR_MINOR}" ]] && { log_error "Missing python binary"; exit 1; }

After symlinking, verify the expected binary exists. If uv changes its structure, this catches it before your pipeline does.

Usage

Basic Setup

cd /path/to/ado-agent
./setup-python-ado.sh 3.12

Installs Python 3.12 and configures both full version and major.minor.

Multiple Versions

./setup-python-ado.sh 3.11
./setup-python-ado.sh 3.12
./setup-python-ado.sh 3.13

All versions coexist. Pipelines choose which to use.

Update Workflow

When Python 3.12.13 releases:

# Step 1: Install the specific new version without updating major.minor
./setup-python-ado.sh 3.12.13

# Step 2: Test in dev pipeline using versionSpec: '3.12.13'

# Step 3: Promote to major.minor after validation
./setup-python-ado.sh 3.12.13 --update

Pipeline Configuration

# Development - gets latest 3.12.x automatically
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.12'

# Production - pinned version, never changes
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.12.12'

Troubleshooting

"uv not installed"

curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc

"No write permission"

sudo chown -R $USER:$USER /path/to/agent/_work

"Failed to install Python X.Y"

Invalid version: Check Python releases
Network issue: Verify connectivity
Use --verbose for detailed error

Pipeline still can't find Python

Restart the agent:

# Method 1: Using svc.sh
cd /path/to/agent && sudo ./svc.sh restart

# Method 2: Using systemctl
sudo systemctl restart vsts.agent*
# or: sudo systemctl restart azpipelines.agent*

Verify installation:

ls -la /path/to/agent/_work/_tool/Python/
/path/to/agent/_work/_tool/Python/3.12/x64/bin/python --version

Check for .complete files:

find /path/to/agent/_work/_tool/Python -name "*.complete"

Credits

Inspired by Alex Kaszynski's "Create an Azure Self-Hosted Agent with Python without going Insane". The original 2021 approach used Python venv. This script automates the process and uses uv for faster installation and better version management.

Quick Reference

# Install version
./setup-python-ado.sh 3.12

# Update to latest patch
./setup-python-ado.sh 3.12 --update

# Debug issues
./setup-python-ado.sh 3.12 --verbose

# Check versions
ls -la /path/to/agent/_work/_tool/Python/

# Verify
/path/to/agent/_work/_tool/Python/3.12/x64/bin/python --version

# Restart
cd /path/to/agent && sudo ./svc.sh restart

Full Script

The complete, production-ready script with error handling and verbose logging. Save as setup-python-ado.sh:

#!/bin/bash
set -e

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

PYTHON_VERSION=""
UPDATE_MODE=false
VERBOSE=false

log_error() { echo -e "${RED}Error: $1${NC}" >&2; }
log_success() { echo -e "${GREEN}$1${NC}"; }
log_info() { echo -e "${BLUE}$1${NC}"; }
log_warning() { echo -e "${YELLOW}$1${NC}"; }

show_usage() {
    echo "Usage: $0 <python-version> [--update] [--verbose]"
    echo ""
    echo "Arguments:"
    echo "  <python-version>  Python version (e.g., 3.12, 3.11, 3.13)"
    echo "  --update          Force update major.minor symlink"
    echo "  --verbose         Show detailed output"
    exit 1
}

# Parse arguments
while [[ $# -gt 0 ]]; do
    case $1 in
        --update) UPDATE_MODE=true; shift ;;
        --verbose) VERBOSE=true; shift ;;
        -h|--help) show_usage ;;
        *)
            if [[ -z "$PYTHON_VERSION" ]]; then
                PYTHON_VERSION=$1
            else
                log_error "Unknown argument '$1'"
                show_usage
            fi
            shift
            ;;
    esac
done

# Validate version
if [[ -z "$PYTHON_VERSION" ]]; then
    log_error "Python version required"
    show_usage
fi

if ! [[ "$PYTHON_VERSION" =~ ^[0-9]+\.[0-9]+(\.[0-9]+)?$ ]]; then
    log_error "Invalid version format. Use X.Y or X.Y.Z"
    exit 1
fi

echo ""
echo "Python $PYTHON_VERSION Setup for ADO Agent"
echo "=========================================="
echo ""

# Setup directories
AGENT_ROOT_DIR="${ADO_AGENT_DIR:-$(pwd)}"
log_info "Agent directory: $AGENT_ROOT_DIR"

if [[ ! -d "$AGENT_ROOT_DIR/_work" ]]; then
    log_warning "No '_work' folder found"
    read -p "Continue? (y/N): " -n 1 -r
    echo
    [[ ! $REPLY =~ ^[Yy]$ ]] && exit 0
fi

AGENT_TOOL_DIR="$AGENT_ROOT_DIR/_work/_tool"

# Check permissions
if [[ ! -w "$(dirname "$AGENT_TOOL_DIR")" ]]; then
    log_error "No write permission"
    echo "Fix: sudo chown -R \$USER:\$USER $AGENT_ROOT_DIR/_work"
    exit 1
fi

# Verify uv is installed
if ! command -v uv &> /dev/null; then
    log_error "uv not installed"
    echo "Install: curl -LsSf https://astral.sh/uv/install.sh | sh"
    exit 1
fi

UV_VERSION=$(uv --version 2>/dev/null || echo "unknown")
log_success "Found uv ($UV_VERSION)"
echo ""

# Install Python if needed
log_info "Checking for Python $PYTHON_VERSION..."
if ! uv python find $PYTHON_VERSION &> /dev/null; then
    log_info "Installing Python $PYTHON_VERSION..."

    if [[ "$VERBOSE" == true ]]; then
        uv python install $PYTHON_VERSION || {
            log_error "Installation failed"
            echo "Possible causes: invalid version, network issues, disk space"
            exit 1
        }
    else
        uv python install $PYTHON_VERSION > /dev/null 2>&1 || {
            log_error "Installation failed"
            echo "Run with --verbose: $0 $PYTHON_VERSION --verbose"
            exit 1
        }
    fi

    log_success "Installed Python $PYTHON_VERSION"
else
    log_success "Found Python $PYTHON_VERSION"
fi

# Get Python info
PYTHON_PATH=$(uv python find $PYTHON_VERSION)
[[ ! -x "$PYTHON_PATH" ]] && { log_error "Not executable: $PYTHON_PATH"; exit 1; }

PYTHON_FULL_VERSION=$($PYTHON_PATH --version 2>&1 | awk '{print $2}')
[[ -z "$PYTHON_FULL_VERSION" ]] && { log_error "Could not determine version"; exit 1; }

PYTHON_MAJOR_MINOR=$(echo $PYTHON_FULL_VERSION | cut -d. -f1,2)
UV_PYTHON_DIR=$(dirname $(dirname $PYTHON_PATH))

log_info "Full version: $PYTHON_FULL_VERSION"
log_info "Major.minor: $PYTHON_MAJOR_MINOR"
echo ""

[[ ! -d "$UV_PYTHON_DIR/bin" ]] && { log_error "Missing bin directory"; exit 1; }

# Check existing installation
MAJOR_MINOR_EXISTS=false
if [[ -d "$AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR" ]]; then
    MAJOR_MINOR_EXISTS=true
    EXISTING_LINK=$(readlink -f "$AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR/x64/bin/python$PYTHON_MAJOR_MINOR" 2>/dev/null || echo "")

    if [[ -n "$EXISTING_LINK" && -x "$EXISTING_LINK" ]]; then
        EXISTING_VERSION=$($EXISTING_LINK --version 2>&1 | awk '{print $2}')
        log_warning "Python $PYTHON_MAJOR_MINOR exists (currently: $EXISTING_VERSION)"

        if [[ "$UPDATE_MODE" == false ]]; then
            log_info "Use --update to replace with $PYTHON_FULL_VERSION"
            log_info "Configuring only: $PYTHON_FULL_VERSION"
            echo ""
        else
            log_info "Update mode: replacing with $PYTHON_FULL_VERSION"
            echo ""
        fi
    fi
fi

# Configure full version
log_info "Configuring $PYTHON_FULL_VERSION..."
mkdir -p $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64

ln -sf $UV_PYTHON_DIR/* $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64/ || {
    log_error "Symlink failed for $PYTHON_FULL_VERSION"
    exit 1
}

cd $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64/bin
[[ ! -e "python${PYTHON_MAJOR_MINOR}" ]] && { log_error "Missing python binary"; exit 1; }

test -L python || ln -sf python${PYTHON_MAJOR_MINOR} python
test -L python3 || ln -sf python${PYTHON_MAJOR_MINOR} python3

touch $AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64.complete
log_success "Configured $PYTHON_FULL_VERSION"

# Configure major.minor if needed
if [[ "$MAJOR_MINOR_EXISTS" == false ]] || [[ "$UPDATE_MODE" == true ]]; then
    echo ""
    log_info "Configuring $PYTHON_MAJOR_MINOR..."

    [[ "$UPDATE_MODE" == true ]] && [[ "$MAJOR_MINOR_EXISTS" == true ]] && rm -rf $AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR

    mkdir -p $AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR/x64

    ln -sf $UV_PYTHON_DIR/* $AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR/x64/ || {
        log_error "Symlink failed for $PYTHON_MAJOR_MINOR"
        exit 1
    }

    cd $AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR/x64/bin
    test -L python || ln -sf python${PYTHON_MAJOR_MINOR} python
    test -L python3 || ln -sf python${PYTHON_MAJOR_MINOR} python3

    touch $AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR/x64.complete
    log_success "Configured $PYTHON_MAJOR_MINOR"
fi

# Verify
echo ""
echo "Verification"
echo "============"
$AGENT_TOOL_DIR/Python/$PYTHON_FULL_VERSION/x64/bin/python --version || {
    log_error "Verification failed"
    exit 1
}

if [[ "$MAJOR_MINOR_EXISTS" == false ]] || [[ "$UPDATE_MODE" == true ]]; then
    $AGENT_TOOL_DIR/Python/$PYTHON_MAJOR_MINOR/x64/bin/python --version || {
        log_error "Verification failed"
        exit 1
    }
fi

echo ""
log_success "Setup complete"
echo ""
echo "Available versions:"
echo "  $PYTHON_FULL_VERSION"
if [[ "$MAJOR_MINOR_EXISTS" == false ]] || [[ "$UPDATE_MODE" == true ]]; then
    echo "  $PYTHON_MAJOR_MINOR -> $PYTHON_FULL_VERSION"
elif [[ "$MAJOR_MINOR_EXISTS" == true ]]; then
    echo "  $PYTHON_MAJOR_MINOR -> $EXISTING_VERSION (unchanged)"
fi
echo ""
echo "Next steps:"
echo "  1. Restart: cd /path/to/agent && sudo ./svc.sh restart"
echo "  2. Use in pipeline: UsePythonVersion@0"
echo ""

Forem: Rich Evans

Setting Up Side-by-Side Python Versions on Azure DevOps Self-Hosted Agents

Setting Up Side-by-Side Python Versions on Azure DevOps Self-Hosted Agents

The Problem

The Solution & Why It Works

Quick Start

Key Implementation Details

Usage

Basic Setup

Multiple Versions

Update Workflow

Pipeline Configuration

Troubleshooting

Credits

Quick Reference

Full Script