Forem: Alexandre Vazquez

XSLT grouping with xsl:for-each-group: complete guide

Alexandre Vazquez — Mon, 11 May 2026 09:00:00 +0000

Grouping is one of the most powerful features introduced in XSLT 2.0. Before it, grouping in XSLT 1.0 required the Muenchian method — a clever but verbose technique involving keys and node-set comparisons. In 2.0, xsl:for-each-group makes grouping straightforward.

Basic grouping with group-by

group-by groups nodes that share the same value for a given expression. The result is one iteration per distinct group value.

Input:


  DE120
  US85
  DE200
  FR60
  US140

Stylesheet:

Output:


  2320
  160
  2225

Key functions inside for-each-group:

current-grouping-key() — returns the value that defines the current group
current-group() — returns the sequence of all nodes in the current group

Nested grouping

Groups can be nested. Group orders by country, then within each country by status:

group-adjacent

Groups consecutive nodes that share the same key value. Unlike group-by, it starts a new group when the key changes, even if the same key appeared earlier. This is useful for processing structured text or segmented data.


  Starting
  Processing
  Failed
  Retrying
  Done

This produces three blocks: two INFO (positions 1-2), one ERROR (3-4), one INFO (5). With group-by, the two INFO groups would be merged into one.

group-starting-with and group-ending-with

These group nodes based on a pattern match rather than a key value. Every time a node matches the pattern, a new group starts (or ends).

group-starting-with example — treat every ## as the start of a section:



    <xsl:value-of select="self::h2"/>

group-ending-with example — group lines until a blank line:

Computing aggregates

current-group() returns a sequence, so you can apply any XPath aggregate function directly:

Try it in XSLT Playground

Paste any of the examples above into XSLT Playground with version set to 2.0 or 3.0. Grouping is one of the features that benefits most from live testing — you can immediately see how changing the grouping key or switching between group-by and group-adjacent affects the output structure.

XSLT template matching explained with examples

Alexandre Vazquez — Thu, 07 May 2026 09:00:00 +0000

Template matching is the mechanism that drives every XSLT transformation. Understanding how the processor selects templates — and what happens when multiple templates could match — is the difference between a stylesheet that works reliably and one that produces surprising output. This post covers everything you need to know.

How match patterns work

When the processor visits a node, it evaluates every template's match attribute as an XPath pattern. A pattern is a restricted form of XPath that tests properties of a node rather than selecting nodes from a starting point. If the pattern is satisfied, that template is a candidate.

Priority and conflict resolution

More than one template can match the same node. The processor resolves the conflict using priority. Each pattern has a default priority calculated by the spec:

Pattern type	Default priority
`node()` or `*`	-0.5
`element-name`	0
`prefix:element-name`	0
`a/b` (path)	0.5
`a[predicate]`	0.5
`@attr`	0

More specific patterns automatically get higher priority. You can override this with the priority attribute:

If two templates have equal computed priority, the processor signals an error (or picks the last one, depending on implementation — Saxon issues an error by default). Always assign explicit priorities when you have competing templates.

The built-in templates

XSLT has default templates for every node type. If no explicit template matches a node, the built-in fires. For elements and the document root, the built-in calls apply-templates on all children. For text and attribute nodes, it outputs the string value.

This means that without any templates at all, the processor will walk the entire tree and output all text content. Understanding this explains why simple stylesheets can produce unexpected extra text — a text node matched nothing explicit, and the built-in output it.

To suppress text output globally, add:

This overrides the built-in with an empty template, producing no output for text nodes that are not handled elsewhere.

Modes

Modes let you have multiple templates for the same node that serve different purposes. A mode is a named context for a set of templates.

##

Call with mode:

Modes are especially useful when you need to process the same nodes in multiple places in the output with different logic each time.

apply-templates vs for-each

Both iterate over a set of nodes. The difference is that apply-templates dispatches to the best matching template for each node, while for-each stays in the current context and does not do template lookup.

Use apply-templates when you want polymorphism — different node types handled differently. Use for-each when you are doing a simple iteration over a homogeneous set and do not need dispatch.

Testing patterns in XSLT Playground

XSLT Playground is a fast way to experiment with matching rules. Paste a stylesheet with multiple competing templates and check which one fires. Add `` in each template to trace which one the processor picks. The trace panel shows all messages in order so you can follow the dispatch chain.

Solid understanding of template matching pays off every time you work on a complex stylesheet. Once you know how priorities and built-ins interact, most "unexpected output" bugs become obvious.

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Alexandre Vazquez — Tue, 05 May 2026 11:00:00 +0000

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Introduction

Most observability stacks running in production for over a year end up with alerting spread across two systems: Prometheus Alertmanager handling metric-based alerts and Grafana Alerting managing everything else. This creates the "alerting consolidation problem" where on-call teams receive duplicated pages, silencing rules live in two places, and nobody is certain which system is authoritative.

The question is straightforward: should you standardize on Prometheus Alertmanager, move everything into Grafana Alerting, or deliberately run both? The answer depends on your datasource mix, your GitOps maturity, and how your organization manages on-call routing.

Architecture Overview

Prometheus Alertmanager: The Standalone Receiver

Alertmanager is a dedicated, standalone component in the Prometheus ecosystem. It does not evaluate alert rules itself. Instead, Prometheus (or compatible senders like Thanos Ruler, Cortex, or Mimir Ruler) evaluates PromQL expressions and pushes firing alerts to the Alertmanager API. Alertmanager then handles deduplication, grouping, inhibition, silencing, and notification delivery.

# Simplified Prometheus → Alertmanager flow
#
# [Prometheus] --evaluates rules--> [firing alerts]
#        |
#        +--POST /api/v2/alerts--> [Alertmanager]
#                                      |
#                          +-----------+-----------+
#                          |           |           |
#                       [Slack]    [PagerDuty]  [Email]

The entire configuration lives in a single YAML file (alertmanager.yml). This includes the routing tree, receiver definitions, inhibition rules, and silence templates. There is no database, no UI-driven state — just a config file and an optional local storage directory for notification state and silences. This makes it trivially reproducible and ideal for GitOps workflows.

For high availability, you run multiple Alertmanager instances in a gossip-based cluster. They use a mesh protocol to share silence and notification state, ensuring that failover does not result in duplicate or lost notifications.

Grafana Alerting: The Integrated Platform

Grafana Alerting (sometimes called "Grafana Unified Alerting," introduced in Grafana 8) takes a different architectural approach. It embeds the entire alerting lifecycle — rule evaluation, state management, routing, and notification — inside the Grafana server process. Under the hood, it uses a fork of Alertmanager for the routing and notification layer.

# Simplified Grafana Alerting flow
#
# [Grafana Server]
#   ├── Rule Evaluation Engine
#   │     ├── queries Prometheus
#   │     ├── queries Loki
#   │     ├── queries CloudWatch
#   │     └── queries any supported datasource
#   │
#   ├── Alert State Manager (internal)
#   │
#   └── Embedded Alertmanager (routing + notifications)
#           |
#           +-----------+-----------+
#           |           |           |
#        [Slack]    [PagerDuty]  [Email]

The critical distinction is that Grafana Alerting evaluates alert rules itself, querying any configured datasource — not just Prometheus. It can fire alerts based on Loki log queries, Elasticsearch searches, CloudWatch metrics, PostgreSQL queries, or any of the 100+ datasource plugins available in Grafana. Rule definitions, contact points, notification policies, and mute timings are stored in the Grafana database (or provisioned via YAML files and the Grafana API).

Feature Comparison

Feature	Prometheus Alertmanager	Grafana Alerting
Datasources	Prometheus-compatible only (Prometheus, Thanos, Mimir, VictoriaMetrics)	Any Grafana datasource (Prometheus, Loki, Elasticsearch, CloudWatch, SQL databases, etc.)
Rule evaluation	External (Prometheus/Ruler evaluates rules and pushes alerts)	Built-in (Grafana evaluates rules directly)
Routing tree	Hierarchical YAML-based routing with match/match_re, continue, group_by	Notification policies with label matchers, nested policies, mute timings
Grouping	Full support via group_by, group_wait, group_interval	Full support via notification policies with equivalent controls
Inhibition	Native inhibition rules (suppress alerts when a related alert is firing)	Supported since Grafana 10.3 but less flexible than Alertmanager
Silencing	Label-based silences via API or UI, time-limited	Mute timings (recurring schedules) and silences (ad-hoc, label-based)
Notification channels	Email, Slack, PagerDuty, OpsGenie, VictoriaOps, webhook, WeChat, Telegram, SNS, Webex	All of the above plus Teams, Discord, Google Chat, LINE, Threema, Oncall, and more via contact points
Templating	Go templates in notification config	Go templates with access to Grafana template variables and functions
Multi-tenancy	Not built-in; achieved via separate instances or Mimir Alertmanager	Native multi-tenancy via Grafana organizations and RBAC
High availability	Gossip-based cluster (peer mesh, well-proven)	Database-backed HA with peer discovery between Grafana instances
Configuration model	Single YAML file, fully declarative	UI + API + provisioning YAML files, stored in database
GitOps compatibility	Excellent — config file lives in version control natively	Possible via provisioning files or Terraform provider, but requires extra tooling
External alert sources	Any system that can POST to the Alertmanager API	Supported via the Grafana Alerting API (external alerts can be pushed)
Managed service	Available via Grafana Cloud (as Mimir Alertmanager), Amazon Managed Prometheus	Available via Grafana Cloud

Alertmanager Strengths

Alertmanager has been a production staple since 2015. Over a decade of use across thousands of organizations has made it one of the most battle-tested components in the CNCF ecosystem.

Declarative, GitOps-Native Configuration

The entire Alertmanager configuration is a single YAML file. There is no hidden state in a database, no click-driven configuration that someone forgets to document. You check it into Git, review it in a pull request, and deploy it through your CI/CD pipeline like any other infrastructure code.

# alertmanager.yml — everything in one file
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/T00/B00/XXX"

route:
  receiver: platform-team
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
      group_wait: 10s
    - match_re:
        team: "^(payments|checkout)$"
      receiver: payments-slack
      continue: true

receivers:
  - name: platform-team
    slack_configs:
      - channel: "#platform-alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: ""
  - name: payments-slack
    slack_configs:
      - channel: "#payments-oncall"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, cluster]

Every change is auditable. Rollbacks are a git revert away. This matters enormously when you are debugging why an alert did not fire at 3 AM.

Lightweight and Single-Purpose

Alertmanager does one thing: route and deliver notifications. It has no dashboard, no query engine, no datasource plugins. This single-purpose design makes it operationally simple. Resource consumption is minimal — a small Alertmanager instance handles thousands of active alerts on a few hundred megabytes of memory. It starts in milliseconds and requires almost no maintenance.

Mature Inhibition and Routing

Alertmanager's inhibition rules are first-class citizens. You can suppress downstream warnings when a critical alert is already firing, preventing alert storms from overwhelming your on-call team. The hierarchical routing tree with continue flags allows for nuanced delivery: send to the team channel AND escalate to PagerDuty simultaneously, with different grouping strategies at each level.

Proven High Availability

The gossip-based HA cluster has been stable for years. Running three Alertmanager replicas behind a load balancer (or using Kubernetes service discovery) gives you reliable notification delivery without shared storage. The protocol handles deduplication across instances automatically, which is the hardest part of distributed alerting.

Grafana Alerting Strengths

Grafana Alerting has matured considerably since its rocky introduction in Grafana 8. By Grafana 11 and 12, it has become a legitimate production alerting platform with capabilities that Alertmanager cannot match on its own.

Multi-Datasource Alert Rules

This is Grafana Alerting's strongest differentiator. You can write alert rules that query Loki for error log spikes, CloudWatch for AWS resource utilization, Elasticsearch for application errors, or a PostgreSQL database for business metrics — all from the same alerting system. If your observability stack includes more than just Prometheus, this eliminates the need for separate alerting tools per datasource.

# Grafana alert rule provisioning example — alerting on Loki log errors
apiVersion: 1
groups:
  - orgId: 1
    name: application-errors
    folder: Production
    interval: 1m
    rules:
      - uid: loki-error-spike
        title: "High error rate in payment service"
        condition: C
        data:
          - refId: A\            datasourceUid: loki-prod
            model:
              expr: 'sum(rate({app="payment-service"} |= "ERROR" [5m]))'
          - refId: B
            datasourceUid: "__expr__"
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: "__expr__"
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [10]
        for: 5m
        labels:
          severity: warning
          team: payments

This is something Alertmanager simply cannot do. Alertmanager only receives pre-evaluated alerts — it has no concept of datasources or query execution.

Unified UI for Alert Management

Grafana provides a single pane of glass for alert rule creation, visualization, notification policy management, contact point configuration, and silence management. For teams where not every engineer is comfortable editing YAML routing trees, the visual notification policy editor significantly reduces the barrier to entry. You can see the state of every alert rule, its evaluation history, and the exact notification path it will take — all without leaving the browser.

Native Multi-Tenancy and RBAC

Grafana's organization model and role-based access control extend naturally to alerting. Different teams can manage their own alert rules, contact points, and notification policies within their organization or folder scope, without seeing or interfering with other teams. Achieving this with standalone Alertmanager requires either running separate instances per tenant or using Mimir's multi-tenant Alertmanager.

Mute Timings and Richer Scheduling

While Alertmanager supports silences (ad-hoc, time-limited suppressions), Grafana Alerting adds mute timings — recurring time-based windows where notifications are suppressed. This is useful for scheduled maintenance windows, business-hours-only alerting, or suppressing non-critical alerts on weekends. Alertmanager requires external tooling or manual silence creation for recurring windows.

Grafana Cloud as a Managed Option

For teams that want to avoid managing alerting infrastructure entirely, Grafana Cloud provides a fully managed Grafana Alerting stack. This includes HA, state persistence, and notification delivery without any self-hosted components. The Grafana Cloud alerting stack also includes a managed Mimir Alertmanager, which means you can use Prometheus-native alerting rules if you prefer that model while still benefiting from the managed infrastructure.

When to Use Prometheus Alertmanager

Alertmanager is the right choice when the following conditions describe your environment:

Your metrics stack is Prometheus-native. If all your alert rules are PromQL expressions evaluated by Prometheus, Thanos Ruler, or Mimir Ruler, Alertmanager is the natural fit. There is no added value in routing those alerts through Grafana.
GitOps is non-negotiable. If every infrastructure change must go through a pull request and be fully declarative, Alertmanager's single-file configuration model is significantly easier to manage than Grafana's database-backed state. Tools like amtool provide config validation in CI pipelines.
You need fine-grained routing with inhibition. Complex routing trees with multiple levels of grouping, inhibition rules, and continue flags are more naturally expressed in Alertmanager's YAML format. The routing logic has been stable and well-documented for years.
You run microservices with per-team routing. If each team owns its routing subtree and the routing logic is complex, Alertmanager's hierarchical model scales better than UI-driven configuration. Teams can own their section of the config file via CODEOWNERS in Git.
You want minimal operational overhead. Alertmanager is a single binary with minimal resource requirements. There is no database to back up, no migrations to run, and no UI framework to keep updated.

When to Use Grafana Alerting

Grafana Alerting is the right choice when these conditions apply:

You alert on more than just Prometheus metrics. If you need alert rules based on Loki logs, Elasticsearch queries, CloudWatch metrics, or database queries, Grafana Alerting is the only option that handles all of these natively. The alternative is running separate alerting tools per datasource, which is worse.
Your team prefers UI-driven configuration. Not every engineer wants to edit YAML routing trees. If your organization values a visual interface for managing alerts, contact points, and notification policies, Grafana's UI is a major productivity advantage.
You are using Grafana Cloud. If you are already on Grafana Cloud, using its built-in alerting is the path of least resistance. You get HA, managed notification delivery, and a unified experience without running any additional infrastructure.
Multi-tenancy is a requirement. If multiple teams need isolated alerting configurations with RBAC, Grafana's native organization and folder-based access model is significantly easier to set up than running per-tenant Alertmanager instances.
You want mute timings for recurring maintenance windows. If your team regularly needs to suppress alerts during scheduled windows (deploy windows, batch processing hours, weekend non-critical suppression), Grafana's mute timings feature is more ergonomic than creating and managing recurring silences in Alertmanager.

Running Both Together: The Hybrid Pattern

In practice, many production environments run both Alertmanager and Grafana Alerting. This is not necessarily a mistake — it can be a deliberate architectural choice when done with clear boundaries.

Common Hybrid Architecture

The most common pattern looks like this:

Prometheus Alertmanager handles all metric-based alerts. PromQL rules are evaluated by Prometheus or a long-term storage ruler (Thanos, Mimir). Alertmanager owns routing, grouping, and notification for these alerts.
Grafana Alerting handles non-Prometheus alerts: log-based alerts from Loki, business metrics from SQL datasources, and cross-datasource correlation rules.

The key to making this work without chaos is establishing clear ownership rules:

# Ownership boundaries for hybrid alerting
#
# Prometheus Alertmanager owns:
#   - All PromQL-based alert rules
#   - Infrastructure alerts (node, kubelet, etcd, CoreDNS)
#   - Application SLO/SLI alerts based on metrics
#
# Grafana Alerting owns:
#   - Log-based alert rules (Loki, Elasticsearch)
#   - Business metric alerts (SQL datasources)
#   - Cross-datasource correlation rules
#   - Alerts for teams that prefer UI-driven management
#
# Shared:
#   - Contact points / receivers use the same Slack channels and PagerDuty services
#   - On-call rotations are managed externally (PagerDuty, Grafana OnCall)

Both systems can deliver to the same notification channels. The critical discipline is ensuring that silencing and maintenance windows are applied in both systems when needed. This is the primary operational cost of the hybrid approach.

Grafana as a Viewer for Alertmanager

Even if you use Alertmanager exclusively for routing and notification, Grafana can serve as a read-only viewer. Grafana natively supports connecting to an external Alertmanager datasource, allowing you to see firing alerts, active silences, and alert groups in the Grafana UI. This gives you the operational visibility of Grafana without moving your alerting logic into it.

# Grafana datasource provisioning for external Alertmanager
apiVersion: 1
datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://alertmanager.monitoring.svc:9093
    access: proxy
    jsonData:
      implementation: prometheus

Migration Considerations

If you are moving from one system to the other, here are the practical considerations to plan for.

Migrating from Alertmanager to Grafana Alerting

Rule conversion. Your PromQL-based recording and alerting rules defined in Prometheus rule files need to be recreated as Grafana alert rules. Grafana provides a migration tool that can import Prometheus-format rules, but complex expressions may need manual adjustment.
Routing tree translation. Alertmanager's hierarchical routing tree maps to Grafana's notification policies, but the semantics are not identical. Test the notification routing thoroughly — the continue flag behavior and default routes may differ.
Silence and inhibition migration. Active silences are ephemeral and do not need migration. Inhibition rules need to be recreated in Grafana's format. Recurring maintenance windows should be converted to mute timings.
Run in parallel first. The safest migration strategy is to run both systems in parallel for two to four weeks, sending notifications from both, then cutting over when you have confidence in the Grafana setup. Accept the temporary noise of duplicate alerts — it is far cheaper than missing a critical page during migration.

Migrating from Grafana Alerting to Alertmanager

Datasource limitation. You can only migrate alerts that are based on Prometheus-compatible datasources. Alerts querying Loki, Elasticsearch, or SQL datasources have no equivalent in Alertmanager — you will need an alternative solution for those.
Rule export. Export Grafana alert rules and convert them to Prometheus-format rule files. The Grafana API (GET /api/v1/provisioning/alert-rules) provides structured output that can be transformed with a script.
Contact point mapping. Map Grafana contact points to Alertmanager receivers. The configuration format is different, but the concepts are equivalent.
State loss. Alertmanager does not carry over Grafana's alert evaluation history. You start fresh. Plan for a brief period where alerts may re-fire as Prometheus evaluates rules that were previously managed by Grafana.

Decision Framework

If you want a quick decision path, use this framework:

Start here:
│
├── Do you alert on non-Prometheus datasources (Loki, ES, SQL, CloudWatch)?
│   ├── YES → Grafana Alerting (at least for those datasources)
│   └── NO ↓
│
├── Is GitOps/declarative config a hard requirement?
│   ├── YES → Alertmanager
│   └── NO ↓
│
├── Do you need multi-tenancy with RBAC?
│   ├── YES → Grafana Alerting (or Mimir Alertmanager)
│   └── NO ↓
│
├── Are you on Grafana Cloud?
│   ├── YES → Grafana Alerting (path of least resistance)
│   └── NO ↓
│
└── Default → Alertmanager (simpler, lighter, well-proven)

For many teams, the honest answer is "both" — Alertmanager for the Prometheus-native metric pipeline, Grafana Alerting for everything else. That is a valid architecture as long as the ownership boundaries are documented and the on-call team knows where to look.

Frequently Asked Questions

What is the difference between Alertmanager and Grafana Alerting?

Prometheus Alertmanager is a standalone notification routing engine that receives pre-evaluated alerts from Prometheus and delivers them to receivers like Slack, PagerDuty, or email. Grafana Alerting is an integrated alerting platform embedded in Grafana that both evaluates alert rules and handles notification routing. Alertmanager is configured entirely via YAML, while Grafana Alerting offers a UI, API, and file-based provisioning. The fundamental difference is scope: Alertmanager handles only the routing and notification phase, while Grafana Alerting handles the full lifecycle from query evaluation to notification.

Can Grafana Alerting replace Prometheus Alertmanager?

Yes, for many use cases. Grafana Alerting can evaluate PromQL rules directly against your Prometheus datasource, so you do not strictly need a separate Alertmanager instance. However, there are scenarios where Alertmanager remains the better choice: heavily GitOps-driven environments, teams that need Alertmanager's mature inhibition rules, or architectures where Prometheus rule evaluation happens externally (Thanos Ruler, Mimir Ruler) and a dedicated Alertmanager is already in the pipeline. If your only datasource is Prometheus and you value declarative configuration, Alertmanager is still simpler and lighter.

Is Grafana Alertmanager the same as Prometheus Alertmanager?

Not exactly. Grafana Alerting uses a fork of the Prometheus Alertmanager code internally for its notification routing engine, but it is not the same product. The Grafana "Alertmanager" visible in the UI is a managed, embedded component with a different configuration interface (notification policies, contact points, mute timings) compared to the standalone Prometheus Alertmanager (routing tree, receivers, inhibition rules in YAML). Grafana can also connect to an external Prometheus Alertmanager as a datasource, which adds to the confusion.

What are the best alternatives to Prometheus Alertmanager?

The most direct alternative is Grafana Alerting, which can receive and route Prometheus alerts while also supporting other datasources. Beyond that: Grafana OnCall for on-call management and escalation, PagerDuty or Opsgenie as managed incident response platforms, Keep as an open-source AIOps alert management platform, and Mimir Alertmanager for multi-tenant environments running Grafana Mimir.

Should I use Prometheus alerts or Grafana alerts for Kubernetes monitoring?

For Kubernetes monitoring specifically, the kube-prometheus-stack (which includes Prometheus, Alertmanager, and a comprehensive set of pre-built alerting rules) remains the industry standard. These rules are PromQL-based and are designed to work with Alertmanager. If you are deploying kube-prometheus-stack, using Alertmanager for metric-based alerts is the straightforward choice. Add Grafana Alerting on top if you also need to alert on logs (via Loki) or non-metric datasources.

Final Thoughts

The Alertmanager vs Grafana Alerting debate is not really about which tool is better — it is about which tool fits your operational context. Alertmanager is simpler, lighter, and more GitOps-friendly. Grafana Alerting is more versatile, more accessible to UI-oriented teams, and the only option if you need multi-datasource alerting. Running both is perfectly valid when the boundaries are clear.

The worst outcome is not picking the "wrong" tool. The worst outcome is running both accidentally, with overlapping coverage, duplicated notifications, and no clear ownership. Whatever you choose, document the decision, define the ownership boundaries, and make sure your on-call team knows exactly where to go when they need to silence an alert at 3 AM.

Originally published at alexandre-vazquez.com/alertmanager-vs-grafana-alerting

Prometheus Alertmanager Vs Grafana Alerting (2026): Architecture, Features, And When To Use Each

Alexandre Vazquez — Tue, 05 May 2026 10:00:00 +0000

Prometheus Alertmanager Vs Grafana Alerting (2026): Architecture, Features, And When To Use Each

Originally published at alexandre-vazquez.com

Read the full article on my blog: https://alexandre-vazquez.com/alertmanager-vs-grafana-alerting/

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Alexandre Vazquez — Tue, 05 May 2026 08:00:01 +0000

Originally published at alexandre-vazquez.com/debugging-distroless-containers/

The container works fine in CI. It deploys successfully to staging. Then something goes wrong in production and you type the command you always type: kubectl exec -it my-pod -- /bin/bash. The response is immediate: OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory.

You try /bin/sh. Same error. You try ls. Same error. The container image is distroless — it ships only your application binary and its runtime dependencies, with no shell, no package manager, no debugging tools of any kind. This is intentional and correct from a security standpoint. It is also a significant operational challenge the first time you face it in production.

This article covers every practical technique for debugging distroless containers in Kubernetes: kubectl debug with ephemeral containers (the standard approach), pod copy strategy (for Kubernetes versions without ephemeral container support, or when you need to modify the running pod spec), debug image variants (the pragmatic developer shortcut), cdebug (a purpose-built tool that simplifies the process), and node-level debugging (the last resort with the most power). For each technique I will explain what it can and cannot do, what Kubernetes version or RBAC permissions it requires, and in which scenario — developer in local, platform engineer in staging, ops in production — it is the appropriate choice.

Why Distroless Breaks the Normal Debugging Workflow

Traditional container debugging assumes you can exec into the container and use shell tools: ps, netstat, strace, curl, a text editor. Distroless images remove all of this by design. The Google distroless project, Chainguard's Wolfi-based images, and the broader minimal image ecosystem deliberately exclude everything that is not required to run the application. The result is a dramatically smaller attack surface: no shell means no RCE via shell injection, no package manager means no easy escalation path, fewer binaries means fewer CVEs in the image scan.

The tradeoff is operational: when something goes wrong, you cannot use the tools that the process itself is not allowed to run. A Java application in gcr.io/distroless/java17-debian12 has the JRE and nothing else. A Go binary compiled with CGO disabled and shipped in gcr.io/distroless/static-debian12 has literally only the binary and the necessary CA certificates and timezone data. There is no wget to download a debug binary, no apt to install one, no bash to run a script.

Kubernetes solves this at the platform level with ephemeral containers , added as stable in Kubernetes 1.25. The principle is that a debug container — which can have a full shell and any tools you want — can be injected into a running pod and share its process namespace, network namespace, and filesystem mounts without modifying the original container or restarting the pod.

Option 1: kubectl debug with Ephemeral Containers

Ephemeral containers are the canonical solution. Since Kubernetes 1.25 (stable), kubectl debug can inject a temporary container into a running pod. The container shares the target pod's network namespace by default, and with --target it can also share the process namespace of a specific container, allowing you to inspect its running processes and open file descriptors.

The basic invocation is:

kubectl debug -it my-pod \
  --image=busybox:latest \
  --target=my-container

The --target flag is the critical piece. Without it, the ephemeral container gets its own process namespace. With it, it shares the process namespace of the specified container — meaning you can run ps aux and see the application's processes, use ls -la /proc//fd to inspect open file descriptors, and read the application's environment via cat /proc//environ.

For a more capable debug environment, replace busybox with a richer image:

kubectl debug -it my-pod \
  --image=nicolaka/netshoot \
  --target=my-container

nicolaka/netshoot includes tcpdump, curl, dig, nmap, ss, iperf3, and dozens of other network diagnostic tools, making it the standard choice for network debugging scenarios.

What You Can and Cannot Do

Ephemeral containers share the pod's network namespace and, when --target is used, the process namespace. This gives you:

Full visibility into the application's network traffic from inside the pod (tcpdump, ss, netstat)
Process inspection via /proc/ — open files, memory maps, environment variables, CPU/memory usage
Access to the pod's DNS resolution context — exactly the same /etc/resolv.conf the application sees
Ability to make outbound network calls from the same network namespace (testing service endpoints, DNS resolution)

What you do not get with ephemeral containers:

Access to the application container 's filesystem. The ephemeral container has its own root filesystem. You cannot cat /app/config.yaml from the application container's filesystem unless you access it via /proc//root/.
Ability to remove the container once added. Ephemeral containers are permanent until the pod is deleted. This is by design — the Kubernetes API does not allow removing them after creation.
Volume mount modifications via CLI. You cannot add volume mounts to an ephemeral container via kubectl debug (though the API spec supports it, the CLI does not expose this).
Resource limits. Ephemeral containers do not support resource requests and limits in the kubectl debug CLI, though this is evolving.

Accessing the Application Filesystem

The most common surprise for developers new to ephemeral containers is that they cannot directly browse the application container's filesystem. The workaround is the /proc filesystem:

# Find the application's PID
ps aux

# Browse its filesystem via /proc
ls /proc/1/root/app/
cat /proc/1/root/etc/config.yaml

# Or set the root to the application's root
chroot /proc/1/root /bin/sh  # only if /bin/sh exists in the app image

The /proc//root path is a symlink to the container's root filesystem as seen from the process namespace. Because the ephemeral container shares the process namespace with --target, the application's PID is typically 1, and /proc/1/root gives you full read access to its filesystem.

RBAC Requirements

Ephemeral containers require the pods/ephemeralcontainers subresource permission. This is separate from pods/exec, which controls kubectl exec. A common mistake is to grant pods/exec for debugging purposes without realizing that ephemeral containers require an additional grant:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ephemeral-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

In production environments, this permission should be tightly scoped: time-limited via RoleBinding rather than permanent ClusterRoleBinding, restricted to specific namespaces, and ideally gated behind an approval workflow. The debug container runs as root by default, which can create privilege escalation paths if the application container runs as a non-root user with shared process namespace — the debug container can attach to the application's processes with higher privileges.

Option 2: kubectl debug -copy-to (Pod Copy Strategy)

When you need to modify the pod's container spec — replace the image, change environment variables, add a sidecar with a shared filesystem — the --copy-to flag creates a full copy of the pod with your modifications applied:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=my-app:debug \
  --share-processes

This creates a new pod named my-pod-debug that is a copy of my-pod but with the container image replaced by my-app:debug. If my-app:debug is your application image built with debug tooling included (or a debug variant from your registry), this lets you interact with the exact same binary in the exact same configuration as the original pod.

A more common use of --copy-to is to attach a debug container alongside the existing application container while keeping the original image unchanged:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  --container=debugger

This creates the copy-pod with both the original containers and a new debugger container sharing the process namespace. Unlike ephemeral containers, this approach supports volume mounts and resource limits, and the debug pod can be deleted cleanly when you are done.

Limitations of the Copy Strategy

The pod copy approach has a critical limitation: it is not debugging the original pod. It creates a new pod that may behave differently because:

It does not share the original pod's in-memory state — if the issue is a goroutine leak or heap corruption that has been accumulating for hours, the fresh copy will not exhibit it immediately
It creates a new Pod UID, which means any admission webhooks, network policies, or pod-level security contexts that depend on pod identity may apply differently
If the original pod is crashing (CrashLoopBackOff), the copy will also crash — this technique does not help for crash debugging unless you also change the entrypoint

For crash debugging specifically, combine --copy-to with a modified entrypoint to keep the container alive:

kubectl debug my-crashing-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  -- sleep 3600

Option 3: Debug Image Variants

The most pragmatic approach — and the one most appropriate for developer workflows — is to maintain a debug variant of your application image that includes shell tooling. Both the Google distroless project and Chainguard provide this pattern officially.

Google distroless images have a :debug tag that adds BusyBox to the image:

# Production image
FROM gcr.io/distroless/java17-debian12

# Debug variant — identical but with BusyBox shell
FROM gcr.io/distroless/java17-debian12:debug

Chainguard images follow a similar convention with :latest-dev variants that include apk, a shell, and common utilities:

# Production (zero shell, minimal footprint)
FROM cgr.dev/chainguard/go:latest

# Development/debug variant
FROM cgr.dev/chainguard/go:latest-dev

If you build your own base images, the recommended approach is to use multi-stage builds and maintain separate build targets:

FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Production: static distroless image
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

# Debug variant: same binary, with shell tools
FROM gcr.io/distroless/static-debian12:debug AS debug
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

In your CI/CD pipeline, build both targets and push my-app:${VERSION} (production) and my-app:${VERSION}-debug (debug variant) to your registry. The debug image is never deployed to production by default, but it exists and is ready to be used with kubectl debug --copy-to when needed.

Security Considerations for Debug Variants

Debug image variants defeat much of the security benefit of distroless if they are used in production, even temporarily. Track usage carefully: log when debug images are deployed, require explicit approval, and ensure they are removed after the debugging session. In regulated environments, consider whether deploying a debug variant to production namespaces is permitted by your security policy — in many cases it is not, and you must use ephemeral containers (which add a debug process to the pod without modifying the application image) instead.

Option 4: cdebug

cdebug is an open-source CLI tool that simplifies distroless debugging by wrapping kubectl debug with more ergonomic defaults and additional capabilities. Its primary value is in making ephemeral container debugging feel like a native shell experience:

# Install
brew install cdebug
# or: go install github.com/iximiuz/cdebug@latest

# Debug a running pod
cdebug exec -it my-pod

# Specify a namespace and container
cdebug exec -it -n production my-pod -c my-container

# Use a specific debug image
cdebug exec -it my-pod --image=nicolaka/netshoot

What cdebug adds over raw kubectl debug:

Automatic filesystem chroot. cdebug exec automatically sets the filesystem root of the debug container to the target container's filesystem, so you browse / and see the application's files — not the debug image's files. This addresses the most common friction point with kubectl debug.
Docker integration. cdebug exec works identically for Docker containers (cdebug exec -it), making it the same muscle memory for local and cluster debugging.
No RBAC complications for Docker-based local development — useful for developer workflows before the code reaches Kubernetes.

The tradeoff: cdebug is a third-party dependency and requires installation. In environments with strict tooling policies (regulated industries, air-gapped clusters), it may not be an option. In those cases, the raw kubectl debug workflow with /proc/1/root filesystem navigation is the baseline.

Option 5: Node-Level Debugging

When everything else fails — the pod is in CrashLoopBackOff too fast to attach to, the issue is a kernel-level problem, or you need tools like strace that require elevated privileges — node-level debugging gives you direct access to the container's processes from the host node.

kubectl debug node/ creates a privileged pod on the target node that mounts the node's root filesystem under /host:

kubectl debug node/my-node-name \
  -it \
  --image=nicolaka/netshoot

From this privileged pod, you can use nsenter to enter the namespaces of any container running on the node:

# Find the container's PID on the node
# (from within the node debug pod)
crictl ps | grep my-container
crictl inspect  | grep pid

# Enter the container's namespaces
nsenter -t  -m -u -i -n -p -- /bin/sh

# Or just the network namespace (for network debugging)
nsenter -t  -n -- ip a

The nsenter approach lets you run tools from the node's or debug container's toolset while operating in the namespaces of the target container. This is how you run strace against a distroless process: strace is not in the application container, but you can run it from the node level while targeting the application's PID.

# Trace all syscalls from the application process
nsenter -t  -- strace -p  -f -e trace=network

RBAC and Security for Node Debugging

Node-level debugging requires nodes/proxy and the ability to create privileged pods, which in most production clusters is restricted to cluster administrators. The debug pod runs with hostPID: true and hostNetwork: true, giving it visibility into all processes and network traffic on the node — not just the target container. This is significant: every process running on the node, including those in other tenants' namespaces, is visible.

This technique should be treated as a break-glass procedure: log the access, require dual approval in production environments, and clean up immediately after the debugging session with kubectl delete pod --selector=app=node-debugger.

Choosing the Right Approach: Access Profile and Environment Matrix

The technique you should use depends on two axes: who you are (developer, platform engineer, ops/SRE) and where the issue is (local development, staging, production). The requirements and constraints differ significantly across these combinations.

Developer — Local or Development Cluster

Goal: Reproduce and understand a bug, inspect configuration, verify network connectivity to services.

Constraints: None material — full cluster admin on local or personal dev namespace.

Recommended approach: Debug image variants or cdebug.

In local development (Minikube, Kind, Docker Desktop), the fastest path is to build the debug variant of your image and deploy it directly. If you are working with another team's service, cdebug exec gives you a shell in the container with automatic filesystem root without any special RBAC. The goal is speed and iteration — reserve the more structured approaches for higher environments.

Developer — Staging Cluster

Goal: Debug integration issues, inspect live configuration, verify environment-specific behavior.

Constraints: Shared cluster — cannot deploy arbitrary workloads to other teams' namespaces, but has pods/ephemeralcontainers in own namespace.

Recommended approach: kubectl debug with ephemeral containers (--target), scoped to own namespace.

Staging is where ephemeral containers earn their keep. You can attach to a running pod without restarting it, without modifying the deployment spec, and without affecting other users of the same cluster. Grant developers pods/ephemeralcontainers in their team's namespaces and they can self-service debug without needing ops involvement.

Platform Engineer / SRE — Production

Goal: Diagnose a live production incident. The pod is behaving unexpectedly — high latency, memory growth, unexpected connections, incorrect responses.

Constraints: Changes to running pods are high-risk. Any debug image deployment must be gated. The issue is live and affecting users.

Recommended approach: kubectl debug with ephemeral containers (ephemeral containers do not restart the pod, do not modify the deployment, and are auditable via API audit logs).

The key production requirements are auditability and minimal blast radius. Ephemeral containers satisfy both: they are recorded in the Kubernetes API audit log (who attached, when, to which pod), they do not modify the running application container, and they are limited to the pod's own network and process namespaces. Document the debug session in your incident ticket: pod name, time, what was observed, who ran the debug container.

The --copy-to strategy is generally inappropriate for production incident response: it creates a new pod that may or may not exhibit the issue, it adds load to the cluster during an incident, and if it is attached to the same services (databases, downstream APIs), it produces additional traffic that complicates forensics.

Platform Engineer — Production, Node-Level Issue

Goal: Diagnose a kernel-level issue, a container runtime problem, a networking issue that spans multiple pods, or a situation where the pod is crashing too fast to attach to.

Constraints: Maximum privilege required. High operational risk.

Recommended approach: Node-level debug pod with nsenter. Treat as break-glass.

For this scenario, create a dedicated RBAC role that grants nodes/proxy access and the ability to create pods with hostPID: true in a dedicated debug namespace. Bind it only to specific users, require a separate authentication step (e.g., kubectl auth can-i check against a time-limited binding), and log all access. This level of access should generate a PagerDuty-style alert so that the security team knows a privileged debug session is active in production.

Common Errors and Solutions

Error: "ephemeral containers are disabled for this cluster"

Ephemeral containers require Kubernetes 1.16+ (alpha, behind feature gate) and are stable from 1.25. If you are on 1.16–1.22, you need to enable the EphemeralContainers feature gate on the API server and kubelet. From 1.23 it was beta and enabled by default. From 1.25 it is stable and always on. On managed Kubernetes services (EKS, GKE, AKS), check the cluster version — versions older than 1.25 may still have it disabled depending on your configuration.

Error: "cannot update ephemeralcontainers" (RBAC)

You have pods/exec but not pods/ephemeralcontainers. Add the grant shown in the RBAC section above. Note that pods/exec and pods/ephemeralcontainers are separate subresources — having one does not imply the other.

Error: "container not found" with -target

The container name in --target must match exactly the container name as defined in the Pod spec — not the image name. Check with kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' to get the exact container names.

Error: Can see processes but cannot read /proc/1/root

The application container runs as a non-root user (e.g., UID 1000) and the ephemeral container runs as root. The application's filesystem may have files owned by UID 1000 that are not readable by other UIDs depending on permissions. The /proc//root path itself requires CAP_SYS_PTRACE capability. If your cluster's PodSecurityStandards (PSS) are set to restricted, the debug container may not have this capability. Use the Baseline PSS profile for debug namespaces or explicitly add SYS_PTRACE to the ephemeral container's securityContext.

Error: tcpdump shows no traffic

When using nicolaka/netshoot for network debugging, ensure the ephemeral container is created without --target if your goal is to capture all traffic on the pod's network interface (not just the specific container's process). With --target, you share the process namespace but the network namespace is shared at the pod level regardless. Run tcpdump -i any to capture on all interfaces including loopback, which is where inter-container traffic within a pod travels.

Decision Framework

Use this as a starting point to select the right technique for your situation:

Scenario	Technique	Requirement
Active production incident, pod running	kubectl debug + ephemeral container	pods/ephemeralcontainers RBAC, k8s 1.25+
Pod crashing too fast to attach	kubectl debug -copy-to + modified entrypoint	Ability to create pods in namespace
Developer debugging in dev/staging	cdebug exec or kubectl debug	pods/ephemeralcontainers or pod create
Need full filesystem access	kubectl debug -copy-to + debug image variant	Debug image in registry, pod create
Need strace or kernel tracing	Node-level debug with nsenter	nodes/proxy, cluster admin equivalent
Network packet capture	kubectl debug + nicolaka/netshoot	pods/ephemeralcontainers
Local Docker debugging	cdebug exec	Docker socket access
CI-reproducible debug environment	Debug image variant in separate build target	Separate image tag in registry

Production RBAC Design

A clean RBAC design for production distroless debugging separates three roles with different privilege levels:

# Tier 1: Developer self-service in team namespaces
# Allows attaching ephemeral containers, no node access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: distroless-debugger
  namespace: team-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 2: SRE production incident access
# Ephemeral containers across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-distroless-debugger
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 3: Break-glass node access
# Only for platform team, time-limited binding recommended
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-debugger
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "delete"]
  # Restrict to debug namespace via RoleBinding, not ClusterRoleBinding

Bind Tier 1 permanently to your developers. Bind Tier 2 to SREs permanently but with audit alerts on use. Bind Tier 3 only on-demand (via a Kubernetes operator that creates time-limited RoleBindings) and never as a permanent ClusterRoleBinding.

Summary

Distroless containers are the correct choice for production workloads. They reduce attack surface, eliminate unnecessary CVEs, and force a cleaner separation between application and tooling. The operational cost is that your traditional debugging workflow — exec into the container, run some commands — no longer works by default.

Kubernetes provides a clean answer with ephemeral containers and kubectl debug: inject a debug container with whatever tools you need into the running pod, sharing its network and process namespaces, without restarting or modifying the application. For scenarios where ephemeral containers are insufficient — filesystem access, crash debugging, kernel-level investigation — the copy strategy and node-level debug fill the remaining gaps.

The key to making this work at scale is not the technique itself but the access model : developers get self-service ephemeral container access in their own namespaces, SREs get cluster-wide ephemeral container access for production incidents, and node-level access is a break-glass procedure with audit trail and time limits. With that model in place, distroless becomes an operational non-issue rather than an obstacle.

XSLT 3.0 new features: what changed from 2.0

Alexandre Vazquez — Mon, 04 May 2026 09:00:01 +0000

XSLT 3.0 is a significant step beyond 2.0. If you are running Saxon on the backend — as XSLT Playground does — you have access to the full 3.0 feature set today. This post covers the additions that matter most in practice and shows how to try each one directly in the playground.

Streaming

The most impactful change in 3.0 is streaming. In earlier versions, the processor loads the entire source document into memory before any template can run. With 3.0 streaming, selected templates can consume the document as a stream, which drastically reduces memory usage for large inputs.

To enable streaming, declare it on the stylesheet and on the mode:

Not every expression is streamable. Saxon will tell you at compile time if a pattern is not compatible. The key restriction is that you can only visit each node once — no backward axes, no variables that hold nodes for later inspection.

Maps and arrays

XSLT 3.0 adds maps and arrays as first-class values, borrowed from XPath 3.1. A map is a collection of key-value pairs; an array is an ordered sequence that can hold any value including other maps or arrays.

Arrays work similarly:

Arrays use 1-based indexing. Use array:size(), array:get(), and array:append() from the array namespace for common operations.

JSON input and output

XSLT 3.0 can parse and produce JSON natively via json-to-xml() and xml-to-json(). This eliminates the need for a pre-processing step when your source is JSON.

The function converts JSON into a predictable XML representation defined by the W3C. Objects become elements, arrays become, and primitives become typed ,, or `elements. You transform this intermediate XML normally and then serialize it back withxml-to-json()` if needed.

Higher-order functions

You can now pass functions as arguments using xsl:function and the function() type. This enables patterns like map, filter, and fold over sequences without recursion.

`xml

The #1 notation creates a function reference with arity 1.

Packages

XSLT 3.0 introduces packages, which let you split a large stylesheet into independently compiled units that expose explicit interfaces. This is the equivalent of modules or libraries in other languages.

`xml

...

Packages reduce coupling and enable reuse across projects without copy-paste.

Try it in XSLT Playground

All the examples above run in XSLT Playground with version set to 3.0. Maps and JSON support are the quickest to explore. Paste the json-to-xml() example, provide a JSON string as input, and see the intermediate representation immediately.

XSLT 3.0 is available today. If your integration still targets 2.0, the features above are the best reasons to upgrade.

XSL online tester: run XSL and XSLT transforms in your browser

Alexandre Vazquez — Thu, 30 Apr 2026 09:00:00 +0000

XSL (Extensible Stylesheet Language) is an umbrella term that covers three related specifications: XSLT for transformations, XPath for node selection, and XSL-FO for formatting objects. When developers search for an "XSL tester" or "XSL online editor", they are usually looking for a way to run XSLT stylesheets against XML input without a local install. XSLT Playground does exactly that.

XSL vs XSLT — what is the difference?

XSL is the full family of W3C specifications:

XSLT (XSL Transformations) — transforms XML documents into other formats
XPath — the path language used inside XSLT to navigate XML trees
XSL-FO (XSL Formatting Objects) — describes page layout for print and PDF output

In practice, when people say "XSL" in an integration or development context, they almost always mean XSLT. The stylesheet file extension .xsl and .xslt are interchangeable — Saxon and most processors accept both.

Running XSL transforms online

XSLT Playground supports XSLT 1.0, 2.0, and 3.0 via the Saxon processor. To run an XSL transform:

Paste your XML source document in the input panel
Paste your XSL stylesheet in the stylesheet panel
Select the XSLT version (1.0, 2.0 or 3.0)
Click Run

The output appears immediately. If the stylesheet has errors, the error panel shows the exact line and message from Saxon.

Common XSL use cases

XML to HTML — the most common use. An XSL stylesheet walks an XML document tree and emits HTML tags:

XML to XML — reshaping or filtering a document structure:

XML to plain text or CSV — using xsl:output method="text":

XSL file extensions: .xsl vs .xslt

Both .xsl and .xslt are valid. The .xsl extension is older and more common in enterprise systems (SAP, Oracle, IBM DataPower). The .xslt extension is more explicit. Saxon accepts either. XSLT Playground accepts any content regardless of what you call it.

Testing XSL stylesheets online

The main advantage of an online XSL tester is speed of iteration. You can:

Paste a real XML payload from a production system and see what the stylesheet produces
Add parameters and test different code paths
Enable trace mode to see which templates fired and in what order
Export the entire test case (input + stylesheet + parameters) as a JSON workspace to share with a colleague

All of this is available at XSLT Playground without creating an account or installing anything.

XSL transform online vs local Saxon

For most development and debugging tasks, the online tester is faster than running Saxon locally. Use local Saxon when:

You are processing confidential data that cannot leave your network
Your input files are very large (several MB or more)
You need to integrate the transform into a build pipeline

For everything else — prototyping, debugging, sharing test cases — the online XSL tester is quicker.

Istio ServiceEntry Explained: External Services, DNS, and Traffic Control

Alexandre Vazquez — Tue, 28 Apr 2026 11:00:01 +0000

Istio ServiceEntry Explained: External Services, DNS, and Traffic Control

What Is a ServiceEntry

Istio maintains an internal service registry that merges Kubernetes Services with additional entries you declare. When a sidecar proxy needs to route a request, it consults this registry. Services inside the mesh are automatically registered, but external services require a ServiceEntry to be added to the registry.

A ServiceEntry is a custom resource that registers external services in the mesh's service registry. Once registered, external services become first-class citizens with access to Istio features including metrics, access logs, distributed traces, mTLS origination, retries, timeouts, and circuit breaking.

ServiceEntry Anatomy: All Fields Explained

hosts

A list of hostnames associated with the service. For external services, this is typically the DNS name your application uses (e.g., api.stripe.com). For HTTP protocols, the hosts field is matched against the HTTP Host header. For non-HTTP protocols, you can use synthetic hostnames paired with addresses or static endpoints.

addresses

Optional virtual IP addresses associated with the service. Useful for TCP services where you want to assign a VIP that the sidecar will intercept. Not required for HTTP/HTTPS services that use hostname-based routing.

ports

The ports on which the external service is exposed. Each port needs a number, name, and protocol. The protocol setting determines how Envoy handles the connection—TLS for pass-through without termination, HTTPS for HTTP over TLS, and TCP for database connections.

location

Either MESH_EXTERNAL or MESH_INTERNAL. Use MESH_EXTERNAL for services outside your cluster (third-party APIs, managed databases). Use MESH_INTERNAL for services inside your infrastructure without a sidecar, such as VMs in the same VPC or unmeshed Kubernetes Services. This affects mTLS application and metrics labeling.

resolution

How the sidecar resolves endpoint addresses. Options include NONE, STATIC, DNS, and DNS_ROUND_ROBIN. This is the most critical field for ServiceEntry configuration.

endpoints

An explicit list of network endpoints. Required when resolution is STATIC. Each endpoint can have an address, ports, labels, network, locality, and weight.

exportTo

Controls visibility across namespaces. Use "." for the current namespace only, "*" for all namespaces. In multi-team clusters, restrict exports to avoid namespace pollution.

Resolution Types: NONE vs STATIC vs DNS vs DNS_ROUND_ROBIN

The resolution field determines how Envoy discovers IP addresses behind the service.

Resolution	How It Works	Best For
`NONE`	Envoy uses the original destination IP from the connection. No DNS lookup by the proxy.	Wildcard entries, pass-through scenarios, services where the application already resolved the IP.
`STATIC`	Envoy routes to the IPs listed in the `endpoints` field. No DNS involved.	Services with stable, known IPs (e.g., on-prem databases, VMs with fixed IPs).
`DNS`	Envoy resolves the hostname at connection time and creates an endpoint per returned IP. Uses async DNS with health checking per IP.	External APIs behind load balancers, managed databases with DNS endpoints (RDS, CloudSQL).
`DNS_ROUND_ROBIN`	Envoy resolves the hostname and uses a single logical endpoint, rotating across returned IPs. No per-IP health checking.	Simple external services, services where you do not need per-endpoint circuit breaking.

When to Use NONE

Use NONE when registering a range of external IPs or wildcard hosts without Envoy performing address resolution. This is common for broad egress policies like "allow traffic to *.googleapis.com on port 443." The downside is that Envoy has limited ability to apply per-endpoint policies.

When to Use STATIC

Use STATIC when the external service has known, stable IP addresses that rarely change. This avoids DNS dependencies entirely. Classic use case: a legacy Oracle database on a fixed IP in your data center.

When to Use DNS

Use DNS for most external API integrations. Envoy performs asynchronous DNS resolution and creates a cluster endpoint for each returned IP address. This enables per-endpoint health checking and circuit breaking—critical for production reliability.

When to Use DNS_ROUND_ROBIN

Use DNS_ROUND_ROBIN when the external hostname returns many IPs and you do not need per-IP circuit breaking. Envoy treats all resolved IPs as a single logical endpoint and round-robins across them, which is lighter weight than DNS mode.

Practical Patterns

Pattern 1: External HTTP API (api.stripe.com)

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: stripe-api
  namespace: payments
spec:
  hosts:
    - api.stripe.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: tls
      protocol: TLS
  resolution: DNS

The protocol is TLS, not HTTPS, because the application initiates the TLS handshake directly. Envoy handles this as opaque TLS using SNI-based routing.

Pattern 2: External Managed Database (RDS / CloudSQL)

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: orders-database
  namespace: orders
spec:
  hosts:
    - orders-db.abc123.us-east-1.rds.amazonaws.com
  location: MESH_EXTERNAL
  ports:
    - number: 5432
      name: postgres
      protocol: TCP
  resolution: DNS

For TCP services, the DNS resolution mode ensures Envoy periodically re-resolves the hostname and updates its endpoint list, which is critical for RDS multi-AZ failover scenarios.

Pattern 3: Legacy Internal Service Not in the Mesh

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: legacy-monitoring
  namespace: observability
spec:
  hosts:
    - legacy-monitoring.internal
  location: MESH_INTERNAL
  ports:
    - number: 8080
      name: http
      protocol: HTTP
  resolution: STATIC
  endpoints:
    - address: 10.0.5.10
    - address: 10.0.5.11
    - address: 10.0.5.12

The location is MESH_INTERNAL because the service lives inside your network, and resolution is STATIC because the IPs are known. The hostname is synthetic—your application uses it, and Istio's DNS proxy resolves it to one of the listed endpoints.

Pattern 4: TCP Services with Multiple Ports

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-elasticsearch
  namespace: search
spec:
  hosts:
    - es.example.com
  location: MESH_EXTERNAL
  ports:
    - number: 9200
      name: http
      protocol: HTTP
    - number: 9300
      name: transport
      protocol: TCP
  resolution: DNS

Each port gets its own Envoy listener configuration. The HTTP port benefits from full Layer 7 telemetry, while the TCP port gets Layer 4 metrics and connection-level policies.

Combining ServiceEntry with DestinationRule

A ServiceEntry alone registers the external service. To apply traffic policies—connection pooling, circuit breaking, TLS origination, load balancing—pair it with a DestinationRule.

Connection Pooling and Circuit Breaking

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: stripe-api
  namespace: payments
spec:
  hosts:
    - api.stripe.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: tls
      protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: stripe-api-dr
  namespace: payments
spec:
  host: api.stripe.com
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 50
        connectTimeout: 5s
      http:
        h2UpgradePolicy: DO_NOT_UPGRADE
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 60s
      maxEjectionPercent: 100

This configuration caps outbound connections at 50, sets a 5-second connection timeout, and ejects endpoints that return 3 consecutive 5xx errors, preventing a degraded external API from consuming all connection slots.

TLS Origination

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
  namespace: default
spec:
  hosts:
    - api.external-service.com
  location: MESH_EXTERNAL
  ports:
    - number: 80
      name: http
      protocol: HTTP
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: external-api-tls
  namespace: default
spec:
  host: api.external-service.com
  trafficPolicy:
    portLevelSettings:
      - port:
          number: 443
        tls:
          mode: SIMPLE

The application sends HTTP to port 80. A VirtualService redirects that to port 443. The DestinationRule initiates TLS to the external endpoint. The application never knows TLS happened.

Combining ServiceEntry with VirtualService

VirtualService provides Layer 7 traffic management for external services: retries, timeouts, fault injection, header-based routing, and traffic shifting.

Retries and Timeouts

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: stripe-api-vs
  namespace: payments
spec:
  hosts:
    - api.stripe.com
  http:
    - route:
        - destination:
            host: api.stripe.com
            port:
              number: 443
      timeout: 10s
      retries:
        attempts: 3
        perTryTimeout: 3s
        retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
        retryRemoteLocalities: true

This applies a 10-second overall timeout with up to 3 retry attempts (3 seconds each) for specific failure conditions. This only works for HTTP-protocol ServiceEntries. For TLS-protocol entries, you are limited to TCP-level connection retries via the DestinationRule.

Traffic Shifting Between External Providers

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: geocoding-primary
  namespace: geo
spec:
  hosts:
    - geocoding.internal
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: tls
      protocol: TLS
  resolution: STATIC
  endpoints:
    - address: api.old-geocoding-provider.com
      labels:
        provider: old
    - address: api.new-geocoding-provider.com
      labels:
        provider: new
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: geocoding-dr
  namespace: geo
spec:
  host: geocoding.internal
  trafficPolicy:
    tls:
      mode: SIMPLE
  subsets:
    - name: old-provider
      labels:
        provider: old
    - name: new-provider
      labels:
        provider: new
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: geocoding-vs
  namespace: geo
spec:
  hosts:
    - geocoding.internal
  http:
    - route:
        - destination:
            host: geocoding.internal
            subset: old-provider
          weight: 80
        - destination:
            host: geocoding.internal
            subset: new-provider
          weight: 20

This sends 80% of geocoding traffic to the old provider and 20% to the new one. Adjust weights as you gain confidence.

DNS Resolution Patterns: Istio DNS Proxy vs kube-dns

Istio DNS resolution involves two layers: how your application resolves the hostname (kube-dns / CoreDNS) and how the sidecar resolves the hostname (Envoy's async DNS or Istio's DNS proxy).

Default Flow (Without Istio DNS Proxy)

Your application calls api.stripe.com. kube-dns resolves it to an IP. The application opens a connection to that IP. The sidecar intercepts the connection and—if the ServiceEntry uses DNS resolution—Envoy independently resolves api.stripe.com. Two separate DNS lookups happen, which can lead to inconsistencies if DNS records change between the two resolutions.

With Istio DNS Proxy (dns.istio.io)

Istio's sidecar includes a DNS proxy that intercepts DNS queries from the application. When enabled, the proxy can:

Auto-allocate virtual IPs for ServiceEntry hosts that do not have addresses defined, which is critical for TCP ServiceEntries.
Resolve ServiceEntry hosts directly, avoiding the round-trip to kube-dns for known mesh services.
Ensure consistency between the application's DNS resolution and the sidecar's endpoint resolution.

In modern Istio installations (1.18+), DNS capture is enabled by default.

When DNS Proxy Matters Most

The DNS proxy is especially important for TCP ServiceEntries without an explicit addresses field. Without a VIP, Envoy cannot match an incoming TCP connection to the correct ServiceEntry. The DNS proxy solves this by auto-allocating a VIP from the 240.240.0.0/16 range and returning that VIP when the application resolves the hostname.

Sticky Sessions with ServiceEntry

Some external services require session affinity. Istio supports sticky sessions for external services through consistent hashing in a DestinationRule.

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: legacy-session-service
  namespace: default
spec:
  hosts:
    - legacy-session.internal
  location: MESH_INTERNAL
  ports:
    - number: 8080
      name: http
      protocol: HTTP
  resolution: STATIC
  endpoints:
    - address: 10.0.1.10
    - address: 10.0.1.11
    - address: 10.0.1.12
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: legacy-session-dr
  namespace: default
spec:
  host: legacy-session.internal
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: SERVERID
          ttl: 3600s

This configuration hashes on an HTTP cookie named SERVERID. If the cookie does not exist, Envoy generates one and sets it on the response. You can also hash on:

HTTP header: consistentHash.httpHeaderName: "x-user-id" — useful when your application sends a user identifier in every request.
Source IP: consistentHash.useSourceIp: true — simplest option but breaks in environments with NAT or shared egress IPs.
Query parameter: consistentHash.httpQueryParameterName: "session_id" — for REST APIs that include a session identifier in the URL.

The ServiceEntry must use STATIC or DNS resolution for sticky sessions to work. With DNS_ROUND_ROBIN, there is only one logical endpoint, so consistent hashing has no effect.

Troubleshooting Common Issues

503 Errors When Calling External Services

Start with this diagnostic sequence:

# Check if the ServiceEntry is applied and visible to the proxy
istioctl proxy-config cluster  -n  | grep 

# Check the listeners
istioctl proxy-config listener  -n  --port 

# Look at Envoy access logs for the specific request
kubectl logs  -n  -c istio-proxy | grep

Common causes of 503 errors:

Wrong protocol: Setting protocol: HTTPS when your application initiates TLS. Use TLS for pass-through.
Missing ServiceEntry in REGISTRY_ONLY mode: Any host without a ServiceEntry is blocked.
exportTo restriction: The ServiceEntry is in namespace A, exported only to ".", and the calling pod is in namespace B.
DNS resolution failure: Envoy cannot resolve the hostname. Check that DNS servers are reachable from the pod.

DNS Resolution Failures

When Envoy's async DNS resolver fails, you will see UH (upstream unhealthy) or UF (upstream connection failure) flags in access logs.

# Verify DNS works from inside the sidecar
kubectl exec  -n  -c istio-proxy -- \
  pilot-agent request GET /dns_resolve?proxyID=.&host=api.stripe.com

# Check Envoy cluster health
istioctl proxy-config endpoint  -n  | grep

If the endpoint shows UNHEALTHY, Envoy resolved the DNS but outlier detection ejected the host. If no endpoint appears, DNS resolution is failing. Ensure your pods can reach an external DNS server, or that CoreDNS is configured to forward queries for the external domain.

TLS Origination Not Working

If you configured TLS origination via a DestinationRule but traffic still fails:

Ensure the ServiceEntry port protocol is HTTP, not TLS.
Verify the DestinationRule's host field exactly matches the ServiceEntry's hosts entry.
Check that the VirtualService routes to the correct port number.

TCP ServiceEntry Not Intercepting Traffic

For TCP-protocol ServiceEntries without the DNS proxy, Envoy cannot match traffic by hostname. You must either:

Set an explicit addresses field with a VIP that your application targets.
Enable Istio's DNS proxy to auto-allocate VIPs.
Ensure the destination IP matches what the ServiceEntry resolves to.

Without one of these, TCP traffic goes through the PassthroughCluster and bypasses your ServiceEntry entirely.

Frequently Asked Questions

Do I need a ServiceEntry if outboundTrafficPolicy is set to ALLOW_ANY?

You do not need one for connectivity. But you should create ServiceEntries anyway. Without them, outbound traffic goes through the PassthroughCluster, which means no detailed metrics per destination, no access logging with the external hostname, no circuit breaking, no retries, and no timeout policies.

What is the difference between protocol TLS and HTTPS in a ServiceEntry port?

TLS tells Envoy to treat the connection as opaque TLS. Envoy reads the SNI header to determine routing but does not decrypt the payload. Use this when your application initiates TLS directly. HTTPS tells Envoy the protocol is HTTP over TLS. In practice, for external services where the application manages its own TLS, use TLS.

Can I use wildcards in ServiceEntry hosts?

Yes, but with limitations. You can use *.example.com to match any subdomain of example.com. However, wildcard entries only work with resolution: NONE because Envoy cannot perform DNS lookups for wildcard hostnames. Wildcard ServiceEntries are best used for broad egress access control rather than fine-grained traffic management.

How do I configure sticky sessions for an external service behind a ServiceEntry?

Create a ServiceEntry with STATIC or DNS resolution so Envoy has multiple endpoints. Pair it with a DestinationRule that configures consistentHash under trafficPolicy.loadBalancer. You can hash on an HTTP cookie, header, source IP, or query parameter.

How does ServiceEntry interact with NetworkPolicy and Istio AuthorizationPolicy?

A ServiceEntry does not bypass Kubernetes NetworkPolicy. If a NetworkPolicy blocks egress to the external IP, traffic will be dropped at the CNI level before Envoy can route it. Istio AuthorizationPolicy can also restrict which workloads are allowed to call specific ServiceEntry hosts. Use ServiceEntry for traffic management and observability, AuthorizationPolicy for workload-level access control, and NetworkPolicy for network-level enforcement.

Wrapping Up

ServiceEntry transforms opaque outbound connections into managed, observable, policy-controlled traffic without requiring changes to your application code. Start with the basics: create a ServiceEntry for each external dependency, set the correct resolution type, and pair it with a DestinationRule for connection limits and circuit breaking. As you mature, add VirtualServices for retries and timeouts, configure sticky sessions where needed, and enable the DNS proxy for seamless TCP service integration. Every external dependency you formalize with a ServiceEntry is one fewer blind spot in your production mesh.

Originally published at alexandre-vazquez.com/istio-serviceentry-explained

Istio ServiceEntry Explained: External Services, DNS, and Traffic Control

Alexandre Vazquez — Tue, 28 Apr 2026 10:00:00 +0000

Istio ServiceEntry Explained: External Services, DNS, and Traffic Control

Originally published at alexandre-vazquez.com

Read the full article on my blog: https://alexandre-vazquez.com/istio-serviceentry-explained/

XSLT validator online: catch errors before running your transform

Alexandre Vazquez — Mon, 27 Apr 2026 09:00:02 +0000

A broken XSLT stylesheet can fail in several ways: a syntax error stops the processor immediately, a namespace mismatch silently produces empty output, or an undefined variable causes a runtime error that only appears with specific inputs. Catching these issues early, before the stylesheet reaches a test environment, saves significant debugging time.

What XSLT validation actually checks

XSLT validation happens at two levels:

Compile-time checks happen when the processor parses the stylesheet. These catch:

Malformed XML in the stylesheet itself
References to undefined named templates or functions
Type errors in static expressions
Invalid XSLT element usage (wrong attributes, missing required children)

Runtime errors only appear when the stylesheet runs against actual input:

Missing nodes that are assumed to exist
Type errors in dynamic expressions
Namespace mismatches between the stylesheet and the input document

A good validator runs both levels.

Using XSLT Playground as a validator

XSLT Playground runs Saxon, which is one of the most thorough XSLT processors available. When you paste a stylesheet and click Run, Saxon compiles it first and reports compile errors with exact line numbers before attempting execution.

For runtime errors, the trace mode shows you exactly which template fired, which node was being processed, and where the failure occurred. This is more useful than a bare error message because it gives you the execution context.

To validate a stylesheet quickly:

Paste the stylesheet into the editor
Provide a minimal XML input — even an empty `` catches most compile errors
Run with trace enabled
Check the error panel for compile-time issues and the trace panel for runtime behaviour

Common XSLT errors and how to spot them

Namespace mismatch
Your input uses xmlns="http://example.com/ns" but your stylesheet matches element-name without the namespace. The match never fires, output is empty.

Fix: declare the namespace in the stylesheet and use the prefix in match patterns:
`xml

Using xpath-default-namespace (XSLT 2.0+) avoids having to prefix every element name in your XPath expressions.

Undefined variable
You reference $config but it is only defined inside a conditional branch that did not execute for this input. Saxon reports: Variable $config has not been assigned a value.

Fix: move variable declarations to the template root or provide a default:
`xml

Wrong output method
You are generating HTML but the processor serialises as XML, adding self-closing tags that browsers reject. Declare the output method explicitly:
`xml

Template priority conflict
Two templates match the same node with equal priority. Saxon signals an error rather than silently picking one. Assign explicit priority attributes to resolve the conflict:
`xml

Validating before deploying to production

If you run XSLT as part of an integration pipeline (MuleSoft, Tibco, IBM DataPower, or a custom backend), test the stylesheet in XSLT Playground against representative inputs before deploying. Saxon in the playground uses the same processor your backend may be running, so errors caught here are errors caught before production.

Export the workspace as JSON and keep it as a regression test artifact. If a future change breaks the transform, you have the original inputs and expected output to compare against.

XSLT online editor: how to test transformations without installing anything

Alexandre Vazquez — Thu, 23 Apr 2026 09:00:00 +0000

Testing XSLT locally means installing a processor, configuring classpaths, and running command-line tools every time you want to check a change. For most day-to-day work — writing a new transform, debugging an output, or verifying a colleague's stylesheet — that overhead is unnecessary. A browser-based XSLT editor removes all of it.

What to look for in an online XSLT editor

Not all browser-based tools are equal. The things that matter in practice:

XSLT version support. Many older tools only support XSLT 1.0. If your integration targets 2.0 or 3.0 (grouping, functions, maps, JSON support), you need a tool that runs a proper processor, not a JavaScript port. XSLT Playground uses Saxon on the backend, which gives you full XSLT 2.0 and 3.0 support including extension functions.

Multiple inputs and parameters. Real transforms rarely take a single XML document. You often need a main input plus a reference document, or you need to pass runtime parameters to control output. A good editor lets you define as many inputs and parameters as your stylesheet needs.

Trace output. When a transform produces wrong output, you need to see what the processor did. Trace mode shows template firings and variable values step by step, which is far more useful than reading the final output and guessing what went wrong.

Workspace persistence. If you close the browser and come back later, your inputs and stylesheet should still be there. Saving to localStorage means you can pick up where you left off without copying everything into a text file.

Using XSLT Playground

XSLT Playground covers all of the above. Here is the basic workflow:

Paste your XML source into the input panel.
Paste your XSLT stylesheet into the stylesheet panel.
Set the XSLT version (1.0, 2.0, or 3.0) in the toolbar.
Click Run. The result appears in the output panel within a second or two.

If the transform fails, error messages appear immediately with line references. Enable trace to see the execution log.

For transforms with parameters, open the parameters panel, add key-value pairs, and they are passed to the stylesheet as external parameters on each run. No need to hardcode them in the stylesheet.

Sharing and exporting setups

Each workspace in XSLT Playground can be exported as a JSON file. The export includes the stylesheet, input document, parameters, and any trace output. You can send this file to a colleague, and they import it directly — no copy-pasting required.

This is useful for bug reports: instead of describing what went wrong, export the workspace and share the file. The recipient can reproduce the exact input and output in one click.

When to use an online editor vs a local setup

Use the online editor when:

You are exploring a new XSLT feature or syntax
You need to reproduce or share a specific transform issue
You are working away from your main machine
You want to quickly verify a change before committing it

Use a local setup when:

You are processing files that are sensitive or cannot leave your network
You need to transform very large documents (megabytes or more)
You are integrating XSLT into a CI pipeline

For everything else, the browser editor is faster and easier to use.

Kubernetes HPA Best Practices: When CPU Works, Why Memory Almost Never Does

Alexandre Vazquez — Tue, 21 Apr 2026 11:00:00 +0000

Kubernetes HPA Best Practices: When CPU Works, Why Memory Almost Never Does

How HPA Actually Decides to Scale

The HPA controller uses a formula to determine desired replicas: desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue)). A critical detail is that "the metric value is expressed relative to the resource request, not the resource limit." This distinction explains many HPA failures.

HPA polls metrics every 15 seconds by default, scaling up within one to three polling cycles when thresholds are exceeded. Scale-down is deliberately slow, waiting 5 minutes by default to prevent oscillation.

CPU-Based HPA: When It Works and When It Doesn't

Where CPU HPA Works Well

CPU-based HPA succeeds with stateless request-processing workloads where CPU consumption correlates with request volume. Prerequisites include:

Accurate CPU requests set to actual sustained consumption, not placeholders
Reasonable request-to-limit ratios (1:4 or less)
CPU consumption that tracks user load linearly

Where CPU HPA Fails

CPU HPA struggles with:

Latency-sensitive services with sharp spikes — by the time HPA detects and reacts to peaks, the burst may be over
I/O-bound workloads — showing low CPU even under heavy load
Workloads with cold-start costs — requiring earlier scaling decisions than CPU metrics can trigger

Memory-Based HPA: Why It Almost Always Breaks

The Core Problem

Memory is incompressible; exhausting it causes OOM termination. Unlike CPU, "memory consumption is relatively stable" for well-architected services. A Go service or JVM application maintains a consistent memory footprint regardless of traffic volume from 10 to 10,000 requests per second.

This creates two outcomes: memory HPA either never triggers (useless) or always triggers (permanently scaled out).

The Request Misconfiguration Trap

A Java service needing 512Mi heap but configured with a 256Mi request will immediately consume 200% of its request. An HPA with 70% memory threshold will scale such workloads to maximum replicas permanently. The solution is right-sizing requests, not adjusting thresholds.

JVM and Go Runtime Memory Behavior

The JVM allocates heap up to its maximum and doesn't release it aggressively, even after garbage collection. Go's garbage collector prioritizes low latency over minimal memory use, potentially holding memory above strict necessity.

When Memory HPA Is Actually Appropriate

Memory-based HPA is defensible only in narrow cases:

Workloads where memory consumption tracks load linearly
As a secondary safety valve (not primary) at 85-90% threshold for protecting against memory leaks
Caching services where avoiding eviction before scaling out is critical

Right-Sizing Requests Before Adding HPA

No HPA strategy works without accurate resource requests. Run workloads under representative load and measure actual consumption. VPA in recommendation mode provides data-driven baselines:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-service-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  updatePolicy:
    updateMode: "Off"   # Recommendation only

Critical note: VPA and HPA cannot both auto-manage the same resource metric simultaneously.

Better Signals: What to Scale On Instead

Shift from resource consumption metrics (describing the past) to demand metrics (describing current needs).

Requests Per Second (RPS)

For HTTP services, "requests per second per replica is usually the most accurate proxy for load." RPS measures demand directly, working for CPU-bound, memory-bound, or I/O-bound services.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "500"

Queue Depth and Lag

For consumer workloads reading from message queues, "consumer lag: how many messages are waiting to be processed" is the right scaling signal. KEDA was built for this use case, reading consumer group lag directly.

Latency

P99 latency per replica is an excellent signal for latency-sensitive services, requiring custom metrics from service meshes or APM tools.

Scheduled and Predictive Scaling

For predictable traffic patterns, proactive scaling outperforms reactive scaling. KEDA's Cron scaler enables time-based scaling rules.

HPA Configuration Best Practices

Always Set minReplicas ≥ 2 for Production

A single-replica HPA creates a single point of failure during scale-in events.

Tune Stabilization Windows

The default 5-minute scale-down stabilization is too aggressive for workloads with cyclical patterns. Increase it to match your workload's natural cycle:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
      - type: Percent
        value: 25
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

The behavior block (available in HPA v2) enables independent control over scale-up and scale-down.

Use a Lower CPU Threshold Than You Think

If scale-up takes 45 seconds, a 70% threshold leaves existing pods throttled during that window. Set CPU targets at 50-60% for services where scaling latency matters.

Combine HPA with PodDisruptionBudgets

HPA scale-down terminates pods. Without a PodDisruptionBudget, multiple replicas can be terminated simultaneously during maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-service-pdb
spec:
  minAvailable: "50%"
  selector:
    matchLabels:
      app: my-service

Don't Mix VPA Auto-Update with HPA on the Same Metric

VPA auto-updating requests while HPA scales on those metrics creates conflicting control loops.

Decision Framework: Which Autoscaler for Which Workload

Workload type	Recommended signal	Tool
Stateless HTTP API, CPU-bound	CPU utilization at 50-60%	HPA
Stateless HTTP API, I/O-bound	RPS per replica or P99 latency	HPA + custom metrics
Message queue consumer	Consumer lag / queue depth	KEDA
Event-driven / Kafka / SQS	Event rate or lag	KEDA
Predictable traffic pattern	Schedule (time-based)	KEDA Cron scaler
Workload with memory leak risk	CPU primary + memory at 85% secondary	HPA (v2 multi-metric)
Right-sizing before HPA	Historical CPU/memory recommendations	VPA recommendation mode

Going Beyond HPA: KEDA and Custom Metrics

KEDA provides a Kubernetes-native autoscaling framework supporting over 60 built-in scalers. The key architectural point: "KEDA does not replace HPA — it feeds it." KEDA creates and manages HPA resources while consuming signals HPA cannot access natively.

FAQ

Can I use both CPU and memory in the same HPA?

Yes. HPA v2 supports multiple metrics simultaneously, scaling to satisfy the most demanding metric. Use CPU at 60% threshold and memory at 85% threshold so memory only triggers in genuine overconsumption.

Why does my workload scale up immediately after deployment?

Resource request misconfiguration. Check actual consumption against requests using kubectl top pods. If consuming 200% of request by simply running, adjust requests to match actual usage before enabling HPA.

Why does HPA scale down too aggressively and cause latency spikes?

Increase scaleDown.stabilizationWindowSeconds in the HPA behavior block. Also add a Percent policy limiting scale-down to 25% of replicas per minute.

Should I set HPA on every deployment?

No. HPA fits stateless services, consumers, and request handlers. It's inappropriate for stateful workloads requiring more than replica addition, singleton controllers, or batch jobs that should run to completion.

What is the minimum CPU request for reliable HPA?

No absolute minimum, but requests below 100m make percentage thresholds coarse-grained. At 50m and 70% threshold, scaling triggers at 35m consumption. For lower needs, use RPS or custom metrics instead.

How do I debug HPA scaling decisions?

Use kubectl describe hpa to see current metrics and last scaling events. Check HPA events with kubectl get events --field-selector involvedObject.kind=HorizontalPodAutoscaler. For custom metrics, verify the metrics server returns expected values.

Originally published at alexandre-vazquez.com/kubernetes-hpa-best-practices