Forem: Sodiq Jimoh

Deploying Backstage on Kubernetes with the Helm Chart: The Infrastructure-First Guide

Sodiq Jimoh — Mon, 06 Apr 2026 02:02:26 +0000

Who this is for: Engineers deploying Backstage on Kubernetes via the
official Helm chart who want a working portal, not just a running pod.
This guide starts where most tutorials end — after helm install succeeds
but before anything actually works.

A few weeks ago I published an article called
"Nine Ways Backstage Breaks Before Your Developer Portal Works".
A Backstage maintainer read it and gave me structured feedback. The core of
it was this: several of the failures I documented were caused by not
following the official getting-started documentation before using the Helm
chart, and by using the demo image as if it were a production-ready base.

They were right. This article is the follow-up they suggested — and the one
I should have written first.

It does not repeat the previous article. It starts earlier, goes deeper on
Helm-specific configuration, and correctly attributes failures to their
actual causes rather than blaming Backstage for things that are ArgoCD,
Traefik, or operator error.

Official resources you should read alongside this guide:

Project repo referenced throughout:
github.com/sodiq-code/neuroscale-platform

The one thing you must understand before installing the Helm chart

The Backstage Helm chart uses a demo image by default. The chart README
contains this explicit warning:

The Backstage chart is not an official Backstage project and is not
supported by the Backstage core team. The default image used in this chart
is for demo purposes only.

This single fact explains most of the configuration friction you will
encounter. The demo image does not behave like a real Backstage application
built with backstage new app. It has different startup characteristics,
different configuration defaults, and different failure modes.

What this means practically:

If you are building a real developer portal — not just running a demo — you
should follow the official getting started guide
to create your own Backstage application first, build a custom Docker image
from it, and then use the Helm chart to deploy that image. The chart's
image.repository and image.tag values are where you point to your
own image.

If you are experimenting, learning, or building an integration platform
where Backstage is one component (as in the NeuroScale project), the demo
image path is workable — but you need to understand its limitations and
configure it correctly.

This guide covers the Helm chart path specifically, with the official docs
as the reference point throughout.

The values hierarchy that breaks everything silently

This is the most important configuration concept in the entire Helm chart.
Get this wrong and every override you write will be silently ignored.

The Backstage Helm chart is a wrapper chart — Backstage itself is a
dependency inside it. The dependency is named backstage. This means
configuration for the Backstage application container must be nested under
backstage.backstage.*, not backstage.*.

Wrong — values are silently ignored:

# This looks correct but is placed at the wrong hierarchy level
backstage:
  appConfig:
    app:
      title: My Platform
  startupProbe:
    initialDelaySeconds: 120
  resources:
    requests:
      cpu: 100m

Correct — values reach the Backstage container:

backstage:
  backstage:           # <-- this second level is required
    appConfig:
      app:
        title: My Platform
    startupProbe:
      initialDelaySeconds: 120
    resources:
      requests:
        cpu: 100m

The Helm chart processes the outer backstage key as the dependency name.
Values placed directly under backstage.* are interpreted as chart-level
configuration, not as container configuration. Kubernetes then uses chart
defaults — including probe timings — rather than your overrides.

How to verify your values are actually applied:

Render the Helm chart before applying it and inspect the output Deployment
spec directly:

helm template neuroscale-backstage backstage/backstage \
  -f infrastructure/backstage/values.yaml \
  --namespace backstage \
  | grep -A 30 "startupProbe"

If you see initialDelaySeconds: 120 in the output, your probe override
reached the container. If you see initialDelaySeconds: 5 or a very small
number, your values are at the wrong nesting level.

This verification step should be part of your CI pipeline. In the NeuroScale
platform, scripts/ci/render_backstage.sh runs this check on every PR:

#!/bin/bash
# scripts/ci/render_backstage.sh
helm template neuroscale-backstage backstage/backstage \
  -f infrastructure/backstage/values.yaml \
  --namespace backstage \
  | grep "initialDelaySeconds" \
  | grep -q "120" || {
    echo "ERROR: startupProbe initialDelaySeconds not set correctly"
    exit 1
  }
echo "Helm values nesting verified"

Required configuration keys for the demo image

The demo image requires specific configuration keys to be present at
startup. Missing any of them causes the frontend to crash on load with a
JavaScript error that is only visible in browser developer tools — the page
itself shows a blank white screen with no visible error.

The minimum required appConfig block:

backstage:
  backstage:
    appConfig:
      app:
        title: Your Platform Name    # required — crash if absent
        baseUrl: http://localhost:7010
      backend:
        baseUrl: http://localhost:7010
        cors:
          origin: http://localhost:7010
        database:
          client: better-sqlite3
          connection: ':memory:'

Why baseUrl matters:

The app.baseUrl and backend.baseUrl values must match the URL you are
actually using to access Backstage. If you port-forward on port 7010 but
the config says port 7007, the frontend React app loads but all API calls
fail — the UI appears to work while the backend connection is broken.

Why better-sqlite3 for local deployments:

The demo image ships with SQLite support. For local Kubernetes deployments
where you want zero external dependencies, the in-memory SQLite connection
is sufficient. For production, replace this with a PostgreSQL connection
pointing at a managed database service. The chart includes optional
PostgreSQL deployment — see
the chart's database configuration docs.

Probe timings: the demo image starts slowly

Backstage is a Node.js application. The demo image takes approximately 60
to 90 seconds to complete startup on a typical Kubernetes node. Kubernetes
default probe timings assume a 2-second initial delay. The result is
predictable: the startup probe fires before the application is ready, the
pod fails the probe, Kubernetes kills it, and the pod enters
CrashLoopBackOff.

This is not a Backstage bug. It is a configuration requirement that the
Helm chart does not prominently surface. The correct probe settings:

backstage:
  backstage:
    startupProbe:
      httpGet:
        path: /healthcheck
        port: 7007
      initialDelaySeconds: 120    # give Node.js time to start
      periodSeconds: 10
      failureThreshold: 30        # 30 × 10s = 5 minutes maximum wait
    readinessProbe:
      httpGet:
        path: /healthcheck
        port: 7007
      initialDelaySeconds: 120
      periodSeconds: 10
      failureThreshold: 3
    livenessProbe:
      httpGet:
        path: /healthcheck
        port: 7007
      initialDelaySeconds: 300    # only check liveness after 5 minutes
      periodSeconds: 30
      failureThreshold: 3

How to diagnose probe failures:

# Watch pod status in real time
kubectl get pods -n backstage -w

# When you see CrashLoopBackOff, describe the pod
kubectl describe pod -n backstage <pod-name>

# Look for this in Events:
# Warning  Unhealthy  kubelet  Startup probe failed: connection refused

# Check logs from the previous container instance
kubectl logs -n backstage <pod-name> --previous --tail=100

If you see Startup probe failed: connection refused in events but the
previous container logs show normal Node.js startup messages, the
application is starting correctly — the probe is just firing too early.
Increase initialDelaySeconds.

A full incident postmortem for this specific failure, including the exact
Kubernetes events and the Helm values diff before and after the fix, is in
infrastructure/INCIDENT_BACKSTAGE_CRASHLOOP_RCA.md.

Authentication: local dev vs production

The Backstage new backend architecture (introduced in version 1.x) includes
an internal authentication policy that requires all service-to-service calls
to include a valid Backstage token. This affects how the scaffolder frontend
talks to the scaffolder backend — a call that was unauthenticated in older
versions.

For local development only, the quickest fix is to use the guest auth
provider:

backstage:
  backstage:
    appConfig:
      auth:
        providers:
          guest:
            dangerouslyAllowOutsideDevelopment: true

This keeps the auth subsystem active and provides a real
user:default/guest identity to all plugins — which is safer than
disabling auth entirely with dangerouslyDisableDefaultAuthPolicy: true.
Plugins that assume a user context will behave correctly.

For production, use the GitHub OAuth provider:

# infrastructure/backstage/values-prod.yaml
backstage:
  backstage:
    appConfig:
      auth:
        environment: production
        providers:
          github:
            production:
              clientId: ${GITHUB_CLIENT_ID}
              clientSecret: ${GITHUB_CLIENT_SECRET}

Store GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET as Kubernetes secrets,
not in values.yaml. The Helm chart's extraEnvVarsSecrets field handles
this:

backstage:
  backstage:
    extraEnvVarsSecrets:
      - backstage-secrets

Then create the secret:

kubectl create secret generic backstage-secrets \
  -n backstage \
  --from-literal=GITHUB_CLIENT_ID="your-client-id" \
  --from-literal=GITHUB_CLIENT_SECRET="your-client-secret"

How to verify auth is configured correctly:

# Check the scaffolder actions API directly
curl http://localhost:7010/api/scaffolder/v2/actions

# If you get 401: auth is not configured for your environment
# If you get 200 with a JSON list of actions: auth is working

If you get a 401 with {"error":{"name":"AuthenticationError","message":"Missing credentials"}},
the scaffolder form will load but render blank — the page returns HTTP 200
but has no data to display. This is only visible in browser developer tools.

Catalog configuration: registering templates

The Backstage catalog applies security rules to what entity kinds are
accepted from each registered location. The default allow list for
repository-based locations does not include Template.

This is documented in
the catalog rules documentation
and the
adding templates documentation.

The registration pattern that works:

backstage:
  backstage:
    appConfig:
      catalog:
        locations:
          - type: url
            target: https://github.com/your-org/your-repo/blob/main/backstage/templates/your-template/template.yaml
            rules:
              - allow: [Template]

Without the rules: - allow: [Template] block, the entity is silently
rejected at ingestion time. The only signal is a warning in the Backstage
server logs — nothing appears in the UI.

How to diagnose catalog ingestion failures:

kubectl logs -n backstage deploy/backstage --tail=100 \
  | grep -i "warn\|error\|forbidden\|NotAllowedError"

Look for NotAllowedError: Forbidden: entity of kind Template is not allowed from that location. If you see this, your rules block is missing
or at the wrong YAML nesting level.

After updating the config, restart Backstage to re-ingest:

kubectl rollout restart deploy/backstage -n backstage
kubectl rollout status deploy/backstage -n backstage --timeout=300s

The template should appear in /create within 60 seconds of the pod
becoming ready.

You can validate your app-config.yaml structure using the Backstage CLI:

npx @backstage/cli config:check --config app-config.yaml

GitHub integration: the token secret

The scaffolder requires a GitHub token to open pull requests. The token
must be present as an environment variable in the running Backstage pod.

backstage:
  backstage:
    appConfig:
      integrations:
        github:
          - host: github.com
            token: ${GITHUB_TOKEN}

Store the token as a Kubernetes secret:

# Create or update the secret
kubectl create secret generic backstage-github-token \
  -n backstage \
  --from-literal=GITHUB_TOKEN="ghp_your_token_here" \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart to reload the environment variable
kubectl rollout restart deploy/backstage -n backstage

Critical: environment variables from Kubernetes secrets are injected at
pod start time. Updating the secret does not update the running pod. You
must restart the deployment after updating the secret for the new value
to take effect.

How to verify the token is present without exposing the value:

# Check character length — a valid GitHub token is 40+ characters
kubectl exec -n backstage deploy/backstage -- \
  sh -c 'echo "Token length: ${#GITHUB_TOKEN}"'

If this returns Token length: 0 or Token length: 16 (the length of a
placeholder like <YOUR_TOKEN_HERE>), the secret was not updated correctly
or the pod was not restarted after the update.

A working minimal values.yaml for local development

This is the minimum configuration that produces a functioning Backstage
portal on a local Kubernetes cluster with the demo image:

# infrastructure/backstage/values.yaml
backstage:
  backstage:
    image:
      registry: ghcr.io
      repository: backstage/backstage
      tag: latest           # pin to a specific version in production

    appConfig:
      app:
        title: Your Platform
        baseUrl: http://localhost:7010

      backend:
        baseUrl: http://localhost:7010
        cors:
          origin: http://localhost:7010
        database:
          client: better-sqlite3
          connection: ':memory:'

      auth:
        providers:
          guest:
            dangerouslyAllowOutsideDevelopment: true

      integrations:
        github:
          - host: github.com
            token: ${GITHUB_TOKEN}

      catalog:
        locations:
          - type: url
            target: https://github.com/your-org/your-repo/blob/main/backstage/templates/your-template/template.yaml
            rules:
              - allow: [Template]

    startupProbe:
      httpGet:
        path: /healthcheck
        port: 7007
      initialDelaySeconds: 120
      periodSeconds: 10
      failureThreshold: 30

    readinessProbe:
      httpGet:
        path: /healthcheck
        port: 7007
      initialDelaySeconds: 120
      periodSeconds: 10
      failureThreshold: 3

    livenessProbe:
      httpGet:
        path: /healthcheck
        port: 7007
      initialDelaySeconds: 300
      periodSeconds: 30
      failureThreshold: 3

    resources:
      requests:
        cpu: 100m
        memory: 512Mi

    extraEnvVarsSecrets:
      - backstage-github-token

  postgresql:
    enabled: false    # using in-memory SQLite for local dev

Deploying and verifying

Install:

helm repo add backstage https://backstage.github.io/charts
helm repo update

kubectl create namespace backstage

# Create the GitHub token secret first
kubectl create secret generic backstage-github-token \
  -n backstage \
  --from-literal=GITHUB_TOKEN="your-token"

# Install
helm install backstage backstage/backstage \
  -n backstage \
  -f infrastructure/backstage/values.yaml

Watch the startup:

kubectl get pods -n backstage -w

Expect the pod to stay in Running 0/1 for 60–120 seconds while Node.js
starts. Do not interpret this as a failure. The startup probe will not
pass until the application is ready.

Access the portal:

kubectl -n backstage port-forward svc/backstage 7010:7007
# Open: http://localhost:7010

Verify the backend is responding:

curl http://localhost:7010/healthcheck
# Expected: {"status":"ok"}

curl http://localhost:7010/api/scaffolder/v2/actions
# Expected: JSON list of available scaffolder actions

Verify catalog ingestion:

kubectl logs -n backstage deploy/backstage --tail=50 \
  | grep -i "processed\|warn\|error"

Look for Processed N entities with no NotAllowedError lines.

The production values profile

Separate your dev and prod configuration into two files. The difference is
significant enough that sharing a single file creates dangerous defaults
in production.

# infrastructure/backstage/values-prod.yaml
backstage:
  backstage:
    image:
      registry: ghcr.io
      repository: your-org/your-backstage-app   # your own image
      tag: "1.2.3"                               # pinned, never latest

    replicaCount: 2

    appConfig:
      app:
        baseUrl: https://backstage.your-domain.com
      backend:
        baseUrl: https://backstage.your-domain.com
        database:
          client: pg
          connection:
            host: ${POSTGRES_HOST}
            port: 5432
            user: ${POSTGRES_USER}
            password: ${POSTGRES_PASSWORD}
            database: backstage

      auth:
        environment: production
        providers:
          github:
            production:
              clientId: ${GITHUB_CLIENT_ID}
              clientSecret: ${GITHUB_CLIENT_SECRET}

    startupProbe:
      initialDelaySeconds: 60     # your own image starts faster
      failureThreshold: 18        # 3 minutes maximum

    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: "1"
        memory: 1Gi

Apply both files together:

helm upgrade backstage backstage/backstage \
  -n backstage \
  -f infrastructure/backstage/values.yaml \
  -f infrastructure/backstage/values-prod.yaml

The prod values file overrides only what it specifies. Everything else
comes from the base values.yaml.

Diagnostic command reference

# Pod status and events
kubectl get pods -n backstage
kubectl describe pod -n backstage <pod-name>

# Application logs
kubectl logs -n backstage deploy/backstage --tail=100
kubectl logs -n backstage deploy/backstage --previous --tail=100

# Catalog ingestion errors
kubectl logs -n backstage deploy/backstage --tail=200 \
  | grep -i "warn\|error\|forbidden"

# Verify rendered Helm values
helm template backstage backstage/backstage \
  -f infrastructure/backstage/values.yaml \
  --namespace backstage \
  | grep -A 5 "startupProbe"

# Verify token is loaded in the running container
kubectl exec -n backstage deploy/backstage -- \
  sh -c 'echo "GITHUB_TOKEN length: ${#GITHUB_TOKEN}"'

# Health check endpoints
curl http://localhost:7010/healthcheck
curl http://localhost:7010/api/catalog/entities?limit=1
curl http://localhost:7010/api/scaffolder/v2/actions

What this guide does not cover

This guide covers the Helm chart deployment path specifically. It does not
cover:

Building your own Backstage application — start with the official getting started guide and backstage new app for a real production portal
Writing custom plugins — see plugin development docs
TechDocs integration — covered separately in the TechDocs docs
Production ingress and TLS — specific to your cloud provider and ingress controller

9 Failures That Hit Me Building a Backstage Golden Path for KServe — Every Error, Every Fix

Sodiq Jimoh — Mon, 30 Mar 2026 23:12:36 +0000

Edit (Apr 2026): Updated title and added framing context based on community feedback.

Series context: This is Part 3 of building a production-hardened AI inference platform.

Part 1: Why Your KServe InferenceService Won't Become Ready

Part 2: 5 GitOps Failure Modes That Break KServe Deployments

Project repo: github.com/sodiq-code/neuroscale-platform

If you have ever deployed Backstage and stared at a blank /create page wondering what went wrong, this article is for you.

Most Backstage tutorials end at "the portal is running." This one starts there.

One important framing note before we begin: this article documents the path starting from the official Backstage Helm chart, not from backstage new app. If you're building a real Backstage application from source, some of these failures won't apply. But if you're doing what a lot of platform engineers do, reaching for helm install first, then every single one of these will.

This is a complete production failure log from implementing a Backstage Golden Path that deploys KServe model inference endpoints on Kubernetes. Nine distinct failures. Every one with exact error output, root cause, and the fix that worked.

The goal: a developer fills a Backstage form, a GitHub PR opens, the PR merges, ArgoCD deploys a KServe InferenceService, and the endpoint responds to predictions.
Getting there took nine failures across three days.

What I was trying to build

The Golden Path demo contract:

Backstage form → PR opened → merge → ArgoCD sync → InferenceService Ready=True → curl returns {"predictions":[1,1]}

Stack:

Backstage (Helm chart, self-hosted on k3d)
ArgoCD (GitOps reconciliation)
KServe (model inference endpoints)
GitHub (scaffolder target)
Kyverno (admission policies)

Failure 1: Template Not Visible in Catalog — Silent Rejection With No UI Error

Time lost: 30 minutes

Symptom

After adding the template file and registering it in infrastructure/backstage/values.yaml, the template did not appear in Backstage's /create page. No error was visible in the UI. The page simply showed an empty catalog.

Digging In

$ kubectl -n backstage logs deploy/neuroscale-backstage --tail=50
...
[backstage] warn  Failed to process location
  {"location":{"type":"url","target":"https://github.com/sodiq-code/
  neuroscale-platform/blob/main/backstage/templates/model-endpoint/
  template.yaml"},
  "error":"NotAllowedError: Forbidden: entity of kind Template
  is not allowed from that location"}

The error only appears in server logs. The UI shows nothing.

Root Cause

Backstage's catalog configuration allows only specific entity kinds from each registered location. The default allow list for repository-based locations does not include Template. Without an explicit allow: [Template] rule, entities of kind Template are silently rejected. This is security-by-default behavior — but the complete silence in the UI makes it look like a misconfiguration rather than a permission issue.

Fix

# infrastructure/backstage/values.yaml
backstage:
  backstage:
    appConfig:
      catalog:
        locations:
          - type: url
            target: https://github.com/sodiq-code/neuroscale-platform/blob/main/backstage/templates/model-endpoint/template.yaml
            rules:
              - allow: [Template]

After rolling out the updated Backstage deployment:

$ kubectl -n backstage rollout restart deploy/neuroscale-backstage
$ kubectl -n backstage rollout status deploy/neuroscale-backstage --timeout=300s
deployment "neuroscale-backstage" successfully rolled out

The template appeared in /create within 60 seconds.

Lesson

For a platform team deploying Backstage for internal users, this silent failure means developers see an empty template catalog and assume the platform is broken — not that a config rule is missing. Always check server logs, not just the UI, when Backstage catalog ingestion seems to fail.

Failure 2: Scaffolder /create Page Loads Blank — 401 on Actions API

Time lost: 45 minutes

Symptom

After the template was visible, clicking into it showed a blank form. The browser developer console revealed:

GET /api/scaffolder/v2/actions HTTP/1.1 401 Unauthorized
{"error":{"name":"AuthenticationError","message":"Missing credentials"}}

The page route returned HTTP 200 — the React app loaded — but the actions API returned 401, so the form had no data to render.

Root Cause

Backstage's new backend architecture (introduced in 1.x) adds an internal authentication policy requiring all service-to-service calls to include a valid Backstage token. The scaffolder frontend makes an internal API call to list available actions. Because no auth provider was configured for local development, this internal call was rejected. This is a breaking change from older Backstage versions where the actions endpoint was unauthenticated.

Fix

# infrastructure/backstage/values.yaml
backstage:
  backstage:
    appConfig:
      backend:
        auth:
          dangerouslyDisableDefaultAuthPolicy: true

Production note: dangerouslyDisableDefaultAuthPolicy: true is acceptable for local development only. For production, configure GitHub OAuth via values-prod.yaml with a proper sign-in policy. The production profile uses auth.providers.guest.dangerouslyAllowOutsideDevelopment: true instead — which keeps the auth subsystem active and provides a real user:default/guest identity, rather than disabling auth entirely.

Lesson

An empty scaffolder form is indistinguishable from a misconfigured form to an end user. The 401 error is only visible in browser developer tools. This is the second failure in this series that generated zero visible error in the UI.

Failure 3: Frontend Crashes With Blank White Screen — Missing Required Config Key

Time lost: 20 minutes

Symptom

After the auth policy fix, reloading Backstage showed a blank white screen. The browser console:

Uncaught Error: Missing required config value at 'app.title' in 'app'
    at validateConfigSchema (config.esm.js:234)
    at BackstageApp.render (app.esm.js:891)

Root Cause

The Backstage frontend requires app.title to be present in the runtime configuration. This key was absent from the appConfig section of values.yaml. The React application crashed on initialization before any content could render. This is a required configuration key not documented prominently as "required on first boot."

Fix

# infrastructure/backstage/values.yaml
backstage:
  backstage:
    appConfig:
      app:
        title: NeuroScale Platform
        baseUrl: http://localhost:7010
      backend:
        baseUrl: http://localhost:7010
        cors:
          origin: http://localhost:7010

Note: app.baseUrl and backend.baseUrl also needed to match the port used for port-forwarding (7010), not the default 7007.

Lesson

A blank white screen with no network errors means the JavaScript runtime crashed before rendering. Always check the browser console — not just network requests — for Backstage frontend failures.

Failure 4: Backstage CrashLoopBackOff — Helm Dependency Values Mis-Nesting

Time lost: 2 hours | Impact: Developer portal completely unavailable

Symptom

$ kubectl get pods -n backstage -w
NAME                                    READY   STATUS             RESTARTS
neuroscale-backstage-7d9f5b8c4-xqr2m   0/1     CrashLoopBackOff   8          12m

$ kubectl describe pod neuroscale-backstage-7d9f5b8c4-xqr2m -n backstage
...
Events:
  Warning  Unhealthy  30s  kubelet
    Startup probe failed: connect: connection refused

Root Cause

The Backstage Helm chart is a wrapper chart with backstage as a dependency. Configuration for the Backstage container itself must be nested under backstage.backstage.*, not backstage.*. The misconfiguration meant probe settings and resource requests were silently ignored, so Kubernetes used default probe timings — a 2-second initial delay — that were far too aggressive for Backstage's ~90-second startup time.

The pod was killed before it could become healthy, triggering CrashLoopBackOff.

Backstage requires:

startupProbe:
  initialDelaySeconds: 120
  failureThreshold: 30

The default gives it 2 seconds.

Fix

Correct the values hierarchy and harden probe timings:

# infrastructure/backstage/values.yaml
backstage:
  backstage:           # <-- must be nested here, not at backstage.*
    appConfig:
      ...
    startupProbe:
      initialDelaySeconds: 120
      failureThreshold: 30
    readinessProbe:
      initialDelaySeconds: 120
    livenessProbe:
      initialDelaySeconds: 300
    resources:
      requests:
        cpu: 100m
        memory: 512Mi

Lesson

If a Helm chart is a wrapper with a dependency, configuration for the dependency must be nested under the dependency's alias key. Values placed at the wrong hierarchy level are silently ignored — Kubernetes uses chart defaults, not your overrides. This incident directly motivated adding CI validation for rendered Helm manifests: if the final Deployment spec had been checked in CI, the wrong probe values would have been caught before deployment. Full RCA: infrastructure/INCIDENT_BACKSTAGE_CRASHLOOP_RCA.md

Failure 5: PR Creation Fails — GitHub Token Secret Contains Placeholder Value

Time lost: 30 minutes

Symptom

After the portal was stable, the scaffolder's "Open pull request" step spun for 30 seconds then failed:

Error: Request failed with status 401: Bad credentials

No PR was created in GitHub.

Root Cause

The Kubernetes Secret neuroscale-backstage-secrets contained a placeholder GITHUB_TOKEN value — literally <YOUR_TOKEN_HERE>. The environment variable was present, satisfying kubectl describe secret output, but the value was not a valid token.

A secondary issue: after updating the secret with the correct token, the running pod did not pick up the change. Environment variables from Secrets are injected at pod start time, not dynamically. The pod needed a restart.

Fix

# Update the secret with a valid token
read -s GITHUB_TOKEN
kubectl -n backstage create secret generic neuroscale-backstage-secrets \
  --from-literal=GITHUB_TOKEN="$GITHUB_TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart to reload env vars
kubectl -n backstage rollout restart deploy/neuroscale-backstage
kubectl -n backstage rollout status deploy/neuroscale-backstage --timeout=300s

# Verify token is present — check length, never the value
kubectl -n backstage exec deploy/neuroscale-backstage -- \
  sh -c 'echo ${#GITHUB_TOKEN} chars'

Lesson

kubectl describe secret shows the key exists and has bytes. It does not show whether the value is a valid token or a placeholder string. Always verify token presence by checking character length in the running container, never by reading the secret value directly.

Failure 6: PR Merged But ArgoCD Stays OutOfSync — Fix Not Committed to Git

Time lost: 1 hour of confusion

Symptom

The Backstage scaffolder created the PR correctly. CI passed. The PR was merged. ArgoCD detected the new application. But the child app immediately showed OutOfSync/Degraded:

$ kubectl -n argocd get application demo-iris-2
NAME          SYNC STATUS   HEALTH STATUS
demo-iris-2   OutOfSync      Degraded

$ kubectl -n argocd describe application demo-iris-2
...
Message: Internal error occurred: failed calling webhook
  "inferenceservice.kserve-webhook-server.validator.webhook":
  no endpoints available for service "kserve-webhook-server-service"

Root Cause

This was the kube-rbac-proxy ImagePullBackOff failure from earlier — reappearing after a cluster restart. The fix had been applied with kubectl patch directly, not committed to Git. ArgoCD's selfHeal: true reverted it on the next sync cycle. The cluster restart exposed that the fix was never persisted.

Fix

# Verify the patch is in kustomization.yaml
cat infrastructure/serving-stack/kustomization.yaml | grep -A2 patches

# Commit and push
git add infrastructure/serving-stack/
git commit -m "serving-stack: persist kube-rbac-proxy removal patch"
git push origin main

ArgoCD picked up the change within 3 minutes.

Lesson

Any fix applied with kubectl directly in a GitOps-managed cluster is temporary. The next sync cycle will revert it. Every fix must be committed to Git to survive. The PR-merged-but-nothing-deployed experience is the worst possible failure for a Golden Path demo — the developer did everything correctly and the platform failed silently.

Failure 7: Inference Endpoint Returns HTTP 307 Redirect — Traefik Intercepts Before Kourier

Time lost: 45 minutes

Symptom

After demo-iris-2 became Ready=True, the inference test returned an unexpected redirect:

$ curl -v \
  -H 'Content-Type: application/json' \
  -d '{"instances":[[6.8,2.8,4.8,1.4]]}' \
  http://172.20.0.3/v1/models/demo-iris-2:predict

< HTTP/1.1 307 Temporary Redirect
< Location: https://172.20.0.3/v1/models/demo-iris-2:predict

Root Cause

k3d's built-in Traefik ingress was intercepting the request and applying an HTTP-to-HTTPS redirect before it reached Kourier. The request never reached the Knative routing layer at all.

Fix

Use direct pod port-forward for canonical local verification, bypassing Traefik and Kourier entirely:

# Find predictor pod
kubectl -n default get pods \
  -l serving.knative.dev/revision=demo-iris-2-predictor-00001 \
  -o jsonpath='{.items[0].metadata.name}'

# Port-forward directly to the pod
kubectl -n default port-forward \
  pod/demo-iris-2-predictor-00001-deployment-<hash> 18080:8080

# Predict
curl -sS \
  -H "Content-Type: application/json" \
  -d '{"instances":[[6.8,2.8,4.8,1.4],[6.0,3.4,4.5,1.6]]}' \
  http://127.0.0.1:18080/v1/models/demo-iris-2:predict

{"predictions":[1,1]}

Lesson

A healthy inference endpoint can look completely broken if your test path hits an unexpected intermediary. For local k3d clusters, disable Traefik at cluster creation:

k3d cluster create neuroscale \
  --k3s-arg "--disable=traefik@server:0"

Failure 8: Catalog Ingestion Silently Rejects Template After Values Update

Time lost: 20 minutes

Symptom

After updating values.yaml and rolling out a new Backstage deployment, the template disappeared from /create again — the same symptom as Failure 1, but after it had been working.

Root Cause

The rolling update caused a brief period where the new pod was starting and the old pod was terminating. During this window, the catalog re-ingested all locations. The updated values.yaml had a YAML indentation error in the catalog.locations block, which caused the allow rule for Template to be silently dropped during parsing.

Fix

# Check catalog ingestion in the new pod logs immediately after rollout
kubectl -n backstage logs deploy/neuroscale-backstage --tail=100 | \
  grep -i "warn\|error\|fail\|forbidden"

Fixed the YAML indentation:

# Correct indentation
catalog:
  locations:
    - type: url
      target: https://github.com/...
      rules:
        - allow: [Template]   # must be under rules:, not misaligned

Lesson

YAML indentation errors in Backstage config values are never surfaced as errors — the field is simply ignored. After every Backstage rollout that touches appConfig, immediately verify catalog ingestion by checking server logs and confirming the template appears in /create.

Failure 9: Scaffolder Task Hangs Then Fails — Port-Forward Session Died Mid-Task

Time lost: 15 minutes

Symptom

The scaffolder task started successfully, progress spinner ran for 60 seconds, then failed with a network error. The Backstage UI showed the task as failed with no specific error message. A second attempt worked immediately.

Root Cause

The kubectl port-forward session for Backstage had silently died between opening the browser and submitting the scaffolder form. The React app was loaded from cache — so the page appeared fully functional — but all API calls were failing because the backend was unreachable. The scaffolder task started, sent the first API call, and failed on the network layer.

Fix

# Before running any Backstage scaffolder task, verify the port-forward is alive
curl -s http://localhost:7010/api/catalog/entities?limit=1 | head -c 100

# If it returns nothing or errors, restart the port-forward
kubectl -n backstage port-forward svc/neuroscale-backstage 7010:7007

Use scripts/port-forward-all.sh from the repository which starts all required port-forwards as background processes with clean shutdown handling.

Lesson

A React app loaded from browser cache looks fully functional even when the backend is unreachable. Always verify the backend API is responding before running a scaffolder task, not just that the UI loaded.

What the Golden Path Actually Proves After Nine Failures

Final working state:

$ kubectl -n default get inferenceservice demo-iris-2
NAME          URL                                       READY   AGE
demo-iris-2   http://demo-iris-2.default.example.com   True    25m

$ curl -sS \
  -H "Content-Type: application/json" \
  -d '{"instances":[[6.8,2.8,4.8,1.4],[6.0,3.4,4.5,1.6]]}' \
  http://127.0.0.1:18080/v1/models/demo-iris-2:predict

{"predictions":[1,1]}

The Golden Path demo is a chain of seven moving parts: Backstage config, GitHub auth, ArgoCD app-of-apps, KServe controller, Knative routing, Kourier gateway, and the predictor pod. In production, any link in that chain can fail independently.

The debugging process for these nine failures is a direct map to what a platform SRE does on an on-call shift.

Debugging Commands Reference

# Backstage catalog ingestion errors
kubectl -n backstage logs deploy/neuroscale-backstage | \
  grep -i "warn\|error\|fail\|forbidden"

# Backstage runtime config
kubectl -n backstage describe configmap neuroscale-backstage-app-config

# Verify GitHub token is present (check length only)
kubectl -n backstage exec deploy/neuroscale-backstage -- \
  sh -c 'echo ${#GITHUB_TOKEN} chars'

# ArgoCD child app sync status
kubectl -n argocd get applications
kubectl -n argocd describe application demo-iris-2

# InferenceService conditions
kubectl -n default describe inferenceservice demo-iris-2

# Admission webhook endpoints
kubectl -n kserve get endpoints kserve-webhook-server-service

The Pattern Across All Nine Failures

Looking back at the nine failures, they fall into three categories:

Silent failures (no UI error, log only):
Failures 1, 2, 8 — catalog ingestion rejections and auth failures that show nothing in the UI. Rule: always check server logs, not just the browser.

Configuration hierarchy failures:
Failures 3, 4 — missing required keys and wrong Helm nesting. Rule: validate rendered manifests in CI before applying them.

State and dependency failures:
Failures 5, 6, 7, 9 — stale secrets, unreversioned fixes, intercepting proxies, dead sessions. Rule: verify the complete dependency chain before debugging the thing that appears broken.

Beyond InferenceService Readiness: 5 GitOps Failure Modes That Break KServe Deployments

Sodiq Jimoh — Mon, 30 Mar 2026 22:52:16 +0000

A sequel to my KServe readiness post — five GitOps control-plane failure modes with exact terminal output, diagnostics, and repeatable fixes for ArgoCD + KServe stacks.

This post is a follow-up to my earlier KServe piece on endpoint readiness:

👉 Why Your KServe InferenceService Won't Become Ready: Four Production Failures and Fixes

That article focused on why an InferenceService may not become Ready.

This one zooms out to a broader question:

What breaks when the GitOps control plane itself is unstable?

Most GitOps + AI serving tutorials still focus on the happy path — install ArgoCD, apply KServe, deploy InferenceService, done. But in real platform work, the happy path is the easy part.

The hard part is when your app is OutOfSync, the webhook has no endpoints, and everything looks healthy except the thing you actually need.

This post covers the five failure modes that repeatedly broke KServe deployments in a real production-grade platform build, with exact terminal output, root causes, and the fixes that worked.

All failures come from hands-on implementation work documented here:
Project repo: github.com/sodiq-code/neuroscale-platform

The platform context

Stack:

ArgoCD — GitOps reconciliation
KServe — model serving (InferenceService, runtimes)
Knative + Kourier — serving networking
Kyverno — policy guardrails
Backstage — self-service PR generation

GitOps root app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: neuroscale-infrastructure
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/sodiq-code/neuroscale-platform.git
    targetRevision: main
    path: infrastructure/apps

Failure Mode 1: Webhook Has No Endpoints — Sync Fails Cluster-Wide

Time lost: ~1 hour | Impact: All InferenceService operations blocked

Symptom

ArgoCD syncs child apps and hits this:

$ kubectl -n argocd describe application ai-model-alpha
...
Message: admission webhook
  "inferenceservice.kserve-webhook-server.validator.webhook"
  denied the request: Internal error occurred:
  no endpoints available for service "kserve-webhook-server-service"

Meanwhile the KServe controller pod shows only 1 of 2 containers ready:

$ kubectl -n kserve get pods
NAME                                        READY   STATUS
kserve-controller-manager-8d7c5b9f4-xr2lm  1/2     Running

$ kubectl -n kserve describe pod kserve-controller-manager-8d7c5b9f4-xr2lm
...
  kube-rbac-proxy:
    State:   Waiting
    Reason:  ImagePullBackOff
    Image:   gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1
Events:
  Warning  Failed  kubelet
    Failed to pull image: unexpected status code 403 Forbidden

Root Cause

The kube-rbac-proxy sidecar inside kserve-controller-manager was pulling from gcr.io/kubebuilder/ — a registry that restricted access in late 2025. The manager container was healthy but because the sidecar was not running, the webhook server had no valid certificate endpoint. Result: every InferenceService apply or update was blocked cluster-wide.

Fix

Remove the sidecar via Kustomize strategic merge patch:

# infrastructure/serving-stack/patches/
#   kserve-controller-kube-rbac-proxy-image.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kserve-controller-manager
  namespace: kserve
spec:
  template:
    spec:
      containers:
        - name: kube-rbac-proxy
          $patch: delete

Verify webhook endpoints are restored after re-sync:

$ kubectl -n kserve get endpoints kserve-webhook-server-service
NAME                           ENDPOINTS          AGE
kserve-webhook-server-service  10.42.0.23:9443    45s

Lesson

When webhook endpoints are missing, your app YAML is never the real problem. Diagnose the controller first. An external registry access change can silently kill your entire admission layer cluster-wide with no obvious error in the app itself.

Failure Mode 2: CRD Deleted by a Misapplied Patch — All Endpoints Gone Instantly

Time lost: 4 minutes recovery | Impact: SEV-1 equivalent — all InferenceServices deleted

Symptom

All InferenceService objects disappeared silently:

$ kubectl -n default get inferenceservices
No resources found in default namespace.

$ kubectl -n argocd get application demo-iris-2
NAME          SYNC STATUS   HEALTH STATUS
demo-iris-2   OutOfSync      Missing

Root Cause

A Kustomize patch file named remove-inferenceservice-crd.yaml was mistakenly applied directly with kubectl apply -f instead of being used as a build-time patch inside kustomization.yaml. The file contained a $patch: delete directive:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: inferenceservices.serving.kserve.io
$patch: delete

When applied directly, it deleted the actual CRD from Kubernetes. When a CRD is deleted, Kubernetes immediately garbage-collects every custom resource of that type. Every InferenceService was gone within seconds.

Fix

Restore the CRD immediately:

kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.12.1/kserve.yaml

kubectl wait crd/inferenceservices.serving.kserve.io \
  --for=condition=Established --timeout=60s

kubectl -n argocd patch application demo-iris-2 \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

Lesson

$patch: delete in a Kustomize file is a build-time instruction — it tells kustomize build to omit that resource from output. It must never be applied directly with kubectl apply -f. Ambiguous filenames like remove-inferenceservice-crd.yaml are dangerous footguns. In a production cluster with 50 deployed models this would be a full SEV-1.

⚠️ Rule: Any file containing $patch: delete must only ever be referenced inside a kustomization.yaml patches block, never applied directly.

Failure Mode 3: Permanent OutOfSync Due to Label Key Mismatch

Time lost: 2 weeks undetected | Impact: CI was green while policy enforcement was silently broken

Symptom

A PR is merged, ArgoCD syncs, but the InferenceService stays OutOfSync/Degraded:

$ kubectl -n argocd get application demo-iris-2
NAME          SYNC STATUS   HEALTH STATUS
demo-iris-2   OutOfSync      Degraded

Kyverno denies the resource at admission:

Error from server: error when creating "STDIN":
  admission webhook "clusterpolice.kyverno.svc" denied the request:
  resource InferenceService/default/test-model was blocked due to the following policies
  require-standard-labels-inferenceservice:
    check-owner-and-cost-center-on-isvc: 'validation error:
    InferenceService resources must set metadata.labels.owner and
    metadata.labels.cost-center.
    rule check-owner-and-cost-center-on-isvc failed at path
    /metadata/labels/cost-center/'

But the label is present in the manifest:

$ kubectl -n default get inferenceservice demo-iris-2 \
    -o jsonpath='{.metadata.labels}' | python3 -m json.tool
{
    "owner": "platform-team",
    "costCenter": "ai-platform"
}

Root Cause

costCenter (camelCase) and cost-center (kebab-case) are completely different Kubernetes label keys. The Backstage template skeleton was generating costCenter. The Kyverno policy required cost-center. CI passed because CI used the same manifest that would pass — the mismatch only surfaced at admission time.

Additionally, kyverno-cli apply exits with code 0 even when policy violations are found. CI was checking $? rather than ${PIPESTATUS[0]}, so the CI step appeared green while enforcement was completely broken for two weeks.

Fix

Standardize on kebab-case throughout (Kubernetes convention):

# Backstage template skeleton
# apps/${{ values.name }}/inference-service.yaml
labels:
  owner: platform-team
  cost-center: ai-platform   # was: costCenter

Fix the CI Kyverno check to catch actual violations:

set +e
docker run --rm -v "$PWD:/work" -w /work ghcr.io/kyverno/kyverno-cli:v1.12.5 \
  apply infrastructure/kyverno/policies/*.yaml \
  --resource "${app_files[@]}" \
  2>&1 | tee /tmp/kyverno-output.txt
kyverno_exit="${PIPESTATUS[0]}"
set -e

if [ "${kyverno_exit}" -ne 0 ] \
    || grep -qE "^FAIL" /tmp/kyverno-output.txt \
    || grep -qE "fail: [1-9][0-9]*" /tmp/kyverno-output.txt; then
  echo "Kyverno policy violations detected. Failing CI."
  exit 1
fi

Lesson

$? captures the exit code of tee, not kyverno. ${PIPESTATUS[0]} captures kyverno's actual exit code. "Guardrails exist" and "guardrails enforce" are different states. The most dangerous failure mode for a policy system is silent false positives — everything looks green while nothing is actually being enforced.

Failure Mode 4: Kyverno Install Breaks ArgoCD Reconciliation Loop

Time lost: 2–5 minutes per cluster | Impact: All ArgoCD apps enter Unknown state

Symptom

After adding Kyverno to the platform, previously healthy apps enter Unknown state:

$ kubectl -n argocd get applications
NAME                       SYNC STATUS   HEALTH STATUS
neuroscale-infrastructure  Synced         Healthy
serving-stack              Unknown        Unknown    # was Healthy 10 minutes ago
policy-guardrails          Synced         Healthy

$ kubectl -n argocd describe application serving-stack
...
Message: rpc error: code = Unavailable desc = connection refused

Root Cause

Kyverno installs its own ValidatingWebhookConfiguration and MutatingWebhookConfiguration during install. While Kyverno is initializing, the webhook configurations are registered but point to endpoints that do not exist yet. During this window, any kubectl apply operation — including ArgoCD's sync reconciliation loop — times out waiting for a response from a not-yet-running webhook server. This cascades into the ArgoCD repo-server losing its connection.

Fix

Add a Kyverno webhookAnnotations ConfigMap patch to suppress automatic webhook registration during the initialization window:

# infrastructure/kyverno/kustomization.yaml
patches:
  - target:
      kind: ConfigMap
      name: kyverno
    patch: |-
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: kyverno
        namespace: kyverno
      data:
        webhookAnnotations: "{}"

After Kyverno reaches Running state, force a hard refresh:

kubectl -n argocd patch application serving-stack \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

Lesson

Adding a policy engine to an existing cluster disrupts all other ArgoCD-managed applications during the install window. In production, this requires a maintenance window or a canary install strategy. Kyverno must be fully healthy before any other component syncs.

Failure Mode 5: Stale Admission Webhook Blocks All Resource Creation

Time lost: 30+ minutes | Impact: All Deployments in the namespace silently blocked

Symptom

After fixing the repo-server, apps sync but Deployments never appear:

$ kubectl get applications -n argocd
NAME                       SYNC STATUS   HEALTH STATUS
neuroscale-infrastructure  Synced         Healthy
test-app                   Synced         Progressing   # stuck

$ kubectl get deploy -n default
No resources found in default namespace.

ArgoCD shows the Deployment as "synced" but it does not exist — a contradiction. Checking conditions:

$ kubectl -n argocd get application test-app -o yaml | grep -A 20 conditions
  conditions:
  - message: 'Failed sync attempt: one or more objects failed to apply,
      reason: Internal error occurred: failed calling webhook
      "validate.nginx.ingress.kubernetes.io":
      failed to call webhook: Post
      "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/...":
      dial tcp 10.96.x.x:443: connect: connection refused'
    type: SyncError

Root Cause

A ValidatingWebhookConfiguration from a previous cluster experiment was still registered but pointing to a service that no longer existed. Kubernetes admission webhooks are cluster-scoped. The stale ingress-nginx webhook was intercepting every resource creation attempt and failing them — the error only appears in ArgoCD events, not on the Deployment itself.

Fix

# Discover stale webhooks
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Delete the stale one
kubectl delete validatingwebhookconfiguration ingress-nginx-admission

# Force ArgoCD to retry
kubectl -n argocd patch application test-app \
  --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

After deletion:

$ kubectl get deploy -n default
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
nginx-test    1/1     1            1           23s

Lesson

A stale webhook from a previous workload silently blocks all resource creation in the affected namespace for hours without any obvious error message. The admission error only appears in ArgoCD events logs, not on the resource itself. Always check for stale webhooks before blaming manifests.

The Triage Sequence That Saves Hours

When a KServe app is failing in ArgoCD, run this exact order before touching any manifest:

# 1. Environment gate — if this fails, stop and fix environment first
kubectl get nodes
kubectl -n argocd get applications

# 2. Control-plane health
kubectl -n kserve get deploy,pods,svc,endpoints
kubectl get crd | grep serving.kserve.io

# 3. Controller logs
kubectl -n kserve logs deploy/kserve-controller-manager --tail=100

# 4. Webhook availability
kubectl -n kserve get endpoints kserve-webhook-server-service

# 5. Stale webhooks
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# 6. App-level sync error detail
kubectl -n argocd get application <app-name> -o yaml | grep -A 20 conditions

Only after every step above passes should you edit app manifests.

Why This Matters for Platform Teams

A platform is credible when it supports both:

Self-service delivery — the Golden Path works
Self-service recovery — failures are understandable and fixable without a platform expert

Most teams build the first and postpone the second. That creates operational debt fast.

The fix is not more dashboards. It is better failure-model documentation, tighter GitOps guardrails, and the discipline to document what breaks — not just what works.

A platform is not "done" when the happy path works. It's done when the failure path is understandable and recoverable.

What I Would Improve Next

Pre-merge CI assertions for probe and resource fields in rendered manifests
Explicit dependency ordering using ArgoCD sync waves to prevent Kyverno install disruption
Conformance checks for Helm dependency values nesting to catch silently ignored overrides
Policy test fixtures that verify both pass and fail cases in CI

Why Your KServe InferenceService Won't Become Ready: Four Production Failures and Fixes

Sodiq Jimoh — Mon, 30 Mar 2026 03:37:24 +0000

A practitioner's account of the errors the KServe getting-started documentation doesn't tell you about — with exact terminal output, root causes, and working Kustomize patches.

This article documents four production failures I encountered while deploying KServe on a local k3d cluster as part of building NeuroScale — a self-service AI inference platform. None of these failures appear in the official KServe getting-started documentation. If you are deploying KServe without Istio, this will save you several hours of debugging.

What I Was Building

NeuroScale is a self-service AI inference platform on Kubernetes. The goal was simple: one InferenceService named sklearn-iris reaches Ready=True and responds to a prediction request.

The install had to be GitOps-managed via ArgoCD — not "I ran some scripts." Getting there took two days and four distinct failures. Here is every one of them.

Stack: k3d (local Kubernetes) · KServe 0.12.1 · ArgoCD · Kourier (no Istio) · Knative Serving

📝 Author's Note: This article was originally documented in the NeuroScale platform repository.
File: docs/REALITY_CHECK_MILESTONE_2_KSERVE_SERVING.md
Repo: github.com/sodiq-code/neuroscale-platform

Failure 1: KServe InferenceService Stuck Not Ready — Istio vs Kourier Ingress Mismatch Causes ReconcileError Loop

Time lost: ~3 hours

Symptom

After applying the KServe installation via ArgoCD (serving-stack app), the InferenceService was created but never became Ready:

$ kubectl -n default get inferenceservice sklearn-iris
NAME           URL   READY   PREV   LATEST   AGE
sklearn-iris         False          100      8m

# READY=False with no URL = KServe controller did not complete ingress setup.
# No Knative Route was created. No external URL was assigned.

Digging In

$ kubectl -n default describe inferenceservice sklearn-iris
...
Status:
  Conditions:
    Message: Failed to reconcile ingress
    Reason:  ReconcileError
    Status:  False
    Type:    IngressReady

$ kubectl -n kserve logs deploy/kserve-controller-manager --tail=50
...
ERROR controller.inferenceservice Failed to reconcile ingress
  {"error": "virtual service not found: sklearn-iris.default.svc.cluster.local"}

The error referenced a virtual service — that is an Istio concept. But we were running Kourier. The KServe controller was attempting to create an Istio VirtualService in a cluster that had no Istio control plane.

Root Cause: Default KServe Ingress Mode Assumes Istio

KServe's default inferenceservice-config ConfigMap expects Istio as the ingress provider. It sets ingressClassName: istio and the key disableIstioVirtualHost defaults to false. When Istio is absent, the controller enters an error loop trying to create resources that will never exist.

Setting disableIstioVirtualHost: true tells KServe to skip Istio and fall back to Knative route objects that Kourier can handle.

Why Kourier instead of Istio: Istio adds ~1 GB of memory overhead. On a local k3d cluster shared with Docker Desktop, Backstage, and the KServe controller, that exhausts available RAM. Kourier's entire footprint is under 200 MB.

The Fix: ConfigMap Patch in serving-stack

# infrastructure/serving-stack/patches/inferenceservice-config-ingress.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: inferenceservice-config
  namespace: kserve
data:
  ingress: |-
    {
      "ingressGateway": "knative-serving/knative-ingress-gateway",
      "ingressDomain": "example.com",
      "ingressClassName": "istio",
      "urlScheme": "http",
      "disableIstioVirtualHost": true,
      "disableIngressCreation": false
    }

After this patch was applied and the KServe controller restarted:

$ kubectl -n default get inferenceservice sklearn-iris
NAME           URL                                       READY   AGE
sklearn-iris   http://sklearn-iris.default.example.com   True    2m

Business impact: This failure cost approximately 3 hours. The KServe documentation does not prominently state that the default configuration requires Istio. The error message "virtual service not found" is Istio-specific vocabulary that only makes sense if you already know Istio is the default — a classic undocumented assumption in infrastructure tooling.

Failure 2: ArgoCD Serving-Stack Sync Fails — Duplicate Knative CRD Exceeds 256 KB Annotation Size Limit

Time lost: ~30 minutes

Symptom

$ kubectl -n argocd get application serving-stack
NAME            SYNC STATUS   HEALTH STATUS
serving-stack   OutOfSync     Degraded

$ kubectl -n argocd describe application serving-stack
...
Message: CustomResourceDefinition "services.serving.knative.dev"
  is invalid: metadata.annotations:
  Too long: may not be more than 262144 bytes

Root Cause

ArgoCD stores kubectl.kubernetes.io/last-applied-configuration in the annotation. For large CRDs, this annotation plus the apply payload exceeds Kubernetes' 256 KB annotation size limit. The Knative CRD is approximately 400 KB as a YAML object.

A rendering overlap compounded the issue: the kserve.yaml bundle already includes its own version of the Knative Serving CRDs, and we were also referencing serving-core.yaml directly. This created two attempts to manage the same CRDs, causing comparison instability.

Fix

# infrastructure/serving-stack/kustomization.yaml

# 1. Use server-side apply to bypass the annotation size limit
commonAnnotations:
  argocd.argoproj.io/sync-options: ServerSideApply=true

# 2. Ignore runtime-mutated fields on Knative CRDs
#    (In ArgoCD Application spec)
ignoreDifferences:
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    name: services.serving.knative.dev
    jsonPointers:
      - /spec/preserveUnknownFields

Business impact: ArgoCD's error says "Too long" but does not tell you which annotation or why it got too long. Debugging requires knowing ArgoCD's internal server-side apply mechanism.

Failure 3: kube-rbac-proxy ImagePullBackOff Blocks KServe Admission Webhook — gcr.io Access Restriction

Time lost: ~1 hour | Cluster-wide impact

Symptom

$ kubectl -n argocd describe application ai-model-alpha
...
Message: admission webhook
  "inferenceservice.kserve-webhook-server.validator.webhook"
  denied the request: no endpoints available for
  service "kserve-webhook-server-service"

$ kubectl -n kserve get pods
NAME                            READY   STATUS
kserve-controller-manager-xxx   1/2     Running   # only 1 of 2 ready

$ kubectl -n kserve describe pod kserve-controller-manager-xxx
  kube-rbac-proxy:
    State:   Waiting
    Reason:  ImagePullBackOff
    Image:   gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1
Events:
  Warning  Failed  kubelet
    Failed to pull image: unexpected status code 403 Forbidden

Root Cause

KServe 0.12.1's kserve-controller-manager Deployment includes a kube-rbac-proxy sidecar from gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1. Google Container Registry restricted access to kubebuilder images in late 2025.

The manager container itself was healthy (1 of 2 ready). But without the sidecar, the webhook server certificate was not being served, so the admission webhook had no healthy endpoints. The alternative registry.k8s.io/kube-rbac-proxy:v0.13.1 did not exist at the new location either.

Fix: Remove the Sidecar via Kustomize Strategic Merge Patch

# infrastructure/serving-stack/patches/
#   kserve-controller-kube-rbac-proxy-image.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kserve-controller-manager
  namespace: kserve
spec:
  template:
    spec:
      containers:
        - name: kube-rbac-proxy
          $patch: delete

After this patch and a re-sync:

$ kubectl -n kserve get pods
NAME                            READY   STATUS
kserve-controller-manager-yyy   1/1     Running   # fixed

$ kubectl -n kserve get endpoints kserve-webhook-server-service
NAME                            ENDPOINTS          AGE
kserve-webhook-server-service   10.42.0.23:9443    45s

Known tradeoff: Removing kube-rbac-proxy disables the Prometheus metrics proxy endpoint for the KServe controller. In production, source a verified replacement image from an accessible registry before deploying.

Business impact: An external registry access change cascaded into a complete admission webhook outage. Any InferenceService creation or update was blocked cluster-wide while the sidecar was failing. This class of failure has no good solution without upstream monitoring of your image dependencies.

Failure 4: Inference Request Returns HTTP 405 — IngressDomain Placeholder Resolves to Public Internet

Time lost: ~1 hour

Symptom

$ kubectl -n default get inferenceservice sklearn-iris \
    -o jsonpath='{.status.url}'
http://sklearn-iris.default.example.com

$ curl -sS \
  -H "Content-Type: application/json" \
  -d '{"instances":[[5.1,3.5,1.4,0.2]]}' \
  http://sklearn-iris.default.example.com/v1/models/sklearn-iris:predict
<html><head><title>405 Not Allowed</title></head>...

# The request hit the public example.com server, not our Kourier gateway.

Root Cause

The ingressDomain in the KServe ConfigMap was set to example.com — a literal placeholder. The generated URL resolves publicly to Cloudflare/IANA servers, not the local cluster.

Additionally, Kourier routes by Host header, not by IP. Just port-forwarding Kourier and hitting 127.0.0.1 does not work without the correct Host header.

Fix: Direct Predictor Pod Port-Forward

Bypass Knative routing and Kourier entirely for local verification:

# Step 1: Get the predictor pod name
kubectl -n default get pods \
  -l serving.knative.dev/revision=sklearn-iris-predictor-00001

# Step 2: Port-forward directly to the predictor container
kubectl -n default port-forward \
  pod/sklearn-iris-predictor-00001-deployment-<hash> 18080:8080

# Step 3: Predict (no Host header, no Kourier, no DNS needed)
curl -sS \
  -H "Content-Type: application/json" \
  -d '{"instances":[[5.1,3.5,1.4,0.2],[6.2,3.4,5.4,2.3]]}' \
  http://127.0.0.1:18080/v1/models/sklearn-iris:predict

{"predictions":[0,2]}

For the full Kourier routing path, always pass the Host header:

kubectl -n kourier-system port-forward svc/kourier 18080:80

curl -sS \
  -H 'Host: sklearn-iris-predictor.default.127.0.0.1.sslip.io' \
  -H "Content-Type: application/json" \
  -d '{"instances":[[5.1,3.5,1.4,0.2]]}' \
  http://127.0.0.1:18080/v1/models/sklearn-iris:predict

Business impact: False-negative inference verification. A healthy endpoint looked broken because the test URL resolved to the wrong server. Always verify the complete network path — DNS resolution, ingress routing, pod health — as separate steps rather than assuming a single curl test is conclusive.

What This Proves After the Failures

After working through the above failures, the inference baseline worked:

$ kubectl -n default get inferenceservice sklearn-iris
NAME           URL                                       READY   AGE
sklearn-iris   http://sklearn-iris.default.example.com   True    45m

$ curl -sS \
  -H "Content-Type: application/json" \
  -d '{"instances":[[5.1,3.5,1.4,0.2],[6.2,3.4,5.4,2.3]]}' \
  http://127.0.0.1:18080/v1/models/sklearn-iris:predict

{"predictions":[0,2]}

The Istio/Kourier mismatch is the canonical example of why "default configuration" is dangerous in complex systems. KServe's default assumes a specific network topology that is not disclosed in the getting-started docs. Recognizing this class of failure — configuration that works in the tool author's environment but not yours — is a senior platform engineering competency.

What This Setup Does NOT Solve (Known Tradeoffs)

No Istio service mesh: No mTLS between services, no advanced traffic management. Acceptable for local dev; requires a replacement security layer in production.
kube-rbac-proxy removed: Prometheus metrics from the KServe controller are unavailable. Re-add this sidecar from a working registry before any production deployment.
Port-forward for inference: The Host-header workaround is local only. Cloud deployment requires a real ingress with DNS and TLS. On EKS, swap Kourier for an ALB and set ingressDomain to your real domain. See the Cloud Promotion Guide in the repository.

Debugging Commands Reference

Run these in order when an InferenceService will not become Ready.

1 — InferenceService Conditions

kubectl -n default describe inferenceservice sklearn-iris
kubectl -n kserve logs deploy/kserve-controller-manager --tail=50
kubectl -n kserve logs deploy/kserve-controller-manager -c manager --tail=50

2 — Webhook Endpoint Availability

kubectl -n kserve get endpoints kserve-webhook-server-service
kubectl -n kserve describe endpoints kserve-webhook-server-service
kubectl -n default get ksvc
kubectl -n default get route

3 — ConfigMap and Pod Status

kubectl -n kserve get configmap inferenceservice-config -o yaml
kubectl -n kserve get pods -o wide
kubectl -n kserve describe pod <pod-name>

The One Thing to Remember

KServe's default configuration assumes Istio is installed. This assumption is not prominently stated in the getting-started documentation. Every engineer running KServe on k3d, k3s, GKE Autopilot, or any non-Istio cluster will hit ReconcileError and see error messages referencing "virtual services" — an Istio concept — with no obvious resolution path.

The fix is one ConfigMap patch. It takes 30 seconds to apply. Finding it took three hours.

The kube-rbac-proxy 403 from gcr.io is an external dependency failure that silently kills your admission webhook cluster-wide. The $patch: delete Kustomize strategy is the fastest recovery path when no alternative registry image is available.

Full platform source — all six Reality Check documents, Backstage Golden Path, Kyverno policy enforcement, cost attribution, and a Cloud Promotion Guide to EKS/GKE: Check out the full NeuroScale repo here.

Forem: Sodiq Jimoh

Deploying Backstage on Kubernetes with the Helm Chart: The Infrastructure-First Guide

The one thing you must understand before installing the Helm chart

The values hierarchy that breaks everything silently

Required configuration keys for the demo image

Probe timings: the demo image starts slowly

Authentication: local dev vs production

Catalog configuration: registering templates

GitHub integration: the token secret

A working minimal values.yaml for local development

Deploying and verifying

The production values profile

Diagnostic command reference

What this guide does not cover

See Also

9 Failures That Hit Me Building a Backstage Golden Path for KServe — Every Error, Every Fix

What I was trying to build

Failure 1: Template Not Visible in Catalog — Silent Rejection With No UI Error

Symptom

Digging In

Root Cause

Fix

Lesson

Failure 2: Scaffolder /create Page Loads Blank — 401 on Actions API

Symptom

Root Cause

Fix

Lesson

Failure 3: Frontend Crashes With Blank White Screen — Missing Required Config Key

Symptom

Root Cause

Fix

Lesson

Failure 4: Backstage CrashLoopBackOff — Helm Dependency Values Mis-Nesting

Symptom

Root Cause

Fix

Lesson

Failure 5: PR Creation Fails — GitHub Token Secret Contains Placeholder Value

Symptom

Root Cause

Fix

Lesson

Failure 6: PR Merged But ArgoCD Stays OutOfSync — Fix Not Committed to Git

Symptom

Root Cause

Fix

Lesson

Failure 7: Inference Endpoint Returns HTTP 307 Redirect — Traefik Intercepts Before Kourier

Symptom

Root Cause

Fix

Lesson

Failure 8: Catalog Ingestion Silently Rejects Template After Values Update

Symptom

Root Cause

Fix

Lesson

Failure 9: Scaffolder Task Hangs Then Fails — Port-Forward Session Died Mid-Task

Symptom

Root Cause

Fix

Lesson

What the Golden Path Actually Proves After Nine Failures

Debugging Commands Reference

The Pattern Across All Nine Failures

See Also

Beyond InferenceService Readiness: 5 GitOps Failure Modes That Break KServe Deployments

The platform context

Failure Mode 1: Webhook Has No Endpoints — Sync Fails Cluster-Wide

Symptom

Root Cause

Fix

Lesson

Failure Mode 2: CRD Deleted by a Misapplied Patch — All Endpoints Gone Instantly

Symptom

Root Cause

Fix

Lesson

Failure Mode 3: Permanent OutOfSync Due to Label Key Mismatch