Forem: Alexis Roberson

OpenTelemetry for LLM Applications: A Practical Guide with LaunchDarkly and Langfuse

Alexis Roberson — Wed, 04 Mar 2026 03:11:34 +0000

Originally published in the LaunchDarkly Docs

LLM applications have a telemetry problem. Unlike traditional software where you can trace a bug to a specific line of code or a failed API call, LLM failures are a bit more nuanced. A response that's slightly off, a prompt that worked yesterday but not today, or a model swap can quietly degrade your user experience. OpenTelemetry gives you a structured way to pull back the curtain by capturing token usage, model metadata, latency, and agent responses so you truly know what's happening inside your application.

This tutorial walks you through instrumenting a real LLM application with OTel spans, capturing the right attributes, and fanning out those traces simultaneously to Langfuse and LaunchDarkly's Guarded Releases. Both are LLM observability tools, but they give you different lenses on the same trace data. Langfuse is purpose-built for prompt debugging and cost analysis — surfacing prompt content, completions, and per-agent token usage.

LaunchDarkly connects that same trace data to the specific model variant that was active during a request, giving you flag-correlated observability with automated rollback if a variant starts degrading your users' experience. One OTel collector, two complementary views, no custom integrations required.

Guarded releases are LaunchDarkly's observability solution that encompasses application performance thresholds, release auto-remediation, and release monitoring, along with error monitoring and session replay.

The WorkLunch App

In order to see the full process of instrumenting an LLM application, I added a new feature in an app called WorkLunch where users were able to create/join office communities and swap lunches based on preference. Now they're also able to improve the description field of their lunch post to make it more appealing to potential swappers and receive recommendations for compatible swaps.

So in the initial description you may write, "Grilled cheese sandwich", then click the AI Suggest button. The app replaces it with, "Golden, buttery grilled cheese with perfectly melted cheese sandwiched between crispy white bread. This comfort food classic is grilled to perfection with a satisfying crunch on the outside and gooey, cheesy goodness on the inside. Simple, delicious, and guaranteed to hit the spot!"

Now which lunch post are you more than likely to click on?

This subtle addition takes the app from a fun, simple lunch swap experience to a viable LLM application that still requires the same visibility and observability of traditional systems. OpenTelemetry allows you to extract the necessary data like token count, model name, agent responses, etc in order to properly debug system failures.

Multi-Agent Architecture

The WorkLunch backend uses 3 agents to rewrite the lunch post description and find good lunch swaps.

The orchestrator coordinates the other two agents. It receives the user's request and the model type, calls the description agent first, then passes the generated description into the match agent. It acts as the parent span that ties the whole chain together.
The description agent takes the user's sparse lunch post input and calls Claude to generate an appealing 2-3 sentence description.
The match agent takes the user's lunch post (including the description just generated) plus a list of other active posts in the community, and uses AI to suggest 2-3 posts that would make good swaps.

These features are controlled by two feature flags, one for enabling the AI suggest feature and the other to control which model version the app uses. Every layer gets its own OTel span, creating a trace tree that shows the full request lifecycle.

Prerequisites

Before you start, you'll need the following installed locally and accounts set up with the services below.

Docker Desktop
Node.js / Npm
An Anthropic account (or openai)
A LaunchDarkly account (sdk key and access token)
A Langfuse account (public and secret key)
A Supabase account (database configuration)

Environment variables

Once you're all setup, clone the WorkLunch repo. Copy the example .env file and fill in your values:

cp .env.example .env

Your .env should contain:

# .env

# Supabase (required for the app)
EXPO_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
EXPO_PUBLIC_SUPABASE_ANON_KEY=your-anon-key

# LaunchDarkly client-side (required for feature flags in the frontend)
EXPO_PUBLIC_LAUNCHDARKLY_SDK_KEY=mob-your-mobile-key
EXPO_PUBLIC_LAUNCHDARKLY_CLIENT_SIDE_ID=your-client-side-id

# AI Backend URL (where docker compose runs the Python backend)
EXPO_PUBLIC_AI_BACKEND_URL=http://localhost:8000

# --- Docker Compose vars (used by the backend + otel-collector) ---

# Anthropic API key for Claude
ANTHROPIC_API_KEY=sk-ant-your-key-here

# LaunchDarkly server-side SDK key (starts with sdk-, NOT mob-)
LD_SDK_KEY=sdk-your-key-here

# Langfuse auth — Base64 of "public_key:secret_key" (keep on one line)
LANGFUSE_AUTH_HEADER=your-base64-encoded-string

Supabase setup

Create a new Supabase project and grab your Project URL and Anon key from Dashboard → Settings → API
Run the migration files in supabase/migrations/ to create the database schema. Execute them in order in the Supabase Dashboard → SQL Editor:

supabase/migrations/

20240101000000_initial_schema.sql        ← tables: profiles, spaces, posts, proposals, messages, trades
20240101000001_rls_policies.sql          ← row-level security policies
20240101000002_storage_setup.sql         ← storage bucket for post photos
20240101000003_disable_email_confirmation.sql  ← simplifies local dev auth
20240205000000_fix_space_memberships_rls_recursion.sql
20240205000001_spaces_delete_policy.sql
20240206000000_space_creator_as_admin.sql
20240206100000_delete_space_rpc.sql

Be sure to run each file in order as later migrations depend on tables and policies from earlier ones.

LaunchDarkly setup

Create two feature flags in your new LaunchDarkly worklunch project:

ai-suggest-enabled — Boolean flag, client-side. Gates visibility of the AI Suggest button in the frontend. Set it to true for users you want to test with.
llm-model-variant — String flag, server-side. Controls which Claude model the backend uses. Set the default value to claude-sonnet-4-20250514. Add a variation for claude-haiku-4-5-20251001 if you want to experiment with a faster/cheaper model.

Langfuse setup

Create a new project in Langfuse (note whether your project URL starts with us.cloud.langfuse.com or cloud.langfuse.com — this determines your region)
Go to Project Settings → API Keys and create a new key pair
Generate your Base64 auth header and place it inside your .env file:

echo -n "pk-lf-your-public-key:sk-lf-your-secret-key" | base64

Quick Start

Once your .env is configured:

# Install frontend dependencies
npm install

# Start the OTel Collector + Python backend
docker compose up --build

# In a separate terminal, start the Expo dev server
npm run web

Verify traces are flowing by checking the collector logs:

docker compose logs -f otel-collector

You should see spans with gen_ai.* attributes and feature_flag events printed by the debug exporter.

Now, let's take a look at how each agent is instrumented to send spans to LaunchDarkly.

Step 1: Instrument your LLM application

Initialize the Tracer and Application

The FastAPI app sets up OTel, LaunchDarkly, CORS, and auto-instrumentation in a single lifespan handler:

# backend/app/main.py
from contextlib import asynccontextmanager

import ldclient
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from app.config import settings
from app.routers.suggest import router as suggest_router

def setup_otel() -> None:
    """Configure OpenTelemetry with OTLP gRPC exporter."""
    provider = TracerProvider()
    provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=settings.OTEL_EXPORTER_ENDPOINT, insecure=True)
        )
    )
    trace.set_tracer_provider(provider)

def setup_launchdarkly() -> None:
    """Initialize LaunchDarkly server SDK."""
    config = ldclient.Config(settings.LD_SDK_KEY)
    ldclient.set_config(config)

@asynccontextmanager
async def lifespan(app: FastAPI):
    setup_otel()
    setup_launchdarkly()
    yield
    ldclient.get().close()
    provider = trace.get_tracer_provider()
    if hasattr(provider, "shutdown"):
        provider.shutdown()

app = FastAPI(title="WorkLunch AI Backend", lifespan=lifespan)

# CORS
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Instrument FastAPI with OTel
FastAPIInstrumentor.instrument_app(app)

# Routes
app.include_router(suggest_router, prefix="/api/v1")

@app.get("/health")
async def health():
    return {"status": "ok"}

The Route: Flag evaluation + feature flag span event

The FastAPI route is where the LaunchDarkly flag gets evaluated. The feature_flag span event on this span is what LaunchDarkly's observability layer looks for when correlating traces with flag evaluations.

# backend/app/routers/suggest.py
import ldclient
from fastapi import APIRouter
from opentelemetry import trace

from app.agents import orchestrator
from app.models import SuggestRequest, SuggestResponse

router = APIRouter()
tracer = trace.get_tracer("worklunch.routers.suggest")

DEFAULT_MODEL = "claude-sonnet-4-20250514"


@router.post("/suggest", response_model=SuggestResponse)
async def suggest(request: SuggestRequest) -> SuggestResponse:
    with tracer.start_as_current_span("suggest.endpoint") as span:
        # Evaluate the model variant flag
        ld_client = ldclient.get()
        context = ldclient.Context.builder("worklunch-backend").kind("service").build()
        model = ld_client.variation("llm-model-variant", context, DEFAULT_MODEL)

        # Emit the feature_flag span event — this is what LD correlates with
        span.add_event(
            "feature_flag",
            {
                "feature_flag.key": "llm-model-variant",
                "feature_flag.provider.name": "LaunchDarkly",
                "feature_flag.variant": str(model),
            },
        )
        span.set_attribute("gen_ai.request.model", model)

        # The flag-controlled model flows into the orchestrator
        description, matched_posts = await orchestrator.run(request, model)

    return SuggestResponse(
        suggested_description=description,
        matched_posts=matched_posts,
    )

The Orchestrator: Parent span for the agent chain

The orchestrator creates a parent span and calls each sub-agent sequentially. Because the sub-agent spans are created while the orchestrator span is active, OTel automatically nests them as children.

# backend/app/agents/orchestrator.py
from opentelemetry import trace

from app.agents.description_agent import generate_description
from app.agents.match_agent import find_matches
from app.models import MatchedPost, SuggestRequest

tracer = trace.get_tracer("worklunch.orchestrator")

async def run(
    request: SuggestRequest, model: str
) -> tuple[str, list[MatchedPost]]:
    with tracer.start_as_current_span("orchestrator.run") as span:
        span.set_attribute("orchestrator.model", model)
        span.set_attribute("orchestrator.title", request.title)
        span.set_attribute("orchestrator.active_posts_count", len(request.active_posts))

        # Step 1: Generate description
        description = await generate_description(request, model)

        # Step 2: Find matches using the generated description
        matched_posts = await find_matches(
            title=request.title,
            description=description,
            category=request.category,
            dietary_preferences=request.dietary_preferences,
            active_posts=request.active_posts,
            model=model,
        )

        span.set_attribute("orchestrator.matches_found", len(matched_posts))

    return description, matched_posts

The Description Agent: LLM Call with genAI semantic conventions

This is where the OTel GenAI Semantic Conventions come in. The conventions define a standard schema for LLM spans — gen_ai.system, gen_ai.request.model, gen_ai.usage.*, and prompt/completion content as span events.

# backend/app/agents/description_agent.py
import json

import anthropic
from opentelemetry import trace

from app.config import settings
from app.models import SuggestRequest

tracer = trace.get_tracer("worklunch.agents.description")

async def generate_description(request: SuggestRequest, model: str) -> str:
    client = anthropic.Anthropic(api_key=settings.ANTHROPIC_API_KEY)

    system_prompt = (
        "You are a helpful assistant that writes appealing, concise lunch descriptions "
        "for a lunch-swapping app. Given a title and optional details, write a 2-3 sentence "
        "description that makes the lunch sound appetizing and highlights what makes it special. "
        "Mention any dietary info naturally if provided. Keep it friendly and casual."
    )

    user_content_parts = [f"Lunch title: {request.title}"]
    if request.description:
        user_content_parts.append(f"Current description: {request.description}")
    if request.category:
        user_content_parts.append(f"Category: {request.category}")
    if request.dietary_preferences:
        user_content_parts.append(f"Dietary preferences: {request.dietary_preferences}")
    if request.allergies:
        user_content_parts.append(f"Allergies to note: {request.allergies}")

    user_content = "\n".join(user_content_parts)
    messages = [{"role": "user", "content": user_content}]

    with tracer.start_as_current_span("description_agent.generate") as span:
        # GenAI semantic conventions — provider and request attributes
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 256)
        span.set_attribute("gen_ai.request.temperature", 0.7)

        # Log prompt as a span event (keeps large payloads out of the attribute index)
        span.add_event(
            "gen_ai.content.prompt",
            {"gen_ai.prompt": json.dumps(messages)},
        )

        response = client.messages.create(
            model=model,
            max_tokens=256,
            temperature=0.7,
            system=system_prompt,
            messages=messages,
        )

        result = response.content[0].text

        # Response attributes — model identity, finish reason, token usage
        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute(
            "gen_ai.response.finish_reasons", [response.stop_reason or "end_turn"]
        )
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

        # Log completion as a span event
        span.add_event(
            "gen_ai.content.completion",
            {"gen_ai.completion": result},
        )

    return result

The Match Agent: Structured JSON output from an LLM

The match agent follows the same GenAI span pattern but with different parameters (lower temperature for more deterministic output, higher token budget for JSON) and post-processing to parse structured JSON from the LLM response.

# backend/app/agents/match_agent.py
import json

import anthropic
from opentelemetry import trace

from app.config import settings
from app.models import ActivePost, MatchedPost

tracer = trace.get_tracer("worklunch.agents.match")

async def find_matches(
    title: str,
    description: str,
    category: str | None,
    dietary_preferences: str | None,
    active_posts: list[ActivePost],
    model: str,
) -> list[MatchedPost]:
    if not active_posts:
        return []

    client = anthropic.Anthropic(api_key=settings.ANTHROPIC_API_KEY)

    system_prompt = (
        "You are a lunch-matching assistant. Given a user's lunch post and a list of "
        "active posts from other users, suggest 2-3 posts that would make good swaps. "
        "Consider complementary flavors, dietary compatibility, and variety. "
        "Respond with valid JSON only — an array of objects with keys: "
        '"post_id", "title", "reason". Keep reasons to one short sentence.'
    )

    posts_text = "\n".join(
        f"- ID: {p.id}, Title: {p.title}, Description: {p.description}, "
        f"Category: {p.category}, By: {p.user_name}"
        for p in active_posts
    )

    user_parts = [
        f"My lunch: {title}",
        f"Description: {description}",
    ]
    if category:
        user_parts.append(f"Category: {category}")
    if dietary_preferences:
        user_parts.append(f"My dietary preferences: {dietary_preferences}")
    user_parts.append(f"\nAvailable posts to match with:\n{posts_text}")

    user_content = "\n".join(user_parts)
    messages = [{"role": "user", "content": user_content}]

    with tracer.start_as_current_span("match_agent.find") as span:
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 512)
        span.set_attribute("gen_ai.request.temperature", 0.3)

        span.add_event(
            "gen_ai.content.prompt",
            {"gen_ai.prompt": json.dumps(messages)},
        )

        response = client.messages.create(
            model=model,
            max_tokens=512,
            temperature=0.3,
            system=system_prompt,
            messages=messages,
        )

        raw = response.content[0].text

        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute(
            "gen_ai.response.finish_reasons", [response.stop_reason or "end_turn"]
        )
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)

        span.add_event(
            "gen_ai.content.completion",
            {"gen_ai.completion": raw},
        )

    # Parse the JSON response
    try:
        cleaned = raw.strip()
        if cleaned.startswith("```

"):
            cleaned = cleaned.split("\n", 1)[1]
            cleaned = cleaned.rsplit("

```", 1)[0]
        matches_data = json.loads(cleaned)
        return [MatchedPost(**m) for m in matches_data[:3]]
    except (json.JSONDecodeError, KeyError, TypeError):
        return []

For each of these agents, Langfuse receives the full trace including prompt/completion content for debugging. LaunchDarkly receives the same trace and correlates the feature_flag event with the HTTP span for experimentation metrics.

Step 2: Configure the OTel collector

This is where the fan-out happens. The collector receives traces over OTLP and exports them to both backends simultaneously. Two pipelines from the same receiver is the key: you configure one receivers block and reference it in multiple pipelines — no duplication of ingestion, no changes needed in application code.

otel-collector-config.yaml

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

  # Stamp traces with the LD project identifier so the endpoint
  # knows which project they belong to
  resource/launchdarkly:
    attributes:
      - key: launchdarkly.project_id
        value: "${env:LD_SDK_KEY}"
        action: upsert

exporters:
  # Langfuse — LLM-specific traces with full prompt content
  otlphttp/langfuse:
    endpoint: https://us.cloud.langfuse.com/api/public/otel
    headers:
      Authorization: "Basic ${env:LANGFUSE_AUTH_HEADER}"

  # LaunchDarkly — flag-correlated observability
  # No auth header needed; identification is via the
  # launchdarkly.project_id resource attribute
  otlphttp/launchdarkly:
    endpoint: https://otel.observability.app.launchdarkly.com

  # Debug exporter for local development
  debug:
    verbosity: detailed

service:
  pipelines:
    # Pipeline 1: Full LLM traces to Langfuse (includes prompt content)
    traces/llm-observability:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/langfuse]

    # Pipeline 2: Flag-correlated traces to LaunchDarkly
    traces/feature-flags:
      receivers: [otlp]
      processors: [resource/launchdarkly, batch]
      exporters: [otlphttp/launchdarkly]

    # Pipeline 3: Debug output for development
    traces/debug:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Now if we rerun our application, we should see LaunchDarkly Traces capturing the Otel spans.

docker compose up --build
npm run web # in a separate terminal

How LaunchDarkly processes OTel traces

LaunchDarkly receives traces for logging and converts OTel span data into events for use with Experimentation and Guarded Rollouts. The process works like this:

Your application emits a span that covers an HTTP request (or LLM call). This span carries standard HTTP attributes: http.response.status_code, http.route, latency derived from span duration.
On that same span (or a parent span in the same trace), you've emitted a feature_flag span event with feature_flag.key and feature_flag.variant.
LaunchDarkly's collector ingests the trace and looks for HTTP spans that overlap with spans containing at least one feature_flag event. When it finds a match, it produces a metric event associating the flag variant with the observed latency and error rate (5xx status codes).
Those metric events flow into Experimentation, where they become the outcome metrics for your flag-controlled A/B test — for example, comparing claude-sonnet-4-20250514 vs claude-haiku-4-5-20251001 on p95 latency and error rate without writing a single line of custom metric instrumentation.

Every span in the agent chain is nested under a single trace. The collector fans out that trace to both backends simultaneously. Langfuse gets the full LLM details for prompt debugging and cost analysis. LaunchDarkly gets the flag-correlated signal it needs for automated rollout decisions.

Key attributes from gen_ai trace spans

Step 3: Trigger a guarded rollout

With traces flowing into LaunchDarkly and span events carrying your flag evaluations, you can now configure a Guarded Rollout that automatically rolls back the AI Suggest feature if token costs spike or response truncation increases as you increase the percentage of users who see it.

In the LaunchDarkly UI, navigate to your flag (ai-suggest-enabled), under Default rule click Edit and select Guarded Rollout.

You'll need to create two new custom metrics to attach to the guarded rollout. The first will is the AI tokens total rollout metric. This will measure cost-per-request as a gate for releasing the feature to a wider audience and alert if average tokens per request exceeds your baseline by more than 25%. And the second is AI completion truncated error metric, which will catch prompt truncation before users notice degraded output quality. This metric will halt the rollout if the rate climbs above your control baseline.

For ai.tokens.total:

Event kind: Custom
Event key: ai.tokens.total
What do you want to measure?: Value / Size (Numeric) — you're passing the actual token count as the magnitude
Metric name: AI tokens total

For ai.completion.truncated:

Event kind: Custom
Event key: ai.completion.truncated
What do you want to measure?: Occurrence (Binary) — you're tracking whether truncation happened at least once, not how many times
Metric name: AI completion truncated

Select the two newly created metrics.

Set the threshold to 25 percent for 1 week.

Click Save.

LaunchDarkly will now monitor both metrics against the ai-suggest-enabled flag and trigger an automatic rollback if either threshold is breached.

What You've Built

At this point you have a fully instrumented LLM application where every layer of the stack tells a story. The FastAPI route evaluates a LaunchDarkly flag and stamps the result onto the trace. The orchestrator creates a parent span that ties the entire agent chain together. Each agent makes a Claude API call and records exactly what was sent, what came back, and how many tokens it cost. The OTel Collector fans all of that out to two backends simultaneously without a single line of application code changing between them.

Langfuse gives you the LLM-specific view: prompt content, completions, token usage, and latency per agent so you can debug why a description came out wrong or whether the match agent is consistently burning more tokens than expected. LaunchDarkly gives you the experimentation view: which model variant was active during a given request, how latency and error rates compare between claude-sonnet-4-20250514 and claude-haiku, and the automated safety net to roll back if a new variant starts degrading your users' experience. Both tools are consuming the same trace. Neither required a custom integration.

Conclusion

LLM applications fail in ways that traditional monitoring wasn't designed to catch. OpenTelemetry gives you the standard schema to do that, and the collector architecture gives you the flexibility to route that signal wherever it's most useful.

If you're building anything with LLMs in production, start here. Instrument at the agent level, follow the GenAI semantic conventions, and build your observability pipeline before you need it.

The full source code for the WorkLunch app is available here. Clone it, swap in your API keys, and you'll have a working multi-agent trace pipeline running locally in under ten minutes.

Additional resources

Detection to Resolution: Real world debugging with rage clicks and session replay

Alexis Roberson — Wed, 04 Mar 2026 03:00:54 +0000

Originally published in the LaunchDarkly Docs

Part 3 of 3: Rage Click Detection with LaunchDarkly

Overview

In Part 1, we set up rage click detection using LaunchDarkly's Session Replay. In Part 2, we connected those frustration signals to guarded releases for automated rollback protection.

Now it's time to put it all together. In this final installment, we'll walk through real-world debugging scenarios using our WorkLunch app—a cross-platform application built with React Native and Expo where coworkers can swap lunches.

These scenarios demonstrate how the integrated system of feature flags, session replay, and guarded releases can transform the way you diagnose and fix production issues.

All code for this blog post can be found here.

The Debugging Workflow: An Observability Loop

Before diving into scenarios, let's understand the complete workflow we've built:

This workflow enables you to:

Detect frustration signals automatically as they happen.
Alert when thresholds are breached during rollouts.
Investigate with full session context, not just error logs.
Fix with confidence knowing the exact user experience.
Verify the fix works by monitoring the new rollout.

When a guarded release detects a spike in rage clicks, it automatically correlates the frustration with specific flag variations:

The alert tells you exactly which feature caused the problem: "Rage clicks increased 10x for users on join-community-redesign: true" It’s important to note that rage clicks aren’t the only measure for frustration signals around user experience.

When items are out of place, confusing or navigation is not consistent across mobile and web, it can cause users to rage scroll. Unlike rage clicks that can be measured and tracked, rage scrolls are a little more nuanced, which will result in relying heavily on session replay and drops and traffic to diagnose.

Just as llm observability tools monitor model outputs for hallucinations or drift, rage click detection monitors the human layer, catching UX failures that no server-side metric would reveal.

Using feature flags, session replay and guarded releases, let’s take a closer look at both scenarios to debug in a real-world worklunch app.

Scenario 1: Form Validation Frustration

The Setup

Your team ships a new "Create Community" button redesign and places the feature behind a feature flag called (inline-form-validation).

After the feature flag is toggled on, you notice users on this form are rage clicking the Create Community button. Your metrics show community creation conversions dropped 15 percent since enabling the flag, but there are no errors in the logs.

Thankfully, you connected your feature flag with a Guarded Release and can use it to debug the issue.

Investigation Steps

Step 1: Search for sessions with rage clicks

Since we attached a Guarded Rollout to your feature flag using a metric for detecting rage clicks, we can navigate to the LaunchDarkly’s Session page and filter for those specific sessions.

In the LaunchDarkly Sessions tab search bar, apply this filter:

has_rage_clicks=true featureflag.key=inline-form-validation

This query filters for sessions with rage clicks and where the inline-form-validation flag is enabled.

Step 2: Watch for user behavior patterns

In the session replay, you see:

User fills out the form completely.
User clicks Create Community.
User sees a loading spinner and nothing else.
User clicks button repeatedly.
User abandon forms altogether.

Step 3: Identify the UX failure
The community was being created successfully meaning the API returned a 200 and the new community existed. The new feature was intended to simply change the button color (baby blue → purple), but it broke the code that allows the success card to be shown.

The Root Cause

When the flag is off, the code does the right thing after a successful create:

Success → Set createdSpace → Show verification card (name, join code, "Back to community list")
User taps Back → Return to community list

When the flag is on, the success path was missing:

Success → Nothing. The if (!useInlineValidation) block runs only for the old flow, so when the flag is on, setCreatedSpace is never called.
The user stays on the form with no confirmation, no join code, and no way to know the create worked.

// Only the control (flag OFF) got success UI; the treatment (flag ON) did not
if (!useInlineValidation) {
  setCreatedSpace({ name: space.name, join_code: space.join_code });
}

The Fix

To fix the issue, you can either rollback to previous working code or fix the issue and toggle flag back on.

Option A – Roll back the flag: Set inline-form-validation to off in LaunchDarkly. Users are no longer on the new feature; they get the existing, working code path and see the verification card after Create Community.

Option B – Fix the new feature so it no longer breaks the success card: The new feature (flag on) breaks the code that shows the success card. Fix it by adding the success handling to the flag-on path. Also, when the flag is on, also call setCreatedSpace after a successful create so the success card is shown again.

Resolution

With the root cause identified from Session Replay, you update the code and resolve the rollout:

Roll back inline-form-validation to false so users get the verification card and can return to the list (instant, no deployment needed).
Fix the flag-on success path so it shows the verification card (or redirects) after create.
Deploy and re-enable the flag with the guarded rollout (rage click metrics) still attached, then monitor to confirm rage clicks return to baseline.

Time to resolution: 30 minutes to an hour (compared to potentially days of user complaints and support tickets).

Scenario 2: The Infinite Scroll Frustration

The Setup

Your team ships a feature flag (new-filter-location) that moves the existing Category and Sort controls on the "Lunches for Swap" feed from the top of the list to the bottom.

Everything works fine until you notice a drop in traffic.

Since you see no uptick in rage clicks for your Guarded Release, you know it must be another user signal you’re missing that could indicate user frustration. So you decide to investigate further by placing yourself in the user's shoes using session replay.

Investigation Steps

Step 1: Filter by flag variation

feature_flag.key=new-filter-location feature_flag.result.value=true

This query will filter for sessions where the feature flag new-filter-location is set to true.

In session replay, you see:

users scroll down the feed, then back up, then down again.
Minimal item taps.
Constantly moving through the list without making selections.
User navigates to a different page.

The pattern suggests they're looking for something rather than browsing items.

Step 3: Identify UX Failure
Replay shows the issue: “Category” and “Sort” are not at the top where users expect them. With the flag on, the controls were moved to below every post in the feed.

The Root Cause

The feature flag moves the existing filter bar from the top to the bottom of the list (ListFooterComponent when the flag is on). Same controls, different position.

// Flag ON: same filter bar rendered as ListFooterComponent — breaks user expectation
ListFooterComponent={
  useFiltersAtBottom ? (
    <View style={styles.filterBar}>...</View>
  ) : null
}

The Fix

Step 1 – Roll back the flag: Set new-filter-location to false. Filters return to the top (original position); rage scrolls drop.

Step 2 – Fix the experiment: Don't move the filters to the bottom. Keep the filter bar at the top regardless of the flag, or remove the flag and leave filters in their original place.

// Fix: always show filter bar at top (original place)
{/* Filters always above the list */}
<View style={styles.filterBar}>
  <TouchableOpacity style={styles.filterButton} onPress={openCategoryFilter}>
    <Text style={styles.filterButtonText}>Category: {categoryLabel}</Text>
  </TouchableOpacity>
  <TouchableOpacity style={styles.filterButton} onPress={openSortMenu}>
    <Text style={styles.filterButtonText}>Sort: {sortOrder === 'newest' ? 'Newest' : 'Oldest'}</Text>
  </TouchableOpacity>
</View>
<FlatList ... ListFooterComponent={null} />

Resolution

Roll back new-filter-location to false so filters are at the top again
Fix keep the filter bar at the top.
Deploy and re-enable the flag with monitoring.

Building your debugging playbook

Based on these scenarios, here's a systematic approach to rage click debugging:

Step 1: Triage with Filters
| What to look for | Search query |
| --- | --- |
| All frustrated users | has_rage_clicks=true |
| Specific page issues | has_rage_clicks=true visited-url="*/spaces/create*" |
| Browser-specific | has_rage_clicks=true browser_name=Chrome |
| High frustration | has_rage_clicks=true active_length>180s |

Step 2: Identify the Pattern
When reviewing session replays, look for:

Visual feedback gaps (Did the UI acknowledge the click?)
Loading states (Is there a spinner? Does it ever resolve?)
Error visibility (If there's an error, can the user see it?)
Touch target issues (On mobile, are elements too small or overlapping?)
Timing problems (Does the click happen before the element is ready?)

Step 3: Correlate with Technical Data
Session replay shows you the user's experience. Pair it with:

Network tab (API response codes and payloads.)
Console errors (JavaScript exceptions.)
Feature flag state (Which variation was the user seeing?)
Timing (When in the session did frustration peak?)

Step 4: Fix and Verify

Roll back using your feature flag (instant, no deployment needed).
Fix the root cause in code.
Re-deploy with the guarded rollout active.
Monitor rage click metrics to confirm the fix worked.

Whether you're shipping traditional features or AI-powered experiences, this playbook reflects a devops for ai mindset: continuous monitoring, fast rollback, and data-driven debugging that closes the gap between deployment and user impact.

Bringing it all together

The combination of rage click detection, session replay, and guarded releases creates something powerful: observability that starts with the human experience.

Traditional monitoring asks: "Is the system healthy?"

This approach asks: "Are users successful?"

When you can detect frustration in real-time, watch exactly what users experienced, and roll back problematic features instantly, you fundamentally change how fast you can ship with confidence.

This workflow fits into the broader ai deployment lifecycle, treating user frustration signals as first-class deployment metrics, just like error rates or latency. By embedding rage click detection into your ai lifecycle management strategy, every release becomes a feedback loop that improves both the product and the process.

The next time your users are frustrated, you'll know exactly what went wrong, which users were affected, and why. And the best part is you'll fix it before most users even notice.

Additional Resources

Day 7 | 🎄✨The Rockefeller tree in NYC: SLOs that actually drive decisions

Alexis Roberson — Wed, 17 Dec 2025 04:47:02 +0000

Originally published in the LaunchDarkly Docs

Most Subject Level Objectives or SLOs sit in dashboards gathering dust. Seeing that SLOS are performance targets that can be measured, they're extremely important in understanding the quality of a service or system. You define them, measure them, but when your conditions are not met, there’s no followup.

The biggest drawback is that SLOs are created to add value but if they’re never reinforced, it’s impossible to drive decisions, influence roadmaps, or help during incidents.

When it comes to defining SLOs, many folks often start at the top of the funnel by picking general metrics to measure, but in order to create SLOs that work, it’s important to understand how the roots impact the leaves.

In this post, we'll cover the pitfalls leading to out of sync SLOs with a few tips and tricks to ensure what you measure produces business value. You'll also see an example of how to set SLOs in realtime for a flag evaluation feature that you can implement in your own planning process.

But first, we'll explore a tree metaphor to recap key observability components and how they're influence expands from the roots all the way to the leaves. What if the popular Rockefeller tree in NYC represented the relationship between telemetry data and SLOs?

Understanding the Observability Tree

SLOs would essentially be the leaves on the Response branch as shown in the above image, meaning they are visible, measurable targets that everyone can see. The leaves would not be possible without the support of the trunk.

This is your telemetry data or the traces, logs, events you collect from your system. The data you collect acts as the foundation and support for the branches and leaves.

The roots represent the things you cannot see but are still vital to overall health of your system. This is the hard part of understanding system behavior, debugging unknown unknowns, and making data-driven decisions.

Most teams skip the roots entirely. They define SLOs using only the trunk (logs, traces, events), measuring things that can already be measured. However, the business outcomes and user behaviors are buried in what would be defined as the roots. Sometimes you have to dig through the soil to ensure your SLOs don't end up technically accurate yet strategically ineffective.

What Makes an SLO Decision-Worthy

So what makes a good SLO? The goal of a SLO is to bridge engineering and business needs to support a high quality user experience. And a good SLO depends on three things, business clarity or asking the right questions, measurability, or can these components actually be measured, and actionable targets, or the game plan for when things go wrong.

First, you need business clarity, which are the roots of the previously mentioned observability tree. This means articulating why something matters in concrete terms like dollars, users, retention and also avoiding vague statements like "uptime is important." For instance, if I were measuring the impact of downtime on a checkout feature, I could establish the SLO scope with “each minute of checkout downtime costs us $12,000 in lost revenue based on our average transaction volume.” It is essential to be able to explain the business impact in one clear sentence.

Second, you need measurability. This is like the trunk of the tree. Your SLO must connect to your golden signals such as latency, traffic, errors, saturation. This is where a lot of aspirational SLOs fall apart. Upper management might want to measure user happiness, but how can engineering translate this into actual metrics? Try to express the business impact in one clear sentence. If that's difficult, it’s usually a sign the problem definition needs a bit more shaping before defining the SLO.

Third, you need actionable targets, which represent the leaves on the observability tree. This is where most SLOs fail even when they get the first two right. There's a number, maybe even a threshold, but no clear action plan. What happens when you miss it? Who gets paged? What gets paused? Decision-worthy SLOs specify exactly what happens at different levels of degradation, and more importantly, they give everyone the confidence to make decisions based on those levels.

Building production resilient SLOs: LaunchDarkly’s Flag evaluation example

We can apply these same principles of building a production-worthy SLO using LaunchDarkly’s flag evaluation feature.

The flag evaluation feature in the monitoring tab is an extension of observability where it tracks how often each flag variation is served to different contexts over time, and highlights flag changes that might affect evaluation patterns.

Now, let’s build a SLO.

Step 1: Start with the business question

What would be impacted if the flag evaluations monitoring feature broke? Customers use these charts to understand rollout progress, debug targeting issues, and verify that their flags are working as expected. If evaluation data is delayed or missing, they can't trust what they're seeing. They might roll back a working feature thinking it's broken, or fail to catch a real problem because the charts show stale data. This undermines confidence in the platform and increases support load.

Step 2: Translate to user experience terms

What does "working well" look like? When a customer makes a flag change and checks the monitoring tab, they see updated evaluation counts within a couple minutes. The charts load quickly (under 3 seconds). The data is accurate meaning evaluation counts match what's actually happening in their application. If there's a delay, we tell them explicitly rather than showing stale data as if it's current.

Step 3: Connect to telemetry

We track several golden signals for this feature.

Data pipeline latency: time from evaluation event to appearing in charts.
Chart load time: how long it takes to render the monitoring page.
Data accuracy: comparing our recorded evaluations against a known sample.
Error rate: failed queries or chart rendering errors.

For the sake of this example will set arbitrary numbers for these signals. Let’s say you had a median pipeline latency of 45 seconds, p95 at 2 minutes, p99 at 5 minutes. And a chart load time averages 1.2 seconds. Data accuracy is 99.7 percent (some evaluations drop due to sampling) and error rate is 0.3 percent.

Using this data, we can set the target.

Step 4: Set the target

Based on that data, here's our SLO: 98 percent of flag evaluation events will appear in monitoring charts within 3 minutes, with chart load times under 3 seconds at p95.

Why these numbers? Customer research shows they expect "near real-time" monitoring, which they define as 2-3 minutes. Anything longer feels like stale data. Three seconds for chart loading is the threshold where users perceive delay and start questioning if something's broken.

We chose 98 percent instead of 99.9 percent because some evaluation events get sampled out intentionally for cost reasons, and occasional data pipeline delays from third-party dependencies are acceptable.

Now that we have our targets, we can use those thresholds to set conditional responses based on alerts or indicators.

Step 5: Define operational responses

Responses for Green, Red, or Yellow indicators in production:

If Green (>98%, <3 min, <3 sec load), continue normal operations.
If Yellow (95-98%, or 3-5 min, or 3-5 sec load), alert on-call, investigate within 4 hours.
If Red (<95%, or >5 min, or >5 sec load), page immediately, update status page if widespread.

Step 6: Drive decisions

Now the SLO becomes your decision-making framework. When engineering proposes adding a new feature like "evaluations by SDK" breakdown, the first question is: "Will this keep us within our 3-second chart load SLO?" If the answer is no, we either optimize the implementation or push back on the feature.

Infrastructure changes get evaluated the same way. Before migrating the data pipeline to a new system, we load tests against both our latency and accuracy targets. If the migration risks our SLO, we either fix the architecture or delay the migration. Another way I've seen SLOs used is planning future work. ie. if a team knows they are in the yellow this month, they may avoid picking up other risky work.

The SLO transforms from a monitoring target into a decision filter, helping to determine what gets shipped and what doesn’t.

Bringing it all together

Great SLOs aren't just leaves you pluck and add to dashboards. They're connected to everything below them from the trunk of solid telemetry to the roots of understanding what actually matters to your business and users. If you skip those foundational layers, your SLOs become technically accurate but strategically useless.

Start with the roots. Ask what would be impacted if this feature were to break. Work your way up through user experience and technical measurement. Build SLOs that bridge engineering and business with clear thresholds and clear consequences. And finally, make them specific enough to drive real decisions.

Day 6 | 💸 The famous green character that stole your cloud budget: the cardinality problem

Alexis Roberson — Tue, 16 Dec 2025 01:17:54 +0000

Originally published in the LaunchDarkly Docs

Every December, engineering teams unwrap the same unwanted gift: their annual observability bill. And every year, it's bigger than the last.

You know the pattern. Services multiply. Traffic grows. Someone discovers OpenTelemetry and suddenly every microservice is emitting 50 spans per request instead of 5. Then January rolls around and your observability platform sends an invoice that's 30% higher than last quarter.

Your VP of Engineering wants to know why.

You could blame it on the famous green character who hates Christmas, or you could join other teams who are getting serious about cost-efficient observability. That is, collecting telemetry data based on value, not volume.

Why "collect everything" no longer works

The old playbook was simple: instrument everything, store it all, figure out what you need later. Storage was cheap enough. Queries were fast enough. No need to overthink it.

Then, three things happened:

OpenTelemetry went mainstream. Teams migrated from vendor agents to OTel and began adding spans for everything. This added more visibility, but with 10x the data.
AI observability tools arrived. Platforms started using LLMs to analyze traces and suggest root causes. Powerful, but also expensive to run against terabytes of unfiltered trace data.
CFOs started asking questions. "Our_traffic grew 15% but observability costs grew 40%. Explain."

To stop instrumenting wouldn't be an option and also you want to make informed decisions, but still the biggest culprit, hiding in your telemetry stack is cardinality.

Cardinality will eat your budget

Cardinality is the observability villain. It sneaks in quietly, one innocent-looking label at a time, and before you know it, it's stolen your entire cloud budget. What is cardinality? It's just the number of unique time series your metrics generate, but it's also the main driver of observability costs that nobody sees coming.

Low cardinality: http_requests_total tracked by method and status_code. Maybe 20 unique combinations. Fairly manageable.

High cardinality: Same counter, but now you've added user_id, request_id, and session_token as labels. By simply adding these labels, you’ve just created millions of unique time series. Each one needs storage, indexing, and query compute. This will compound your bill faster than you can say deck the halls, except you wouldn’t be able to deck the halls, you’d be paying off your usage bill.

Stopping the Green character: set cardinality budgets

Most teams don't set limits on how many time series a service can create even though they should., but you can.

Start by auditing what you're currently generating. Look for metrics with >100K unique time series, or labels that include UUIDs, request IDs, or email addresses. These are your problem children.

Then set budgets. Give each service a limit, like 50K time series max. Assign team quotas so the checkout team knows they get 200K total across all their services. Create attribute allowlists that define exactly which labels are allowed in production. Yes, this feels restrictive at first. Your developers will complain. They'll argue that they need that user_id label for debugging. And sometimes they're right. But forcing that conversation up front means they have to justify the cost, not just add labels reflexively.

Finally, enforce budgets through linters that flag high-cardinality attributes in code review, CI checks that fail if estimates get too high, and dashboards that alert when cardinality spikes. This isn't about being restrictive. It's about being intentional. If you're adding a label, you should know why and what it'll cost.

Cardinality budgets solve the metrics problem, but what about traces? That's where sampling comes in.

Sampling: without the guilt

Not all sampling strategies are created equal, and picking the right one depends on what you're trying to protect.

Head-based sampling is pretty strict. You decide whether to keep a trace at the very start of a request, before you know if it'll be interesting. Fast checkout gets dropped. Slow checkout that timeout also gets dropped, because the decision happened too early. Not great.

Tail-based sampling is smarter. Wait until the trace completes, then decide based on what actually happened. Keep errors, high latency, or specific user cohorts. Sample down the boring stuff. This costs more (you have to buffer complete traces) but you keep what matters.

Probabilistic sampling is the middle ground. Keep 10% of everything, regardless of content. Predictable cost reduction, but you'll still lose some critical events. Works fine for stable services where trends matter more than individual traces.

Now rule-based sampling is where things get interesting, and honestly where most teams should be spending their energy. The idea is dead simple: different traffic deserves different sampling rates. You keep 100% of traces during feature rollouts because you actually care about every request when you're validating a new flow.

If you're using LaunchDarkly for progressive rollouts, you can tie sampling rates directly to flag evaluations. 100% sampling for users in the new variant, 10% for the control group. Your main API endpoints can run at 50% since they're stable and high-volume. Internal health checks that just verify the service is alive need maybe 5%, or even less. I've seen teams go down to 1% for health checks and never miss it.

The key is that you're making these decisions based on the actual value of the signal, not just applying a blanket rate across everything. Adjust based on context: feature flags, experiments, specific endpoints, user cohorts, whatever makes sense for your system.

Sampling isn't about compromising visibility. It's about amplifying signals. The noisy 90% of traces you're storing never get looked at anyway.

Once you've decided what to keep, you still need to decide how long to keep it and at what resolution.

Downsample vs. Discard: know when to do which

Not all data reduction is the same, and mixing up downsampling with discarding is how teams accidentally delete data they actually need.

Downsample when you need historical context but not full precision. SLO burn rates don't need second-by-second granularity so you can downsample to 1-minute intervals and still catch every trend. An additional practice is to keep high-res data for a week, then downsample to hourly for long-term retention.

Discard when the data is redundant or has served its purpose. For instance, debug spans from a canary that passed three days ago can be deleted. Or if you captured an error in both a trace and a log, you can pick one source of truth and drop the duplicate.

The rule of thumb here is If you'll never query it, don't store it. If you might need it for trends in six months, downsample it. If you need it immediately when something breaks, keep it at full resolution with an aggressive retention policy.

What this actually looks like

Cost-efficient observability isn't about cutting capabilities. It's about cutting waste.

Start by auditing your cardinality. Find the metrics generating hundreds of thousands of time series because someone added user_id as a label. Then, set budgets like 50K per service, 200K per team and enforce them through linters and CI checks. Create ways to encourage developers to justify high-cardinality labels before they ship, not after the bill arrives.

Then you’ll be ready to tackle sampling. Drop the blanket 10% probabilistic rate and switch to rule-based sampling tied to actual value. Keep 100% of traces during feature rollouts. Sample stable endpoints at 10%. Go as low as 1% for health checks. If you're running feature flags, tie sampling to flag evaluations so you capture what matters and discard what doesn't.

Finally, clean up retention, downsample SLO metrics to 1-minute intervals, discard debug spans from canaries that passed days ago and delete duplicate error data.

This not only leads to lower bills, but also cleaner dashboards, faster queries, fewer noisy alerts, and teams that spend less time swimming through telemetry and more time fixing actual problems.

Observability ROI isn't measured in data volume. It's measured in how fast you detect and resolve issues.

The teams figuring this out in 2025 aren't collecting everything. They're collecting what matters.

Day 3 | 🔔 Jingle All the Way to Zero-Config Observability

Alexis Roberson — Wed, 10 Dec 2025 19:10:26 +0000

Originally published in the LaunchDarkly Docs

For years, auto-instrumentation promised effortless observability but kept falling short. You'd still end up manually adding spans to business logic, hunting down missing metadata, or trying to piece together how a feature rollout was affecting customers.

That finally shifted in 2025. With OTel auto-instrumentation maturing and LaunchDarkly adding built-in OTel support to server-side SDKs, teams started getting feature flag context baked into their traces without writing instrumentation code. The zero-config promise actually started delivering.

Auto-instrumentation has always had a blind spot: it shows you what happened, but not why. You'd see a latency spike, but had no idea which feature flag was active, which users hit it, or what experiment was running.

Without that context, you're doing detective work. Digging through logs, matching up timestamps, guessing at what caused what. Manual instrumentation helped, but you paid for it in engineering time, inconsistent coverage, and mounting technical debt.

Auto-instrumentation that actually knows about your features

The game changed when OTel auto-instrumentation actually got good. Instead of just capturing basic HTTP calls, it now handles:

Framework-level request tracing.
Automatic context propagation across services.
Runtime metadata and environment details.
Errors and exceptions without manual try-catch blocks.

LaunchDarkly takes this further by injecting flag evaluation data straight into OTel spans. Every time you evaluate a flag, you automatically get the flag key, user context, which variation served, and the targeting rule that fired. That data feeds into your existing OTel pipeline, so your traces finally show which features were active and who was affected - not just database queries and API calls.

So how do you actually set this up?

To get started with Otel trace hooks and feature flag data, simply add the hooks to your LaunchDarkly client config.

import ldclient
from ldclient import Config
from ldotel.tracing import Hook

config = Config('YOUR_SDK_KEY', hooks=[Hook()])
ldclient.set_config(config=config)

client = ldclient.get()

This flows into your existing OpenTelemetry pipeline, enriching every trace with feature-aware context.

The TracingHook automatically decorates your OpenTelemetry spans with flag evaluation events. When your application evaluates flags during a request, those evaluations become part of the trace along with the full context about what was evaluated and for whom.

You can also configure your OpenTelemetry collector or exporter to point to LaunchDarkly's OTLP endpoint, and you're done.

For HTTP:

https://otel.observability.app.launchdarkly.com:4318

For gRPC:

https://otel.observability.app.launchdarkly.com:4317

This feature is also available in .Net, Go, Java, Node.JS and Ruby.

Auto-instrumentation handles the rest, HTTP spans, database calls, framework-level tracing, error capture, and now, feature flag context.

What auto-instrumentation unlocks

When you ship a new feature variant, you immediately see how it performs per cohort. If there's a latency spike in the "new-checkout-flow" variation, you'll know within minutes before it affects user experience.

That same visibility matters during incidents. When an outage hits, filter traces by flag evaluation to see which features were active when errors occurred. The trace shows you whether it was the new recommendation engine, the optimized query path, or something else entirely.

This is especially powerful for experimentation. LaunchDarkly processes your OTel traces into metrics automatically, so when you run an A/B test, you get latency, error rate, and throughput calculated per variation without extra config. The same telemetry powering your dashboards powers your experiments.

The best part of this setup is that it scales without additional work. As teams ship more features behind flags, the telemetry gets more valuable without getting more expensive to maintain. New services inherit feature-aware tracing just by initializing the SDK.

When to add custom spans

Zero-config doesn't mean never-config. You'll still want custom spans for:

Business logic milestones. If you need to measure time-to-recommendation or search-to-purchase, custom spans make that explicit.
ML pipeline stages. Feature extraction, model inference, and post-processing often warrant their own spans for detailed performance analysis.
Cross-service boundaries. Queue producers, stream processors, and async workers may need manual context propagation and span creation.
Experiment-specific KPIs. If your A/B test measures "items added to cart" or "video completion rate," you'll instrument those as custom metrics.

The important part is you're writing these spans to capture business value, not to patch holes in your instrumentation.

Delivering real value

Combining mature auto-instrumentation with feature-aware enrichment changes how teams approach observability. It's no longer a separate investment that competes with feature development. It's a byproduct of how you ship features.

When you evaluate a flag, you get telemetry. When you roll out a feature, you get performance data segmented by variation. When you run an experiment, you get metrics derived from production traces. The instrumentation you would have written manually is now embedded in the tools you already use.

Observability stops being something you retrofit after launch and becomes something you inherit by default. Which means teams spend less time debugging instrumentation gaps and more time acting on insights.

That's the promise of zero-config, finally delivered.

Ready to try it? Explore LaunchDarkly's OpenTelemetry integration documentation or sign up for a free trial account.

Enable Observability or Experimentation in your LaunchDarkly dashboard and start seeing feature-aware telemetry from your existing traces.

Day 2 | 🎅 He knows if you have been bad or good... But what if he gets it wrong?

Alexis Roberson — Tue, 09 Dec 2025 20:21:47 +0000

Originally published in the LaunchDarkly Docs

"He knows if you've been bad or good..."

As kids, we accepted the magic. As engineers in 2025, we need to understand the mechanism. So let's imagine Santa's "naughty or nice" system as a modern AI architecture running at scale. What would it take to make it observable when things go wrong?

The architecture: Santa's distributed AI system

Santa's operation would need three layers. The input layer handles behavioral data from 2 billion children on a point system. "Shared toys with siblings" gets +10 points, "Threw tantrum at store" loses 5.

The processing layer runs multiple AI agents working together. A Data Agent collects and organizes behavioral events. A Context Agent retrieves relevant history: letters to Santa, past behavior, family situation. A Judgment Agent analyzes everything and makes the Nice/Naughty determination. And a Gift Agent recommends appropriate presents based on the decision.

The integration layer connects to MCP servers for Toy Inventory, Gift Preferences, Delivery Routes, and Budget Tracking.

It's elegant. It scales. And when it breaks, it's a nightmare to debug.

The Problem: A good child on the Naughty List

It's Christmas Eve at 11:47 PM.

A parent calls, furious. Emma, age 7, has been a model child all year. She should be getting the bicycle she asked for. Instead, the system says: Naughty List - No Gift.

You pull up the logs:

Emma's judgment: 421 NICE points vs 189 NAUGHTY points
Gift Agent tries to check bicycle inventory → TIMEOUT
Gift Agent retries → TIMEOUT  
Gift Agent retries again → TIMEOUT
Gift Agent checks inventory again → Count changed
Gift Agent reasoning: "Inventory uncertain, cannot fulfill request"
Gift Agent defaults to: NAUGHTY LIST

Emma wasn't naughty. The Toy Inventory MCP was overloaded from Christmas Eve traffic. But the agent's reasoning chain interpreted three timeouts as "this child's request cannot be fulfilled" and failed to the worst possible default.

With traditional APIs, you'd find the bug on line 47, fix it, and deploy. With AI agents, it's not that simple. The agent decided to interpret timeouts that way. You didn't code that logic. The LLM's 70 billion parameters did.

This is the core challenge of AI observability: You're debugging decisions, not code.

Why AI Systems are hard to debug

Black box reasoning and reproducibility go hand in hand. With traditional debugging, you step through the code and find the exact line that caused the problem. With AI agents, you only see inputs and outputs. The agent received three timeouts and decided to default to NAUGHTY_LIST. Why? Neural network reasoning you can't inspect.

And even if you could inspect it, you couldn't reliably reproduce it. Run Emma's case in test four times and you might get:

Run 1: NICE LIST, gift = bicycle ✓
Run 2: NICE LIST, gift = video game ✓
Run 3: NICE LIST, gift = art supplies ✓
Run 4: NAUGHTY LIST, no gift ✗

Temperature settings and sampling introduce randomness. Same input, different results every time. Traditional logs show you what happened. AI observability needs to show you why, and in a way you can actually verify.

Then there's the question of quality. Consider this child:

Refused to eat vegetables (10 times) but helped put away dishes
Yelled at siblings (3 times) but defended a classmate from a bully
Skipped homework (5 times) but cared for a sick puppy

Is this child naughty or nice? The answer depends on context, values, and interpretation. Your agent returns NICE (312 points), gift = books about empathy. A traditional API would return 200 OK and call it success. For an AI agent, you need to ask: Did it judge correctly?

Costs can spiral out of control. Mrs. Claus (Santa's CFO) sees the API bill jump from 5,000 in Week 1 to 890,000 on December 24th. What happened? One kid didn't write a letter. They wrote a 15,000-word philosophical essay. Instead of flagging it, the agent processed every last word, burning through 53,500 tokens for a single child. At scale, this bankrupts the workshop.

And failures cascade in unexpected ways. The Gift Agent doesn't just fail when it hits a timeout. It reasons through failure. It interpreted three timeouts as "system is unreliable," then saw the inventory count change and concluded "inventory is volatile, cannot guarantee fulfillment." Each interpretation fed into the next, creating a chain of reasoning that led to: "Better to disappoint than make a promise I can't keep. Default to NAUGHTY_LIST."

With traditional code, you debug line by line. With AI agents, you need to debug the entire reasoning chain. Not just what APIs were called, but why the agent called them and how it interpreted each result.

What Santa Actually Needs

The answer isn't to throw out traditional observability, but to build on top of it. Think of it as three layers.

This is exactly what we've built at LaunchDarkly. Our platform combines AI observability, online evaluations, and feature management to help you understand, measure, and control AI agent behavior in production. Let's walk through how each layer works.

Start with the fundamentals. You still need distributed tracing across your agent network, latency breakdowns showing where time is spent, token usage per request, cost attribution by agent, and tool call success rates for your MCP servers. When the Toy Inventory MCP goes down, you need to see it immediately. When costs spike, you need alerts. This isn't optional. It's table stakes for running any production system.

For Santa's workshop, this means tracing requests across Data Agent → Context Agent → Judgment Agent → Gift Agent, monitoring MCP server health, tracking token consumption per child evaluation, and alerting when costs spike unexpectedly. It’s important to note, LaunchDarkly's AI observability captures all of this out of the box, providing full visibility into your agent's infrastructure performance and resource consumption.

Then add semantic observability. This is where AI diverges from traditional systems. You need to capture the reasoning, not just the results. For every decision, log the complete prompt, retrieved context, tool calls and their results, the agent's reasoning chain, and confidence scores.

When Emma lands on the Naughty List, you can replay the entire decision. The Gift Agent received three timeouts from the Toy Inventory MCP, interpreted "inventory uncertain" as "cannot fulfill request," and defaulted to NAUGHTY_LIST as the "safe" outcome. Now you understand why it happened. And more importantly, you realize this isn't a bug in your code. It's a reasoning pattern the model developed. Reasoning patterns require different fixes than code bugs.

LaunchDarkly's trace viewer lets you inspect every step of the agent's decision-making process, from the initial prompt to the final output, including all tool calls and the reasoning behind each step.

Finally, use online evals. Where observability shows what happened, online evals automatically assess quality and take action. Using the LLM-as-a-judge approach, you score every sampled decision. One AI judges another's work:

{
  "accuracy": {
    "score": 0.3,
    "reasoning": "Timeouts should trigger retry logic, not default to 
      worst-case outcome. System error conflated with behavioral judgment."
  },
  "fairness": {
    "score": 0.4,
    "reasoning": "Similar timeout patterns resulted in NICE determination 
      for other children. Inconsistent failure handling."
  }
}

This changes the conversation from vague to specific.

Without evals: "Let's meet tomorrow to review Emma's case and decide if we should rollback."

With evals: "Accuracy dropped below 0.7 for the 'timeout cascade defaults to NAUGHTY' pattern. Automatic rollback triggered. Here are the 23 affected cases."

LaunchDarkly's online evaluations run continuously in production, automatically scoring your agent's decisions and alerting you when quality degrades. You can define custom evaluation criteria tailored to your use case and set thresholds that trigger automatic actions.

This is where feature management and experimentation come in. Feature flags paired with guarded rollouts let you control deployments and roll back bad ones. Experimentation lets you A/B test different approaches. With AI agents, you're doing the same thing, but instead of testing button colors or checkout flows, you're testing prompt variations, model versions, and reasoning strategies. When your evals detect accuracy has dropped below threshold, you automatically roll back to the previous agent configuration.

Use feature flags to control which model version, prompt template, or reasoning strategy your agents use and seamlessly roll back when something goes wrong. Our experimentation platform lets you A/B test different agent configurations and measure which performs better on your custom metrics. Check out our guide on feature flagging AI applications.

You're not just observing decisions. You're evaluating quality in real-time and taking action.

Debugging Emma: all three layers in action

Traditional observability shows the Toy Inventory MCP experienced three timeouts that triggered retry logic. Token usage remained average. From an infrastructure perspective, nothing looked catastrophic.

Semantic observability reveals where the reasoning went wrong. The Gift Agent interpreted the timeouts as "inventory uncertain" and made the leap to "cannot fulfill requests." Rather than recognizing this as a temporary system issue, it treated the timeouts as a data problem and defaulted to NAUGHTY_LIST.

Online evals reveal this isn't just a one-off problem with Emma, but a pattern happening across multiple cases. The accuracy judge flagged this decision at 0.3, well below acceptable thresholds. Querying for similar low-accuracy decisions reveals 23 other cases where timeout cascades resulted in NAUGHTY_LIST defaults.

Each layer tells part of the story. Together, they give you everything you need to fix it before more parents call.

With LaunchDarkly, all three layers work together in a single platform. You can trace the infrastructure issue, inspect the reasoning chain, evaluate the decision quality, and automatically roll back to a safer configuration, all within minutes of Emma's case being flagged.

Conclusion

Every AI agent system faces these exact challenges. Customer service agents making support decisions. Code assistants suggesting fixes. Content moderators judging appropriateness. Recommendation engines personalizing experiences. They all struggle with the same problems.

Traditional observability tools weren't built for this. AI systems make decisions, and decisions need different observability than code.

Santa's system says "He knows if you've been bad or good." But how he knows matters. Because when Emma gets coal instead of a bicycle due to a timeout cascade at 11:47 PM on Christmas Eve, you need to understand what happened, find similar cases, measure if it's systematic, fix it without breaking other cases, and ensure it doesn't happen again.

You can't do that with traditional observability alone. AI agents aren't APIs. They're decision-makers. Which means you need to observe them differently.

LaunchDarkly provides the complete platform for building reliable AI agent systems: observability to understand what's happening, online evaluations to measure quality, and feature management to control and iterate safely. Whether you're building Santa's naughty-or-nice system or a production AI application, you need all three layers working together.

Ready to make your AI agents more reliable? Start with our AI quickstart guide to see how LaunchDarkly can help you ship AI agents with confidence.

Day 1 | 🎄 Observability under the Tree: What Changed in 2025

Alexis Roberson — Mon, 08 Dec 2025 20:21:33 +0000

Originally published in the LaunchDarkly Docs

Sometimes you need distance to see what really changed. And that’s exactly how 2025 felt in regards to observability. While the focus was rightfully on AI agents and MCP servers, the observability space was experiencing its own maturation. And if you feel you missed this shift, no need to panic, Santa left some gifts under the tree. Those gifts being a universal standard for instrumentation, practical platform consolidation, and AI-native capabilities that finally deliver on the hype.

Gift #1: OpenTelemetry or the foundation nobody wanted to build

For years, observability felt like the Wild West. Every vendor had their own way of doing things, their own agents, instrumentation libraries. If you wanted to switch tools, you basically had to re-instrument your entire application. It was exhausting, but nobody wanted to be the first to blink (so to speak).

OpenTelemetry had been building toward something for years, but 2025, in my opinion, was when it reached a tipping point. After gradual adoption since the early 2020s, it crossed from "early adopters only" to "the default choice." The specs had stabilized, the major players finally committed, and the tooling matured enough that adoption became low-risk rather than bleeding-edge.

The change was subtle at first. A cloud provider here, a major vendor there, all quietly releasing native OTel support. Then it accelerated. Suddenly the question wasn't "should we adopt OpenTelemetry?" but "why haven't we adopted it yet?" Teams started instrumenting once and sending data anywhere they wanted. The instrumentation lock-in that had defined observability for years started to ease. Now, you still had considerations around data migration and historical context, but the path forward got clearer.

Even more, here's what made this interesting: standardization didn't just reduce vendor lock-in. It unlocked something bigger, and that is tool scrutinization by reducing scope and being more selective about what tool goes in your toolbox.

Gift #2: Platform Consolidation after the dust has settled

With a common data standard gaining real traction, something practical happened: the tooling chaos started to make less sense.

Think about the typical observability stack from a few years ago. Metrics in one place, logs in another, traces somewhere else. APM here, synthetic monitoring there, and probably two or three other specialized tools for edge cases. Each with its own interface, its own query language, its own billing model, and its own person on the team who understood how it worked.

2025 was the year teams started asking "why are we doing this to ourselves?"

Platform consolidation had been promised for years. However, this time felt different. With OTel providing a standard data layer, platforms could actually focus on the analysis and visualization layer instead of fighting about ingestion formats. They could genuinely unify telemetry data in ways that reduced cognitive load.

It is becoming common practice for teams to cut their tool count from eight or nine down to two or three, and actually improve coverage in the process. It’s important to note that the cost story varied with some teams seeing real savings from eliminating redundant ingestion and storage and others mainly reducing operational overhead rather than raw dollar spend. At the end of the day the consistent theme was relief.

With tool consolidation, came the freedom to innovate in the world of query execution, where instead of building complex queries to retrieve specific data, teams could interact with data sets using natural language.

Gift #3 Natural language query execution

I’m no stranger to learning and mastering query languages like the Elasticsearch Query DSL and though optimized for improving latency and data processing, there still remained a continual learning curve.

2025 brought natural language interfaces that crossed the "actually useful" threshold. Though not perfect as complex queries still benefit from knowing the underlying language, it has proven genuinely helpful for common questions. You could ask questions like:

"Show me API latency for checkout over the last hour broken down by region."

And get back related data without memorizing aggregation functions or time window syntax. The true value is removing the barrier of entry, so less technical team members like product managers and business-focused teams can also understand the type of performance issues that were generally understood by DevOps or SREs. In other words, the observability data that had been locked behind specialized knowledge became accessible to more people who needed it.

But accessibility alone doesn't solve the deeper problem, which is knowing what to look for in the first place to understand real business impact.

Gift #4 Aligned Business KPIs with measurable metrics

Historically, observability was about system health. Is the service up? What's the error rate? How's latency looking? All important questions, but that unfortunately rarely answered the question of why any of this should matter to other teams. The shift toward business outcomes had been building, but 2025 is when it moved from buzzword to practical framework.

Tools made it easier to connect technical metrics to business impact. That latency spike? You could now configure dashboards and alerts that showed the correlation with conversion drops and revenue impact, rather than just knowing the service was slow. This error rate? You could track it against transaction volume and calculate lost revenue. This performance degradation?

You could filter by customer segment and see who was actually affected. Going as far as detecting rage clicks on a page affecting user experience.

The setup still required work like defining what business metrics mattered, instrumenting those alongside technical metrics, building the correlation logic. But the platforms made it possible without custom data pipelines and analytics infrastructure.

This changed prioritization too. Instead of treating every alert as equally urgent, teams could focus on what actually moved the needle. The 99th percentile latency issue that only affected a low-value endpoint? Still worth fixing eventually, but not at 2 AM. The intermittent error affecting checkout? All hands on deck.

The business-outcome framing made observability feel less like a tax and more like a strategic capability. But it also created a new opportunity that begged the question: how can AI truly be used to not just automate processes but make real decisions around data interpretation and root cause analysis with the intent of self healing.

Gift #5 AI-Native observability, when the promises started delivering

Which brings us to AI, the talk of the town that has finally started delivering in meaningful ways in 2025.

Not the "AI-powered" marketing speak that got slapped on every dashboard, but real AI-native observability that changed how teams work. Anomaly detection that's actually tuned for your environment and doesn't require a data science degree to configure. Root cause analysis that connects the dots across services, traces, logs, and business metrics without requiring manual correlation. Predictive insights that warn you about capacity issues before they become incidents. All built into dashboards like Vega AI for thorough investigations.

The capabilities themselves weren't entirely new as statistical anomaly detection has existed for years. What changed was the integration, the usability, and the accuracy. The systems got good enough that you could trust them to surface real issues without drowning you in false positives.

All the previous observability gifts created the perfect foundation for AI to enter the chat. From OTel's standardized data formats creating consistent structure leading to more reliable pattern recognition. To the consolidated platforms that provide a unified view making natural language interfaces even more powerful because you can ask not just "what happened" but "why did this happen and what should I do about it." By the time you add connections between metrics and business impact, you already have a full observability system primed and ready for AI.

Bonus: What LaunchDarkly left in your stockings this year

While the industry was catching up on the observability fundamentals, LaunchDarkly was actively building toward the same vision, with one crucial difference: observability unified with feature flag context. Here's what landed in 2025 that aligns with these broader trends.

Vega AI: Making AI-Native Observability Real

Speaking of AI capabilities that actually deliver, LaunchDarkly introduced Vega, an AI-powered debugging companion that goes beyond generic copilots. Vega combines two integrated capabilities: an AI debugging assistant that summarizes errors, identifies root causes, and suggests code fixes with GitHub integration, and a natural language search assistant that lets you query observability data without memorizing syntax. The key difference? Context. Vega understands feature flags alongside telemetry data, connecting what changed in your release with what broke in production, enabling faster triage and smarter fixes.

OpenTelemetry Support: Embracing the Standard

In the spirit of OTel's tipping point, LaunchDarkly's C++ Server SDK added native OpenTelemetry tracing support. Teams can now enrich their distributed traces with LaunchDarkly context, sending observability data to power Guardian while maintaining the flexibility to route telemetry wherever they need. It's exactly the kind of integration that becomes possible when standardization reduces the friction.

Session Replay for Mobile: Closing the Visibility Gap

Web session replay has been table stakes, but mobile remains a black box. LaunchDarkly's new native iOS and Android SDKs for Session Replay now capture UI interactions, gestures, and network events tied directly to feature flag evaluations. When an issue occurs, you can see exactly what your users experienced and which flag variation they encountered, bringing the same level of observability to mobile that web teams have enjoyed.

Regression Debugging: Automated Error Detection

The Guarded Releases capability now includes regression debugging that automatically surfaces frontend errors tied to new feature flag variations without manual instrumentation. When a rollout triggers a regression, relevant errors appear directly in the monitoring view, with one-click access to deeper analysis in the observability view. It's the kind of integration that makes detecting issues faster and connecting dots easier.

LLM Observability: AI-Specific Telemetry

As GenAI applications moved to production, teams needed visibility beyond standard metrics. LaunchDarkly's LLM observability tracks not just performance metrics like latency and error rates, but semantic details including prompts, token usage, and responses. It addresses the unique challenge of monitoring AI features where understanding quality matters as much as performance.

Guardrail Metrics and Feature Flag Monitoring: Business Impact by Default

Two updates reinforced the shift toward business-outcome observability. Guardrail metrics let teams define default trusted metrics that automatically apply to any guarded rollout, ensuring consistent measurement across releases. Feature Flag Monitoring provides a unified view showing errors, web vitals, and the impact of feature changes in one place, with flag change annotations overlaid to see correlation. You ship a feature, you immediately see how it's performing, all with the context of which flags are active.

Together, these updates represent LaunchDarkly's commitment to the same principles driving industry-wide change: standardized instrumentation, consolidated platforms, natural language accessibility, business-outcome alignment, and AI-native capabilities. The difference is in the integration, where feature flags provide the missing context that helps teams understand not just what happened, but what they changed that caused it to happen.

What next?

When I look at these changes together, I see something more significant than individual feature releases or product launches. The observability landscape of 2025 represents a meaningful shift in what's practical and accessible.

It's more accessible seeing that more people on the team can explore data and ask questions without specialized training. It's more focused because we have better tools for measuring what actually matters for business outcomes, not just what's easy to instrument. It's more intelligent in that AI handles more of the grunt work of correlation and pattern matching. And it's more practical because costs are more manageable and the tool sprawl that was slowly crushing teams can hopefully start to recede.

As we head into 2026, I'm not thinking about what new features will launch. I'm thinking about what becomes possible when these foundations are in place. When observability isn't a specialized skill but a shared capability. When understanding system behavior is directly tied to business strategy. When the friction between having a question and getting an answer continues to decrease.

That's the real gift 2025 left us. Not just better tools, but a better way of working. One that's still evolving, but noticeably more mature than where we were.

Join us for Day 2 of the Holiday Campaign where we'll discuss the reality of observability for AI agents.

Detecting User Frustration: Understanding rage clicks and session replay

Alexis Roberson — Wed, 26 Nov 2025 00:02:09 +0000

Originally published in the LaunchDarkly Docs

Overview

Part 1 of 3: Rage Click Detection with LaunchDarkly

The holidays are around the corner and with it comes the expected uptick in traffic, and as traffic increases the need to preserve user experience becomes that much more imperative. You can ship out a new feature, everything seems to be going as planned, and then all of a sudden you start to see a spike in support tickets. The error logs aren’t helpful or show nothing and the metrics look fine.

So, what happened?
Wouldn’t you like to literally put yourself in the user's shoes by seeing exactly what happened in a user session? Enter session replay and more specifically rage clicks as a barometer for user experience.

In this three-part series, we'll explore how LaunchDarkly's session replay and observability features help you detect, diagnose, and fix user experience issues in real-time. Part 1 covers the fundamentals: what rage clicks are, how to detect them, and how to get started with session replay.

What is a rage click?

A rage click occurs when a user rapidly clicks the same element multiple times in frustration, usually because the element appears clickable but isn't responding as expected. It's one of the strongest behavioral signals that something is wrong with your user experience.

Why rage clicks matter:

Silent failures: Many bugs don't throw errors or trigger alerts.
Real user impact: Unlike synthetic monitoring or load tests, rage clicks show you exactly what real users experienced in production conditions.
Early warning system: Rage click spikes often appear before users start filing support tickets or abandoning your app entirely.
Actionable insights: Unlike vague complaints like 'the site is slow,' rage clicks point to specific UI elements that need attention.

The Anatomy of a rage click

Not every series of rapid clicks qualifies as rage clicks. LaunchDarkly uses three criteria to distinguish genuine frustration from normal user behavior:

These defaults work well for most applications, but LaunchDarkly lets you adjust them based on your specific use case. We'll cover customization later in this post.

Beyond rage clicks: Other Frustration Signals

While rage clicks are the most obvious indicator of user frustration, LaunchDarkly's session replay automatically captures several other behavioral patterns:

Together, these signals paint a complete picture of user frustration, which gives you qualitative context that complements your quantitative metrics.

Let's walk through implementing session replay in a complete application, from installation to viewing your first rage click detection.

Getting Started with LaunchDarkly session replay

HolisticSelf App

This health tracking app, written in Javascript, demonstrates LaunchDarkly session replay implementation for detecting user frustration through rage click monitoring.

Find all the code for this demo here, with the LaunchDarkly integration instructions in this .md file.

Implementation Architecture

File Structure:

src/
├── services/
│   └── launchdarkly.js    # LaunchDarkly initialization service
├── main.jsx                # App entry point (initializes LaunchDarkly)
└── App.jsx                 # Main React component

Key Features:

Automatic initialization on app load.
Anonymous user tracking (no login required).
Strict privacy mode for health data protection.
Network recording for API debugging.
Zero custom rage click code needed.

Step 1: Install Dependencies
First, install the required LaunchDarkly packages:

npm install launchdarkly-js-client-sdk @launchdarkly/observability @launchdarkly/session-replay

< Note: The observability plugin requires JavaScript SDK version 3.7.0 or later. >

Step 2: Configure Environment Variables.
Create a .env file in your project root. You can get your client-side ID from LaunchDarkly: Project Settings > Environments.

VITE_LAUNCHDARKLY_CLIENT_ID=your-client-side-id-here

Step 3: Initialize LaunchDarkly Client.
Create src/services/launchdarkly.js:

// LaunchDarkly session replay and Observability Client Initialization
// Based on: https://launchdarkly.com/docs/sdk/observability/javascript
import { initialize } from 'launchdarkly-js-client-sdk';
import Observability from '@launchdarkly/observability';
import SessionReplay, { LDRecord } from '@launchdarkly/session-replay';

let ldClient = null;
let isInitialized = false;

export async function initializeLaunchDarkly(
  clientSideId,
  user = null,
  options = {}
) {
  if (isInitialized) {
    console.warn('LaunchDarkly client already initialized');
    return ldClient;
  }

  // Default to anonymous user if none provided
  const userContext = user || {
    kind: 'user',
    key: `anonymous-${Date.now()}`,
    anonymous: true,
  };

  const {
    manualStart = false,
    privacySetting = 'default', // 'none', 'default', or 'strict'
    startSessionReplay = true,
  } = options;

  try {
    // Initialize LaunchDarkly client with observability and session replay plugins
    // Reference: https://launchdarkly.com/docs/sdk/observability/javascript
    ldClient = initialize(clientSideId, userContext, {
      plugins: [
        new Observability({
          manualStart: manualStart,
        }),
        new SessionReplay({
          manualStart: manualStart,
          privacySetting: privacySetting, // Redacts PII based on setting
        }),
      ],
    });

    // Wait for client to be ready
    await ldClient.waitForInitialization();

    isInitialized = true;
    console.log('LaunchDarkly client initialized with observability and session replay');

    // Start plugins if not using manual start
    if (!manualStart) {
      if (startSessionReplay) {
        // Start session replay recording
        // Note: Rage click detection is automatically handled by LaunchDarkly
        // Configure thresholds in Project Settings > Observability > Session settings
        // Reference: https://launchdarkly.com/docs/home/observability/settings
        LDRecord.start({
          forceNew: false, // Continue existing session if available
          silent: false, // Show console warnings
        });
        console.log('Session replay started');
      }
    }

    return ldClient;
  } catch (error) {
    console.error('Failed to initialize LaunchDarkly:', error);
    throw error;
  }
}

Step 4: Implement the LaunchDarkly Client in Main.
Initialize LaunchDarkly in src/main.jsx for:

import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import './styles.css';
import { initializeLaunchDarkly } from './services/launchdarkly';

// Get LaunchDarkly client-side ID from environment variable
const LAUNCHDARKLY_CLIENT_ID = import.meta.env.VITE_LAUNCHDARKLY_CLIENT_ID || '';

// Initialize LaunchDarkly before React renders
// This ensures session replay starts as early as possible
if (LAUNCHDARKLY_CLIENT_ID) {
  initializeLaunchDarkly(
    LAUNCHDARKLY_CLIENT_ID,
    null, // User context (null = anonymous user)
    {
      serviceName: 'my-health-app',
      privacySetting: 'strict', // Recommended for health apps
      enableNetworkRecording: true,
    }
  ).catch((error) => {
    console.error('Failed to initialize LaunchDarkly:', error);
  });
} else {
  console.warn('LaunchDarkly client-side ID not configured. Set VITE_LAUNCHDARKLY_CLIENT_ID environment variable.');
}

ReactDOM.createRoot(document.getElementById('root')).render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

Now that we have initialized our LaunchDarkly client and configured session replay at the start of the app, we can configure rage detection in the LaunchDarkly UI.

Configuring Rage Click Detection

Rage click detection is configured in the LaunchDarkly UI, not in code. This makes it easy to adjust thresholds without redeploying.

Access Settings:

Log in to LaunchDarkly.
Navigate to Project Settings > Observability > Session settings.
Find the Rage clicks section.

Adjust the rage click settings to reflect the user clicking 5+ times within 2 seconds in the same area:

Minimum clicks -> Set number of clicks required to trigger to 5 clicks.
Click radius -> Set the pixel radius for click proximity to 8 pixels.
Elapsed time -> Set the time window for detecting rage clicks to 2 seconds.

These settings apply to all sessions automatically and require no code changes needed.

How it works:

Session recording
When the session replay plugin is initialized, LaunchDarkly automatically:

Records all DOM changes.
Captures every click with precise coordinates and timestamps.
Tracks scroll events, form inputs, and navigation.
Sends data to LaunchDarkly servers in the background.

Automatic detection
LaunchDarkly's backend analyzes recorded sessions:

Identifies rapid click patterns.
Applies your configured thresholds.
Marks sessions with has_rage_clicks=true attribute.
Associates rage clicks with specific elements and pages.

Viewing session replay Results

In order to test this integration, you can do the following:

Create a test button that intentionally does nothing:
Rapidly click the button 5+ times within 2 seconds in the same spot.
Wait a few minutes for LaunchDarkly to process the session
Check LaunchDarkly dashboard by navigating to Monitor > Sessions with the option to Filter by has_rage_clicks=true.

To start, your app should look something like this:

And when you navigate to Launchdarkly UI -> Sessions and you should be able to see the complete session replay.

In order to test the rage clicks integration, we can add a custom button by adding the following code to the AilmentsListScreen.jsx file within the header div.

<h1>Health Tracker</h1>
{/* Test button for rage click detection - click rapidly 5+ times within 2 seconds */}
<button
  style={{
    marginTop: '10px',
    padding: '8px 16px',
    backgroundColor: '#ff4444',
    color: 'white',
    border: 'none',
    borderRadius: '4px',
    cursor: 'pointer',
    fontSize: '12px',
  }}
  onClick={(e) => {
    // Intentionally does nothing - for testing rage clicks
    e.preventDefault();
    console.log('Test button clicked (intentionally non-functional for rage click testing)');
  }}
  title="Test rage click detection: Click rapidly 5+ times within 2 seconds in the same spot"
>
  🧪 Test Rage Click (Click Rapidly)
</button>

Now that you have your test button, you can try clicking it 5+ times within 2 seconds in the same spot as shown in the session below.

Finally, you should see your full session replay under the sessions tab.

It’s important to note that all sessions for a specific app are appended in the session replay tab. So if a user is inactive and comes back to the same tab, the video will be longer. However, if the user starts a new tab or its been more than 4 hours a new video will be created for that session.

After enabling rage click, you can now:

Filter sessions by has_rage_clicks=true to find frustrated users.
Replay sessions to see exactly what caused frustration.
Identify specific UI elements that trigger rage clicks.
View correlated errors and network requests in the timeline.
Prioritize fixes based on real user frustration data.

Advanced Search Queries

# Basic: All sessions with rage clicks
has_rage_clicks=true

# Filter by page URL
has_rage_clicks=true AND visited-url contains "/checkout"

# Filter by specific HTML element clicked
clickSelector="button.submit-order"

# Filter by button text
clickTextContent contains "Place Order"

# Rage clicks with long session duration
has_rage_clicks=true AND active_length > 120s

# Combine multiple conditions
has_rage_clicks=true AND browser="Chrome" AND device_type="Desktop"

Troubleshooting

Rage clicks Not detected

Verify rage click detection is enabled in Project Settings > Observability > Session settings.
Check that your click pattern meets the configured thresholds (default: 5 clicks within 8 pixels in 2 seconds).
Ensure the LaunchDarkly client initialized successfully (check browser console).
Wait a few minutes for LaunchDarkly to process sessions.

Session replay Not Working

Verify LaunchDarkly account has observability features enabled.
Check that both plugins are properly initialized.
Ensure Content Security Policy allows connections to LaunchDarkly.
Check browser console for initialization errors.
Verify the client-side ID is correct.

Privacy and Security Considerations

Session replay is powerful, but it comes with important privacy responsibilities. LaunchDarkly provides several layers of protection to ensure you're capturing useful debugging data without exposing sensitive user information.

Default Privacy Mode: Strict Protection
By default, LaunchDarkly operates in strict privacy mode, which provides the safest option:

All text inputs are obfuscated: Form fields, text areas, and input boxes show as masked characters.
PII regex matching: Text matching patterns for emails, phone numbers, social security numbers, addresses, and credit cards are automatically masked.
Images and media preserved: Visual elements remain visible for UX debugging.

This means you can safely record sessions without worrying about capturing passwords, credit card numbers, or other sensitive data.

Key Takeaways

Implementing rage click detection with LaunchDarkly session replay is a pretty straightforward process and involves installing the observability sdk with specific plugins. The real power comes from LaunchDarkly's automatic detection and the ability to replay sessions with full context (errors, logs, network requests) to understand exactly what frustrated users.

By detecting rage clicks, we can:

Identify broken or confusing UI elements.
Understand user frustration patterns.
Prioritize fixes based on real user data.
Improve user experience proactively.

This was accomplished without any additional code changes. LaunchDarkly handled everything automatically, and you were able to adjust sensitivity through the dashboard.

What's Next: From Detection to Action

You now have the foundation for detecting user frustration with LaunchDarkly's session replay. You can:

Automatically capture rage clicks, rage scrolls, and form abandons.
Search for sessions with specific frustration patterns.
Watch full session replays with correlated errors and logs.

But detection alone isn't enough. The real magic comes from connecting these frustration signals directly to your feature releases, so you can catch issues during progressive rollouts and roll back instantly if something breaks.

In Part 2, we'll explore Guarded Releases: how to automatically monitor rage clicks during feature rollouts, set up alerts for frustration spikes, and enable automated rollback when metrics exceed thresholds.

You'll learn how to create a closed-loop system where user frustration signals trigger immediate action, which prevents small issues from becoming widespread problems.

Additional Resources:

From DevOps to Developer Advocacy: Finding My Path in the Age of AI

Alexis Roberson — Thu, 06 Nov 2025 15:22:14 +0000

When my college professor challenged me to try computer science for just one semester, I never imagined it would lead me here. I was planning to be an English major. Fast forward through a CS degree, multiple tech internships, teaching at GirlsWhoCode, and a stint as a DevOps engineer—I found myself at a crossroads, seeking something that married my technical skills with my passion for communication and teaching.

That's when a friend mentioned developer advocacy. I had never heard of it in school. But once I understood the role, I knew I'd found my calling.

The Pivot: Why DevRel?

After working as a DevOps engineer, I started exploring product management. I was drawn to the big-picture thinking: What should we build? Why should we build it? What's next? I wanted something more people-facing, something that let me connect with the broader tech community.

But here's what drew me to developer advocacy over product management: freedom and creativity.

In product management, you focus intensely on one product. In developer advocacy, you work across multiple products and solutions. You get to determine what you focus on, what content you create, and how you show developers what's possible. You're not just conceptualizing ideas, but you're building solutions, writing code, creating content, and directly engaging with the community.

For me, DevRel was the perfect marriage of my computer science background and my original desire to study English. I could code and communicate. I could build and teach.

What Does a Developer Advocate Actually Do?

When people ask me what I do, I break it down into these core areas:

Technical Writing: Creating documentation, tutorials, and guides that help developers succeed.
Coding: Building demos, sample applications, and proof-of-concepts that showcase what's possible.
Teaching/Speaking: Breaking down complex concepts and making them digestible, whether on stage, on video, or in writing.
Community Engagement: Building authentic relationships with developers, listening to their feedback, and being their voice internally.
Learning in Public: Constantly exploring new technologies and sharing the journey with others.
Bridge the gap between community and internal teams: Ensuring feedback flows both ways.

Every day is different. It's goal-driven, but you set your own metrics. As I like to say, it feels like getting a PhD—you're doing research and providing deliverables against a hypothesis you've developed.

For the Introverts: You Don't Have to Be a Social Butterfly

One question I get a lot: "Do you have to be an extrovert to do DevRel?"

Absolutely not. You don't need to be a social butterfly. What you need is authenticity and the ability to connect meaningfully with people, whether that's one-on-one, in writing, or through video content. Some of the best developer advocates I know are introverts who've found their own channels for community engagement.

It's funny because people often assume I'm naturally outgoing, which in some cases is accurate especially if I'm excited about something or an event, but the truth is I'm a mix of both an introvert and an extrovert.

I have no problem talking to people and putting myself out there when networking. At the same time, I find a lot of rest and ideas in solitude and deep thought, which, in the age of AI, is so necessary for creativity and original thinking.

If you consider yourself an introvert, I wouldn't automatically assume DevRel isn't for you. Rather, you should consider whether you enjoy teaching others, building in public (mostly online), and can translate business goals into content strategy.

What It Takes to Thrive in DevRel

Based on my journey and what I've seen work for others, here's what helps you succeed:

Comfort with ambiguity. If you're self-driven, you'll thrive. There's not a lot of hand-holding. You chart your own course.

Goal-setting discipline. Break down big goals into a roadmap. Set metrics for yourself, no matter how big or small.

Technical foundation and love for learning. A huge part of this role is learning new technologies so you can connect with different tech communities. If you don't genuinely enjoy learning, this will burn you out. Most DevRel folks have spent years working as engineers before transitioning into the field. The reason is simple—empathy.

The best way to relate to developer problems and to speak from experience is being in the trenches: writing code, debugging, searching for solutions, getting frustrated, resisting the urge to throw your laptop, going for a walk instead, coming back with one piece of insight that changes your perspective, getting a working solution, and getting to close out your 50 tabs spread across 2-3 screens.

It's a rite of passage and will help you understand how developers learn best.

Understanding your strengths. This role centers around your personal brand and building your credibility. Your company's awareness grows by association with your authentic voice.

The ability to simplify complexity. You have to take the super cool things and build bridges to people who don't understand them yet. This is where teaching ability matters most.

Authentic relationship building. People want to know you care and that what they say is valued. Authenticity beats perfection every time.

DevRel's Biggest Stage Yet: The AI Revolution

I recently came across a talk by Angie Jones, VP of Engineering at Block, where she refocused our attention to the impact of AI education on DevRel strategy and how AI should not be seen as a threat, but an opportunity to lead the charge forward.

It completely changed how I think about developer advocacy in the age of AI. She outlined how her team navigated a massive pivot when Block shifted focus to Goose, their AI agent. The lessons she shared resonated deeply with my own experience watching AI transform not just software development, but the entire DevRel landscape.

Angie's talk crystallized several truths about how AI education has reshaped DevRel's goals and focus:

Vulnerability builds trust. We all have our areas of expertise in which we can show up as the expert, but AI leveled the playing field, where the only way of maintaining authenticity was to admit that we are not AI experts. We are fellow learners who are in the trenches with you.

You do have to have a level of confidence in what you're talking about to build trust, but trust is maintained through honesty. Saying "I don't know, but I'll find out" is better than what my aunt often referred to as "speaking from the hip" and leading people down the wrong path.

Which leads to the next point that resonated with me from the talk.

Approachability is a superpower. Developers don't want gurus on pedestals. They want fellow travelers who are a few steps ahead, willing to share the messy process of tinkering with new technology. The emphasis is on being a few steps ahead.

That's what informs strategy. When we touch on the pulse of the developer community and are able to see how fast or slow the heart is beating for certain topics, we can create content that is ready for the impending demand.

Build in public. I like when Angie mentions this as the concept of showing your work. This is also tied to trust. Because AI can do so many things including write code, it can be easy to just show the final product, but the true value is in the process.

This can be intimidating at first, but what I've found is in doing so you answer the question of how, so users can move quickly to building stuff themselves without getting stuck in theory jail.

Focus on what matters. AI forced us to focus on what matters. Yes, this was the case before, but now that we can essentially build anything, we have to finetune our skills as teachers.

Now that AI is here and has delivered on its promises from previous years—where AI is no longer stuck in a chat window, but now has arms and legs to move with agency across the internet, performing tasks—we have to focus on AI education and AI enablement.

Finding Your Path Forward

Looking back at my journey from that first computer science class to where I am now, at the risk of being overly sentimental, I realize that developer advocacy found me as much as I found it. It gave me a way to combine everything I love, including building, teaching, learning, and connecting with people.

If you're considering DevRel, my advice is simple: start where you are. You don't need to have it all figured out. Share what you're learning. Build something small in public. Connect authentically with one developer at a time. The path will reveal itself.

And if you're already in DevRel, remember that we're in a unique position right now. We get to shape how an entire generation of developers thinks about and uses AI. That's not just a job. it's a responsibility and an opportunity.

The industry will keep changing. New technologies will emerge. But the core of what we do, which is helping developers succeed, building trust through authenticity, and making complex things accessible will always matter.

A Deeper Look at LaunchDarkly Architecture: More than Feature Flags

Alexis Roberson — Tue, 28 Oct 2025 21:15:14 +0000

Originally published in the LaunchDarkly Docs

Overview

When developers first encounter LaunchDarkly, they often see it as a feature flag management tool. Turns out calling LaunchDarkly a feature flag tool is like calling a Swiss Army knife "a device for opening wine bottles." Even though that would still be useful. Although technically true, you're missing about 90% of the picture.

LaunchDarkly has quietly evolved into a full feature delivery platform that happens to use flags as the foundation for four interconnected pillars: Release Management, Observability & Monitoring, and Analytics & Experimentation, and AI Configs.
Understanding how these pillars work together, including the backend infrastructure reveals why LaunchDarkly has become mission-critical for modern software delivery.

The Foundation: Feature Flag Management

At the heart of LaunchDarkly lies its feature flag management system. Think of feature flags as the control switches for your application's behavior. But unlike traditional configuration management, LaunchDarkly's flags are dynamic, real-time, and incredibly sophisticated.

Feature flag management serves as the foundation layer because it enables everything else. Without the ability to control feature visibility and behavior at runtime, none of the other pillars could function. This foundation includes:

Feature Flags: Binary or multi-variant toggles that control feature availability.
AI Configs: Dynamic configuration for AI model parameters and behaviors.
Targeting Rules: Sophisticated logic for determining who sees what features.
Context Management: User, device, and organizational context for personalized experiences.

I spent an embarrassing amount of time in my hammock thinking about why this is the foundation layer. The answer is simple: without runtime control over features, you're back to deploying code every time you want to change something. And if you've ever been on-call during a Friday deployment that went sideways, you know that's its own level of trauma.

The Four Pillars (Or how to sleep through deployments)

1. Release Management (Yellow)

The Release Management pillar focuses on safely delivering features to production. This includes:

Releases: Traditional feature rollouts with full control over timing and audience.

Guarded Rollouts: Progressive rollouts combined with real-time monitoring and automatic rollback capabilities. This is the feature that will single-handedly help you get more sleep. When you enable a guarded rollout, LaunchDarkly monitors metrics like error rates, latency, and custom business metrics. If it detects a regression, it can automatically roll back the change before users are impacted.

Progressive Rollouts: Automated gradual rollouts that increase traffic to a new feature over time (e.g., 10% -> 25% -> 50% -> 100%)

The key insight here is Release Management isn't about deploying code anymore. It's about deploying business value while your code sits safely in production, waiting for permission to run.

2. Observability & Monitoring (Blue)

This pillar answers life's most important production question: "Wait, what's happening right now?" This includes:

Session Replay: Record and replay user sessions to understand exactly what users experienced. For instance, if the user says a button didn’t work, then you can literally watch what they did.

Feature Monitoring: Track feature health, performance, and adoption in real-time.

Alerts: Proactive notifications when metrics breach thresholds.

Errors, Logs, Traces: The ultimate trio of debugging, all in one place, all correlated with which flags were active when things went sideways.

Dashboards: Customizable visualizations of all observability data.

What makes LaunchDarkly's observability unique is the feature-level granularity. Traditional monitoring says "error rate increased at 2:47pm." LaunchDarkly says "error rate increased at 2:47pm when you toggled the new-payment-processor flag to 30% rollout." One of these lets you fix the problem from your hammock. The other leads you down a git rabbit hole.

3. Analytics & Experimentation (Green)

The Analytics & Experimentation pillar helps teams make data-driven decisions:

Experimentation: Full-featured A/B testing and multivariate experiments. Run controlled experiments to measure the impact of features on business metrics.

Product Analytics: Warehouse-native analytics that integrates with your data infrastructure (like Snowflake) to provide deep insights into user behavior.

Metrics: Track both engineering metrics (error rates, latency) and business metrics (conversion, revenue, engagement).

Guarded Rollouts (also appears here): While primarily a release mechanism, Guarded Rollouts use Experimentation methodology to automatically detect regressions during rollouts.

The Experimentation pillar transforms feature flags from simple on/off switches into scientific instruments for measuring impact.

4. AI Configs (Purple)

The unveiling of AI Configs marks a shift in LaunchDarkly's shift from simply creating and storing feature flag values to storing values related to Large Language Models (LLMs). This gives way to pretty neat opportunities like customizing, testing, and rolling out new LLMs.

How the Pillars are Interconnected

The four pillars aren't just sitting next to each other making awkward small talk. They're in a deeply committed relationship with constant communication. Here's how:

Release Management -> Observability: When you toggle a flag or start a rollout, observability tools immediately begin tracking the impact. Error rates, traces, and logs are automatically correlated with the flag change.
Observability -> Analytics: The data collected through monitoring feeds directly into Experimentation and analytics. You're not just watching for errors; you're measuring business impact.
Analytics -> Release Management: Experiment results inform which variations to roll out. Metrics from guarded rollouts trigger automatic decisions (rollback or continue).
AI Configs -> All Pillars: AI configurations add a dynamic layer across the ecosystem:
To Release Management: Model versions, prompts, and parameters can be toggled like features, enabling safe AI deployments.
To Observability: Track model performance, latency, token usage, and output quality in real-time.
To Analytics: A/B test different prompts, models, or parameters to optimize AI outcomes and measure business impact.
Feature Flags Enable Everything: Without the foundational flag management system, none of these capabilities would work. Flags are the control point that makes progressive delivery, real-time monitoring, and controlled Experimentation possible.

The Infrastructure: Flag Delivery Network (FDN)

![Image of Flag Delivery Network Infrastructure.

Understanding the pillars is only half the story. The infrastructure that delivers flags to your applications is equally crucial. LaunchDarkly's Flag Delivery Network is what makes real-time feature control possible at massive scale.

What is the FDN?

The Flag Delivery Network is LaunchDarkly's proprietary infrastructure combining:

LaunchDarkly's core infrastructure: Central flag management and rule evaluation engine.
Content Delivery Network (CDN): Over 100 points of presence globally.
Multi-region streaming service: Real-time flag updates via persistent connections.

Think of it as a specialized CDN, but instead of delivering static assets, it delivers feature flag configurations and streams real-time updates.

How the FDN Works

Flag Creation: A developer, PM, or operator creates or modifies a flag in the LaunchDarkly dashboard.
Global Distribution: The flag configuration is immediately pushed to all CDN edge locations worldwide.
SDK Connection: Your application's LaunchDarkly SDK connects to the nearest CDN point of presence.
Initial Load: The SDK retrieves all flag configurations and stores them in memory.
Real-time Updates: The SDK maintains a streaming connection; when flags change, updates arrive in milliseconds.
Local Evaluation: Flag rules are evaluated locally in your application, which means no round-trip to LaunchDarkly required for each flag check.

SDK Modes: Polling vs. Streaming

LaunchDarkly SDKs can operate in two modes: streaming mode or polling mode.

Streaming Mode (recommended for server-side SDKs):

Maintains a persistent connection to the FDN.
Receives flag updates in real-time.
Ideal for long-running applications like backend services.

Polling Mode (common for mobile/client-side SDKs):

Periodically checks for flag updates.
Lower resource usage on mobile devices.
Configurable polling interval.

The Six Layers of Resilience

LaunchDarkly's architecture includes multiple layers of failover to ensure flags are always available:

In-memory cache: SDKs store flags locally; if the network fails, your app continues working.
Fallback values: Every flag evaluation includes a default fallback value.
CDN redundancy: 100+ global Points of Presence (POPs) ensure low latency and high availability.
Multi-region infrastructure: LaunchDarkly's core systems span multiple cloud regions.
SDK resilience: Automatic retry logic and circuit breakers.
Relay Proxy option: Deploy your own local flag cache for maximum control.

What this means in practice: your app never depends on LaunchDarkly being reachable. Flags work even during a complete LaunchDarkly outage.

Putting It All Together: A Real-World Scenario

Let me walk you through how this actually works in practice. Picture this: your team wants to launch a new checkout flow. In the old days (2023), this would involve:

Extensive staging testing.
A carefully planned deployment window.
Someone's weekend.
Hoping for the best.

Now, it looks more like:

Step 1: Release

Create feature flag new-checkout-flow.
Configure AI configs for the personalized product recommendation model (prompt version, temperature, model selection).
Configure targeting rules (start with your internal test accounts, because you're not a monster).
Start a guarded rollout to 10% of production traffic.

Step 2: Flag Delivery

Flag configuration and AI Config hit the FDN globally (this happens in milliseconds, which is faster than you can say "Jenkins pipeline").
Your mobile and web SDKs receive the update via their streaming connections.
Flags are evaluated locally in-memory, adding exactly zero latency to your checkout flow.

Step 3: Observability Does Its Thing

Session replay starts capturing user interactions with the new flow.
Traces show checkout API performance in real-time.
Error monitoring tracks any exceptions (there will be exceptions, there are always exceptions).
Dashboards update with adoption metrics and AI performance indicators.
Everything comes back as green.

Step 4: Analytics & Decision Making

Metrics automatically track conversion rate, cart abandonment, purchase value.
AI-specific metrics measure recommendation click-through rates and relevance scores.
Guarded rollout monitors for regressions (error rate, latency, AI hallucinations, angry user emails).
A/B test different prompt variations or model parameters to optimize recommendations.
If metrics look good: automatic progression to 25% -> 50% -> 100%.
If metrics look bad: automatic rollback before your on-call engineer finishes their coffee.
You receive a Slack notification.

Step 5: The Result

Feature safely released with automated safeguards.
AI models deployed and optimized without risking production.
Full visibility into user experience and system health.
Data-driven decision on whether to keep the feature and which AI configuration performs best.
Zero code deploys after the initial setup.
You solved this entire problem without leaving your hammock.

Why This Architecture Matters

LaunchDarkly's architecture represents a fundamental shift in how we think about software delivery:

Traditional Approach: Deploy -> Hope -> React to incidents -> Debug -> Fix -> Deploy again.

LaunchDarkly Approach: Deploy once -> Control via flags -> Monitor continuously -> Experiment safely -> Optimize based on data.

The key innovation is making the feature flag the control point for delivery, observability, Experimentation and AI Configs. This creates a closed feedback loop:

Release Management provides the controls (the steering wheel).
Observability provides the visibility (the headlights and dashboard).
Analytics provides the insights (the GPS telling you which route is actually faster).
AI Configs provide the intelligence (the adaptive cruise control that adjusts based on conditions).
Feature flags tie it all together (they're the guardrails, ensuring the car stays on the road).

Conclusion

LaunchDarkly isn't just a feature flag tool, but it's a complete feature delivery platform built on four interconnected pillars. The Flag Delivery Network ensures that control decisions made in the dashboard reach your applications globally in milliseconds, while multiple layers of resilience guarantee availability even during outages.
Understanding this architecture helps explain why LaunchDarkly has become essential infrastructure for companies that need to ship software quickly without breaking things.

The combination of real-time control, comprehensive observability, data-driven experimentation, and intelligent AI configuration management, built on a foundation of reliable, fast feature flag evaluation, enables true continuous delivery.

Whether you're managing a handful of flags, orchestrating complex progressive rollouts across a global user base, or safely deploying and iterating on AI models with the same rigor as your traditional features, LaunchDarkly's architecture scales to meet your needs while keeping the developer experience simple and the operational risk low.

Add observability to your React Native application in 5 minutes

Alexis Roberson — Wed, 01 Oct 2025 19:31:44 +0000

Originally published in the LaunchDarkly Docs

Overview

In modern application development, feature flags are the guardrails that keep experiments controlled and rollbacks safe when conditions shift. If feature flags act as the guardrails, observability provides the visibility: the headlights (traces), mirrors (logs), and dashboard instruments (metrics) that reveal what’s happening in the environment and how well a feature is performing. Together, feature flags and observability unlock powerful insights by correlating code changes with real-time system behavior. This combination reduces time-to-diagnosis and builds greater confidence when rolling out new features.

In this post, we’ll walk through just how to add observability to a React Native application using LaunchDarkly’s observability SDK. To demonstrate the process, we’ll build on the PlusOne app, a simple counter app that includes increment (+1), reset, and error-triggering buttons. This lightweight demo provides a clean foundation to showcase how logs, traces, and errors can seamlessly flow into LaunchDarkly for monitoring and debugging.

Prerequisites

LaunchDarkly account. Sign up for a free one here.
Visual Studio or another code editor of choice.

All code from this tutorial can be found on GitHub.

Setting up your environment

Before running a React Native app, make sure your development environment is set up correctly. You can find the full setup instructions for both Android and iOS here.

In this tutorial, we'll be running iOS, but keep in mind Expo Orbit, the platform we'll be using to run our iOS simulator, requires both Xcode and Android Studio to be installed.

After going through the instructions you should have the following installed:

Node JS (preferably via nvm)
Watchman for file monitoring
JDK via zulu package manager.
Android Studio. Don’t forget to set your Android_Home environment variables.
Xcode for the iOS simulator.
Cocoapods for iOS dependency management.
Expo orbit for running expo apps Android or iOS.

If you're using Android, don't forget to add your environment variables to bash or zsh profile.

export ANDROID_HOME=$HOME/Library/Android/sdk
export PATH=$PATH:$ANDROID_HOME/emulator
export PATH=$PATH:$ANDROID_HOME/platform-tools

Starting up the PlusOne app

To get started, let’s clone the repo for the PlusOne app and run npm install to ensure the proper dependencies are present in our node_modules file.

Clone the repo.

git clone https://github.com/arober39/PlusOne

Install dependencies using npm

cd PlusOne
npm install

We’ll also need to run both the prebuild command to generate the ios file and the expo run command to run the iOS simulator.

Prebuild for iOS

npx expo prebuild

Run expo app

npm expo run:ios

Now we can view the iOS app in the iPhone simulator using npm.

# iOS
npm run ios

# Android
npm run android

The app should look something like this:

Feel free to interact with the app to ensure all is working as expected.

As you can see in the code, we have three buttons: one that adds one to the displayed number, one to bring the count back to zero and an intentional Error button to test error monitoring within the LaunchDarkly UI.

// app/index.tsx

import { useState } from "react";
import { StyleSheet, Text, TouchableOpacity, View } from "react-native";

export default function Index() {
 const [count, setCount] = useState(0);

 const handleReset = () => setCount(0);
 const handleIncrement = () => setCount((prev) => prev + 1);

 const triggerRecordedError = () => {
   try {
     throw new Error("Simulated controlled error from Plus One app")
   } catch (e) {
     alert("You intentionally threw an error")
   }
 };

 return (
   <View style={styles.container}>
     <View style={styles.header}>
       <Text style={styles.headerText}>Plus One</Text>
     </View>
     <View style={styles.counterWrapper}>
       <Text style={styles.counterText}>{count}</Text>
     </View>
     <View style={styles.actionsRow}>
       <ButtonBox label="Reset" onPress={handleReset} />
       <ButtonBox label="+1" onPress={handleIncrement} />
       <ButtonBox label="Error" onPress={triggerRecordedError} />
     </View>
   </View>
 );
}

type ButtonBoxProps = {
 label: string;
 onPress: () => void;
};

function ButtonBox({ label, onPress }: ButtonBoxProps) {
 return (
   <TouchableOpacity onPress={onPress} style={styles.button} activeOpacity={0.8}>
     <Text style={styles.buttonText}>{label}</Text>
   </TouchableOpacity>
 );
}


/* The rest of the application code */

Now that we have verified a working app, we can add observability support by downloading the observability React Native SDK.

Install LaunchDarkly SDK dependencies.

npm install @launchdarkly/react-native-client-sdk
npm install @launchdarkly/observability-react-native

Next, you’ll need to initialize the React Native LD client in the _layout file. Replace the in the layout file by pasting the following code.

// app/_layout.tsx

import { Observability } from '@launchdarkly/observability-react-native';
import { AutoEnvAttributes, LDOptions, LDProvider, ReactNativeLDClient } from '@launchdarkly/react-native-client-sdk';
import { Stack } from 'expo-router';
import { useEffect, useState } from 'react';

const options: LDOptions = {
 applicationInfo: {
   id: 'Plus-One',
   name: 'Sample Application',
   version: '1.0.0',
   versionName: 'v1',
 },
 debug: true,
 plugins: [
   new Observability({
     serviceName: 'my-react-native-app',
     serviceVersion: '1.0.0',
   })
 ],
};

const userContext = { kind: 'user', key: 'test-hello' };

export default function RootLayout() {
 const [client, setClient] = useState<ReactNativeLDClient | null>(null);
  useEffect(() => {
   // Initialize client
   const featureClient = new ReactNativeLDClient(
     'mob-abc123',
     AutoEnvAttributes.Enabled,
     options,
   );

   featureClient.identify(userContext).catch((e: any) => console.log(e));

   setClient(featureClient);

   // Cleanup function that runs when component unmounts
   return () => {
     featureClient.close();
   };
 }, []);

 if (!client) {
   return null;
 }
 return (
   <LDProvider client={client}>
     <Stack />
   </LDProvider>
 );
}

First, we’re importing the Observability SDK as well as a few LD libraries to add options and attributes to the LD client.- Initialized the SDK and plugin options.

Defined the user context.
Lastly, you initialized the client.

Now that you have defined your LD React Native client, you can implement different observability methods within your application logic.

We can do this by importing the LDObserve library in the app/_layout.tsx file.

import { LDObserve } from '@launchdarkly/observability-react-native';

Then, add the recordError() method within the triggerRecordedError function inside the app/_layout.tsx file. This will allow for error messages to be sent back to the LD UI.

 const triggerRecordedError = () => {
   try {
     throw new Error("Simulated controlled error from Plus One app")
   } catch (e) {
     LDObserve.recordError(e as Error, {feature: "test-button"})
     alert("You intentionally threw an error")
   }
 };

Before being able to receive data in the LD UI, you’ll need to add your mobile key to the React Native LD client, which can be found by logging in to the LD UI.

Once logged in, tap the settings button at the bottom left.

Navigate to the Projects page and click create to create a new project.

Define the new Project and click Create Project.

Then, define the environment where you would like your data to be sent.

Now, grab the mobile key by pressing the three dots for the environment and selecting the mobile key, which will copy the key to your keyboard.

Then, add it to the app/_layout file.

   const featureClient = new ReactNativeLDClient(
     ‘mob-abc123’,
     AutoEnvAttributes.Enabled,
     options,
   );

Finally, you can generate data by interacting with your app in the iOS app simulator.

Feel free to restart the app to ensure data is displaying in real time.

npm expo run:ios

Once you navigate back to the LD UI, you should be able to see the logs, traces, and errors under the Monitor section.

Logs

Traces

Errors

Conclusion

In just a few minutes, we’ve taken the PlusOne React Native app from a simple counter to a fully observable application connected to LaunchDarkly. By setting up the SDK, initializing observability plugins, and recording errors, we now have a live feedback loop where application behavior is visible in the LaunchDarkly UI. This makes it far easier to diagnose issues, validate feature flag rollouts, and ensure smooth user experiences.

Next Steps

Looking ahead, there are many ways to expand on what we’ve built by including features like recording custom metrics and session replay, which provide even deeper insights into app behavior. By integrating observability at the foundation of your React Native projects, you equip your team with the clarity needed to debug faster, ship features more confidently, and deliver reliable experiences to your users.

You can also read this article to learn more about observability and guarded releases.