Forem: Uptrace

OpenTelemetry OpenAI Instrumentation

Alexandr Bandurchin — Thu, 09 Apr 2026 07:09:53 +0000

By adding OpenTelemetry instrumentation to the OpenAI SDK, you get automatic tracing for every API call — model name, token usage, finish reason, and errors — without modifying your existing OpenAI code. This data flows into any OpenTelemetry APM including Uptrace.

Quick Setup

Step	Action	Code/Command
1. Install	Install the official OTel instrumentation package	`pip install opentelemetry-instrumentation-openai-v2`
2. Instrument	Call instrument() before any OpenAI calls	`OpenAIInstrumentor().instrument()`
3. Configure	Point the exporter at your backend	See exporter setup below
4. Run	Make OpenAI calls as normal	Spans captured automatically

Minimal working example:

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

# Call once at application startup, before any OpenAI API calls
OpenAIInstrumentor().instrument()

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5",
    messages=[{"role": "user", "content": "What is OpenTelemetry?"}]
)

This single call automatically creates a span for every client.chat.completions.create(), client.embeddings.create(), and other OpenAI SDK operations your code makes.

Two packages exist. opentelemetry-instrumentation-openai-v2 is the official OpenTelemetry implementation (maintained in the opentelemetry-python-contrib repo). The older opentelemetry-instrumentation-openai is a community package by Traceloop/OpenLLMetry — still maintained but not the OTel standard. Use the -v2 package for new projects.

What Gets Captured

The instrumentation populates standard OpenTelemetry GenAI semantic convention attributes on each span:

Attribute	Example value	Description
`gen_ai.system`	`openai`	AI provider
`gen_ai.operation.name`	`chat`	Operation type
`gen_ai.request.model`	`gpt-5`	Requested model
`gen_ai.response.model`	`gpt-5`	Actual model version returned by API
`gen_ai.usage.input_tokens`	`142`	Prompt tokens consumed
`gen_ai.usage.output_tokens`	`38`	Completion tokens generated
`gen_ai.response.finish_reason`	`stop`	Why generation ended

These attributes follow the OpenTelemetry GenAI semantic conventions and are supported by Uptrace, Datadog (v1.37+), and Grafana. The gen_ai.usage.input_tokens and gen_ai.usage.output_tokens values are the foundation for LLM cost monitoring — multiply them by model price per token to get per-request spend.

Full Setup with OTLP Exporter

Note on Examples: This guide uses Uptrace as the OpenTelemetry backend in code examples. OpenTelemetry is vendor-neutral and works with any OTLP-compatible backend (Jaeger, Grafana Tempo, Prometheus, etc.). Simply replace the endpoint and headers with your preferred backend configuration.

A complete setup configures the SDK, adds a span processor, and instruments OpenAI before your application logic runs:

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

def setup_tracing():
    exporter = OTLPSpanExporter(
        endpoint=os.environ["OTLP_ENDPOINT"],   # e.g. https://api.uptrace.dev:4317
        headers={"uptrace-dsn": os.environ["UPTRACE_DSN"]},
    )
    provider = TracerProvider()
    provider.add_span_processor(BatchSpanProcessor(exporter))
    trace.set_tracer_provider(provider)

    OpenAIInstrumentor().instrument()

setup_tracing()

After this, all OpenAI SDK calls in the process produce spans automatically — no changes to existing code required.

Capturing Prompt and Completion Content

By default the instrumentation records token counts and metadata but not the actual message content, to avoid accidentally logging sensitive data. To capture prompts and completions as span events, set the environment variable before starting your application:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

With this enabled, each span gets log events containing the full input messages and the model's response text. Use this in development and debugging environments; evaluate carefully before enabling in production where prompts may contain PII.

Streaming Responses

The instrumentation handles streaming responses. Token counts are recorded when the stream completes:

with client.chat.completions.stream(
    model="gpt-5",
    messages=[{"role": "user", "content": "List 5 observability best practices"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

# gen_ai.usage.input_tokens and gen_ai.usage.output_tokens are
# set on the span after the stream closes

Adding Custom Attributes

For context beyond what the auto-instrumentation captures — user ID, feature flag, request source — add attributes to the current span:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def generate_summary(text: str, user_id: str) -> str:
    with tracer.start_as_current_span("summarize") as span:
        span.set_attribute("app.user_id", user_id)
        span.set_attribute("app.input_length", len(text))

        response = client.chat.completions.create(
            model="gpt-5.4-nano",
            messages=[
                {"role": "system", "content": "Summarize the following text."},
                {"role": "user", "content": text},
            ]
        )
        return response.choices[0].message.content

The auto-instrumented gen_ai.* attributes appear as child spans under summarize, giving you both application context and model-level detail in the same trace.

Async Support

The instrumentation works with the async OpenAI client without additional configuration:

from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def async_chat(prompt: str) -> str:
    response = await async_client.chat.completions.create(
        model="gpt-5",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

Sending Traces to Uptrace

Uptrace accepts OpenAI traces via OTLP and stores gen_ai.* attributes as queryable fields. Find your DSN under Project Settings → Connection details in the Uptrace dashboard:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="https://api.uptrace.dev:4317",
    headers={"uptrace-dsn": "https://<token>@api.uptrace.dev?grpc=4317"},
)

In Uptrace you can filter traces by gen_ai.request.model, group by model to compare token usage, and sum gen_ai.usage.input_tokens + gen_ai.usage.output_tokens per trace to track consumption over time. If you're using LangChain, combine this setup with the LangChain observability guide to trace chain and agent structure alongside individual OpenAI calls.

JavaScript / TypeScript

The equivalent package for Node.js is @opentelemetry/instrumentation-openai (community maintained):

import { OpenAIInstrumentation } from '@opentelemetry/instrumentation-openai';
import { NodeSDK } from '@opentelemetry/sdk-node';

const sdk = new NodeSDK({
  instrumentations: [new OpenAIInstrumentation()],
});

sdk.start();

The same gen_ai.* attributes are captured on the Node side.

LLM Cost Monitoring with OpenTelemetry

Alexandr Bandurchin — Thu, 09 Apr 2026 06:37:46 +0000

Teams running LLM applications in production face a cost problem that traditional APM tools were never designed to solve. CPU and memory costs are relatively predictable — a web service processing 1,000 requests per second costs roughly the same week over week. LLM API costs are not. A single user session can cost $0.01 or $5 depending on prompt length, model choice, conversation history, and how many retries happen inside your chain. Without instrumentation, cost anomalies are invisible until the monthly invoice.

The standard pattern: a team launches a feature using GPT-5, everything looks fine in staging, and then production traffic reveals that a small percentage of requests trigger long multi-turn conversations that cost 50× more than the average. By the time the bill arrives, the cost has already happened.

OpenTelemetry's GenAI semantic conventions solve this at the instrumentation layer. The gen_ai.usage.input_tokens and gen_ai.usage.output_tokens attributes are captured automatically per API call, giving you token-level visibility that you can turn into dollar figures, per-request cost breakdowns, and budget alerts — using the same observability stack you already have.

Why Standard APM Misses LLM Costs

Traditional APM tracks latency, error rates, and throughput. These metrics are meaningful for LLM applications too, but they say nothing about financial cost. A request that takes 3 seconds and costs $0.002 looks identical in APM to one that takes 3 seconds and costs $0.40. Both have the same latency. Only token counts tell you the difference.

Three things make LLM costs hard to track without dedicated instrumentation:

Token consumption is buried inside SDK calls. Unless you manually read response.usage after every API call and record it somewhere, the data never appears in your traces or metrics. Most applications don't do this consistently.

Costs happen across chained calls. A LangChain agent might make 8 OpenAI calls to answer a single user question. The cost of the full interaction is the sum of all 8, but standard tracing only shows individual requests — not their aggregate cost under a parent operation.

Model prices vary widely and change. GPT-5.4 costs 12× more per input token than GPT-5.4-nano ($2.50 vs $0.20 per 1M tokens). Reasoning models like o3 and o4-mini bill internal "thinking" tokens that never appear in the response but still cost money. If your application conditionally uses different models, you need model-level attribution to understand your cost structure.

LLM Pricing Reference

Current pricing for the most common models (April 2026 — always verify against provider docs as prices change):

Model	Input (per 1M tokens)	Output (per 1M tokens)	Notes
gpt-5.4	$2.50	$15.00	OpenAI flagship (Mar 2026)
gpt-5	$1.25	$10.00	Good balance of cost and capability
gpt-5.4-mini	$0.75	$4.50	Mid-tier, good for most tasks
gpt-5.4-nano	$0.20	$1.25	Lowest cost in GPT-5.4 family
o3	$2.00	$8.00	Reasoning model — see note below
o4-mini	$1.10	$4.40	Compact reasoning model
claude-sonnet-4.6	$3.00	$15.00	Anthropic recommended
claude-haiku-4.5	$1.00	$5.00	Anthropic budget tier
gemini-2.5-pro	$1.25	$10.00	Contexts under 200K tokens

Reasoning models (o3, o4-mini) require special handling. These models use internal "reasoning tokens" during inference that are billed as output tokens but not returned in the response. gen_ai.usage.output_tokens includes these hidden tokens, so actual cost can be significantly higher than visible completion length suggests. Set conservative alert thresholds for o-series models and treat output token counts as an upper bound on reasoning effort.

Output tokens are consistently more expensive than input tokens — 4–8× for most models. Applications generating long completions (code, detailed explanations) have very different cost profiles from those producing short factual answers.

Capturing Token Usage with OpenTelemetry

The opentelemetry-instrumentation-openai-v2 package automatically records gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on every span. No manual response parsing required:

from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter())
)
trace.set_tracer_provider(provider)

OpenAIInstrumentor().instrument()

# All subsequent OpenAI calls are automatically traced with token counts
from openai import OpenAI
client = OpenAI()

Each span now carries the token breakdown:

Span: gen_ai.operation.name = "chat"
  gen_ai.system              = "openai"
  gen_ai.request.model       = "gpt-5"
  gen_ai.usage.input_tokens  = 312
  gen_ai.usage.output_tokens = 87
  gen_ai.response.finish_reason = "stop"

For Anthropic, the equivalent package is opentelemetry-instrumentation-anthropic. Both emit the same gen_ai.* attributes, so your queries and dashboards work across providers. For full setup instructions and available options, see the OpenAI instrumentation guide.

Calculating Cost Per Request

With token counts on spans, cost calculation is straightforward. Add it as a custom span attribute so it's queryable alongside everything else:

from opentelemetry import trace

# Keep pricing in one place — update when providers change rates
MODEL_PRICING = {
    "gpt-5.4":              {"input": 2.50,  "output": 15.00},
    "gpt-5":                {"input": 1.25,  "output": 10.00},
    "gpt-5.4-mini":         {"input": 0.75,  "output": 4.50},
    "gpt-5.4-nano":         {"input": 0.20,  "output": 1.25},
    "o3":                   {"input": 2.00,  "output": 8.00},
    "o4-mini":              {"input": 1.10,  "output": 4.40},
    "claude-sonnet-4-6":    {"input": 3.00,  "output": 15.00},
    "claude-haiku-4-5":     {"input": 1.00,  "output": 5.00},
    "gemini-2.5-pro":       {"input": 1.25,  "output": 10.00},
}

def calculate_cost_usd(model: str, input_tokens: int, output_tokens: int) -> float:
    pricing = MODEL_PRICING.get(model, {"input": 0.0, "output": 0.0})
    return (input_tokens * pricing["input"] + output_tokens * pricing["output"]) / 1_000_000

tracer = trace.get_tracer(__name__)

def chat_with_cost(prompt: str, model: str = "gpt-5") -> str:
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("gen_ai.request.model", model)

        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}]
        )

        input_tokens = response.usage.prompt_tokens
        output_tokens = response.usage.completion_tokens
        cost = calculate_cost_usd(model, input_tokens, output_tokens)

        span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
        span.set_attribute("llm.cost.usd", cost)

        return response.choices[0].message.content

The llm.cost.usd attribute is now queryable in your observability backend: filter by model, sum over time ranges, group by service or user.

Tracking Total Cost per Agent Run

When a single user operation triggers multiple LLM calls — a LangChain agent, a multi-step chain, any orchestrated workflow — you want the total cost of the full interaction, not just individual calls. Wrap the operation in a parent span and aggregate:

def run_research_agent(question: str, user_id: str) -> str:
    with tracer.start_as_current_span("agent.run") as parent_span:
        parent_span.set_attribute("app.user_id", user_id)
        parent_span.set_attribute("app.operation", "research")

        total_cost = 0.0
        total_input_tokens = 0
        total_output_tokens = 0

        # Step 1: decompose the question (cheap model)
        with tracer.start_as_current_span("agent.decompose") as span:
            response = client.chat.completions.create(
                model="gpt-5.4-nano",
                messages=[{"role": "user", "content": f"Break this into sub-questions: {question}"}]
            )
            step_cost = calculate_cost_usd(
                "gpt-5.4-nano",
                response.usage.prompt_tokens,
                response.usage.completion_tokens
            )
            span.set_attribute("llm.cost.usd", step_cost)
            total_cost += step_cost
            total_input_tokens += response.usage.prompt_tokens
            total_output_tokens += response.usage.completion_tokens
            sub_questions = response.choices[0].message.content

        # Step 2: answer each sub-question (full model)
        with tracer.start_as_current_span("agent.answer") as span:
            response = client.chat.completions.create(
                model="gpt-5",
                messages=[{"role": "user", "content": sub_questions}]
            )
            step_cost = calculate_cost_usd(
                "gpt-5",
                response.usage.prompt_tokens,
                response.usage.completion_tokens
            )
            span.set_attribute("llm.cost.usd", step_cost)
            total_cost += step_cost
            total_input_tokens += response.usage.prompt_tokens
            total_output_tokens += response.usage.completion_tokens
            answer = response.choices[0].message.content

        # Record totals on the parent span
        parent_span.set_attribute("llm.cost.usd", total_cost)
        parent_span.set_attribute("llm.total_input_tokens", total_input_tokens)
        parent_span.set_attribute("llm.total_output_tokens", total_output_tokens)

        return answer

With this structure you can query both individual step costs and total operation cost from the same trace.

Recording Cost as an OpenTelemetry Metric

Spans are good for per-request cost. For aggregate spend over time — daily cost, cost by model, cost rate anomalies — OpenTelemetry metrics are the right tool. A counter accumulates continuously and can be queried for any time window:

from opentelemetry import metrics

meter = metrics.get_meter(__name__)

# Counter for total cost in USD
cost_counter = meter.create_counter(
    name="llm.cost.usd",
    description="Cumulative LLM API cost in USD",
    unit="USD",
)

# Histogram for per-request cost distribution
cost_histogram = meter.create_histogram(
    name="llm.cost.per_request.usd",
    description="Cost distribution per LLM request",
    unit="USD",
)

def tracked_completion(model: str, messages: list) -> str:
    response = client.chat.completions.create(model=model, messages=messages)

    cost = calculate_cost_usd(
        model,
        response.usage.prompt_tokens,
        response.usage.completion_tokens
    )

    labels = {"gen_ai.request.model": model, "service.name": "my-service"}
    cost_counter.add(cost, labels)
    cost_histogram.record(cost, labels)

    return response.choices[0].message.content

The counter gives you cumulative spend that you can diff over any window. The histogram shows your cost distribution — whether you have occasional expensive outlier requests or a uniformly expensive workload. For broader AI metrics patterns — GPU utilization, inference latency histograms, sampling strategies for high-volume workloads — see OpenTelemetry for AI Systems.

Cost Visibility in LangChain Applications

For LangChain chains and agents, LangChainInstrumentor captures spans for each chain step. Combine it with per-call cost attribution using the pattern above. For a deeper walkthrough of LangChain-specific monitoring patterns including silent failure detection, see the LangChain observability guide.

from opentelemetry.instrumentation.langchain import LangChainInstrumentor
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

# Both instrumentors together: LangChain provides chain structure,
# OpenAI instrumentation provides token counts on each LLM call
LangChainInstrumentor().instrument()
OpenAIInstrumentor().instrument()

With both active, your trace shows the chain as the parent span and individual LLM calls — with gen_ai.usage.* attributes — as children. You can sum token counts across children to derive chain-level cost.

Cost Dashboards and Alerts in Uptrace

Uptrace stores gen_ai.usage.* and custom llm.cost.usd attributes as queryable numeric fields in ClickHouse. Once traces and metrics are flowing, useful queries include:

Daily cost by model:
Group llm.cost.usd metric by gen_ai.request.model, sum over 24h. This shows which model drives the most spend and whether usage shifted after a deployment.

P99 cost per agent run:
Filter parent spans with app.operation = "research", take the 99th percentile of llm.cost.usd. High P99 means a small percentage of runs is generating disproportionate cost.

Cost rate alert:
Alert when the rate of llm.cost.usd counter exceeds your threshold — for example, if hourly spend exceeds $50 when normal is under $10. This catches runaway loops or unexpected traffic spikes before they compound.

Configure your Uptrace DSN and OTLP endpoint via the getting started guide to begin streaming telemetry.

Cost Optimization Signals

Collected cost data reveals optimization opportunities that are invisible without instrumentation:

Model downgrade candidates. Compare response quality versus cost across models for your specific use cases. If gpt-5.4-mini handles 80% of your requests acceptably at 3× lower cost than GPT-5.4, routing those requests to the cheaper model has an immediate impact. gpt-5.4-nano reduces input cost by 12× for simple classification or extraction tasks.

Prompt length outliers. High gen_ai.usage.input_tokens on specific endpoints points to prompts that have grown with accumulated context or system prompt bloat. Trimming 200 tokens from a system prompt that runs on every request saves proportionally.

Retry amplification. If your error handling retries failed LLM calls, each retry doubles or triples the cost of that request. Token-level tracing makes retry patterns visible — high input token counts on spans with errors often indicate retry loops.

Conversation history accumulation. Chat applications that include full conversation history in every prompt pay linearly more as conversations grow. Seeing gen_ai.usage.input_tokens increase monotonically across a session identifies this pattern.

7 Best End-to-End Monitoring Tools

Alexandr Bandurchin — Thu, 02 Apr 2026 08:03:55 +0000

What Is End-to-End Monitoring

Quick Answer: The best end-to-end monitoring tools in 2026 are Uptrace (OpenTelemetry-native), Datadog (enterprise), and Grafana Cloud (open source). Choose based on budget, team size, and vendor preference.

End-to-end monitoring (also known as end to end application monitoring, end to end application performance monitoring, or end to end performance monitoring) is the practice of observing and analyzing the full delivery chain of an application — from the end user's browser to backend services, networks, and infrastructure.

Unlike traditional APM tools that focus mainly on backend performance, end-to-end monitoring connects frontend, backend, and infrastructure signals in a single view. This helps teams understand not only what failed but also where and why.

Key data types collected include:

Distributed traces to follow a request across services
Metrics to monitor application and infrastructure health
Logs to provide debugging context
RUM (Real User Monitoring) to capture frontend performance

This holistic visibility allows engineering teams to detect performance degradation early, correlate incidents across multiple layers, and deliver more stable, faster user experiences. End-to-end monitoring is especially valuable in environments with microservices, serverless functions, and multi-cloud deployments.

Why It Matters

Modern systems are complex: microservices, serverless, multi-cloud, and edge architectures. Monitoring only one layer no longer works and problems often appear at the intersection of services.

End-to-end monitoring addresses this by:

Correlating data sources across frontend, backend, and infrastructure
Reducing blind spots that occur with isolated monitoring tools
Supporting open standards like OpenTelemetry for better interoperability

As organizations adopt open standards and hybrid architectures, end-to-end monitoring becomes a foundation for reliable performance and incident response.

Key Capabilities of End-to-End Monitoring Tools

Full-stack visibility across apps, infra, and frontend
Real-time tracing and error analysis
Metrics collection and alerting
Log aggregation and correlation
RUM integration for frontend performance
OpenTelemetry compatibility
Flexible deployment (SaaS or self-hosted)

🏆 Top 7 End-to-End Monitoring Tools in 2026

Uptrace

Type: OpenTelemetry-native observability platform

Deployment: Self-hosted or SaaS

Best for: Teams wanting OpenTelemetry-native observability without vendor lock-in

Uptrace APM focuses on full-stack observability without vendor lock-in. Built around OpenTelemetry from the ground up, it ingests traces, metrics, logs, and RUM data without requiring proprietary agents. The platform uses ClickHouse as a storage engine, enabling fast queries even at scale.

Key strengths:

Native OpenTelemetry support (no translation layers)
Unified view of traces, metrics, logs, and RUM
Lightweight and cost-efficient
Open source + managed cloud

Limitations:

Smaller integration ecosystem compared to enterprise tools
Requires some setup for advanced alerting and automation

Datadog

Type: Enterprise SaaS monitoring platform

Best for: Large enterprises needing comprehensive integrations and support

Datadog offers a comprehensive end-to-end monitoring solution that combines APM, infrastructure metrics, logs, security, RUM, and synthetic monitoring. Known for its broad integration ecosystem and strong dashboards, it's widely used by large engineering teams.

Key strengths:

Rich integrations and plugins
All-in-one platform (APM, infra, RUM, security)
Real-time alerting and anomaly detection

Limitations:

Vendor lock-in and proprietary agents
Can become expensive at scale

Dynatrace

Type: Enterprise observability platform

Best for: Large-scale deployments requiring AI-driven automation

Dynatrace uses AI-powered monitoring and automated instrumentation to provide deep visibility into large-scale distributed systems. Its OneAgent automatically discovers components, and the Davis AI engine helps detect root causes.

Key strengths:

Automatic instrumentation
Strong AI-driven analysis
Deep application and infrastructure insights

Limitations:

High cost for large deployments
Heavily tied to its proprietary agent

Grafana Cloud

Type: Open source observability stack (managed)

Best for: Teams already using Prometheus or Grafana

Grafana Cloud offers a managed version of the popular open source stack: Prometheus (metrics), Loki (logs), Tempo (traces), and Grafana dashboards. It's ideal for teams already using Prometheus or Grafana.

Key strengths:

True open source foundations
Strong visualization capabilities
Good OpenTelemetry compatibility

Limitations:

Requires PromQL/LogQL knowledge
Can get complex at scale

New Relic

Type: Commercial observability platform

Best for: Mid-size teams wanting an all-in-one solution with a generous free tier

New Relic provides a unified observability platform with strong support for APM, infrastructure monitoring, logs, RUM, and synthetic checks. It supports OpenTelemetry ingestion and is widely adopted by mid-size and enterprise teams.

Key strengths:

OpenTelemetry support
Generous free tier
Good coverage across frontend and backend

Limitations:

Per-user pricing can limit access for larger teams
UI complexity

AppDynamics

Type: Enterprise APM

Best for: Enterprises focused on business transaction monitoring

AppDynamics (owned by Cisco) focuses primarily on APM with strong application mapping, business transaction monitoring, and alerting capabilities. It integrates with backend, frontend, and infrastructure layers.

Key strengths:

Strong APM and transaction mapping
Enterprise-grade support
Good visibility into backend performance

Limitations:

Less flexible than modern OpenTelemetry-based tools
High enterprise pricing

Prometheus + Grafana

Type: DIY observability stack

Best for: Teams wanting full control and avoiding vendor lock-in

The combination of Prometheus and Grafana remains a popular choice for teams wanting full control over their stack. It provides powerful metrics collection, alerting, and visualization — and can be extended with tracing (Tempo) and logging (Loki).

Key strengths:

Fully open source
Flexible and extensible
Ideal for infrastructure and backend monitoring

Limitations:

High operational overhead
Requires manual integration of components

Feature Comparison Table

Tool	OpenTelemetry	Traces	Metrics	Logs	RUM	Self-Host	SaaS	Best For
Uptrace	✅ Native	✅	✅	✅	✅	✅	✅	OpenTelemetry-native teams
Datadog	⚠️ Partial	✅	✅	✅	✅	❌	✅	Enterprise all-in-one
Dynatrace	⚠️ Partial	✅	✅	✅	✅	❌	✅	AI-driven automation
Grafana Cloud	✅	✅	✅	✅	✅	⚠️	✅	Open source stack users
New Relic	✅	✅	✅	✅	✅	❌	✅	Startups with free tier needs
AppDynamics	⚠️ Partial	✅	✅	✅	⚠️	❌	✅	Business transaction monitoring
Prometheus+Grafana	✅ (via OTel)	⚠️	✅	⚠️	❌	✅	❌	Full control, no vendor lock-in

How to Choose the Right End-to-End Monitoring Tool

Selecting the right monitoring tool depends on your technical requirements, team size, and budget constraints.

For small teams and startups:

Start with tools that offer generous free tiers or low operational overhead. New Relic and Grafana Cloud are solid choices. If you're already using OpenTelemetry, Uptrace provides a cost-efficient path without vendor lock-in.

For mid-size teams:

Look for platforms that balance features with cost. Uptrace and Grafana Cloud work well here, especially if you value open standards. Datadog is a strong option if you need extensive integrations and are willing to pay for them.

For enterprises:

Prioritize platforms with strong support, compliance features, and enterprise-grade SLAs. Datadog, Dynatrace, and AppDynamics are built for this scale, though they come with higher costs. If avoiding vendor lock-in is critical, consider Uptrace or self-hosted Prometheus + Grafana.

Key decision factors:

Budget: Self-hosted solutions (Uptrace, Prometheus) are cheaper at scale; SaaS tools have predictable pricing but can get expensive
Team expertise: Grafana and Prometheus require more hands-on management; Datadog and Dynatrace are easier to adopt
OpenTelemetry commitment: If you're invested in OpenTelemetry, choose native-first tools like Uptrace or Grafana Cloud
Vendor lock-in tolerance: Proprietary platforms (Datadog, Dynatrace) make migration harder; open standards reduce risk

FAQ

What is an example of end-to-end monitoring?

End-to-end monitoring might include collecting frontend performance metrics through RUM (Real User Monitoring) tools, tracing API calls through microservices, measuring infrastructure resource usage, and correlating it all into one view.

How does end-to-end monitoring differ from APM?

APM typically focuses on application performance. End-to-end monitoring includes frontend, network, infrastructure, and external dependencies — offering a more complete view.

How much does end-to-end monitoring cost?

Pricing varies by platform and data volume. Open source solutions like Prometheus or Uptrace self-hosted can be cost-efficient, while enterprise SaaS tools may charge by data volume, host, or user.

Can I build end-to-end monitoring with OpenTelemetry?

Yes. OpenTelemetry provides the instrumentation layer to collect traces, metrics, and logs from multiple services and send them to a backend like Uptrace, Grafana, or Datadog.

What's the best tool for small teams?

For small teams, Uptrace or Grafana Cloud can be good options due to their flexibility, OpenTelemetry compatibility, and lower operational overhead.

Final Thoughts

End-to-end monitoring has become a critical capability for modern systems. With growing system complexity, teams need visibility across the entire stack — frontend, backend, network, and infrastructure.

Open standards like OpenTelemetry make it possible to build or adopt monitoring solutions without being tied to a single vendor. Platforms like Uptrace offer open, flexible observability, while enterprise SaaS tools like Datadog and Dynatrace deliver integrated, ready-to-use solutions.

The best choice depends on your team's size, budget, and control requirements. But one thing is clear — end-to-end monitoring is no longer optional. It's the backbone of reliable, performant systems in 2026.

Ready to start? Try Uptrace free.

How to Choose an APM Solution: 5 Critical Questions

Alexandr Bandurchin — Thu, 02 Apr 2026 07:49:23 +0000

What is an APM Solution?

An APM solution, or Application Performance Monitoring tool, is a software application that helps businesses monitor and manage the performance and availability of software applications. APM tools gather data from systems, servers, databases, APIs, and end-user devices to provide deep insights into the root causes of performance issues.

APM solutions have evolved far beyond basic monitoring. They now offer AI-driven analytics, predictive performance forecasting, and automated issue resolution to ensure software reliability in real time.

Role of APM

DevOps and Site Reliability Engineering (SRE) teams depend on APM solutions to maintain service-level objectives (SLOs), debug production issues quickly, and improve release velocity. Exploring available SRE tools can help you build a comprehensive monitoring strategy. APM has become the backbone of DevOps pipelines, offering continuous feedback for code quality, performance regressions, and more.

Opentelemetry-native APM platforms combine metrics, traces, and logs into a unified observability pipeline, helping teams identify performance bottlenecks faster while keeping infrastructure visibility simple and cost-effective.

Why Businesses Need APM Solutions

Key Benefits for DevOps and IT

✅ Early detection of anomalies before users are impacted
✅ Reduced mean time to resolution (MTTR) through automated root cause analysis
✅ Enhanced collaboration across dev, ops, and QA teams

Impact on UX and System Uptime

A slow or buggy app can lose customers in seconds. APM solutions ensure seamless performance by proactively identifying and fixing latency, crashes, or errors. This results in higher user satisfaction, better app store ratings, and improved conversion rates.

Cost Reduction and Optimization

By detecting inefficient code or misconfigured infrastructure, APM tools help businesses cut unnecessary cloud spending and avoid downtime-related losses. This significantly improves the organization's bottom line.

Critical Question #1 – What Are Your Business Objectives?

Align APM with Goals

Not every APM solution fits every business. Start by defining your primary goals—whether it's improving load times, ensuring 99.99% uptime, or gaining visibility into microservices. Choose an APM tool that maps to those objectives and scales as your business grows.

Common Use Cases by Industry

E-commerce: Reducing cart abandonment via faster page loads
Healthcare: Ensuring uptime for telehealth portals
SaaS: Monitoring user behavior and onboarding flows
Finance: Detecting latency in real-time transaction systems

Critical Question #2 – What Features Matter Most to You?

Core Features for 2026

When evaluating APM solutions, feature depth and flexibility are paramount. The most critical features include:

End-to-End Transaction Tracing: Visualizes user journeys through every microservice
AI-Powered Anomaly Detection: Identifies issues before they escalate
Dynamic Baseline Monitoring: Adjusts thresholds automatically based on normal behavior
Custom Dashboards & Alerts: Tailor insights and notifications to team needs
Multi-environment Visibility: Monitor performance across staging, production, and hybrid environments

Real-Time Monitoring and AI Insights

Modern APM solutions don't just detect issues—they help resolve them. Look for platforms offering:

✅ Real-time metrics and logs aggregation
✅ Heatmaps and flame charts for deep diagnostics
✅ AI-generated remediation suggestions

These tools empower teams to swiftly pinpoint bottlenecks and apply intelligent fixes, minimizing downtime and business impact.

Critical Question #3 – How Well Does It Integrate with Your Stack?

Environment Compatibility

A good APM solution must integrate smoothly with your existing infrastructure:

Cloud-native: AWS, Azure, GCP
On-premise: Traditional setups
Hybrid: Mixed environments

Ensure the APM tool provides SDKs, agents, or APIs for your technology stack and supports containerized and serverless environments.

DevOps Integration

APM tools that integrate directly into CI/CD pipelines, incident response platforms (like PagerDuty), and collaboration tools (like Slack or Jira) enhance workflow automation and overall team productivity.

Critical Question #4 – What Is the Total Cost of Ownership (TCO)?

Beyond Subscription Fees

When comparing APM solutions, don't just consider the monthly license cost. Factor in:

💰 Onboarding and configuration time
💰 Training costs for engineers
💰 Maintenance and scaling fees
💰 Support plan costs

Understanding the long-term investment helps avoid surprises and ensures sustainable adoption.

ROI Considerations

A strong APM solution can reduce engineering hours spent on debugging, increase uptime, and improve user retention. Calculate expected returns by estimating:

⚡ MTTR reduction
⚡ Developer productivity gains
⚡ Revenue preserved from reduced downtime

Critical Question #5 – What Is the Vendor's Track Record and Support Quality?

Vendor Reliability

Support is a deal-breaker. Choose vendors who offer:

✅ 99.9%+ SLA guarantees
✅ Proven uptime histories
✅ Enterprise-grade encryption and security

A vendor's history with enterprise clients and their ability to adapt quickly to market needs speaks volumes about long-term viability.

Support Ecosystem

An active user community, detailed documentation, and prompt support services can significantly reduce learning curves and enhance troubleshooting. Bonus points if the vendor offers forums, webinars, or certifications.

Comparing Popular APM Solutions in 2026

APM Tool	Strengths	Use Case
Uptrace	Open-source, cost-effective, OpenTelemetry native	Small to medium teams
Datadog	Seamless cloud integration, observability suite	DevOps in multi-cloud
New Relic	Flexible pricing, powerful dashboarding	SMBs and startups
Dynatrace	AI engine (Davis), enterprise scalability	Large enterprises
AppDynamics	Deep code diagnostics, part of Cisco	Hybrid enterprise systems

Each of these platforms brings something unique. Compare them based on your stack, budget, and desired level of automation.

Cloud-Native and AI-Driven APM Trends

The Rise of Observability

The industry is moving from traditional APM to full observability, which provides a holistic view of metrics, logs, traces, and events. Observability platforms help teams:

🎯 Understand why issues occur, not just what went wrong
🎯 Improve cross-functional collaboration between teams
🎯 Achieve faster mean time to recovery (MTTR)

Predictive Analytics and Automation

AI is no longer a luxury—it's a necessity. APM tools now use machine learning to:

🤖 Forecast potential slowdowns before they occur
🤖 Suggest performance optimizations
🤖 Auto-resolve incidents without human input

This trend is revolutionizing incident management and reducing stress on on-call engineers.

Security and Compliance Considerations

Data Privacy and Storage

Ensure your APM provider:

🔒 Stores data in compliant geographic locations
🔒 Encrypts all data in transit and at rest
🔒 Offers role-based access control (RBAC)

This is crucial for industries handling sensitive customer information.

Industry Standards

Depending on your sector, your APM solution should comply with:

GDPR: European data protection
HIPAA: Healthcare data handling
ISO/IEC 27001 and SOC 2: Security standards

Compliance is not optional—it's a core feature in regulated environments.

How to Conduct a Successful APM Trial

Set Clear Benchmarks

Before trialing a tool, define clear success metrics such as:

⏱️ Reduced incident response times
⏱️ Faster application load speeds
⏱️ Fewer unplanned outages

This helps quantify the value and ensures you choose the right tool.

Get Stakeholder Buy-In

Demonstrate how APM aligns with business goals. Use proof-of-value reports and pilot successes to win over engineering leadership and procurement teams.

Implementation Best Practices

Phased Rollouts

Don't try to implement everything at once. Instead:

Roll out by environment or team
Offer hands-on training and Q&A sessions
Appoint APM champions within teams

This phased approach ensures smooth adoption and higher engagement.

Continuous Optimization

APM isn't "set and forget." Make it a living tool by:

📊 Reviewing performance dashboards weekly
📊 Adjusting alert thresholds as systems evolve
📊 Gathering user feedback regularly

Metrics and KPIs to Track Post-Deployment

Performance Metrics

Key metrics to track include:

Apdex scores
Transaction response times
Error rates
Service-level indicators (SLIs)

User Experience Metrics

User expectations are sky-high. Monitor:

⚡ Time to first interaction (TTFI)
⚡ Frontend load times
⚡ API latency across regions

These insights help deliver world-class digital experiences.

Common Mistakes When Choosing an APM Solution

Ignoring Scalability

Many businesses outgrow their tools. Always pick a platform that can scale with your growth, whether that's traffic, features, or team size.

Overlooking Hidden Costs

Don't be lured by low entry pricing. Confirm:

💡 Support availability in your time zone
💡 Transparent pricing for additional data or users
💡 Exit policies and data ownership

FAQ

What does APM stand for?
APM stands for Application Performance Monitoring, a set of tools for observing and optimizing application health and speed.

How is APM different from Observability?
APM focuses on performance metrics and user experience, while observability includes logs, traces, and events for deeper system insight.

What industries benefit most from APM?
Industries like finance, healthcare, e-commerce, and SaaS benefit significantly due to their high uptime requirements and user traffic.

Is open-source APM reliable?
Open-source APM tools like Prometheus and Grafana are widely used but may require more setup and expertise compared to commercial solutions.

How long does it take to implement an APM solution?
Implementation can take a few hours to a few weeks, depending on your environment's complexity and team size.

Can small businesses afford APM tools?
Yes. Tools like New Relic and AppSignal offer affordable plans for startups and SMBs.

Conclusion: Making the Smart Choice

Choosing the right APM solution means thinking beyond just monitoring. It's about adopting a strategic tool that enhances software performance, improves user satisfaction, and aligns with business goals. By asking the five critical questions outlined here, you're already on the right path to selecting a future-proof APM platform that evolves with your technology and your team.

You may also be interested in:

Distributed Tracing: From 100% Error Rate to Root Cause in 60 Seconds

Alexandr Bandurchin — Sat, 14 Mar 2026 16:12:58 +0000

A step-by-step breakdown of microservice incident investigation: service graph, trace timeline, log correlation, and structured debugging using Uptrace

Introduction

In a microservice architecture, a single failing endpoint can hide a problem in an entirely different service. In this article, I'll walk through a step-by-step investigation of a real incident — from detecting 100% error rate to pinpointing the exact root cause — in under a minute.

We'll use Uptrace, an OpenTelemetry-native platform for tracing and monitoring. All examples are based on a real demo application with microservices.

What you'll learn:

How to use the service graph to find bottlenecks

Span filtering techniques for deep, focused analysis

How timeline reconstruction helps isolate the problem

Correlating traces and logs for root cause analysis

Going from investigation to prevention through monitoring

📊 The Symptom: A Broken Endpoint

Let's say users report that recommendations aren't loading. We open the APM dashboard and immediately spot the problem:

Endpoint: GET /api/recommendations
Error rate: 100% 🔴
Latency p90: 10.6ms
Rate: 1.9 spans/min

APM Overview → "Slowest endpoints" tab. The GET /api/recommendations endpoint is highlighted: error_rate = 100%, p90 = 10.6ms

100% error rate is critical. Every single request is failing. But why?

🔍 Phase 1: Service Graph — Where Is the Problem?

The first step is understanding where exactly in the service chain the failure is happening. For this, we use the service graph.

What Is a Service Graph?

A service graph visualizes how services call each other. Each node is a service, each edge is a call between services.

Our graph looks like this:

[frontend] ---> [recommendation] ---> [product-catalog]
   ✅               🔴 100% errors           ❓

Service graph: red nodes (frontend, recommendation) signal errors. Green nodes (product-reviews, cart) are healthy. Edges show latency and error rate.

Already clear:

frontend is making requests (red node — there are issues)
recommendation is returning errors 🔴
Green nodes (product-reviews, cart) are working normally ✅
The problem is somewhere inside or further down the chain

Incoming vs Outgoing Perspective

The service graph supports two views:

Incoming — who calls this service (dependencies on it)

Outgoing — what dependencies this service relies on

Switching between them reveals:

Incoming: frontend → recommendation (6.1ms duration, errors)
Outgoing: recommendation → ??? (this is where the problem is)

Phase 1 conclusion: The problem is in recommendation — it can't reach its downstream dependencies.

🎯 Phase 2: Analyzing Problematic Spans

Now we need to look at specific requests. We navigate to spans.

Filtering: Cutting Through the Noise

In a production system there are millions of spans. We need precise filtering:

Step 1: Focus on one service

service_name = frontend

Step 2: Only server spans (handled requests)

_kind = server

Step 3: Only errors

_status_code = error

Now we see only the problematic requests from frontend.

Aggregations: Understanding the Scale

We group spans by operation and check aggregations:

Aggregations:
  - perMin(count())     # requests per minute
  - p99(_dur_ms)        # 99th percentile latency
  - _error_rate         # error percentage

Group by:
  - _group_id           # group identical operations

Result:

Operation	Rate	p99 Latency	Status
GET /api/recommendations	2.1/min	16ms	🔴 Errors
POST /api/checkout	1.4/min	23ms	🔴 Errors
GET /api/products/{productId}	11/min	5.7ms	🔴 Errors

Spans → Groups: three active filters (service_name, _kind, _status_code) narrowed the list to 5 groups. Aggregations perMin(count()), p99(_dur_ms), and _error_rate show the scale of the problem.

The problem affects multiple operations, but GET /api/recommendations shows a consistent error pattern. Let's drill deeper into this operation and look at specific requests.

The problem is isolated — we can focus on specific operations.

📋 Phase 2.5: Selecting a Specific Span

We click on one of the GET /api/recommendations rows in the Groups table. A detail panel opens with information about that specific request.

Clicking GET /api/recommendations opens a side panel. The ATTRS tab shows all structured attributes as key-value pairs. The VIEW TRACE button (arrow) leads to the full request timeline.

Selecting a span reveals its attributes as structured key-value pairs, making it easy to filter and search for issues.

What we see in the attributes:

http_status_code: 500 — confirms a server error
http_route: /api/recommendations — the specific endpoint
http_target — full URL with query parameters
host_name: play-all-in-one — host running the service
next_span_name — next span in the call chain

This is structured logging in action — every field has a name and type, making debugging far more effective than working with plain text logs.

Now we click VIEW TRACE to see the full picture of the request across all microservices.

📈 Phase 3: Trace Timeline — Where Is the Bottleneck?

Clicking VIEW TRACE opens a waterfall timeline — the complete history of one request across all microservices.

What Is a Trace?

A trace is the full history of a single request through all microservices, visualized as a waterfall timeline.

Our trace looks like this:

GET /api/recommendations                                [9.5ms total]
├─ user_get_recommendations (load-generator)           [0.7ms]
│  └─ GET http://frontend-proxy:8080/...               [1.3ms]
│     └─ Ingress (frontend-proxy)                      [0.2ms]
│        └─ router frontend egress                     [0.8ms]
│           └─ GET http://frontend-proxy:8080/...      [0.3ms]
│              └─ GET /api/recommendations (frontend)  [6.4ms] 🔴
│                 └─ rpc:recommendation                [failures]

Waterfall trace view: the request path from load-generator through proxy to frontend. The last span (6.4ms) is the clear culprit for both latency and errors.

Timeline Analysis:

Total duration: 9.5ms
Load generator: 0.7ms (request initiation)
HTTP calls through proxy: ~2ms
Frontend processing: 6.4ms — this is where most of the time goes! 🔴
- A significant portion is consumed by rpc:recommendation (26% of total)
- Errors are visible (red markers)

Phase 3 conclusion: Bottleneck is in GET /api/recommendations on frontend (6.4ms out of 9.5ms). The request passes through several hops (load-generator → proxy → frontend), but most of the time and all the errors occur at the final stage when calling the recommendation service.

Inspecting the Problematic Span

We click on the longest span, GET /api/recommendations (6.4ms), in the waterfall. A detail panel opens on the right with complete span information.

Attributes panel of the slowest span: status error, http_status_code = 500, duration 6.4ms. All service metadata is visible: host, OS, Next.js library version.

This span shows exactly where the error occurs — frontend is processing the /api/recommendations request but receiving a 500 error.

Key attributes:

http_status_code: 500 — server returned an internal server error
http_target — full URL with parameter productIds=L9ECAV7KIM
next_span_name: GET /api/recommendations — next call in the chain
next_span_type: BaseServer.handleRequest — type of request handler
otel_library_name: next.js — Next.js framework is being used

Now we need to understand why this span is failing. For that, we look at the logs associated with this trace.

📋 Phase 4: Logs — Root Cause

The trace showed where the problem is. Now we need to understand why. We look at the logs tied to the trace.

Trace-Log Correlation

In Uptrace, traces and logs are linked via trace_id. No need to manually search logs by time or service — just click the LOGS & ERRORS (7) tab while in the waterfall view, and all logs for that specific request appear automatically.

"LOGS & ERRORS (7)" tab: all 7 entries belong to one trace_id. The repeating message "DNS resolution failed for product-catalog:3550" with gRPC status UNAVAILABLE (code 14) — this is the root cause.

What we see in the logs:

A repeating error message appears multiple times with identical content:

Exception: _InactiveRpcError of RPC that terminated with:
  status = StatusCode.UNAVAILABLE
  details = "DNS resolution failed for product-catalog:3550;
            C-ares status is not ARES_SUCCESS
            qtype=A name=product-catalog is_balancer=0;
            DNS server returned general failure"
  grpc_status: 14
  grpc_message: "DNS resolution failed for product-catalog:3550"

We also see the HTTP access log confirming the outcome:

GET /api/recommendations?productIds=L9ECAV7KIM HTTP/1.1
Status: 500
User-Agent: python-requests/2.32.4

Each log entry is linked to the trace via trace_id. Clicking the link icon opens the full structured attributes of the entry in key-value format.

Root Cause Found! 🎯

Problem: DNS cannot resolve product-catalog:3550

Technical details:

gRPC Status: UNAVAILABLE (code 14)

DNS Query: A record for product-catalog

DNS Resolver: C-ares (async DNS resolver)

Result: "DNS server returned general failure"

Possible causes:

The product-catalog service is not running

Incorrect DNS record in Kubernetes/service mesh

Network policy blocking DNS access

Wrong hostname in config (should be product-catalog.namespace.svc.cluster.local)

DNS server overloaded or unreachable

Time from symptom to root cause: ~60 seconds ⏱️

🚨 Phase 5: Prevention Through Monitoring

The investigation is complete — we found the root cause. In a production environment the next step would be a fix (correct the DNS record, restart the service, update the configuration). But it's not enough to fix the current problem — we need to prevent recurrence. For that, we create a monitor.

From Trace to Monitor:

Go back to the TRACE tab
Click the problematic span GET /api/recommendations (6.4ms) in the waterfall
The span detail panel opens on the right
Click the MONITOR dropdown
Select Monitor error rate

The MONITOR button in the span panel opens a menu with 7 options. Selecting "Monitor p99 duration" — Uptrace automatically fills all form fields from the current span's attributes.

After selecting Monitor p99 duration, Uptrace automatically creates a form pre-filled with data from the span.

What gets auto-populated:

Monitor: httpserver:frontend > GET /api/recommendations p99 duration

Metric: uptrace_tracing_spans
Aggregation: p99($spans)
Filter:
  - _group_id = "8037161693486813802"
  (this is the span group ID for this operation)

Alert threshold: p99 latency > 15ms
Check interval: last 5 points (5 minutes)
Notification: Email to EVERYONE

The green zone on the chart shows the acceptable range (up to 15ms). When p99 latency exceeds this threshold, the monitor fires an alert automatically.

Now if p99 latency goes above 15ms, the team receives an automatic alert and can respond before the problem becomes critical. The monitor can be configured to send notifications to Slack, Telegram, PagerDuty, or email.

📚 Takeaways: Investigation Techniques

1. Structured Filtering

Don't look at all spans at once. Filter by:

service_name — one service at a time
_kind = server — only handled requests
_status_code = error — only errors

2. Timeline Thinking

The trace timeline is the story of a request. Look for:

Longest spans (the bottleneck)
Failed spans (where it breaks)
Unexpected calls (unnecessary hops)

3. Log-Trace Correlation

Logs without context are useless. trace_id connects:

One log entry → full request flow
Structured attributes → filtering and search

4. Aggregations Matter

Don't rely on a single span. Aggregations reveal:

perMin(count()) — frequency of the issue
p99(_dur_ms) — worst-case latency
error_rate — percentage of failures

5. From Investigation to Prevention

An investigation doesn't end at the fix. Monitors turn knowledge into automation.

🛠️ Tools & Techniques Recap

Tool	Usage	What It Gives You
Service Graph	Dependency visualization	Where in the chain the problem is
Span filtering	service_name, _kind, _status_code	Remove noise, focus on the problem
Aggregations	perMin, p99, error_rate	Scale and frequency of the issue
Trace timeline	Waterfall view	Bottleneck within the request
Log correlation	trace_id linking	Root cause analysis
Monitors	Automated alerting	Prevention

💬 Conclusion

Distributed tracing isn't just "looking at logs." It's a systematic approach to incident investigation:

Service graph — find the problematic service
Span filtering — isolate the failing requests
Trace timeline — understand the bottleneck
Log correlation — identify the root cause
Monitors — prevent recurrence

With the right tools (OpenTelemetry + Uptrace), the path from symptom to resolution takes less than a minute instead of hours of debugging.

What's next?

Try Uptrace: play.uptrace.dev

Open-source APM based on OpenTelemetry: uptrace.dev/get/hosted/open-source-apm

Set up OTEL Collector: uptrace.dev/opentelemetry/collector

Explore best practices: uptrace.dev/opentelemetry

Questions? Comments? Drop them below! 👇

OpenTelemetry Zero-Code Instrumentation

Alexandr Bandurchin — Wed, 14 Jan 2026 12:07:53 +0000

Adding observability to applications traditionally requires extensive code changes—importing SDKs, creating spans, configuring exporters. Zero-code instrumentation eliminates this friction by automatically capturing telemetry data through agents and runtime modifications, enabling observability without touching application code.

What is Zero-Code Instrumentation?

Zero-code instrumentation (also called automatic instrumentation) uses agents that attach to your application at runtime and automatically instrument popular libraries and frameworks. These agents intercept function calls, network requests, and database queries to generate distributed traces, metrics, and logs without requiring source code changes.

How it works:

Java/JVM: Bytecode manipulation at class load time using Java agents
Python/Node.js: Runtime monkey-patching of library functions
.NET: CLR profiling APIs to modify IL code
Go: eBPF-based instrumentation via kernel probes (Kubernetes only)
PHP: Zend extension hooks

The agent captures telemetry from instrumented libraries and exports it to OpenTelemetry backends using the OTLP protocol.

Zero-Code vs Manual Instrumentation

Aspect	Zero-Code Instrumentation	Manual Instrumentation
Setup Time	5-10 minutes	Hours to days
Code Changes	None required	Extensive modifications
Coverage	Framework/library operations only	Full application including business logic
Customization	Limited to agent configuration	Complete control over spans and attributes
Maintenance	Agent updates only	Code changes for new instrumentation
Learning Curve	Minimal	Requires OpenTelemetry SDK knowledge
Performance Impact	Moderate (all libraries instrumented)	Lower (instrument only critical paths)
Best For	Legacy apps, quick start, third-party libraries	Custom business metrics, fine-grained control

Recommendation: Start with zero-code instrumentation for immediate visibility, then add manual instrumentation incrementally for business-specific observability.

Supported Languages

OpenTelemetry provides zero-code instrumentation for these languages:

Java (production-ready, most mature)
.NET (production-ready)
Python (production-ready)
Node.js (production-ready)
PHP (stable)
Go (eBPF-based, Kubernetes only, experimental)

Each language uses different techniques based on runtime capabilities.

Java Zero-Code Instrumentation

Java's bytecode manipulation makes it ideal for zero-code instrumentation. The OpenTelemetry Java agent works with any JVM-based application.

Installation

Download the latest Java agent (v2.23.0):

wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.23.0/opentelemetry-javaagent.jar

Configuration

Configure the agent using environment variables. The examples below use Uptrace as the OTLP backend, but you can use any OpenTelemetry-compatible backend (Jaeger, Grafana Tempo, etc.) by changing the OTEL_EXPORTER_OTLP_ENDPOINT.

export OTEL_SERVICE_NAME=my-java-app
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_uptrace_dsn>

# Run your application with the agent
java -javaagent:opentelemetry-javaagent.jar -jar myapp.jar

Spring Boot Example

No code changes required for Spring Boot applications:

# Download agent
wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.23.0/opentelemetry-javaagent.jar

# Set environment variables
export OTEL_SERVICE_NAME=spring-boot-app
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_dsn>

# Run Spring Boot with agent
java -javaagent:opentelemetry-javaagent.jar -jar target/myapp-0.0.1-SNAPSHOT.jar

The agent automatically instruments:

Spring Web MVC / Spring WebFlux
JDBC database calls
HTTP clients (Apache HttpClient, OkHttp, etc.)
Kafka, RabbitMQ, Redis
RestTemplate, WebClient
JPA/Hibernate queries

Dockerfile Example

FROM openjdk:17-slim

WORKDIR /app

# Download OpenTelemetry Java agent
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.23.0/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar

COPY target/myapp.jar /app/app.jar

# Configure OpenTelemetry
ENV OTEL_SERVICE_NAME=my-service
ENV OTEL_TRACES_EXPORTER=otlp
ENV OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317

CMD ["java", "-javaagent:/app/opentelemetry-javaagent.jar", "-jar", "/app/app.jar"]

.NET Zero-Code Instrumentation

The OpenTelemetry .NET automatic instrumentation uses CLR profiling APIs to instrument .NET applications.

Installation (Windows)

Download and install the latest .NET instrumentation (v1.13.0):

# Download installer
Invoke-WebRequest -Uri "https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v1.13.0/opentelemetry-dotnet-instrumentation-windows.zip" -OutFile "otel-dotnet-auto.zip"
# Extract
Expand-Archive -Path otel-dotnet-auto.zip -DestinationPath "C:\otel-dotnet-auto"
# Install
C:\otel-dotnet-auto\install.cmd

Installation (Linux)

Note: .NET CLR profiling requires many environment variables for runtime hooks. Consider using Docker or Kubernetes Operator to simplify configuration and avoid manual setup errors.

# Download and extract
wget https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/download/v1.13.0/opentelemetry-dotnet-instrumentation-linux-glibc.zip
unzip opentelemetry-dotnet-instrumentation-linux-glibc.zip -d /opt/otel-dotnet-auto

# Set environment variables
export OTEL_DOTNET_AUTO_HOME=/opt/otel-dotnet-auto
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
export CORECLR_PROFILER_PATH=$OTEL_DOTNET_AUTO_HOME/linux-x64/OpenTelemetry.AutoInstrumentation.Native.so
export DOTNET_STARTUP_HOOKS=$OTEL_DOTNET_AUTO_HOME/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll
export DOTNET_ADDITIONAL_DEPS=$OTEL_DOTNET_AUTO_HOME/AdditionalDeps
export DOTNET_SHARED_STORE=$OTEL_DOTNET_AUTO_HOME/store

Configuration

# Service configuration
export OTEL_SERVICE_NAME=my-dotnet-app
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_dsn>

# Run your application
dotnet myapp.dll

ASP.NET Core Example

# Set required environment variables
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}
export CORECLR_PROFILER_PATH=/opt/otel-dotnet-auto/linux-x64/OpenTelemetry.AutoInstrumentation.Native.so
export DOTNET_STARTUP_HOOKS=/opt/otel-dotnet-auto/net/OpenTelemetry.AutoInstrumentation.StartupHook.dll

# Service configuration
export OTEL_SERVICE_NAME=aspnet-app
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_dsn>

# Run ASP.NET Core app
dotnet MyWebApp.dll

Automatically instruments:

ASP.NET Core MVC / Web API
HttpClient requests
Entity Framework Core
SQL Client (SqlConnection)
MongoDB, Redis, Elasticsearch

Python Zero-Code Instrumentation

Python uses the opentelemetry-instrument wrapper to automatically inject instrumentation.

Installation

# Install OpenTelemetry auto-instrumentation
pip install opentelemetry-distro opentelemetry-exporter-otlp

# Bootstrap auto-instrumentation (downloads instrumentation packages)
opentelemetry-bootstrap -a install

Configuration and Usage

Run your Python application with the opentelemetry-instrument wrapper:

export OTEL_SERVICE_NAME=my-python-app
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_dsn>

# Run with automatic instrumentation
opentelemetry-instrument python app.py

Flask Example

Zero-code setup (5 minutes) - just install dependencies and run with the wrapper:

# Install Flask and auto-instrumentation
pip install flask opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

# app.py (no code changes needed)
# from flask import Flask
# app = Flask(__name__)
#
# @app.route('/')
# def hello():
#     return 'Hello World!'

# Run with auto-instrumentation
export OTEL_SERVICE_NAME=flask-app
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
opentelemetry-instrument flask run

Django Example

Zero-code setup (5 minutes) - just install dependencies and run with the wrapper:

# Install dependencies
pip install django opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

# Run Django with auto-instrumentation
export OTEL_SERVICE_NAME=django-app
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
opentelemetry-instrument python manage.py runserver

Automatically instruments:

Flask, Django, FastAPI
Requests, HTTPX, urllib3
psycopg2 (PostgreSQL), pymongo (MongoDB)
Redis, Celery, SQLAlchemy

Node.js Zero-Code Instrumentation

Node.js uses the @opentelemetry/auto-instrumentations-node package for automatic instrumentation.

Installation

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

Configuration

Create tracing.js in your project root:

// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
const { Resource } = require('@opentelemetry/resources')
const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions')

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'nodejs-app',
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
    headers: {
      'uptrace-dsn': process.env.UPTRACE_DSN,
    },
  }),
  instrumentations: [getNodeAutoInstrumentations()],
})

sdk.start()

Usage

Use the --require flag to load instrumentation before your application:

export OTEL_SERVICE_NAME=nodejs-app
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4318
export UPTRACE_DSN=<your_dsn>

# Run with automatic instrumentation
node --require ./tracing.js app.js

Express.js Example

Zero-code setup (10 minutes) - create tracing file and use --require flag:

// app.js (no changes needed)
const express = require('express')
const app = express()

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(3000, () => {
  console.log('Server running on port 3000')
})

Run with instrumentation:

node --require ./tracing.js app.js

Automatically instruments:

Express.js, Fastify, Koa, Nest.js
HTTP/HTTPS clients
MongoDB, MySQL, PostgreSQL, Redis
GraphQL, gRPC

PHP Zero-Code Instrumentation

PHP uses a Zend extension for automatic instrumentation.

Installation

# Install via PECL
pecl install opentelemetry

# Or download from releases
wget https://github.com/open-telemetry/opentelemetry-php-instrumentation/releases/latest/download/opentelemetry.so

# Enable extension in php.ini
echo "extension=opentelemetry.so" >> /etc/php/8.2/cli/php.ini

Configuration

export OTEL_SERVICE_NAME=php-app
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4318
export OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_dsn>

# Run PHP application
php app.php

Laravel Example

Zero-code setup (5 minutes) with PHP extension - no code changes required:

# Install extension
pecl install opentelemetry

# Configure
export OTEL_SERVICE_NAME=laravel-app
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4318

# Run Laravel
php artisan serve

Go Zero-Code Instrumentation (Kubernetes)

Go doesn't support traditional agent-based instrumentation due to its compiled nature. However, the OpenTelemetry Operator for Kubernetes can use eBPF to instrument Go applications without code changes.

Why Kubernetes only: eBPF (extended Berkeley Packet Filter) requires Linux kernel 4.14+ and CAP_SYS_ADMIN capability to attach kernel probes. This level of system access is only safely available in Kubernetes with proper security contexts and Pod Security Policies.

Prerequisites

Kubernetes cluster
OpenTelemetry Operator installed

Install OpenTelemetry Operator

# Install cert-manager (required)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

# Install OpenTelemetry Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Configure Instrumentation

Create an Instrumentation resource:

# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: auto-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: https://api.uptrace.dev:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  go:
    image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest

Apply the configuration:

kubectl apply -f instrumentation.yaml

Instrument Go Deployment

Add the annotation to your deployment:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-app
  template:
    metadata:
      labels:
        app: go-app
      annotations:
        instrumentation.opentelemetry.io/inject-go: "true"  # Enable auto-instrumentation
    spec:
      containers:
      - name: app
        image: my-go-app:latest
        env:
        - name: OTEL_SERVICE_NAME
          value: "go-app"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "https://api.uptrace.dev:4317"
        - name: OTEL_EXPORTER_OTLP_HEADERS
          value: "uptrace-dsn=<your_dsn>"

The operator injects an init container and sidecar that use eBPF to instrument the Go application at runtime.
Limitations:

Kubernetes only (eBPF requires Linux kernel capabilities)
Experimental (not production-ready as of 2025)
Limited coverage compared to manual instrumentation ## Kubernetes Multi-Language Auto-Instrumentation The OpenTelemetry Operator can automatically instrument multiple languages in Kubernetes without code changes. ### Example: Multi-Language Application

# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: multi-lang-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: https://api.uptrace.dev:4317
  propagators:
    - tracecontext
    - baggage
  # Java configuration
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.23.0
  # Node.js configuration
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.67.3
  # Python configuration
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.49b0
  # .NET configuration
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.13.0

Java Service Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-service
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
      - name: app
        image: my-java-service:latest
        env:
        - name: OTEL_SERVICE_NAME
          value: "java-service"

Node.js Service Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodejs-service
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-nodejs: "true"
    spec:
      containers:
      - name: app
        image: my-nodejs-service:latest
        env:
        - name: OTEL_SERVICE_NAME
          value: "nodejs-service"

The operator automatically injects agents for each language without requiring code changes or custom Dockerfiles.

Configuration Options

Common Environment Variables

These variables work across all languages:

# Service identification
OTEL_SERVICE_NAME=my-service
OTEL_SERVICE_VERSION=1.0.0

# Exporter configuration
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.uptrace.dev:4317
OTEL_EXPORTER_OTLP_HEADERS=uptrace-dsn=<your_dsn>

# Sampling
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1  # Sample 10% of traces

# Resource attributes
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=ecommerce

# Propagators
OTEL_PROPAGATORS=tracecontext,baggage

Language-Specific Configuration

Java:

# Disable specific instrumentations
OTEL_INSTRUMENTATION_JDBC_ENABLED=false
OTEL_INSTRUMENTATION_SPRING_WEBMVC_ENABLED=true

# JVM system properties
-Dotel.javaagent.debug=true
-Dotel.instrumentation.common.db-statement-sanitizer.enabled=false

.NET:

# Disable specific instrumentations
OTEL_DOTNET_AUTO_TRACES_INSTRUMENTATION_ENABLED=true
OTEL_DOTNET_AUTO_TRACES_ASPNETCORE_INSTRUMENTATION_ENABLED=true
OTEL_DOTNET_AUTO_TRACES_HTTPCLIENT_INSTRUMENTATION_ENABLED=true

Python:

# Disable specific instrumentations
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=flask,django

# Enable/disable
OTEL_PYTHON_FLASK_EXCLUDED_URLS=/health,/metrics

Node.js:

Disable instrumentations in tracing.js:

instrumentations: [
  getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fs': { enabled: false },
    '@opentelemetry/instrumentation-http': { enabled: true },
  }),
]

Limitations of Zero-Code Instrumentation

While zero-code instrumentation provides quick observability, it has limitations:

Generic Span Names

Automatic instrumentation generates generic span names based on framework operations:

❌ Automatic: "GET /api/users"
✅ Manual:    "fetch_active_users_for_dashboard"

No Business Context

Zero-code instrumentation can't capture domain-specific information:

❌ Missing: order.id, user.subscription_tier, cart.total_value
✅ Manual:  Custom attributes for business logic

Framework Coverage Only

Only instrumented libraries generate spans. Custom code remains invisible:

# Automatically traced (HTTP request)
response = requests.get('https://api.example.com/data')

# NOT traced (custom business logic)
def calculate_discount(user, cart):
    # This logic is invisible without manual instrumentation
    return apply_business_rules(user, cart)

Performance Overhead

Instrumenting all libraries can introduce overhead:

Java agent adds ~50-100ms startup time
Runtime overhead: 1-5% CPU increase (varies by traffic)
Memory overhead: 50-200MB additional heap usage

Version Compatibility

Agents may not support the latest library versions immediately. Check compatibility matrix before deploying.

When to Use Zero-Code Instrumentation

✅ Ideal for:

Legacy applications: No budget for code refactoring
Quick POC: Evaluate OpenTelemetry before committing
Third-party libraries: Instrument dependencies you don't control
Getting started: Immediate visibility while learning OpenTelemetry
Kubernetes environments: Operator-based injection across services

❌ Not ideal for:

Business metrics: Custom KPIs require manual instrumentation
Performance-critical paths: Lower overhead with selective manual instrumentation
Complex transactions: Fine-grained control needs manual spans
Proprietary frameworks: Zero-code only supports popular libraries

Hybrid Solution

The most effective strategy combines zero-code and manual instrumentation:

Phase 1: Zero-Code Foundation

Deploy automatic instrumentation to get immediate visibility into framework operations, HTTP requests, and database queries.

Phase 2: Selective Manual Instrumentation

Add manual instrumentation for:

Business-critical operations
Custom metrics (e.g., checkout success rate)
Domain-specific context (user ID, subscription tier)

Example:

// Spring Boot automatically instrumented (zero-code)
@RestController
public class OrderController {

    // HTTP request automatically traced
    @PostMapping("/orders")
    public Order createOrder(@RequestBody OrderRequest request) {

        // Add manual span for business logic
        Span span = tracer.spanBuilder("process_order")
            .setAttribute("order.type", request.getType())
            .setAttribute("user.subscription", getCurrentUser().getSubscription())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            // Business logic with custom attributes
            Order order = orderService.create(request);
            span.setAttribute("order.id", order.getId());
            span.setAttribute("order.total", order.getTotal());
            return order;
        } finally {
            span.end();
        }
    }
}

This approach provides comprehensive coverage: automatic instrumentation for infrastructure, manual instrumentation for business logic.

Troubleshooting

Issue: Agent Not Loading

Symptom: No traces appear in backend.

Solution:

Check agent is attached:

# Java: Look for "opentelemetry-javaagent" in JVM arguments
jps -v | grep opentelemetry

# .NET: Verify profiler environment variables
env | grep CORECLR_PROFILER

# Python: Ensure bootstrap ran successfully
opentelemetry-bootstrap -a list

Enable debug logging:

# Java
export OTEL_JAVAAGENT_DEBUG=true

# .NET
export OTEL_DOTNET_AUTO_LOG_DIRECTORY=/tmp/otel-logs

# Python
export OTEL_LOG_LEVEL=debug

# Node.js
export OTEL_LOG_LEVEL=debug

Issue: Missing Instrumentations

Symptom: Some libraries not instrumented (e.g., Redis calls missing).

Solution:

Verify instrumentation is enabled:

# Java: Check instrumentation status
java -javaagent:opentelemetry-javaagent.jar -jar app.jar --help | grep redis

# Python: List installed instrumentations
pip list | grep opentelemetry-instrumentation

# Node.js: Check auto-instrumentations version
npm list @opentelemetry/auto-instrumentations-node

Some libraries require specific versions. Check compatibility matrix.

Issue: High Performance Overhead

Symptom: CPU usage increased by >10% after enabling instrumentation.

Solution:

Disable unnecessary instrumentations:

# Java: Disable specific instrumentations
export OTEL_INSTRUMENTATION_JDBC_ENABLED=false
export OTEL_INSTRUMENTATION_LOGBACK_ENABLED=false

# Python
export OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=urllib3,requests

Reduce sampling rate:

export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1  # Sample only 10%

Issue: Conflicting Agents

Symptom: Application fails to start with multiple profiling agents.

Solution:

Remove conflicting APM agents (New Relic, AppDynamics, Datadog) before deploying OpenTelemetry. Most JVM profilers cannot coexist.

What is Uptrace?

Uptrace is an open source APM for OpenTelemetry that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. For language-specific instrumentation guides, see the instrumentation guides and compare with top APM tools.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules, notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

Next Steps

Implement OpenTelemetry sampling strategies for production
Set up OpenTelemetry Collector for advanced processing

OpenTelemetry Gin Monitoring [otelgin]

Alexandr Bandurchin — Thu, 20 Nov 2025 09:21:50 +0000

OpenTelemetry Gin can help developers monitor and diagnose issues with their Gin applications, and can provide valuable insights into the behavior of the applications in production.

What is Gin?

Gin is a lightweight web framework for building web applications and APIs in the Go programming language. It provides a minimalist and fast approach to building web applications, emphasizing simplicity, performance, and ease of use.

Gin's simplicity and performance make it a popular choice for building web applications and APIs in Go. It provides a solid foundation for developing scalable and efficient web services while maintaining a straightforward and intuitive API.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. The OpenTelemetry architecture is built with modularity and flexibility, allowing it to work seamlessly across different programming languages and frameworks.

Gin instrumentation

Gin OpenTelemetry instrumentation allows developers to easily add observability to their Gin applications, providing insights into application performance, behavior, and usage patterns.

OpenTelemetry Gin provides a simple API for instrumenting Gin applications, making it possible for developers to quickly add observability to their applications without having to write a lot of code.

To install otelgin instrumentation:

go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

Usage

You can instrument Gin router by installing OpenTelemetry middleware:

import (
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

router := gin.Default()
router.Use(otelgin.Middleware("service-name"))

Once your Gin application is instrumented with OpenTelemetry and telemetry data is exported, you can use observability tools compatible with your chosen OpenTelemetry backend to visualize distributed traces, analyze performance metrics, and gain insight into the behavior of your application.

Instrumenting templates rendering

To instrument templates rendering, use otelgin.HTML helper:

import "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

func exampleHandler(c *gin.Context) {
    otelgin.HTML(c, http.StatusOK, indexTmpl, gin.H{
        "foo": "bar",
    })
}

What is Uptrace?

Uptrace is a OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's next?

With Gin instrumentation configured, you can track HTTP requests, monitor performance bottlenecks, and correlate traces across your microservices. For database monitoring, check out GORM instrumentation or explore Echo framework for alternative web frameworks.

OpenTelemetry Flask monitoring

Alexandr Bandurchin — Thu, 20 Nov 2025 09:21:05 +0000

By integrating OpenTelemetry with Flask, you can gain valuable insights into the performance, behavior, and dependencies of your application. This comprehensive guide will walk you through setting up observability for your Flask applications, from basic instrumentation to advanced monitoring strategies.

Using OpenTelemetry, you can monitor and troubleshoot issues, optimize performance, and ensure the reliability of your Flask applications across development, staging, and production environments.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that provides a standardized way to collect, process, and export telemetry data from applications. It combines three pillars of observability:

The Three Pillars of Observability:

Traces: Track requests as they flow through distributed systems
Metrics: Quantitative measurements of system performance and behavior
Logs: Discrete records of events that occurred in your application

Benefits of OpenTelemetry:

Vendor Neutrality: Works with multiple observability backends
Standardization: Consistent APIs across languages and frameworks
Automatic Instrumentation: Built-in support for popular libraries
Custom Instrumentation: APIs for adding application-specific telemetry
Performance: Designed for minimal overhead in production environments

OpenTelemetry enables developers to instrument their code to gain insights into application performance, behavior, and dependencies across distributed systems, making it easier to debug issues, optimize performance, and understand system behavior.

Prerequisites and Installation

System Requirements

Python 3.7 or higher
pip package manager
A Flask application (existing or new)

Core Dependencies

Install the essential OpenTelemetry packages for Flask. For comprehensive Python instrumentation, see the OpenTelemetry Python guide.

# Core OpenTelemetry packages
pip install opentelemetry-api opentelemetry-sdk

# Flask-specific instrumentation
pip install opentelemetry-instrumentation-flask

# Optional: Auto-instrumentation for common libraries
pip install opentelemetry-bootstrap opentelemetry-instrumentation

Optional Dependencies

Depending on your needs, you might also want:

# For database instrumentation
pip install opentelemetry-instrumentation-sqlalchemy
pip install opentelemetry-instrumentation-psycopg2  # PostgreSQL
pip install opentelemetry-instrumentation-pymongo  # MongoDB

# For HTTP client instrumentation
pip install opentelemetry-instrumentation-requests
pip install opentelemetry-instrumentation-urllib3

# For different exporters
pip install opentelemetry-exporter-jaeger
pip install opentelemetry-exporter-otlp
pip install opentelemetry-exporter-prometheus

Verification

You can verify your installation by running:

import opentelemetry
print(f"OpenTelemetry version: {opentelemetry.__version__}")

Basic Setup

OpenTelemetry SDK Initialization

Create a dedicated configuration module for OpenTelemetry setup. This approach keeps your instrumentation code organized and reusable.

Create a file called telemetry.py:

"""
OpenTelemetry configuration for Flask application.
This module handles the initialization of tracing, metrics, and exporters.
"""

import os
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION

def configure_opentelemetry(service_name: str = "flask-app", service_version: str = "1.0.0"):
    """
    Configure OpenTelemetry with proper resource attributes and exporters.

    Args:
        service_name: Name of your service
        service_version: Version of your service
    """

    # Create resource with service information
    resource = Resource.create({
        SERVICE_NAME: service_name,
        SERVICE_VERSION: service_version,
        "environment": os.getenv("ENVIRONMENT", "development"),
        "team": os.getenv("TEAM", "backend"),
    })

    # Configure tracing
    trace_provider = TracerProvider(resource=resource)

    # Choose exporter based on environment
    if os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"):
        # Production: Use OTLP exporter
        otlp_exporter = OTLPSpanExporter(
            endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
            headers={"Authorization": f"Bearer {os.getenv('OTEL_API_KEY', '')}"}
        )
        trace_provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
    else:
        # Development: Use console exporter
        console_exporter = ConsoleSpanExporter()
        trace_provider.add_span_processor(BatchSpanProcessor(console_exporter))

    # Set the global tracer provider
    trace.set_tracer_provider(trace_provider)

    # Configure metrics (optional)
    metrics.set_meter_provider(MeterProvider(resource=resource))

    return trace.get_tracer(__name__)

# Initialize tracer
tracer = configure_opentelemetry()

Basic Flask Application Setup

Now integrate OpenTelemetry with your Flask application:

# app.py
from flask import Flask, request, jsonify
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from telemetry import configure_opentelemetry

# Initialize OpenTelemetry before creating Flask app
tracer = configure_opentelemetry(
    service_name="my-flask-api",
    service_version="1.0.0"
)

# Create Flask application
app = Flask(__name__)

# Instrument the Flask app
FlaskInstrumentor().instrument_app(app)

@app.route('/')
def hello_world():
    return jsonify({"message": "Hello, World!", "status": "healthy"})

@app.route('/health')
def health_check():
    return jsonify({"status": "ok", "service": "my-flask-api"})

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Flask Instrumentation

Automatic Instrumentation

OpenTelemetry provides automatic instrumentation for Flask applications, which captures HTTP requests, responses, and basic metadata without requiring code changes.

Basic Instrumentation

from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Simple instrumentation
FlaskInstrumentor().instrument_app(app)

Advanced Configuration Options

from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Configure with custom options
FlaskInstrumentor().instrument_app(
    app,
    # Add SQL comment context to database queries
    enable_commenter=True,
    commenter_options={
        "opentelemetry_values": True,  # Include OpenTelemetry trace context
    },
    # Exclude certain routes from tracing
    excluded_urls="health,metrics,favicon.ico",
    # Custom request hook for adding attributes
    request_hook=request_hook_function,
    # Custom response hook for adding attributes
    response_hook=response_hook_function,
)

Custom Hook Functions

def request_hook_function(span, environ):
    """Add custom attributes to request spans."""
    if span and span.is_recording():
        # Add user information if available
        user_id = environ.get('HTTP_X_USER_ID')
        if user_id:
            span.set_attribute("user.id", user_id)

        # Add API version
        api_version = environ.get('HTTP_X_API_VERSION', 'v1')
        span.set_attribute("api.version", api_version)

def response_hook_function(span, status, response_headers):
    """Add custom attributes based on response."""
    if span and span.is_recording():
        # Add response size
        content_length = next(
            (value for key, value in response_headers if key.lower() == 'content-length'),
            None
        )
        if content_length:
            span.set_attribute("http.response.size", int(content_length))

Auto-Instrumentation with Bootstrap

For a more comprehensive setup, you can use auto-instrumentation to automatically detect and instrument common libraries:

# Install auto-instrumentation
pip install opentelemetry-bootstrap opentelemetry-instrumentation

# Bootstrap (detects installed libraries)
opentelemetry-bootstrap -a install

# Run your application with auto-instrumentation
opentelemetry-instrument --traces_exporter console --metrics_exporter console python app.py

Advanced Configuration

Environment Variables Configuration

OpenTelemetry supports configuration through environment variables:

# Service identification
export OTEL_SERVICE_NAME="my-flask-api"
export OTEL_SERVICE_VERSION="1.2.0"
export OTEL_RESOURCE_ATTRIBUTES="environment=production,team=backend"

# Exporters
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="prometheus"
export OTEL_LOGS_EXPORTER="console"

# OTLP Endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.uptrace.dev"
export OTEL_EXPORTER_OTLP_HEADERS="uptrace-dsn=YOUR_DSN"

# Sampling
export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"  # Sample 10% of traces

Custom Sampling Strategies

from opentelemetry.sdk.trace.sampling import (
    TraceIdRatioBasedSampler,
    ParentBasedSampler,
    NEVER_SAMPLE,
    ALWAYS_SAMPLE
)

# Custom sampler that samples based on route
class RouteBasedSampler:
    def __init__(self, high_volume_routes=None, sample_rate=0.01):
        self.high_volume_routes = high_volume_routes or ['/health', '/metrics']
        self.low_sample = TraceIdRatioBasedSampler(sample_rate)
        self.high_sample = TraceIdRatioBasedSampler(1.0)

    def should_sample(self, parent_context, trace_id, name, kind, attributes, links, trace_state):
        # Sample health checks and metrics at lower rate
        if any(route in name for route in self.high_volume_routes):
            return self.low_sample.should_sample(
                parent_context, trace_id, name, kind, attributes, links, trace_state
            )
        return self.high_sample.should_sample(
            parent_context, trace_id, name, kind, attributes, links, trace_state
        )

# Use custom sampler
trace_provider = TracerProvider(
    sampler=RouteBasedSampler(),
    resource=resource
)

Resource Detection

Automatically detect and include resource information:

from opentelemetry.sdk.resources import Resource
from opentelemetry.resourcedetector.aws_ec2 import AwsEc2ResourceDetector
from opentelemetry.resourcedetector.process import ProcessResourceDetector

# Automatically detect resource attributes
resource = Resource.create().merge(
    AwsEc2ResourceDetector().detect()  # AWS EC2 metadata
).merge(
    ProcessResourceDetector().detect()  # Process information
)

Custom Instrumentation

Manual Span Creation

Add custom spans to trace specific business logic:

from flask import Flask, request
from opentelemetry import trace

app = Flask(__name__)
tracer = trace.get_tracer(__name__)

@app.route('/api/users/<int:user_id>')
def get_user(user_id):
    # Create a custom span for the entire operation
    with tracer.start_as_current_span("get_user_operation") as span:
        # Add attributes to the span
        span.set_attribute("user.id", user_id)
        span.set_attribute("operation.type", "read")

        try:
            # Simulate user validation
            with tracer.start_as_current_span("validate_user_permissions"):
                if not validate_user_permissions(user_id):
                    span.set_attribute("error", True)
                    span.set_attribute("error.type", "permission_denied")
                    return jsonify({"error": "Access denied"}), 403

            # Simulate database query
            with tracer.start_as_current_span("database_query") as db_span:
                db_span.set_attribute("db.operation", "SELECT")
                db_span.set_attribute("db.table", "users")
                user_data = fetch_user_from_database(user_id)
                db_span.set_attribute("db.rows_affected", 1 if user_data else 0)

            # Process user data
            with tracer.start_as_current_span("process_user_data"):
                processed_user = process_user_data(user_data)

            span.set_attribute("operation.success", True)
            return jsonify(processed_user)

        except Exception as e:
            # Record the exception in the span
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            span.set_attribute("error", True)
            span.set_attribute("error.type", type(e).__name__)
            return jsonify({"error": "Internal server error"}), 500

def validate_user_permissions(user_id):
    """Simulate permission validation."""
    # Add custom logic here
    return user_id > 0

def fetch_user_from_database(user_id):
    """Simulate database fetch."""
    return {"id": user_id, "name": f"User {user_id}", "email": f"user{user_id}@example.com"}

def process_user_data(user_data):
    """Simulate data processing."""
    if user_data:
        return {
            "id": user_data["id"],
            "name": user_data["name"],
            "email": user_data["email"],
            "processed_at": "2024-01-15T10:30:00Z"
        }
    return None

Decorators for Automatic Tracing

Create reusable decorators for common patterns:

import functools
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def trace_function(name=None, attributes=None):
    """Decorator to automatically trace function calls."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            span_name = name or f"{func.__module__}.{func.__name__}"
            with tracer.start_as_current_span(span_name) as span:
                # Add default attributes
                span.set_attribute("function.name", func.__name__)
                span.set_attribute("function.module", func.__module__)

                # Add custom attributes
                if attributes:
                    for key, value in attributes.items():
                        span.set_attribute(key, value)

                try:
                    result = func(*args, **kwargs)
                    span.set_attribute("function.success", True)
                    return result
                except Exception as e:
                    span.record_exception(e)
                    span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
                    raise
        return wrapper
    return decorator

# Usage example
@trace_function(name="business_logic.calculate_total", attributes={"operation.type": "calculation"})
def calculate_order_total(items):
    """Calculate the total price of order items."""
    total = sum(item.get('price', 0) * item.get('quantity', 0) for item in items)
    return total

@app.route('/api/orders/total', methods=['POST'])
def calculate_total():
    items = request.json.get('items', [])
    total = calculate_order_total(items)
    return jsonify({"total": total})

Adding Context and Baggage

Use OpenTelemetry context and baggage to pass information across service boundaries:

from opentelemetry import baggage, context
from opentelemetry.trace import set_span_in_context

@app.route('/api/orders', methods=['POST'])
def create_order():
    # Set baggage for cross-service context
    ctx = baggage.set_baggage("user.id", request.headers.get("X-User-ID", "anonymous"))
    ctx = baggage.set_baggage("request.id", request.headers.get("X-Request-ID", "unknown"), ctx)

    with tracer.start_as_current_span("create_order") as span:
        # Get baggage values
        user_id = baggage.get_baggage("user.id", ctx)
        request_id = baggage.get_baggage("request.id", ctx)

        span.set_attribute("user.id", user_id)
        span.set_attribute("request.id", request_id)

        # Process order...
        return jsonify({"status": "created", "user_id": user_id})

Database Instrumentation

SQLAlchemy Integration

Comprehensive SQLAlchemy instrumentation setup:

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from sqlalchemy import event

app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = "postgresql://user:password@localhost/mydb"
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False

db = SQLAlchemy(app)

# Instrument SQLAlchemy with detailed configuration
SQLAlchemyInstrumentor().instrument(
    engine=db.engine,
    # Enable SQL commenter to add trace context to SQL queries
    enable_commenter=True,
    commenter_options={
        "opentelemetry_values": True,
    },
)

# Optional: Add custom database event listeners
@event.listens_for(db.engine, "before_cursor_execute")
def receive_before_cursor_execute(conn, cursor, statement, parameters, context, executemany):
    """Add custom logic before SQL execution."""
    current_span = trace.get_current_span()
    if current_span:
        current_span.add_event("sql.before_execute", {"statement": statement})

@event.listens_for(db.engine, "after_cursor_execute")
def receive_after_cursor_execute(conn, cursor, statement, parameters, context, executemany):
    """Add custom logic after SQL execution."""
    current_span = trace.get_current_span()
    if current_span:
        current_span.add_event("sql.after_execute", {"rowcount": cursor.rowcount})

# Example model
class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True, nullable=False)
    email = db.Column(db.String(120), unique=True, nullable=False)

    def to_dict(self):
        return {
            "id": self.id,
            "username": self.username,
            "email": self.email
        }

@app.route('/api/users')
def get_users():
    # This query will be automatically traced
    users = User.query.all()
    return jsonify([user.to_dict() for user in users])

Multiple Database Support

from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
from opentelemetry.instrumentation.pymongo import PyMongoInstrumentor

# PostgreSQL with psycopg2
Psycopg2Instrumentor().instrument()

# MongoDB
PyMongoInstrumentor().instrument()

# Multiple SQLAlchemy engines
main_engine = create_engine("postgresql://...")
analytics_engine = create_engine("postgresql://...")

SQLAlchemyInstrumentor().instrument(
    engine=main_engine,
    service="user-service-main-db"
)

SQLAlchemyInstrumentor().instrument(
    engine=analytics_engine,
    service="user-service-analytics-db"
)

Best Practices

Performance Optimization

Sampling Strategy: Use appropriate sampling rates for different environments:

# Production sampling configuration
SAMPLING_RATES = {
    "production": 0.01,    # 1% sampling
    "staging": 0.1,        # 10% sampling
    "development": 1.0,    # 100% sampling
}

environment = os.getenv("ENVIRONMENT", "development")
sampling_rate = SAMPLING_RATES.get(environment, 1.0)

tracer_provider = TracerProvider(
    sampler=TraceIdRatioBasedSampler(sampling_rate)
)

Attribute Guidelines:

# Good: Structured, consistent attributes
span.set_attributes({
    "user.id": user_id,
    "user.role": user_role,
    "operation.type": "database_query",
    "database.table": "users",
    "database.operation": "SELECT"
})

# Avoid: High-cardinality attributes
# span.set_attribute("sql.query", full_sql_query)  # Can cause memory issues
# span.set_attribute("user.session_id", session_id)  # Too many unique values

Span Lifecycle Management:

def process_with_proper_span_management():
    with tracer.start_as_current_span("main_operation") as main_span:
        try:
            # Child spans are automatically linked to parent
            with tracer.start_as_current_span("sub_operation_1"):
                result1 = do_something()

            with tracer.start_as_current_span("sub_operation_2"):
                result2 = do_something_else()

            main_span.set_attribute("operation.success", True)
            return combine_results(result1, result2)

        except Exception as e:
            main_span.record_exception(e)
            main_span.set_status(trace.Status(trace.StatusCode.ERROR))
            raise

Security Considerations

Sensitive Data Filtering:

import re

SENSITIVE_PATTERNS = [
    re.compile(r'password', re.IGNORECASE),
    re.compile(r'token', re.IGNORECASE),
    re.compile(r'secret', re.IGNORECASE),
    re.compile(r'key', re.IGNORECASE),
]

def sanitize_attributes(attributes):
    """Remove sensitive data from span attributes."""
    sanitized = {}
    for key, value in attributes.items():
        if any(pattern.search(key) for pattern in SENSITIVE_PATTERNS):
            sanitized[key] = "[REDACTED]"
        else:
            sanitized[key] = value
    return sanitized

# Use in custom instrumentation
def secure_span_decorator(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        with tracer.start_as_current_span(func.__name__) as span:
            # Sanitize function arguments before adding as attributes
            safe_attrs = sanitize_attributes(kwargs)
            for key, value in safe_attrs.items():
                span.set_attribute(f"function.arg.{key}", str(value))

            return func(*args, **kwargs)
    return wrapper

Network Security:

# Use TLS for OTLP exports
otlp_exporter = OTLPSpanExporter(
    endpoint="https://secure-collector.example.com",
    headers={
        "Authorization": f"Bearer {os.getenv('OTEL_API_KEY')}",
        "X-API-Version": "v1"
    },
    insecure=False,  # Ensure TLS is used
)

Error Handling and Monitoring

import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentor

# Configure logging instrumentation
LoggingInstrumentor().instrument(set_logging_format=True)

# Create logger
logger = logging.getLogger(__name__)

@app.errorhandler(500)
def handle_internal_error(error):
    """Custom error handler with tracing."""
    current_span = trace.get_current_span()
    if current_span:
        current_span.set_attribute("error", True)
        current_span.set_attribute("error.type", "internal_server_error")
        current_span.record_exception(error)
        current_span.set_status(trace.Status(trace.StatusCode.ERROR, "Internal server error"))

    logger.error(f"Internal server error: {str(error)}", exc_info=True)
    return jsonify({"error": "Internal server error"}), 500

@app.errorhandler(404)
def handle_not_found(error):
    """Handle 404 errors with tracing."""
    current_span = trace.get_current_span()
    if current_span:
        current_span.set_attribute("http.status_code", 404)
        current_span.set_attribute("error.type", "not_found")

    return jsonify({"error": "Resource not found"}), 404

Troubleshooting

Common Issues and Solutions

No Traces Appearing:

# Debug configuration
import os
os.environ["OTEL_LOG_LEVEL"] = "DEBUG"

# Check if tracer is properly initialized
tracer = trace.get_tracer(__name__)
print(f"Tracer: {tracer}")
print(f"Tracer provider: {trace.get_tracer_provider()}")

# Verify span creation
with tracer.start_as_current_span("test-span") as span:
    print(f"Span context: {span.get_span_context()}")
    span.set_attribute("test", "value")

High Memory Usage:

# Configure batch processor with memory limits
from opentelemetry.sdk.trace.export import BatchSpanProcessor

processor = BatchSpanProcessor(
    exporter,
    max_queue_size=512,         # Reduce queue size
    export_timeout_millis=5000, # Faster exports
    schedule_delay_millis=1000, # More frequent exports
    max_export_batch_size=128   # Smaller batches
)

Performance Issues:

# Profile instrumentation overhead
import time

def profile_instrumentation():
    """Profile the overhead of instrumentation."""

    # Test without instrumentation
    start = time.time()
    for _ in range(1000):
        # Simulate work
        pass
    baseline = time.time() - start

    # Test with instrumentation
    start = time.time()
    for _ in range(1000):
        with tracer.start_as_current_span("test"):
            # Simulate work
            pass
    instrumented = time.time() - start

    overhead = ((instrumented - baseline) / baseline) * 100
    print(f"Instrumentation overhead: {overhead:.2f}%")

# Run profiling
if __name__ == "__main__":
    profile_instrumentation()

Debugging Tools

# Custom debug middleware
class DebugTelemetryMiddleware:
    def __init__(self, app):
        self.app = app

    def __call__(self, environ, start_response):
        def debug_start_response(status, headers, exc_info=None):
            current_span = trace.get_current_span()
            if current_span and current_span.is_recording():
                print(f"Span ID: {current_span.get_span_context().span_id}")
                print(f"Trace ID: {current_span.get_span_context().trace_id}")
                print(f"Status: {status}")
            return start_response(status, headers, exc_info)

        return self.app(environ, debug_start_response)

# Use in development
if app.config.get('DEBUG'):
    app.wsgi_app = DebugTelemetryMiddleware(app.wsgi_app)

Analyzing Observability Data

Key Metrics to Monitor

Request Metrics:
- Request rate (requests per second)
- Response time percentiles (p50, p95, p99)
- Error rate percentage
- Request size distribution
Database Metrics:
- Query execution time
- Connection pool usage
- Database errors and timeouts
- Slow query identification
System Metrics:
- CPU and memory usage
- Garbage collection frequency
- Thread pool utilization

Custom Metrics Collection

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader

# Configure metrics
metric_reader = PeriodicExportingMetricReader(
    ConsoleMetricExporter(),
    export_interval_millis=5000,
)
metrics.set_meter_provider(MeterProvider(metric_readers=[metric_reader]))

# Create meter and instruments
meter = metrics.get_meter(__name__)

# Counter for API requests
request_counter = meter.create_counter(
    name="api_requests_total",
    description="Total API requests",
    unit="1",
)

# Histogram for request duration
request_duration = meter.create_histogram(
    name="api_request_duration_seconds",
    description="API request duration",
    unit="s",
)

# Gauge for active connections
active_connections = meter.create_up_down_counter(
    name="active_connections",
    description="Number of active database connections",
    unit="1",
)

@app.before_request
def before_request():
    """Record metrics before processing request."""
    request.start_time = time.time()
    request_counter.add(1, {"method": request.method, "endpoint": request.endpoint})

@app.after_request
def after_request(response):
    """Record metrics after processing request."""
    if hasattr(request, 'start_time'):
        duration = time.time() - request.start_time
        request_duration.record(
            duration,
            {
                "method": request.method,
                "status_code": str(response.status_code),
                "endpoint": request.endpoint or "unknown"
            }
        )
    return response

FAQ

General Questions

Q: What's the performance impact of OpenTelemetry instrumentation?

A: OpenTelemetry is designed to have minimal performance impact, typically adding less than 5% overhead in production environments. The actual impact depends on:

Sampling rate (lower rates = less overhead)
Number of custom spans created
Attribute complexity and cardinality
Export frequency and batch sizes

Q: Can I use OpenTelemetry with existing monitoring solutions?

A: Yes! OpenTelemetry supports multiple exporters simultaneously. You can export traces to Jaeger, metrics to Prometheus, and logs to your existing centralized logging system all at once.

Q: How do I handle sensitive data in traces?

A: Use attribute sanitization, avoid logging sensitive request/response bodies, and consider using OpenTelemetry's sampling and filtering capabilities. Always review what data is being collected before deploying to production.

Q: Why are my traces not appearing in my observability backend?

A: Common causes include:

Incorrect exporter configuration
Network connectivity issues
Authentication problems
Sampling configuration excluding your traces
Service name mismatches

Use console exporter for debugging and verify your configuration step by step.

Q: How do I trace across different Flask blueprints?

A: OpenTelemetry Flask instrumentation automatically works across blueprints. Each route in any blueprint will be traced individually. You can add custom spans within blueprint handlers just like regular routes.

Q: Can I disable instrumentation for specific routes?

A: Yes, use the excluded_urls parameter:

FlaskInstrumentor().instrument_app(
    app,
    excluded_urls="health,metrics,static/*"  # Exclude multiple patterns
)

Q: How do I correlate logs with traces?

A: Use OpenTelemetry's logging instrumentation:

from opentelemetry.instrumentation.logging import LoggingInstrumentor

LoggingInstrumentor().instrument(set_logging_format=True)

This automatically adds trace and span IDs to log records.

What is Uptrace?

Uptrace is an OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. Compare with other top APM tools for your Flask applications.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's Next?

Explore More Examples

Flask Basic Example - Simple Flask application with OpenTelemetry
Flask Auto-instrumentation - Using automatic instrumentation
Flask and Gunicorn - Production deployment with Gunicorn
Flask and uWSGI - Alternative WSGI server setup

Related Guides

OpenTelemetry Python Tracing API - Deep dive into the tracing API
Instrumenting SQLAlchemy with OpenTelemetry - Database instrumentation guide
OpenTelemetry Backends Comparison - Choosing the right observability platform

Ready to implement observability in your Flask application? Start with the basic setup and gradually add more advanced features as your monitoring needs evolve.

OpenTelemetry Filelog Receiver

Alexandr Bandurchin — Thu, 20 Nov 2025 09:20:29 +0000

📋 Part of the OpenTelemetry ecosystem: The Filelog Receiver is a component of the OpenTelemetry Collector that transforms file-based logs into structured OpenTelemetry format. New to OpenTelemetry? Start with What is OpenTelemetry?

Traditional log collection ties you to specific backends. The OpenTelemetry Filelog Receiver breaks this lock-in by converting file logs into vendor-neutral format, allowing you to send data to any compatible backend (Uptrace, Jaeger, Elasticsearch, Splunk).

This guide takes you from basic file tailing to production-grade log pipelines with parsing, multiline handling, and Kubernetes integration.

What is Filelog Receiver

The Filelog Receiver reads log files from disk, parses their content, and converts them into OpenTelemetry log records. It's part of the OpenTelemetry Collector Contrib distribution.

Why OpenTelemetry format?

OpenTelemetry provides a vendor-neutral standard for telemetry data. Instead of being locked into proprietary formats, your logs become portable:

Vendor-neutral: Send logs to any backend without changing instrumentation
Structured: Consistent format with timestamps, severity, attributes
Correlated: Link logs to distributed traces and metrics
Standardized: Part of CNCF OpenTelemetry specification

💡 Learn more: Read our OpenTelemetry Architecture guide to understand how components fit together.

Core capabilities:

Tails log files continuously
Handles log rotation automatically
Parses JSON, regex, and structured formats
Combines multiline entries (stack traces)
Checkpoints file positions across restarts
Enriches logs with metadata

How it works - the 4-step loop:

flowchart LR
  A[1. Discover Files] --> B[2. Read Lines]
  B --> C[3. Parse & Transform]
  C --> D[4. Emit Logs]
  D --> A

Discover: Scans filesystem using include/exclude patterns
Read: Opens files and reads new lines based on start_at setting
Parse: Applies operators to structure raw text
Emit: Sends structured logs to the pipeline

Quick Start: Your First Log Collection

📚 Prerequisites: This guide assumes you have the OpenTelemetry Collector installed. Need help getting started? See our Collector installation guide.

💡 Backend Configuration: Examples in this guide use Uptrace as the
observability backend. OpenTelemetry is vendor-neutral - you can send logs to
Jaeger, Grafana Cloud, Elasticsearch, Splunk, or any OTLP-compatible platform.
See backend examples at the end of this guide.

Let's start simple. Imagine your app writes JSON logs to /var/log/myapp.log:

{"time":"2025-10-06 14:30:00","level":"ERROR","message":"Connection failed","user_id":"123"}

Here's the minimal configuration:

receivers:
  filelog:
    include: [/var/log/myapp.log]
    start_at: beginning
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'
        severity:
          parse_from: attributes.level

exporters:
  otlp/uptrace:  # OTLP = OpenTelemetry Protocol
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: 'YOUR_UPTRACE_DSN_HERE'

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlp/uptrace]

💡 About OTLP: OpenTelemetry Protocol (OTLP) is the standard protocol for transmitting telemetry data. Learn more about configuring exporters.

What happens:

File is discovered and opened
Each line is parsed as JSON
time becomes the log timestamp
level becomes severity
All fields become attributes

Result: Structured logs sent to your OpenTelemetry APM.

Understanding Operators

Operators are the building blocks of log processing. Each operator does one thing well:

Raw Log → [Operator 1] → [Operator 2] → [Operator 3] → Structured Log

Common Operators

JSON Parser - for structured JSON logs:

receivers:
  filelog:
    include: [/var/log/app/*.json]
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%fZ'

Regex Parser - for text logs:

receivers:
  filelog:
    include: [/var/log/app.log]
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(?P<level>\w+)\] (?P<message>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '2006-01-02 15:04:05'
        severity:
          parse_from: attributes.level

Example log:

2025-10-06 14:30:00 [ERROR] Database connection timeout

Add Attributes - enrich with metadata:

operators:
  - type: add
    field: attributes.environment
    value: production
  - type: add
    field: attributes.service.name
    value: payment-api

Container Parser - for Kubernetes (simplified):

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    include_file_path: true
    operators:
      - type: container  # Handles all container formats automatically

🔗 Kubernetes Logs: For complete Kubernetes log collection, see our Kubernetes Log Management guide.

Automatically extracts:

Kubernetes namespace, pod, container names
Container runtime format (CRI-O, containerd, Docker)
stdout/stderr stream

Multiline Logs: Stack Traces

Stack traces span multiple lines. Without configuration, each line becomes a separate log:

2025-10-06 14:30:00 ERROR Exception occurred
    at com.example.App.main(App.java:42)
    at java.lang.Thread.run(Thread.java:829)

Solution: Use multiline to combine them:

receivers:
  filelog:
    include: [/var/log/app/errors.log]
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}'  # New entry starts with date
    operators:
      - type: regex_parser
        regex: '^(?P<time>\S+) (?P<level>\w+) (?P<message>[\s\S]*)$'

Now the entire stack trace is captured as one log record.

Common patterns:

# Java exceptions
line_start_pattern: '^(\d{4}-\d{2}-\d{2}|Exception|Caused by:)'

# Python tracebacks
line_start_pattern: '^(Traceback|\d{4}-\d{2}-\d{2})'

# Go panics
line_start_pattern: '^(panic:|\d{4}/\d{2}/\d{2})'

Kubernetes Log Collection

🔗 Integration Tip: For automated deployment in Kubernetes, use the OpenTelemetry Operator which simplifies Collector management and auto-instrumentation.

Simple Method (Recommended)

Use the container operator for automatic parsing:

receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    include_file_path: true
    operators:
      - type: container

processors:
  batch:
    timeout: 10s

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: 'YOUR_UPTRACE_DSN_HERE'

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp/uptrace]

This automatically:

Detects container runtime format
Parses timestamps and metadata
Handles partial log lines
Extracts Kubernetes metadata

DaemonSet Deployment

Deploy as DaemonSet to collect from all nodes:

📖 Full Kubernetes Setup: See our complete Kubernetes Monitoring with OpenTelemetry guide for cluster-wide observability.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.136.0
        volumeMounts:
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

State Persistence: Never Lose Logs

Without persistence, restarting the Collector means:

start_at: beginning → Re-read everything (duplicates!)
start_at: end → Miss logs written during downtime

Solution: Checkpointing with storage extension:

extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage
    timeout: 10s

receivers:
  filelog:
    include: [/var/log/app.log]
    storage: file_storage  # Link to storage extension

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlp/uptrace]

What happens:

File positions saved to disk
On restart, reading resumes from last position
No duplicates, no data loss

Essential for production!

Real-World Examples

NGINX Access Logs

receivers:
  filelog/nginx:
    include: [/var/log/nginx/access.log]
    operators:
      - type: regex_parser
        regex: '^(?P<remote_addr>\S+) - (?P<remote_user>\S+) \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<protocol>\S+)" (?P<status>\d{3}) (?P<bytes_sent>\d+)'
        timestamp:
          parse_from: attributes.time
          layout: '02/Jan/2006:15:04:05 -0700'
        severity:
          parse_from: attributes.status
          mapping:
            "401": WARN
            "5": ERROR

Handles:

192.168.1.1 - - [06/Oct/2025:14:30:00 +0000] "GET /api/users HTTP/1.1" 200 1234

PostgreSQL Logs

receivers:
  filelog/postgres:
    include: [/var/log/postgresql/*.log]
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+ \S+ \w+) \[(?P<pid>\d+)\] (?P<level>\w+):  (?P<message>[\s\S]*)$'
        timestamp:
          parse_from: attributes.timestamp
          layout: '2006-01-02 15:04:05.000 MST'
        severity:
          parse_from: attributes.level

Application with Trace Context

receivers:
  filelog/app:
    include: [/var/log/app/*.json]
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%fZ'
        trace:
          trace_id:
            parse_from: attributes.trace_id
          span_id:
            parse_from: attributes.span_id

Log example:

{
  "timestamp": "2025-10-06T14:30:00.123Z",
  "level": "error",
  "msg": "Payment failed",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7"
}

Trace context links logs to distributed traces!

Troubleshooting

Logs Not Appearing

Problem: No logs show up in your backend.

Check 1: Permissions

# Verify collector can read files
ls -la /var/log/myapp.log
# Check collector process user
ps aux | grep otelcol

Fix:

# Add collector user to log group
usermod -a -G adm otelcol
# Or adjust permissions
chmod 644 /var/log/myapp.log

Check 2: start_at Setting

Default is end (only new logs). For testing, use:

start_at: beginning

Check 3: File Glob Pattern

# Test your pattern
ls /var/log/pods/*/*/*.log
# Verify files exist
find /var/log -name "*.log"

Regex Not Matching

Debug with test:

# Test pattern locally
echo "2025-10-06 14:30:00 ERROR Test" | grep -P '^\d{4}-\d{2}-\d{2}'

Add debug exporter:

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [debug]

Check logs:

journalctl -u otelcol -f | grep -i error

Multiline Not Working

Common mistake:

# Bad - pattern too specific
multiline:
  line_start_pattern: '^2025-10-06'

# Good - flexible pattern
multiline:
  line_start_pattern: '^\d{4}-\d{2}-\d{2}'

High Memory Usage

Cause: Too many files or large entries.

Solutions:

Exclude unnecessary files:

exclude:
  - /var/log/debug-*.log
  - /var/log/archive/*.log

Limit entry size:

receivers:
  filelog:
    max_log_size: 1MiB

Add memory limiter:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

Performance Optimization

Batching

Reduce network calls:

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp/uptrace]

Guidelines:

Low latency: timeout: 1-5s
High volume: send_batch_size: 1024-2048
Balance both for production

Resource Limits

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    timeout: 10s

Important: Memory limiter must be first processor!

Polling Interval

receivers:
  filelog:
    poll_interval: 500ms  # Default: 200ms

Lower: More CPU, faster detection
Higher: Less CPU, higher latency
Default (200ms) good for most cases

Advanced Features

Compressed Files

receivers:
  filelog:
    include: [/var/log/archive/*.log.gz]
    compression: auto  # or 'gzip'

Header Metadata

receivers:
  filelog:
    include: [/var/log/app.log]
    start_at: beginning
    header:
      pattern: '^# METADATA:.*$'
      metadata_operators:
        - type: regex_parser
          regex: 'version="(?P<app_version>[^"]+)"'

File example:

# METADATA: version="1.2.3" env="production"
2025-10-06 14:30:00 INFO App started

Metadata added to all logs in the file.

Router Operator

Route logs based on content:

receivers:
  filelog:
    include: [/var/log/app.log]
    operators:
      - type: json_parser
      - type: router
        routes:
          - output: error_logs
            expr: 'attributes.level == "ERROR"'
          - output: default_logs
            expr: 'true'

      - id: error_logs
        type: add
        field: attributes.alert
        value: true

      - id: default_logs
        type: noop

FAQ

Why aren't my logs appearing?
Possible causes include:
- Collector can't read files or directories due to permission issues.
- start_at: end only reads new logs (use beginning for testing).
- The glob pattern may not match files specified in the include setting.
How do I handle log rotation?
Log rotation is handled automatically. The receiver tracks files by fingerprint, not filename.
When app.log rotates to app.log.1, it finishes reading the old file and starts the new one.
What's the difference between attributes and resource?
- Attributes: Log-level metadata (varies per log)
- Resource: Service-level metadata (shared across all logs from the same service)
operators:
- type: add field: attributes.request_id # Per-log value: "req-123"
- type: add field: resource["service.name"] # Per-service value: "api"
Can I parse logs from multiple services?
Yes. Use multiple receivers:

   receivers:
     filelog/app1:
       include: [/var/log/app1/*.log]
       operators:
         - type: add
           field: attributes.service.name
           value: app1

     filelog/app2:
       include: [/var/log/app2/*.log]
       operators:
         - type: add
           field: attributes.service.name
           value: app2

   service:
     pipelines:
       logs:
         receivers: [filelog/app1, filelog/app2]
         exporters: [otlp/uptrace]

How do I test regex patterns?
Use Regex101 with the "Golang" flavor to match the Collector's regex engine.
What if I have mixed compressed and uncompressed files?

   receivers:
     filelog:
       include: [/var/log/app/*.log, /var/log/app/*.log.gz]
       compression: auto  # Auto-detects .gz files

How do I move attributes to resource?

   operators:
     - type: move
       from: attributes["file.name"]
       to: resource["log.file.name"]

Can I filter logs before sending? Yes. Use a filter processor:

   processors:
     filter:
       logs:
         exclude:
           match_type: regexp
           record_attributes:
             - key: level
               value: DEBUG

How do I parse key-value logs?

   operators:
     - type: regex_parser
       regex: '(\w+)=("[^"]*"|\S+)'
       parse_from: body

Example: user=john status=200 duration=123ms

What's the max file size limit?
There’s no hard limit, but you can use max_log_size to prevent memory issues:
```
receivers:
  filelog:
    max_log_size: 1MiB  # Per log entry
```

Backend Examples

This guide uses Uptrace in examples, but OpenTelemetry works with any OTLP-compatible backend. Here are quick configuration examples for other platforms:

Uptrace

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: 'YOUR_UPTRACE_DSN'

Grafana Cloud

exporters:
  otlp:
    endpoint: otlp-gateway.grafana.net:443
    headers:
      authorization: "Bearer YOUR_GRAFANA_TOKEN"

Jaeger

exporters:
  otlp:
    endpoint: jaeger-collector:4317
    tls:
      insecure: true

Datadog

exporters:
  otlp:
    endpoint: trace.agent.datadoghq.com:4317
    headers:
      dd-api-key: "YOUR_DATADOG_API_KEY"

New Relic

exporters:
  otlp:
    endpoint: otlp.nr-data.net:4317
    headers:
      api-key: "YOUR_NEW_RELIC_LICENSE_KEY"

Prometheus (metrics only)

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"

Multiple Backends

Send data to multiple platforms simultaneously:

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: 'YOUR_UPTRACE_DSN'

  otlp/datadog:
    endpoint: trace.agent.datadoghq.com:4317
    headers:
      dd-api-key: "YOUR_DATADOG_API_KEY"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/uptrace, otlp/datadog]  # Send to both!

More backends: See the OpenTelemetry Vendors directory for 40+ compatible platforms.

Next Steps

OpenTelemetry FastAPI monitoring

Alexandr Bandurchin — Thu, 20 Nov 2025 09:19:56 +0000

By integrating OpenTelemetry with FastAPI, you can gain valuable insight into the performance, behavior and dependencies of your API. You can monitor and troubleshoot issues, optimize performance, and ensure the reliability of your FastAPI applications.

What is FastAPI?

FastAPI is a modern, high-performance web framework for building APIs with Python. It is designed to be easy to use, highly efficient, and able to handle high loads.

FastAPI's combination of performance, productivity, and modern features has made it popular among developers building APIs with Python.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments. For comprehensive Python instrumentation, see the OpenTelemetry Python guide.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. Configuration can be easily managed through OpenTelemetry environment variables, providing a standardized way to configure exporters, resource attributes, and sampling behavior.

OpenTelemetry provides detailed instrumentation, enabling you to monitor and measure the performance of your FastAPI applications in real-time. You can identify bottlenecks, optimize code, and ensure your application runs smoothly even under heavy traffic.

FastAPI instrumentation

To install OpenTelemetry instrumentation for FastAPI:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-fastapi

OpenTelemetry SDK

To initialize OpenTelemetry in your Flask application, add the following to your application's startup code, for example, in your app.py or wsgi.py file:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter,
)

provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer("my.tracer.name")

Usage

To instrument FastAPI application:

from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

Also see OpenTelemetry FastAPI example at GitHub.

OpenTelemetry allows you to trace requests as they flow through your FastAPI application, providing a clear picture of how different components and services interact. This end-to-end tracing helps diagnose issues quickly and streamline troubleshooting.

Once your FastAPI application is instrumented with OpenTelemetry, you can use the observability data in a variety of ways. You can visualize distributed traces, analyze performance metrics, and gain insight into the behavior of your API using OpenTelemetry backends such as Uptrace, Jaeger, Prometheus, or Grafana.

Instrumenting your code

OpenTelemetry can automatically trace incoming HTTP requests to your FastAPI application. You can also create custom spans to trace specific parts of your code. For example:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.get("/")
async def read_root():
    # Create a custom span
    with tracer.start_as_current_span("custom-span"):
        # Your code here
        return {"message": "Hello, World!"}

What is Uptrace?

Uptrace is an open source APM for OpenTelemetry that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. For APM capabilities, compare with other top APM tools.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules, notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

uvicorn

If you are using uvicorn with Gunicorn, you need to initialize OpenTelemetry in the post-fork hook:

import uptrace

def post_fork(server, worker):
    uptrace.configure_opentelemetry(...)

workers = 4
worker_class = "uvicorn.workers.UvicornWorker"

See Application servers for details.

Conclusion

With insights from OpenTelemetry, you can make informed decisions about scaling your FastAPI application. Load balancing strategies can be adjusted based on real-time data to handle traffic spikes effectively.

OpenTelemetry also allows you to instrument specific parts of your code for custom telemetry collection. You can use OpenTelemetry Python APIs to manually create spans and add custom attributes, events, or metrics to capture additional information.

Monitor Falcon with OpenTelemetry

Alexandr Bandurchin — Tue, 11 Nov 2025 10:17:35 +0000

Falcon is a minimalist web framework for building RESTful APIs in Python. It is lightweight, fast, and designed to be easy to use.

By integrating OpenTelemetry into your Falcon application, you can capture and export telemetry data such as traces, metrics, and logs. You can use that data for monitoring, troubleshooting, and analyzing the behavior and performance of your application.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. The OpenTelemetry Operator can automatically inject instrumentation into your applications when running in Kubernetes, reducing manual configuration overhead.

OpenTelemetry Falcon

To instrument Falcon app, you need to install the corresponding OpenTelemetry Falcon instrumentation:

pip install opentelemetry-instrumentation-falcon

Usage

Use the following code to integrate OpenTelemetry to capture telemetry data from your Falcon application:

from falcon import API
from opentelemetry.instrumentation.falcon import FalconInstrumentor

FalconInstrumentor().instrument()

app = falcon.API()

class HelloWorldResource(object):
    def on_get(self, req, resp):
        resp.body = 'Hello World'

app.add_route('/hello', HelloWorldResource())

You can exclude certain URLs from being tracked using environment variables:

# excludes requests such as `https://site/client/123/info` and `https://site/xyz/healthcheck`
export OTEL_PYTHON_FALCON_EXCLUDED_URLS="client/.*/info,healthcheck"

You can also extract certain attributes from Falcon’s request object and use them as span attributes:

export OTEL_PYTHON_FALCON_TRACED_REQUEST_ATTRS='query_string,uri_template'

See documentation for more details.

Hooks

You can specify request and response hooks:

# the hook is called right after a Span is created
def request_hook(span, req):
    pass

# the hook is called right before the span is finished
def response_hook(span, req, resp):
    pass

FalconInstrumentation().instrument(request_hook=request_hook, response_hook=response_hook)

What is Uptrace?

Uptrace is a distributed tracing tool that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's next?

Falcon applications now have complete observability with OpenTelemetry instrumentation for API monitoring and distributed tracing. For alternative Python frameworks, explore FastAPI for async support or Django for full-featured web applications.

OpenTelemetry Express.js instrumentation

Alexandr Bandurchin — Tue, 11 Nov 2025 10:16:55 +0000

By combining OpenTelemetry with Express.js, you can collect and export telemetry data, including traces, metrics, and logs, to gain insight into the behavior and performance of your application.

What is Express.js?

Express.js is a fast and minimalist web application framework for Node.js, a JavaScript runtime. It simplifies the development of web applications and APIs by providing a robust set of features and utilities, while maintaining a lightweight and unobtrusive approach.

Express.js is widely recognized for its simplicity, flexibility, and extensibility, making it a popular choice for building server-side applications in Node.js.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. The OpenTelemetry architecture follows a modular design with SDKs, APIs, and exporters as its core components.

Express.js instrumentation

Using the OpenTelemetry Node.js library, you can easily add distributed tracing capabilities to your Express.js applications.

To instrument an Express.js app, you need to install OpenTelemetry Node.js SDK and available instrumentations:

# Using npm
npm install @opentelemetry/sdk-node @opentelemetry/api @opentelemetry/auto-instrumentations-node

# Using yarn
yarn add @opentelemetry/sdk-node @opentelemetry/api @opentelemetry/auto-instrumentations-node

You can let OpenTelemetry to instrument your application automatically or do it explicitly by installing required instrumentations:

# Using npm
npm install @opentelemetry/instrumentation-http @opentelemetry/instrumentation-express

# Using yarn
yarn add @opentelemetry/instrumentation-http @opentelemetry/instrumentation-express

Usage

After installing OpenTelemetry, you need to configure OpenTelemetry SDK to export data to an OpenTelemetry backend for strorage and visualization.

'use strict'

const otel = require('@opentelemetry/api')
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base')
const { Resource } = require('@opentelemetry/resources')
const { NodeSDK } = require('@opentelemetry/sdk-node')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
const { AWSXRayIdGenerator } = require('@opentelemetry/id-generator-aws-xray')
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http')
const {
  ExpressInstrumentation,
} = require('@opentelemetry/instrumentation-express')

const dsn = process.env.UPTRACE_DSN
console.log('using dsn:', dsn)

const exporter = new OTLPTraceExporter({
  url: 'http://localhost:14318/v1/traces',
  headers: { 'uptrace-dsn': dsn },
  compression: 'gzip',
})
const bsp = new BatchSpanProcessor(exporter, {
  maxExportBatchSize: 1000,
  maxQueueSize: 1000,
})

const sdk = new NodeSDK({
  spanProcessor: bsp,
  resource: new Resource({
    'service.name': 'myservice',
    'service.version': '1.0.0',
  }),
  idGenerator: new AWSXRayIdGenerator(),
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
})
sdk.start()

To avoid potential issues, it is strongly recommended to put the OpenTelemetry initialization code into a separate file called otel.js and use the --require flag to load the tracing code before the application code:

# JavaScript
node --require ./otel.js app.js

# TypeScript
ts-node --require ./otel.ts app.ts

See OpenTelemetry Express.js example for details.

What is Uptrace

Uptrace is a OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. For Node.js instrumentation, see the OpenTelemetry JavaScript guide and compare with top APM tools.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's next?

By integrating OpenTelemetry with Express.js, you can gain insight into your application's distributed traces, OpenTelemetry metrics, and logs. This helps you understand the behavior of your Express.js application across multiple services or microservices, and facilitates troubleshooting and performance optimization.